Can't submit data

Using Gen3
1

I have been trying (and failing) to spin up a local instance of Gen3 using docker compose. I'm pretty sure I have everything set up correctly.

o smoke_test passes
o application appears to function
o login thru oAuth works
o created program and project
o updated user.yaml with program, project permissions
o created API key
o configured gen3-client

gen3-client upload --profile=vanetten --upload-path=/Users/vanetten/Desktop/test.png

You don't have permission to upload data, detailed error message: 500 Internal Server error has occurred! Please try again later

logs

fence-service | [Mon May 13 17:41:03.651241 2019] [wsgi:error] [pid 1503:tid 139734837204736] [remote 172.31.0.11:3220] ConnectionError: HTTPSConnectionPool(host='localhost', port=443): Max retries exceeded with url: /user/jwt/keys (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f16770d9710>: Failed to establish a new connection: [Errno 111] Connection refused',))
fence-service | [Mon May 13 17:41:03.651403 2019] [wsgi:error] [pid 1503:tid 139734837204736] [remote 172.31.0.11:3220] [2019-05-13 17:41:03,651] ERROR in error_handler: 500 HTTP error occured. ID: 71e0351c-61b2-40bf-9fc8-104a4f423d8a
fence-service | [Mon May 13 17:41:03.651436 2019] [wsgi:error] [pid 1503:tid 139734837204736] [remote 172.31.0.11:3220] Details: {'message': MaxRetryError("HTTPSConnectionPool(host='localhost', port=443): Max retries exceeded with url: /user/jwt/keys (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f16770d9710>: Failed to establish a new connection: [Errno 111] Connection refused',))",)}
revproxy-service | 172.31.0.1 - - [13/May/2019:17:41:03 +0000] "POST /user/data/upload HTTP/1.1" 500 3132 "-" "Go-http-client/1.1" "-"
fence-service | {"gen3log": "apache2", "date_access": "[13/May/2019:17:41:03 +0000]", "user_id": "-", "request_id": "-", "session_id": "-", "visitor_id": "-", "network_client_ip": "-", "network_bytes_write": "3132", "http_response_time": "0", "http_status_code": "500", "http_request": "/data/upload", "http_verb": "POST", "http_referer": "-", "http_useragent": "Go-http-client/1.1", "message": "POST /data/upload HTTP/1.0"}
fence-service | [Mon May 13 17:41:03.660360 2019] [wsgi:error] [pid 1503:tid 139734820402944] [remote 172.31.0.11:54667] [2019-05-13 17:41:03,660] ERROR in error_handler: Catch exception
fence-service | [Mon May 13 17:41:03.660399 2019] [wsgi:error] [pid 1503:tid 139734820402944] [remote 172.31.0.11:54667] Traceback (most recent call last):
fence-service | [Mon May 13 17:41:03.660409 2019] [wsgi:error] [pid 1503:tid 139734820402944] [remote 172.31.0.11:54667] File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1612, in full_dispatch_request
fence-service | [Mon May 13 17:41:03.660419 2019] [wsgi:error] [pid 1503:tid 139734820402944] [remote 172.31.0.11:54667] rv = self.dispatch_request()
fence-service | [Mon May 13 17:41:03.660429 2019] [wsgi:error] [pid 1503:tid 139734820402944] [remote 172.31.0.11:54667] File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1598, in dispatch_request
fence-service | [Mon May 13 17:41:03.660439 2019] [wsgi:error] [pid 1503:tid 139734820402944] [remote 172.31.0.11:54667] return self.view_functionsrule.endpoint
fence-service | [Mon May 13 17:41:03.660448 2019] [wsgi:error] [pid 1503:tid 139734820402944] [remote 172.31.0.11:54667] File "/src/authutils/authutils/token/validate.py", line 245, in wrapper
fence-service | [Mon May 13 17:41:03.661794 2019] [wsgi:error] [pid 1503:tid 139734820402944] [remote 172.31.0.11:54667] set_current_token(validate_request(aud=aud, purpose=purpose))
fence-service | [Mon May 13 17:41:03.661831 2019] [wsgi:error] [pid 1503:tid 139734820402944] [remote 172.31.0.11:54667] File "/src/authutils/authutils/token/validate.py", line 222, in validate_request
fence-service | [Mon May 13 17:41:03.661843 2019] [wsgi:error] [pid 1503:tid 139734820402944] [remote 172.31.0.11:54667] return validate_jwt(encoded_token, aud, purpose)
fence-service | [Mon May 13 17:41:03.661875 2019] [wsgi:error] [pid 1503:tid 139734820402944] [remote 172.31.0.11:54667] File "/src/authutils/authutils/token/validate.py", line 194, in validate_jwt
fence-service | [Mon May 13 17:41:03.662329 2019] [wsgi:error] [pid 1503:tid 139734820402944] [remote 172.31.0.11:54667] attempt_refresh=attempt_refresh
fence-service | [Mon May 13 17:41:03.662338 2019] [wsgi:error] [pid 1503:tid 139734820402944] [remote 172.31.0.11:54667] File "/src/authutils/authutils/token/keys.py", line 171, in get_public_key_for_token
fence-service | [Mon May 13 17:41:03.662347 2019] [wsgi:error] [pid 1503:tid 139734820402944] [remote 172.31.0.11:54667] return get_public_key(kid, iss=iss, attempt_refresh=attempt_refresh)
fence-service | [Mon May 13 17:41:03.662355 2019] [wsgi:error] [pid 1503:tid 139734820402944] [remote 172.31.0.11:54667] File "/src/authutils/authutils/token/keys.py", line 141, in get_public_key
fence-service | [Mon May 13 17:41:03.662370 2019] [wsgi:error] [pid 1503:tid 139734820402944] [remote 172.31.0.11:54667] refresh_jwt_public_keys(iss)
fence-service | [Mon May 13 17:41:03.662768 2019] [wsgi:error] [pid 1503:tid 139734820402944] [remote 172.31.0.11:54667] File "/src/authutils/authutils/token/keys.py", line 86, in refresh_jwt_public_keys
fence-service | [Mon May 13 17:41:03.662777 2019] [wsgi:error] [pid 1503:tid 139734820402944] [remote 172.31.0.11:54667] jwt_public_keys = requests.get(path).json()['keys']
fence-service | [Mon May 13 17:41:03.662783 2019] [wsgi:error] [pid 1503:tid 139734820402944] [remote 172.31.0.11:54667] File "/usr/local/lib/python2.7/dist-packages/requests/api.py", line 75, in get
fence-service | [Mon May 13 17:41:03.662791 2019] [wsgi:error] [pid 1503:tid 139734820402944] [remote 172.31.0.11:54667] return request('get', url, params=params, **kwargs)
fence-service | [Mon May 13 17:41:03.662801 2019] [wsgi:error] [pid 1503:tid 139734820402944] [remote 172.31.0.11:54667] File "/usr/local/lib/python2.7/dist-packages/requests/api.py", line 60, in request
fence-service | [Mon May 13 17:41:03.662811 2019] [wsgi:error] [pid 1503:tid 139734820402944] [remote 172.31.0.11:54667] return session.request(method=method, url=url, **kwargs)
fence-service | [Mon May 13 17:41:03.662821 2019] [wsgi:error] [pid 1503:tid 139734820402944] [remote 172.31.0.11:54667] File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 533, in request
fence-service | [Mon May 13 17:41:03.663306 2019] [wsgi:error] [pid 1503:tid 139734820402944] [remote 172.31.0.11:54667] resp = self.send(prep, **send_kwargs)
fence-service | [Mon May 13 17:41:03.663341 2019] [wsgi:error] [pid 1503:tid 139734820402944] [remote 172.31.0.11:54667] File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 646, in send
fence-service | [Mon May 13 17:41:03.663351 2019] [wsgi:error] [pid 1503:tid 139734820402944] [remote 172.31.0.11:54667] r = adapter.send(request, **kwargs)
fence-service | [Mon May 13 17:41:03.663362 2019] [wsgi:error] [pid 1503:tid 139734820402944] [remote 172.31.0.11:54667] File "/usr/local/lib/python2.7/dist-packages/requests/adapters.py", line 516, in send
fence-service | [Mon May 13 17:41:03.663372 2019] [wsgi:error] [pid 1503:tid 139734820402944] [remote 172.31.0.11:54667] raise ConnectionError(e, request=request)

2

Hi Bill_Van_Etten!

Compose services currently do not support gen3-client.

3

Actually, let me check this information, I will get back to you soon.

4

Thank you for your prompt response. I (BioTeam, bioteam.net) have been working with most ICs from the NIH to help guide them toward trans-NIH solutions for commons, auth, and data index, etc that span CSPs. Gen3 appears to be a good fit. Have spoken with folks from NCI and Kid's First who seem to love Gen3. I'm simply trying to get a private instance running, either locally or on AWS, such that I can kick the tires and figure out what's all included. After trying for several days, I can't seem to get a fully functional system going with compose-services. I'm about to try cloud-automation, but I imagine I might hit even more pitfalls. Any chance I can get a one-on-one meeting with someone?

5

Hi @Bill_Van_Etten! We are trying to reproduce that compose-services error. Its probably just a configuration error that we can fix.

For cloud-automation, we are about to provide updates that will make it easier for 3rd party (non-UChicago) folks to run. I expect that it a couple of weeks.

We are happy to schedule a one-on-one meeting as well. Are you on our slack in our #gen3-community channel?

6

I would like to get to know Gen3 better. I am generally available for a one-on-one meeting this week or next (vanetten@bioteam.net), please suggest a time. I have joined the Gen3 forum (forums.gen3.org), but I have not joined any slack channels outside of my workspace. I will investigate that.

Bill

7

Hi @Bill_Van_Etten! We sent you an invitation to the slack channel.

8

I got the same error message when running docker compose. I can see I have upload permission but got error when I tried to upload a file

hongping@hongping-Virtual-Machine:~/Downloads$ ./gen3-client auth --profile=hongping.liang@jax.org
2020/05/06 15:04:24
You have access to the following project(s) at http://localhost:
2020/05/06 15:04:24 cube_program [create delete read read-storage update upload write-storage]
hongping@hongping-Virtual-Machine:~/Downloads$ ./gen3-client upload --profile=hongping.liang@jax.org --upload-path=experiment.tsv
2020/05/06 15:04:29 Finish parsing all file paths for "/home/hongping/Downloads/experiment.tsv"

The following file(s) has been found in path "/home/hongping/Downloads/experiment.tsv" and will be uploaded:
        /home/hongping/Downloads/experiment.tsv

2020/05/06 15:04:29 You don't have permission to upload data, detailed error message: 404 Not found error has occurred! The requested url "http://localhost/user/data/upload" cannot be found

2020/05/06 15:04:29 Retry upload has started...
9

Hi @hongpingliang!

Data upload might be tricky in Compose Services. Did you configure your AWS buckets in the Secrets/fence-config.yaml ? Here is documentation https://github.com/uc-cdis/compose-services#enabling-data-upload-to-s3

Would you like to join our slack channel, where Gen3 enthusiasts and developers share their experience in configuring and using Gen3?

10

Thanks, please add me to your slack channel.

I only upload the data to my local machine using compose services, why do need s3 bucket permission?

11

We sent you an invite, @hongpingliang!

Yes, to upload data files in Compose Services you would need s3 bucket where to upload them. I think it is not possible to upload locally.

12

I found this thread experiencing similar issues, kicking the tires on the docker-compose version. Trying to upload with the 2020.10 client I tend to get something along the lines of the log below.

I have program 'NTP' and project 'pilot' configured, and tried delving into users.yaml via the guide I found in fence along the lines of a project_submitter policy

  • id: project_submitter
    role_ids:
    • reader
    • creator
    • updater
    • deleter
    • storage_reader
    • storage_writer
      resource_paths:
    • /programs/NIH/projects/pilot

...

users:
michael.c.conway@gmail.com:
policies:
- project_submitter
- workspace
- file_uploader

So I figure:

  • I'm missing something in user.yaml because I don't see any access to projects when i do a gen3-client auth, tho I can find the project and program out there via Graphql

  • I missed that an s3 bucket was required for file upload, I'll look at that tomorrow

  • I'm confused about the message "The requested url "user/data/upload" cannot be found" which almost makes me think there is a mismatch between the version of the gen3-client and the gen3 installation itself? It sounds like uploads moved at some point in the API?

Anyhow there's lots I'm sorting out, I'd love to get on Slack if that's a possibility.

Best,
MC

(base) ~/Documents/Programs/gen3 @ ALMBP-02010755(conwaymc): gen3-client upload --profile=plp01 --upload-path=/Users/conwaymc/Documents/DCPPC\ APIs\ Overview.pdf
2020/11/03 16:13:37 A new version of gen3-client is available! The latest version is 2020.11.0. You are using version 2020.10
2020/11/03 16:13:37 Please download the latest gen3-client release from/
2020/11/03 16:13:37 Finish parsing all file paths for "/Users/conwaymc/Documents/DCPPC APIs Overview.pdf"

The following file(s) has been found in path "/Users/conwaymc/Documents/DCPPC APIs Overview.pdf" and will be uploaded:
/Users/conwaymc/Documents/DCPPC APIs Overview.pdf

2020/11/03 16:13:37 You don't have permission to upload data, detailed error message: 404 Not found error has occurred! The requested url "/user/data/upload" cannot be found

2020/11/03 16:13:37 Retry upload has started...
2020/11/03 16:13:37 1 records has been sent to the retry channel

2020/11/03 16:13:37 #1 retry of record /Users/conwaymc/Documents/DCPPC APIs Overview.pdf
2020/11/03 16:13:37 Sleep for 2 seconds
2020/11/03 16:13:39 You don't have permission to upload data, detailed error message: 404 Not found error has occurred! The requested url "/user/data/upload" cannot be found
2020/11/03 16:13:39 #2 retry of record /Users/conwaymc/Documents/DCPPC APIs Overview.pdf
2020/11/03 16:13:39 Sleep for 4 seconds
2020/11/03 16:13:43 You don't have permission to upload data, detailed error message: 404 Not found error has occurred! The requested url "user/data/upload" cannot be found
2020/11/03 16:13:43 #3 retry of record /Users/conwaymc/Documents/DCPPC APIs Overview.pdf
2020/11/03 16:13:43 Sleep for 8 seconds
^C
(base) ~/Documents/Programs/gen3 @ ALMBP-02010755(conwaymc):

13

Hello @mikeconwayatniehs! Welcome to the forum :slight_smile: We sent you the invite to the slack.

Regarding the message "The requested url "user/data/upload" cannot be found" please check that you configured s3 bucket credentials in fence-config.yaml file, here is a description https://github.com/uc-cdis/compose-services#enabling-data-upload-to-s3

14

Hi @Viktorija,

I got the same error when using gen3-client to upload data to my local compose-services instance.

You don't have permission to upload data, detailed error message: 404 Not found error has occurred! The requested url "http://localhost/user/data/upload" cannot be found

I'm not using the AWS S3 bucket. so I was wondering if there is an updated version of the compose-services or gen3-client which would allow us to use local storage instead of S3? or do we still have to use S3 to get it working?

Also, can I get the invite to the Gen3 Slack? I'm new to gen3 and like to learn more about it. I've fill out this Google Form last week days ago but haven't got approved.

15

Hi @Chinchien_Lin

Welcome to the Gen3 forum :slight_smile:
The easiest way to make it work is to have an AWS s3. We have contributors from Oregon Health & Science University that used Compose Services in non-AWS environment and generously shared their experience, you can learn more in the Useful Links section (point 1).

We sent you the invite to the slack :slight_smile:

16

Hi @Viktorija,

Thanks for sending the slack invite!

I got it working with the AWS S3. I can now successfully upload files via the gen3-client & S3. However, when I tried to map the files to metadata, the uploaded files were always in the "generating" status.
I found this documentation which says:

If one or multiple data files have been submitted to an S3 bucket and you do not want to set up automation through an SNS and SQS, a simple alternative is to index the data files manually after the upload. The upload command creates a "blank" record in indexd, which should be then updated by adding the file's size and hash. This can be done with a PUT request to index, where the base URL is https://your-commons.org/index/index/blank/{GUID}. A list of URLS to reach other services from the Gen3 Framework is shown here. Only once the uploaded data file is indexed, graph metadata can be submitted to it.

So I sent a put request generated from swagger to my local gen3:

curl -X 'PUT'
'http://localhost/index/blank/c9ae7383-567a-46b2-ba94-ea9d86d5a194?rev=1'
-H 'accept: application/json'
-H 'Content-Type: application/json'
-d '{
"size": 5,
"hashes": {
"md5": "806765349a237334fc415c0ce9b9fd72",
"sha": "2e15c3078e11a5c507d77b1c2118fa143bb24b93",
"sha256": "3f8fc96e48ea954dcf62a225885b9c07b0e6d237de98a05d46c632fb0f999987",
"sha512": "b4768fc6f413084b74a700ce65688128cb6f7a5aadf3c4255268195581386f8b7951d013b39473086f99dda19e2bdf67ccfdc08a01081eaf51b1f4c7047eabc3",
"crc": "e09a70c3",
"etag": "f0a044a5975c6b2e4347d1ece5c484e5"
},
"urls": [
"string"
],
"authz": [
"string"
]
}'

Sorry this might be a silly question, I got another 405 error as below:

405 Method Not Allowed

Method Not Allowed

The method is not allowed for the requested URL.</p

I guess I might do something wrong in my put request.
Hoping if you could please advise what the put request command should look like?

Thanks again for your help so far.

17

Hi @Chinchien_Lin!

I see the word "index" is used twice in the URL from the documentation. Can you try to use 'index/index/blank' instead of 'index/blank' in your URL?

18

Hi @Viktorija ,

I tried adding an extra "/index" to the URL. It returned another error this time:

{"error":"revision mismatch"}

This seems to refer to the last parameter "rev" in the URL, but I have no idea where/how I can get the revision value. so I randomly set it to be "1", then failed apparently...

I'm on Ubuntu 18.04, it doesn't seem to store the file revision history (only the creation and last modification date). or Is this something I can find on S3 or the gen3 GUI?

Thanks for your help so far. really appreciate!!