Compose Services upload issues

Using Gen3
1

I'm trying to test out the gen3 app. I have it running inside the docker-compose framework as documented: here.

Now, I'm trying to upload some data. I'm trying to use the gen3-client.

The following succeeded no problems:

configure --profile=bob  --cred=$HOME/.gen3/credentials.json --apiendpoint=https://localhost

Now when I try to upload a file I'm getting the following issues:

gen3-client upload --profile=bob --upload-path=./testing.txt

2019/07/17 13:45:26 You don't have permission to upload data, detailed error message: Error occurred in RequestNewAccessKey: Error occurred during making HTTP request: Post https://localhost/user/credentials/api/access_token: x509: certificate signed by unknown authority

is there a way to force the client to accept a self signed certificate?

2

Hi @csgeek!

Welcome to the forum! :slight_smile:

As a first step I would suggest to add self-signed certificate to your trusted certificates. Steps may vary depending on OS you use. What system are you running on?

3

is there no option for the gen3-client to accepted self-signed certificates?

I added the SSL certificate as a trusted resource, and now my error changed to:

2019/07/18 09:13:05 You don't have permission to upload data, detailed error message: 403 Forbidden error has occurred! You don't have permission to access the requested url "https://localhost/user/data/upload"

I also didn't see a step where I'm setting up a GS bucket in the documentation. Unless the app does that programmatically, I assume that needs to be setup?

4

Compose Services has limitations comparing to Cloud Automation, file upload was one of these limitations, that is why it is not documented. However I think it is possible to make gen3-client work with Composer Services now. Recent fence-yaml.conf file contains configuration variables for cloud services. Let me check with our developer how to better configure for GC, I will update you.

Regarding upload error - have you granted yourself permission in the user.yaml? Following sections should be present in your configuration:

Policies:

rbac:
    policies:
        - id: 'data_upload'
          description: 'upload raw data files to S3'
          resource_paths: ['/data_file']
          role_ids: ['file_uploader']

Roles:

 roles:
      - id: 'file_uploader'
        description: 'can upload data files'
        permissions:
            - id: 'file_upload'
              action:
                  service: 'fence'
                  method: 'file_upload'

Users:

your@email.com:
  admin: True
  policies: ['data_upload', 'workspace']
  projects:
      ....
5

If there is a guide for Cloud Automation that I've missed I'd love to look that over, but for now looking at compose services.

Confirming the following section exist in my user.yml

 rbac:
     policies:
         - id: 'data_upload'
           description: 'upload raw data files to S3'
           resource_paths: ['/data_file']
           role_ids: ['file_uploader']
         - id: workspace
           description: be able to use workspace
           resource_paths: [/workspace]
           role_ids: [workspace_user]
     roles:
         - id: 'file_uploader'
           description: 'can upload data files'
           permissions:
               - id: 'file_upload'
                 action:
                     service: 'fence'
                     method: 'file_upload'
         - id: workspace_user
           permissions:
           - action: {method: access, service: jupyterhub}
             id: workspace_access
   myemail@gmail.com:  ##this is changed so my email isn't on a public forum but config is updated
     admin: True
     policies: ['data_upload', 'workspace']
6

Thanks for confirming user.yaml sections. I'm checking how to configure Google Cloud for Compose Services, those could be related, I will update you.

Here is a link to Cloud Automation guide FYI: https://github.com/uc-cdis/cloud-automation/blob/master/doc/csoc-free-commons-steps.md

7

Hi @csgeek.

I would like to update you with my findings. Currently Google Cloud support is not implemented in the Compose Services and there is no trivial way to automatically tight them.

8

Noted, Thanks for the update @Viktorija. I'll follow the guide you linked above and see how far that gets me. Thank you for the help provided.