Compose Services - current limitations

Using Gen3
1

Formerly logged issues imply, that there are limitations to the compose services version of gen3 (e.g. upload capabilities), that I do not see reflected in documentation.

What are the current limitations here? Should the upload actually work with the latest version?

Greetings,

Thomas

2

Hi @thomasleyer! Welcome to the forum :slight_smile:

The main difference between Gen3 Compose Services and Gen3 Could Automation is automation and logging. Here are short descriptions of the products for comparision: https://gen3.org/resources/operator/

Compose Services supports upload to AWS, but additional manual steps are required from operators to index uploaded data (to make it available for mapping to the structured data in the Data Explorer) or to automate indexing process.

3

Thank you, Viktorija,

I didn't find the comparison very detailed, and only found the upload limitation in the isse tracker entries. That was the reason for my question.

It would be great, if you could point me to a description of the manual steps required from operators to index uploaded data?

Thank you so much,

Thomas

P.S.: I wanted to send a pull request, do I need to have a Jira ticket for that? It seems, that I do not have access to the Jira project.

4

Hi Thomas!

For additional steps for data upload let me quote our developer:

We use AWS SNS to get notifications when objects are uploaded to a bucket, and we store these notifications in an AWS SQS. Our job dispatcher service (GitHub - uc-cdis/ssjdispatcher: SQS S3 Job Dispatcher) watches the SQS and spins up an indexing job (GitHub - uc-cdis/indexs3client) to update indexd with the file information (size, hash).
Since you are just getting started with Gen3, the simpler alternative is to index your files manually after uploading them  The upload command creates a "blank" record in indexd, which you would need to update by adding the file size and hash. This can be done with a PUT request to indexd Swagger UI

It is great that you want to submit pull request! :heart: You don't need a Jira ticket for that, and don't need to reference ticket number in the pull request, those are internal references. Other than that, please follow our recommended guidelines for GitHub usage: Gen3 - Dev Intro

Would you like to join our slack channel where Gen3 developers, operators and users share ther experience in setting up and configuring Compose Services and Cloud Automation?

5

Thanks so much for the clarification.. sure, I would love to join the slack channel

6

We sent you the invite :slight_smile: