Issue with Compose Services

Using Gen3
#2

Hi @Zander_Giuffrida! Welcome to the forum!

Thanks for providing your dump. Are you using Docker Desktop? I'm asking because our developers mentioned a very similar problem was caused by Docker Desktop settings: it was set to make 4 CPUs available to the Docker Engine, and start working after limiting it to 1 CPU in Docker > Preferences > Advanced. Here is the relevant log from logs-esproxy-service.txt:

[36mesproxy-service      |[0m ERROR: [1] bootstrap checks failed
[36mesproxy-service      |[0m [1]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
[36mesproxy-service      |[0m [2020-01-16T13:59:32,475][INFO ][o.e.n.Node               ] [cogtRnY] stopping ...
[36mesproxy-service      |[0m [2020-01-16T13:59:32,937][INFO ][o.e.n.Node               ] [cogtRnY] stopped
[36mesproxy-service      |[0m [2020-01-16T13:59:32,938][INFO ][o.e.n.Node               ] [cogtRnY] closing ...
[36mesproxy-service      |[0m [2020-01-16T13:59:33,030][INFO ][o.e.n.Node               ] [cogtRnY] closed
#3

Hi @Viktorija ,

No I'm not using Docker Desktop. I'm running Docker on Ubuntu.

#4

I closed all the other applications on my computer and was able to get the portal to work. I monitored the memory usage of the services and noticed that their memory usage was climbing slowly over time. It went from 1.5gb initially to 5gb and then started filling my swap.

The spark-service was logging this line:

spark-service | Re-format filesystem in Storage Directory root= /hadoop/hdfs/data/dfs/namenode; location= null ? (Y or N) Invalid input:

#5

Hi @Zander_Giuffrida,

Thanks for this information, I'm glad you got your portal back again! I'll try to find out what's going on and will let you know my findings.

#6

Hi Zander!

We reproduced the issue you are running into and created a story to fix it. Thank you! I will let you know the progress.

1 Like
#7

Hey @Viktorija could you send me an invite to the slack channel? Thanks!

#8

We sent you an invite @Zander_Giuffrida :slight_smile:

#9

Hi @Zander_Giuffrida!

Our developers suggest to run this command before starting Compose Services:

sudo sysctl -w vm.max_map_count=262144

This would help to start esproxy-service. Could you try this and let me know the result?

#10

Hey @Viktorija,

Yes that seems to be working. I'll let you know if I run into any other issues.

Thanks!

Zander

1 Like
#11

Hello @Viktorija,

I've been trying to get data upload to work on my local gen3 installation, and ran into some issues similar to what's described in this post:

I added the self-signed certificate to trusted certificates on my OS (Ubuntu). After a few tries I was able to configure a profile using the following command:
gen3-client configure --profile=zander --cred=~/compose-services/credentials.json --apiendpoint=https://localhost

When I run
gen3-client auth --profile=zander
it returns:
Screenshot from 2020-02-04 13-42-17

However, when I run
gen3-client upload --profile=zander --upload-path=~/Documents/text.txt
I get the following error:

Screenshot from 2020-02-04 13-47-35
Screenshot from 2020-02-04 13-47-35918×42 7.44 KB

Could you help me resolve this issue?

Thanks!

Zander

#12

File submission is not fully working in Compose Services, but it should not give you Internal Server Error. Could you share your logs?

Also I wonder if you are able to create program/project in your commons like described here

#13

Hello @Viktorija. Sorry for the delay in my response!

It seems that I'm unable to get the portal to show up on localhost again. Here's a link to the zip I generated from the dump.sh script.

#14

Hi @Zander_Giuffrida!

Looks like fence is not happy trying to reach AWS buckets and getting errors. I will consult with developers on this and let you know. Do you experience the same with docker-compose down and then docker-compose up -d ?

#15

Hi @Zander_Giuffrida!

Could you please try this:

  1. Shutdown compose services with docker-compose down
  2. Comment out these lines in the ngninx.conf:
#        location /guppy/ {
#                proxy_pass http://guppy-service/;
#        }
  1. Modify the AWS credentials block and add aws_access_key_id: '' and aws_secret_access_key: '' to credentials in the fence-config.yaml
AWS_CREDENTIALS:
  'CRED1':
    aws_access_key_id: ''
    aws_secret_access_key: ''
  'CRED2':
    aws_access_key_id: ''
    aws_secret_access_key: ''
S3_BUCKETS:
  bucket1:
    cred: 'CRED1'
  bucket2:
    cred: 'CRED2'
  bucket3:
    cred: '*'
  bucket4:
    cred: 'CRED1'
    role-arn: 'arn:aws:iam::role1'
  1. Start docker compose with docker-compose up -d

Please let me know the result :slight_smile:

#16

Hey @Viktorija

I'm able to get the login screen to come up, but when I try to login it throws an Internal Server Error.

Here's a link to the logs

#17

Hi @Zander_Giuffrida,

From your new logs I see Fence complains about AWS buckets and fence-config does not contain aws_access_key_id and aws_secret_access_key fields in the AWS_CREDENTIALS block. Could you please modify fence-config.yaml as described in Step 3 in the post above and see if it helps?

#18

Hey @Viktorija

Those fields were already present in fence-config.yaml. Here's a screenshot of it.

screenshot
screenshot844×494 46.5 KB

#19

It's interesting that fence didn't pick changes up. Did you change fence-config.yaml in the templates folder or in Secrets folder? The change should go to Secrets/fence-config.yaml

#20

I changed Secrets/fence-config.yaml

#21

Do I understand correctly, if you re-run docker compose with modified fence-config.yaml on your new logs you don't see your changes to fence-config.yaml? I will check with developers what's happening.