Issue with Compose Services

Viktorija · February 20, 2020, 10:26pm

Do I understand correctly, if you re-run docker compose with modified fence-config.yaml on your new logs you don't see your changes to fence-config.yaml? I will check with developers what's happening.

Viktorija · February 21, 2020, 3:27pm

Oh, you are right! dump.sh script removes creds information for privacy reasons:

github.com

uc-cdis/compose-services/blob/f1d2be4c3c893bfecfc19ee54bb30d8ea2c9a689/dump.sh#L45


	mkdir -p $dirname/config/
fi


if $get_config; then
	echo "Copying config files"
	cp docker-compose.yml $dirname/config/
	cp Secrets/user.yaml $dirname/config/
	cp Secrets/*config.* $dirname/config/
	cp Secrets/*settings.* $dirname/config/


	# remove lines containing creds
	if [[ "$OSTYPE" == "linux-gnu" ]]; then
		sed -i "/key/Id" $dirname/config/*
		sed -i "/secret/Id" $dirname/config/*
		sed -i "/password/Id" $dirname/config/*
	elif [[ "$OSTYPE" == "darwin"* ]]; then # MacOS
		sed -i "" "/[Kk][Ee][Yy]/d" $dirname/config/*
		sed -i "" "/[Ss][Ee][Cc][Rr][Ee][Tt]/d" $dirname/config/*
		sed -i "" "/[Pp][Aa][Ss][Ss][Ww][Oo][Rr][Dd]/d" $dirname/config/*
	else
		echo "WARNING: did not remove lines with creds (unknown OS $OSTYPE)"

Zander_Giuffrida · February 21, 2020, 5:29pm

I've pulled the latest images but still get 'internal service error' when I try to login. Do you think it would be worthwhile to try a clean install?

Viktorija · February 21, 2020, 6:12pm

I would try
I asked our developers to look at your logs again, it might take some time and I'll let you know their opinion on what could break the work.

Zander_Giuffrida · February 24, 2020, 3:45pm

Unfortunately the clean install didn't work. Here are the latest logs.

Viktorija · February 24, 2020, 5:03pm

Thank you for new logs. Let me check with developers and Ill get back to you.

Viktorija · February 28, 2020, 8:18pm

Hi @Zander_Giuffrida,

I just want to share that we were able to reproduce this behavior. We are looking how to fix it. I'll keep you updated.

steviejacs · March 2, 2020, 1:41pm

Hi @Viktorija - I'm running into similar issues with compose services.

I've got the portal up and running, but I'm still getting a 500 error on the Fence service when I run the smoke script -- I get an Internal Service Error when I try to log-in.

Here's my dump link

Thanks!
Steve

Viktorija · March 2, 2020, 3:20pm

Hi @steviejacs! Welcome to the forum!

It looks the problem is on our side, we are working to fix it. I'll update you when have more information.

steviejacs · March 2, 2020, 5:38pm

@Viktorija super! in the meantime, would you mind sending me an invite to the slack channel?

thanks for the support!
s.

Viktorija · March 2, 2020, 8:58pm

@steviejacs, the invite was sent your way

Zander_Giuffrida · March 11, 2020, 4:28pm

Hey @Viktorija,

Here is a link to my logs with the latest docker images for compose services.

I will be testing out cloud-automation on AWS soon. I saw that you previously mentioned this site for help with making a budget. Are there any resources you'd recommend I look at for making a budget of cloud provider costs with gen3?

Also do you know if users have had issues with cloud-automation similar to what I'm experiencing with compose-services?

Thanks!

Zander

Viktorija · March 11, 2020, 7:22pm

Hi @Zander_Giuffrida and @steviejacs!

I'm sorry it took so long but we have a fix:

shutdown compose services with docker-compose down
update your repository with git pull
update images with docker-compose pull
change fence-config.yaml in your Secrets folder as shown here https://github.com/uc-cdis/compose-services/pull/58/files leaving an empty list for AWS_CREDENTIALS and S3_BUCKETS (we updated template for this file and in the new setup it will automatically create a fixed version in Secrets, but in your current setup you already have a copy of the old fence configuration under your Secrets folder so need to update it manually this time).
start Compose Services with docker-compose up

This should fix a fence error.

Viktorija · March 11, 2020, 7:31pm

Hi @Zander_Giuffrida!

I don't think I have other resources for making a budget at this point. I'll ask other people maybe they will suggest something and I'll share with you.

For Cloud Automation it is hard to predict, it could go easy or not, but my recommendation would be to raise any questions you have on the gen3_community slack channel - there is a lot of expertise from our developers and also from independent groups who already have this experience and you can get a good advice from them.

steviejacs · March 12, 2020, 9:14pm

Hi @Viktorija - I'm still running into the same issue

did this fix for others?

steviejacs · March 12, 2020, 9:22pm

Here's a link to the dump

Thanks again for the help!

Viktorija · March 12, 2020, 10:57pm

I will take a look tomorrow and update you. Did you pull new images? Thanks for the logs!

Viktorija · March 12, 2020, 11:11pm

Actually it is another error from fence:

OSError: missing public key file; expected file to exist: /fence/keys/2020-03-12T20:43:34Z/jwt_public_key.pem
[36mfence-service        |[0m unable to load app 0 (mountpoint='') (callable not found or import error)

Do you have jwt_public_key.pem file in your Secrets/fenceJwtKeys/2020-03-12T20:43:34Z folder? It should be created automatically in this step https://github.com/uc-cdis/compose-services#setting-up-credentials by creds_setup.sh script

steviejacs · March 13, 2020, 1:11pm

Odd - I ran the script (when i first set it up) but it didn't create the jwt_public_key.pem file...

just ran it again and still same issue with the new secrets folder.

Viktorija · March 13, 2020, 2:21pm

Hm, I can not reproduce this behavior on my setup. Also I downloaded fresh compose services and still can not reproduce it. Do you have openssl installed as mentioned here in dependencies? I'm asking because creds_setup.sh script uses openssl to generate keys, please see here.

Topic		Replies	Views
Proxy Failure Compose-Services Using Gen3	16	979	July 18, 2019
A rookie's journey to setup a compose services Using Gen3	3	596	September 22, 2020
Issue in elastic search while building testbed using compose-services Using Gen3	14	1296	February 3, 2020
Evaluating Gen3 on bare metal K8s Using Gen3	3	647	January 23, 2020
Error in 'docker-compose up' Using Gen3	10	1381	July 6, 2020

Issue with Compose Services

Related Topics