The requested url cannot be found

Using Gen3
1

Hello.
I was trying to download some pdc data which as aligned WGS with gen3-client.
I success to configure gen3-client with Profile icgc.

$ gen3-client configure --profile=icgc --cred=credentials-pdc-20231022.json --apiendpoint=https://icgc.bionimbus.org/
2023/11/14 03:50:54 Profile 'icgc' has been configured successfully.

but I was failed download and I got following error

/home/dongjoo/manifest# gen3-client download-multiple --profile=icgc --manifest=gen3_manifest_manifest.test.sh.json
2023/11/14 03:58:16 Reading manifest...
 154 B / 154 B [===================================================================================================================] 100.00% 0s
WARNING: flag "rename" was set to false in "original" mode, duplicated files under "./" will be overwritten
Proceed? [y/n]: y
2023/11/14 03:58:18 Total number of objects in manifest: 1
2023/11/14 03:58:18 Preparing file info for each file, please wait...
 0 / 1 [------------------------------------------------------------------------------------------------------------------------------]   0.00%2023/11/14 03:58:19 Error occurred when querying filename from IndexD: 404 Not found error has occurred! The requested url "https://icgc.bionimbus.org/index/index/5ea6ddae-d51a-5c0b-acb7-741089d6ec37" cannot be found or one of the requested resources cannot be found
2023/11/14 03:58:19 Using GUID for filename instead.
 1 / 1 [===========================================================================================================================] 100.00% 0s
2023/11/14 03:58:19 0 files downloaded.successfully
2023/11/14 03:58:19 1 files have been renamed as the following:
2023/11/14 03:58:19 File "N/A" (GUID: 5ea6ddae-d51a-5c0b-acb7-741089d6ec37) has been renamed as: 5ea6ddae-d51a-5c0b-acb7-741089d6ec37
2023/11/14 03:58:19 1 files have encountered an error during downloading, detailed error messages are:
2023/11/14 03:58:19 Error occurred when getting download URL for object 5ea6ddae-d51a-5c0b-acb7-741089d6ec37
 Details of error: 404 Not found error has occurred! The requested url "https://icgc.bionimbus.org/user/data/download/5ea6ddae-d51a-5c0b-acb7-741089d6ec37" cannot be found or one of the requested resources cannot be found

The manifest is converted with dcc_to_gen3.py tool.

/home/dongjoo/manifest# python3 /home/pdc_tools/1.0/dcc_manifest_conversion/dcc_to_gen3.py --manifest manifest.test.sh
/usr/lib/python3/dist-packages/requests/__init__.py:89: RequestsDependencyWarning: urllib3 (2.0.7) or chardet (3.0.4) doesn't match a supported version!
  warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "
Counting PDC files in manifest: 'manifest.test.sh'
	1 PDC files (s3 locations) found in the DCC manifest.
Found 1 matching files in GUID map from the DCC manifest (1 total files).
	GUID: 8b2d4a85-0239-4205-b5d0-2dcba5abf7e4, location: s3://pcawg-tcga-ucec-us/5ea6ddae-d51a-5c0b-acb7-741089d6ec37 (1/1)
	Done (1/1). Guids for 1 files out of the 1 files listed in the DCC manifest written to gen3-client manifest:
		gen3_manifest_manifest.test.sh.json

I can't understand that the url was not found even though GUID map was matched. Do I need to figure out api endpoint and profile?? I've heard that it works with same flow in our lab last year.

2

Hi, thank you for reaching out to us with this issue. We've identified the problem and are looking into it now. We will follow-up once we've implemented a solution, but no further action on your end at this time is required.
Thank you for your patience!
Chris

3

Hello,
Could you try your download(s) again? I've checked the particular file/GUID mentioned in this post, and I am able to access it using the following command:

$ gen3-client download-single --profile=icgc --guid=8b2d4a85-0239-4205-b5d0-2dcba5abf7e4
WARNING: flag "rename" was set to false in "original" mode, duplicated files under "./" will be overwritten
Proceed? [y/n]: y
2023/11/27 12:58:44 Total number of objects in manifest: 1
2023/11/27 12:58:44 Preparing file info for each file, please wait...
 1 / 1 [==========================================================================================================================] 100.00% 0s
2023/11/27 12:58:44 File info prepared successfully
PCAWG.a0fee2f2-6fdb-4b5d-96fa-df9a4dc0688a.bam  40.11 MiB / 70.76 GiB [>----------------------------------------------------]   0.06% 1h12m08s
^C

Also, I believe you were doing this already, but make sure you use the dcc_to_gen3.py conversion script before trying to download any files. The object_id 5ea6ddae-d51a-5c0b-acb7-741089d6ec37 is specific to the DCC database. The same file has object_id 8b2d4a85-0239-4205-b5d0-2dcba5abf7e4 in the Gen3 system:
https://icgc.bionimbus.org/files/8b2d4a85-0239-4205-b5d0-2dcba5abf7e4

I hope this helps.
Best regards,
Chris

4

Hello,

Now it works well. Thank you for your support!