Skip to content

Use r2-downloader for cognata dataset #668

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 8 commits into from
Oct 9, 2025
Merged

Use r2-downloader for cognata dataset #668

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
431adc9
Initial commit for using r2-downloader for cognata
anandhu-eng Oct 8, 2025
129b8cd
Merge branch 'dev' into r2-migration-2
anandhu-eng Oct 8, 2025
0a14e79
Update download links for cognata r2
anandhu-eng Oct 9, 2025
c015935
Merge branch 'r2-migration-2' of https://github.com/mlcommons/mlperf-…
anandhu-eng Oct 9, 2025
81153e4
enable no same owner flag
anandhu-eng Oct 9, 2025
2aed291
fix order of arg in tar
anandhu-eng Oct 9, 2025
0e99f48
Update needs pat
anandhu-eng Oct 9, 2025
af68416
Merge branch 'dev' into r2-migration-2
anandhu-eng Oct 9, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
61 changes: 49 additions & 12 deletions script/get-preprocessed-dataset-cognata/meta.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,45 +38,56 @@ variations:
env:
MLC_DATASET_COGNATA_EXTRACTED_FOLDER_NAME: val_2d
MLC_DATASET_COGNATA_TAR_FILENAME: val_2d.tar.gz
validation,2d_obj_det,rclone:
env:
MLC_DOWNLOAD_URL: mlc-cognata:mlc_cognata_dataset/preprocessed_2d/<<<MLC_DATASET_COGNATA_TAR_FILENAME>>>
validation,2d_obj_det,r2-downloader:
env:
MLC_DOWNLOAD_URL: https://cognata.mlcommons-storage.org/metadata/val_2d.uri
calibration,2d_obj_det:
env:
MLC_DATASET_COGNATA_EXTRACTED_FOLDER_NAME: calib_2d
MLC_DATASET_COGNATA_TAR_FILENAME: calib_2d.tar.gz
calibration,2d_obj_det,rclone:
env:
MLC_DOWNLOAD_URL: mlc-cognata:mlc_cognata_dataset/preprocessed_2d/<<<MLC_DATASET_COGNATA_TAR_FILENAME>>>
calibration,2d_obj_det,r2-downloader:
env:
MLC_DOWNLOAD_URL: https://cognata.mlcommons-storage.org/metadata/calib_2d.uri
validation,segmentation:
env:
MLC_DATASET_COGNATA_EXTRACTED_FOLDER_NAME: val_seg
MLC_DATASET_COGNATA_TAR_FILENAME: val_seg.tar.gz
validation,segmentation,rclone:
env:
MLC_DOWNLOAD_URL: mlc-cognata:mlc_cognata_dataset/preprocessed_seg/<<<MLC_DATASET_COGNATA_TAR_FILENAME>>>
validation,segmentation,r2-downloader:
env:
MLC_DOWNLOAD_URL: https://cognata.mlcommons-storage.org/metadata/val_seg.uri
calibration,segmentation:
env:
MLC_DATASET_COGNATA_EXTRACTED_FOLDER_NAME: calib_seg
MLC_DATASET_COGNATA_TAR_FILENAME: calib_seg.tar.gz
calibration,segmentation,rclone:
env:
MLC_DOWNLOAD_URL: mlc-cognata:mlc_cognata_dataset/preprocessed_seg/<<<MLC_DATASET_COGNATA_TAR_FILENAME>>>
calibration,segmentation,r2-downloader:
env:
MLC_DOWNLOAD_URL: https://cognata.mlcommons-storage.org/metadata/calib_seg.uri
prebuilt:
default: true
group: dataset-src
env:
MLC_NUSCENES_DATASET_TYPE: prebuilt
MLC_DOWNLOAD_FINAL_ENV_NAME: MLC_PREPROCESSED_DATASET_COGNATA_PATH
MLC_EXTRACT_FINAL_ENV_NAME: MLC_PREPROCESSED_DATASET_COGNATA_PATH
prebuilt,rclone:
env:
MLC_DOWNLOAD_EXTRA_OPTIONS: ' --include '
mlc:
group: download-src
default: true
prehook_deps:
- tags: get,rclone
enable_if_env:
MLC_TMP_REQUIRE_DOWNLOAD:
- yes
- tags: get,rclone-config,_config-name.mlc-cognata
force_cache: true
enable_if_env:
MLC_TMP_REQUIRE_DOWNLOAD:
- yes
env:
MLC_RCLONE_DRIVE_FOLDER_ID: 1u5FDoeXHVtDrd4zClE47Gmyr7iLFidz1
- enable_if_env:
MLC_TMP_REQUIRE_DOWNLOAD:
- 'yes'
Expand All @@ -92,6 +103,19 @@ variations:
- MLC_DOWNLOAD_URL
env:
MLC_DOWNLOAD_SRC: mlcommons
mlc,rclone:
prehook_deps:
- tags: get,rclone
enable_if_env:
MLC_TMP_REQUIRE_DOWNLOAD:
- yes
- tags: get,rclone-config,_config-name.mlc-cognata
force_cache: true
enable_if_env:
MLC_TMP_REQUIRE_DOWNLOAD:
- yes
env:
MLC_RCLONE_DRIVE_FOLDER_ID: 1u5FDoeXHVtDrd4zClE47Gmyr7iLFidz1
rclone:
group: download-tool
add_deps_recursive:
Expand All @@ -105,10 +129,23 @@ variations:
dry-run,rclone:
env:
MLC_DOWNLOAD_EXTRA_OPTIONS: --dry-run
r2-downloader:
group: download-tool
add_deps_recursive:
dae:
tags: _r2-downloader
dry-run,r2-downloader:
env:
MLC_DOWNLOAD_EXTRA_OPTIONS: -x
tests:
needs_pat: true
run_inputs:
- variations_list:
- validation,prebuilt,2d_obj_det,rclone,mlc,dry-run
- calibration,prebuilt,2d_obj_det,rclone,mlc,dry-run
- validation,prebuilt,segmentation,rclone,mlc,dry-run
- calibration,prebuilt,segmentation,rclone,mlc,dry-run
- calibration,prebuilt,segmentation,rclone,mlc,dry-run
- validation,prebuilt,2d_obj_det,r2-downloader,mlc,dry-run
- calibration,prebuilt,2d_obj_det,r2-downloader,mlc,dry-run
- validation,prebuilt,segmentation,r2-downloader,mlc,dry-run
- calibration,prebuilt,segmentation,r2-downloader,mlc,dry-run
2 changes: 1 addition & 1 deletion script/get-preprocessed-dataset-cognata/run.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
if [[ "$MLC_DOWNLOAD_MODE" != "dry" && "$MLC_TMP_REQUIRE_DOWNLOAD" = "yes" ]]; then
cd "${MLC_PREPROCESSED_DATASET_COGNATA_PATH}" || exit
for f in *.tar.gz; do
tar -xzvf "$f" || { echo "Failed to extract $f"; exit 1; }
tar --no-same-owner -xzvf "$f" || { echo "Failed to extract $f"; exit 1; }
done
cd - || exit
fi
Loading