adding download script, goldens and validation_config, updated mapping for latest data - #2136
adding download script, goldens and validation_config, updated mapping for latest data#2136smarthg-gi wants to merge 2 commits into
Conversation
…g for latest data
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces a dynamic data downloader and preprocessor (download.py) for the NCES employed college graduates dataset, replacing a hardcoded URL in manifest.json. It also adds golden data files and a validation configuration to verify the imported data, and simplifies the TMCF mapping. The reviewer recommended increasing the retry attempts to 10 when resolving the URL and downloading the file to ensure robust downloads from the external government website.
| def resolve_url(landing_url=LANDING_PAGE_URL, | ||
| file_pattern=FILE_PATTERN, | ||
| headers=None, | ||
| tries=3, |
There was a problem hiding this comment.
According to the repository's general rules, when downloading data from unstable or unreliable external sources (such as this external government website), we should configure a higher number of retries (e.g., tries=10) to ensure download success. Please update the default tries parameter in resolve_url to 10.
| tries=3, | |
| tries=10, |
References
- When downloading data from unstable or unreliable external sources, configure a higher number of retries (e.g., tries=10) to ensure download success, even if it results in a long cumulative wait time.
| logging.error("Failed to resolve URL from landing page.") | ||
| sys.exit(1) | ||
|
|
||
| if not download_file(resolved_url, OUTPUT_FOLDER, False, None): |
There was a problem hiding this comment.
According to the repository's general rules, when downloading data from unstable or unreliable external sources, we should configure a higher number of retries (e.g., tries=10) to ensure download success. Please pass tries=10 to the download_file call.
| if not download_file(resolved_url, OUTPUT_FOLDER, False, None): | |
| if not download_file(resolved_url, OUTPUT_FOLDER, False, None, tries=10): |
References
- When downloading data from unstable or unreliable external sources, configure a higher number of retries (e.g., tries=10) to ensure download success, even if it results in a long cumulative wait time.
1 participant
No description provided.