Download Script Fix + PVmap Changes in Mongolia Imports (Mongolia_Education, Mongolia_Health, Mongolia_Employment, Mongolia_Demographics - #2128
Conversation
Code fix unenergy
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request updates the Mongolian demographics and education import scripts. It refactors common_download_script.py to dynamically resolve table paths and fetch metadata (English and Mongolian) from the API, mapping the POST response data to the expected format. It also extends observation dates up to 2030 in several mapping files and updates place resolution configurations. Feedback focuses on optimizing performance by caching API catalog requests and pre-building lookup dictionaries to avoid inefficient .index() lookups in loops. Additionally, it is recommended to increase the retry count for robust downloads and strip whitespaces from keys to normalize place names in places_resolved.csv.
kartik-s21
changed the title
| Male,#ignore,skip,,,,,,,, | ||
| Female,#ignore,skip,,,,,,,, |
There was a problem hiding this comment.
any particular reason we are skipping gender ?
| Veterinary medicine,bachelorsDegreeMajor,VeterinaryMedicineMajor,,,,,, | ||
| "Agriculture, forestry, fisheries",bachelorsDegreeMajor,AgriculturalEngineering__ForestryEngineering__Fisheries,,,,,, | ||
| Health and social protection,bachelorsDegreeMajor,HealthRelatedMajor,,,,,, | ||
| Health and welfare,bachelorsDegreeMajor,HealthRelatedMajor,,,,,, |
There was a problem hiding this comment.
lets retain mapping for "Health and social protection" previous key as well
| Professional fields,schoolGradeLevel,TertiaryEducation,populationType,Student,observationAbout,country/MNG | ||
| Of which: Female,gender,Female,schoolGradeLevel,TertiaryEducation,, | ||
| Bachelor¹,enrollmentLevel,BachelorsDegree,schoolGradeLevel,TertiaryEducation,#Header,enrollmentLevel | ||
| Bachelor,enrollmentLevel,BachelorsDegree,schoolGradeLevel,TertiaryEducation,#Header,enrollmentLevel |
There was a problem hiding this comment.
lets retain previously mapping as well Bachelor¹
saanikaaa
left a comment
saanikaaa
left a comment
There was a problem hiding this comment.
Pls add differ results with the latest changes
saanikaaa
left a comment
saanikaaa
left a comment
There was a problem hiding this comment.
pls add output_counter
2 participants
This PR addresses data validation failures, point deletions, and execution timeouts across all four Mongolia imports (mongolia_education, mongolia_demographics, mongolia_health, and mongolia_employment).
With these fixes, all four datasets generate cleanly with 0.00% point deletions and execute without API network timeouts or connection dropouts.
Key Changes:
Common Infrastructure (common_download_script.py)
Updated table extraction, parsing logic, and retry formatting for Education, Demographics, and Health datasets to prevent data truncation and column shifts during downloading.
Mongolia Education (mongolia_education)
Added explicit date mappings for survey years 2024 and 2025 across all 6 education PVMAP tables.
Updated run.sh and place mappings to cleanly handle regional groupings, resolving all historical point deletions (+350 newly recovered observation points).
Mongolia Demographics (mongolia_demographics)
Fixed 19.35% record deletion in marital status statistics (resident_population_by_agegroup_15_and_over_and_maritalstatus_pvmap.csv) by re-enabling the missing Age Group:Total mapping.
Added explicit column key prefixes (Marital Status:, Age Group:, Gender:) to prevent ambiguous "Total" matches from creating duplicate observation conflicts.
Mongolia Health (mongolia_health)
Corrected column extraction and monthly time-series parsing logic for births, deaths, infant mortality, and child delivery tables, eliminating all dropped historical observation dates through 2025.
Mongolia Employment (mongolia_employment)
Fixed Cloud Batch container timeout (InvalidDCInstanceError) caused by over 100,000 fallback API calls during place resolution.
Regenerated places_resolved.csv with 134 exhaustive mappings that account for leading whitespace hierarchy indentation.
PR Checklist - link