Add validation config, goldens, counters, and existing statvar mcf for statistics_poland

[abhishekjaisw]

Description

This PR configures validation, golden verification sets, and output counters for the statistics_poland statvar import.

Changes Included:

1. Manifest Configuration (statvar_imports/statistics_poland/manifest.json):

   • Added --output_counters=counters/StatisticsPoland_output_counters.csv to the processor script invocation.
   • Added --existing_statvar_mcf=gs://unresolved_mcf/scripts/statvar/stat_vars.mcf to reuse existing resolved StatVars.
   • Added "counters/*.csv" and "golden_data/*.csv" to source_files so outputs and goldens are preserved and uploaded to GCS.
   • Linked "validation_config_file": "validation_config.json".
   • Enabled invoke_import_validation, invoke_import_tool, invoke_differ_tool, and set "use_autopush_dc_api": false in config_override.
   • Configured resource limits (8 CPU, 32 GB RAM, 100 GB Disk).

2. Golden Files & Validation Config:

   • Created statvar_imports/statistics_poland/validation_config.json with GOLDENS_CHECK and DELETED_RECORDS_PERCENT rules.
   • Generated golden_data/golden_summary_report.csv and golden_data/golden_observations.csv from verified prod runs.

3. Metadata & Script Updates:

   • Added recon_property,"variableMeasured,measurementMethod" in StatisticsPoland_metadata.csv to filter reconciliation to relevant variables and avoid excessive DC API queries on numeric observations.
   • Updated download_input_data.py to address pandas groupby transpose compatibility and removed /tmp fallback on PermissionError.
   • Updated README.md documentation with new CLI flags.

Testing:

• Ran full Cloud Batch execution successfully on Dev (datcom-infosys-dev).
• Formatted and validated with ./run_tests.sh -l.

[gemini-code-assist]

Code Review

This pull request updates the Poland Census import configuration, adds validation configurations with golden data, and introduces a recon_lookup_api flag in the schema reconciler. The review feedback highlights that changing the default value of recon_lookup_api to False globally will break other imports, so it should remain True by default. Additionally, the reviewer advises against silently falling back to /tmp/source_files on permission errors in the download script, and notes that counters/.csv and golden_data/.csv should not be included in the source_files list in manifest.json.