MiCASA Processing — Land Carbon Flux Pipeline

Scripts to take raw MiCASA Land Carbon Flux data (global 0.1° NPP, Rh, FIRE, FUEL fields from the MiCASA model v1) and process it into hourly 1° NEE / NBE products consumed by NOAA's CarbonTracker inverse-modelling system.

Quick start

Required environment variables:

export MAIL_USER=you@example.org
export BASE_DIR=/path/to/GFED-CASA/tree

Then drive a year through the pipeline:

# Full pipeline for one year
./run_year.sh 2026

# Near-real-time stream
./run_year.sh 2026 vNRT

# Skip stages whose inputs already exist
./run_year.sh 2026 v1 --skip-download

# Show what would run, without running it
./run_year.sh 2026 --dry-run

Stage skip flags: --skip-download, --skip-ingest, --skip-aggregate, --skip-piqs, --skip-diurnalize, --skip-daysplit. SBATCH stages are submitted with --wait so the driver blocks until completion.

See docs/PIPELINE.md for full configuration and stage details.

Repository layout

This repo has two branches with unrelated histories:

• main — active development. PCHIP fitter (default), the verify_v2 test suite (60 structural / sign / continuity / sanity checks), bake-off scripts, plus PIQS and MSS as selectable alternative fitters.
• legacy — historical MiCASA_v1 pipeline (classic PIQS only). Preserved for archival reproducibility; do not try to merge into main (separate git inits, unrelated histories).

Work on main. Use legacy only to reproduce a v1-vintage product.

What's where

Document	Contents
docs/V1_TO_V2_JUSTIFICATION.md	The case for switching from V1 to V2 — why V2 is worth adopting despite V1's long verified history (what you keep / gain / risk), backed by an auditable register of every change (behavior-preserving vs intentional improvement) with its verification
docs/PIPELINE.md	Versions (v1 vs vNRT), configuration env vars, flowchart, every program in the tree, data-layout reference, output provenance metadata, NetCDF input schema
docs/METHODOLOGY.md	PIQS / PCHIP / MSS fitter theory; diurnalization with ERA5; why NEE = Rh − NPP and not Rh − NPP − ATMC
docs/FITTER_COMPARISON.md	Full sub-monthly-fitter bake-off (PIQS/PCHIP/PPM/minmod/MSS/ATP), equations, scorecard, uncertainty; why PCHIP is the default
docs/DIURNALIZATION_ALTERNATIVES.md	Diurnal-redistribution survey + soil-temp / Lloyd-Taylor respiration prototypes (opt-in), with shadow-diff results
docs/PRODUCING_A_PIQS_UPDATE.md	Runbook: produce a PIQS-fitted product update on the V2 pipeline via run_year.sh --fitter piqs (incl. the global-refit / re-diurnalize-the-tail caveat)
docs/PROPOSALS.md	Architecture decision records: 18 numbered notes covering landed / proposed / rejected design changes, with rationale
CHANGELOG.md	Dated engineering entries: latent-bug sweep, performance tuning, ATMC integration arc, PCHIP promotion
README.notes	Historical author log (Pera, Jacobson, Weir) — kept for provenance

Verification

The tests/verify_v2.py script runs 60 checks across the pipeline output: schema, mass conservation across re-aggregation, sign-flip rates, polar-night clipping, biome-cell sanity, climate-signal consistency (NEE trend, El Niño anomaly, COVID impact), PCHIP fit invariants, diurnalize timing, v2-vs-v1 cross-product comparison (per-band annual NEE + global NBE carbon-budget context), output-provenance attributes, and a run-manifest / job-log audit.

# Run (requires WORK_DIR set + ERA5 meteo accessible via $CARBONTRACKER)
cd $WORK_DIR && python3 tests/verify_v2.py

It prints PASS=N FAIL=N WARN=N INFO=N (and flags any FAILs); it does not set a non-zero exit code.

License

U.S. Government Work. This is a work of the U.S. Government (NOAA Global Monitoring Laboratory) and is in the public domain in the United States under 17 U.S.C. § 105. See LICENSE.

Citation

If this pipeline supports your work:

• MiCASA Land Carbon Flux v1 — Brad Weir et al., NASA GSFC. DOI 10.5067/ZBXSA1LEN453; data catalog at https://earth.gov/ghgcenter/data-catalog/micasa-carbonflux-grid-v1
• CarbonTracker — https://gml.noaa.gov/ccgg/carbontracker/
• This processing code — cite this repository (U.S. Government work / public domain; attribution appreciated, not required)

Authors: Ash Pera, Andy Jacobson, Brad Weir. See README.notes for the historical author log.