Ngmix v2.0 (CI mirror of #740)

[cailmdaley]

What this is

A same-repo mirror of #740 (@martinkilbinger's "Ngmix v2.0"), pushed to a branch on CosmoStat/shapepipe so that CI actually runs. All 57 commits are authored by Martin (and carry Lucy Baumont's and Axel Guinot's work) — pushing the branch preserves that authorship unchanged; the only thing this PR adds is a same-repo head so GitHub Actions fires the pull_request workflow without the fork-PR approval gate. #740 received no CI runs at all for this reason.

Substance is identical to #740 — see that PR for the full description. In short: upgrade ngmix to 2.4.0 and adopt Lucy's new ngmix classes/interface; overhaul the shape-measurement module; centroid-bias fix + validation; v2.0 production-run plumbing.

Going forward, opening PRs directly on CosmoStat/shapepipe (rather than from a fork) avoids this — fork PRs don't trigger our Docker-image CI without a maintainer approval that wasn't happening.

Closes/supersedes #740 once CI is green (leaving that call to Martin).

Review

A detailed review is on its way (read against Martin's checklist plus a science-quality pass). Headline from exercising the new fitter against real ngmix 2.4.0 on candide: the metacal path runs end-to-end and shear recovery is unbiased at the few×10⁻⁴ level in m. Full notes to follow.

— Claude on behalf of Cail

[cailmdaley]

Review-round summary. Every open thread above now has a reply; here's the shape of what landed on ngmix_v2.0 since the review, and what still awaits your judgment.

On this branch:

• Reproducibility — seeded rng threaded through the noise draws (22f1f0a), guarded by test_metacal_is_reproducible_with_fixed_seed.
• Dead-code cleanup you greenlit (bd60dc8): CHECK_EXISTING_DIR resume path + get_last_id, the unreachable print, unused sextractor_e1e2, scripts/python/fitting.py, and the 51*51 literal (now v_flag_tmp.size). Dangling config entry pruned in 16789eb.
• Inverse-variance weights per [NEW FEATURE] Weight Handling #604 (f466c98 + 6ddea5b, 4aa3b2a, 158986a, f79de46, 0bc6016): optional BKG_RMS_VIGNET_PATH gives per-pixel 1/(Fscale·RMS)² with gating masks; scalar σ_MAD fallback hardened; the chain pinned end-to-end by an integration test. The double-weighting flagged in review is gone.
• Runner contract (d4ada3d, b2dcd79): sextractor_runner's decorator now declares the positional headers input; stale LOG_WCS docs fixed in both packages. No behavior change in any shipped config (all 12 override FILE_PATTERN).

Companion PRs (the r50/T thread):

• feat: size module — Gaussian size-conversion web (T, sigma, r50, FWHM) cs_util#65 — cs_util.size conversion web (T ↔ σ ↔ r50 ↔ FWHM), merged and released as cs_util 0.2.1 on PyPI.
• fix(ngmix): emit true half-light radii in r50 columns; dedupe PSF size columns #743 — honest half-light radii in every r50 column per the UNIONS convention, merged into this branch (d6531b1).
• fix: dimensionally wrong T_to_fwhm distorted the PSF-leakage size axis sp_validation#198 — adopts the cs_util converters and fixes a dimensionally wrong T_to_fwhm that distorted the PSF-leakage size axis (results through the old function should be regenerated). Open, blocked only by shear_psf_leakage's cs-util ^0.1.0 cap.

Awaiting your call (methodology, no code pushed): the *_psfo columns carrying the metacal-reconvolved PSF vs. the documented psfex/mccd input PSF; the galaxy prior/FLUX_AUTO guess reused for the PSF fit; and whether you want an explicit STAMP_SIZE option despite vignetmaker owning the geometry upstream. The zero-pixel any→all change stays as you intended; a fractional-bad-pixel threshold is noted as a possible follow-up.

— Claude on behalf of Cail

[cailmdaley]

Review — part 3 (fresh pass)

Provenance: same convention as parts 1–2 — this is a fresh full-diff pass by Claude working on candide, against head b2dcd79. Every finding below was empirically demonstrated before being fixed, each fix carries a regression test confirmed red on the unfixed code, and the chain is now pushed to this branch (b2dcd793..05f1584e). I haven't hand-edited it line by line, so please read with that lens.

Blockers (2)

Both live in the v2.0 rewrite's tile flow, and both survive a green suite and an easy-object smoke run — which is exactly why they hid: one sits in a path no test exercised, the other only fires when a fit fails. The fixes restore what we read as the v1 contract, but they deserve your sanity-check of intent, Martin.

1. TILE_ID silently truncates the post-process CCD scan (sextractor_script.py, make_post_process — fixed in fcd117f). The tile-level merge_headers (new on this branch) writes TILE_ID as the first key of log_exp_headers<tile>.sqlite; make_post_process derived n_hdu = len(f_wcs[key_list[0]]) — i.e. the length of the tile ID string instead of the CCD count. Every epoch on CCDs ≥ len(tile_string) silently vanishes from N_EPOCH/EPOCH_*, and from there out of ME vignets and shape measurement. Repro: tile "51" (len 2), 3 CCDs → an object on CCD 2 comes back N_EPOCH = 0 instead of 2. The fix filters the metadata key and sizes the scan from a real exposure entry; we audited every other log_exp_headers consumer (psfex_interp, vignetmaker, ngmix, mccd interp) — all do keyed [exp][ccd] access, so make_post_process was the only affected site, and it covers both its callers.

2. compile_results KeyErrors on any failed fit type, discarding the whole tile at save (ngmix.py — fixed in 450f3c1). Re-verified against real ngmix 2.4.0: a failed run_fitter returns normally with flags=512 and keys [errmsg, flags, ier, model, nfev, pars, pars_cov, pars_err] — none of g/g_cov/T/T_err/flux/flux_err/s2n. The old results[idx][name]["flux"] access then KeyErrors, and because it happens at batch/final save, one hard object throws away every measured object in the tile. v1 never saw this because ngmix 1.x raised on failure, which the per-object try/except caught. Now failed types are recorded with their flags and NaN-filled measurement columns (a flags == 0 result missing s2n still KeyErrors, deliberately).

Should-fixes (pushed; please confirm intent)

• mcal_flags was hardcoded to 0 (7a1b3b1). process() set res['mcal_flags'] = 0 unconditionally, so the NGMIX_MCAL_FLAGS column make_cat writes was constant zero and any mcal_flags == 0 quality cut passed everything. Restored to the v1 contract — bitwise OR of the per-type fit flags (new get_mcal_flags helper). Flagging explicitly: confirm this is the intended semantics for that column.
• Failed-PSF epochs crashed (or would have contaminated) the multi-epoch PSF average (00e9f89). With ignore_failed_psf=True a failed-PSF epoch stays in the obs dict carrying only flags/pars; average_multiepoch_psf read its ['T'] → KeyError, dropping the object. It now skips flags != 0 epochs, and n_epoch_model is again the count of epochs that survived the PSF fit (the v1 meaning) rather than the number submitted.
• Config drift, tutorial-facing (3eb6a66). The four example/cfis_simu configs driven by job_sp_simu.bash still used the removed LOG_WCS/ME_LOG_WCS options; they're migrated to the positional WCS-headers input contract, mirroring the migrated example/cfis equivalents. Same commit renames run_sp_exp_Mh → run_sp_tile_Mh_exp in five example/cfis configs (the run name config_tile_Mh_exp.ini actually produces) and adds explicit FILE_EXT where a 3-entry FILE_PATTERN override would trip the new 4-entry decorator default's length check at startup. Related sweep worth doing in this PR: example/cfis/config_exp_psfex.ini has the same latent FILE_PATTERN/FILE_EXT length mismatch.

Noted, not changed

• moments_fail semantic drift (document-only): in v1 the column counted moments-initial-guess (get_guess) failures; on this branch it counts metacal types with nonzero fit flags. Same name, different meaning — flagged so downstream consumers know.
• ntry_fit → nfev_fit (436bcc8): the value written is now ngmix-2.x's nfev (function evaluations), not the v1 retry count (1–5), so the old name was misleading. We verified the column is unconsumed downstream (make_cat never reads it; no reference anywhere in sp_validation), hence an honest rename rather than a compatibility shim. A small dead-code sweep rode along (05f1584): the orphaned copyfile import and the never-read Tile_cat.size/.e/.theta; get_noise was kept — it has a live caller in scripts/jupyter/test_centroid_shift.py.

Branch state, empirically

With the chain applied the container suite is 270 passed / 1 failed — the one failure is the known test_metacal_is_reproducible_with_fixed_seed, which needs ngmix 2.x the test sandbox doesn't have. Under real ngmix 2.4.0 we ran do_ngmix_metacal directly: seed-reproducibility holds, and an injected shear is recovered at g1 = 0.01996 vs 0.02 after response correction.

— Claude on behalf of Cail