Ingest pipeline upgrade

[akhileshh]

Add pychunkedgraph.pipeline — cross-branch chunk-batch core (ingest + meshing)

Adds a self-contained pychunkedgraph/pipeline/ package that runs chunk-grid workloads (ingest, meshing) as stock Kubernetes Indexed Jobs — no Redis/RQ, no scheduler. This is the worker side of the new GKE Autopilot orchestration in CAVEpipelines; each workload is a module entrypoint the pipeline image runs.

Layout

pychunkedgraph/pipeline/
  grid.py          # fixed-seed scatter (Feistel permutation): batch index -> scattered coords
  lock.py          # per-chunk Bigtable claim/done CAS (token-fenced; one writer per chunk)
  exit_codes.py    # SUCCESS / TRANSIENT / FATAL — mapped to the Job podFailurePolicy
  worker.py        # generic harness: run(make_processor) -> exit code
  ingest/          # dispatch (branch shim) + setup (table+meta) + worker (lock+heartbeat)
  meshing/         # meta (MeshConfig) + setup (mesh-metadata) + worker (marching cubes / stitching)

Entrypoints (run in the container)

python -m pychunkedgraph.pipeline.ingest              # ingest worker
python -m pychunkedgraph.pipeline.ingest.setup <id>   # create table + graph meta
python -m pychunkedgraph.pipeline.meshing             # mesh worker (L2 cubes / L>2 stitching)
python -m pychunkedgraph.pipeline.meshing.setup <id>  # one-shot mesh metadata

Workers read the Indexed-Job env contract (JOB_COMPLETION_INDEX, PCG_GRAPH_ID/LAYER/PERM_SEED/BATCH_SIZE), map the index to a scattered batch of chunk coords, and process each — ingest under a per-chunk lock, meshing idempotently (no lock). Graph state is read from Bigtable by graph id (no yaml/Redis at run time).

Design notes

• Self-contained: depends on the rest of the repo only for the actual chunk-compute functions (add_atomic_edges/add_layer, get_atomic_chunk_data/get_active_edges, meshgen.*) and core graph.* classes. Config and setup live in the package (minimal IngestConfig, inlined meta build — no IngestConfig/bootstrap from ingest/).
• Branch-portable: the only branch-specific file is pipeline/ingest/dispatch.py (the chunk-body shim). On pcgv3 it needs a variant using that branch's chunk bodies (called out in the file docstring + pipeline/README.md); everything else cherry-picks unchanged.
• Additive: no existing modules touched; pipeline is not auto-imported by the top-level package, so existing imports/tests are unaffected.

Tests

tests/test_pipeline_grid.py — pure unit tests for the permutation (bijection, invertibility, scatter, batch partition). Lock/worker are validated against the Bigtable emulator outside the committed suite, keeping the package importable on branches without that fixture.

Follow-up

Consumed by the CAVEpipelines PR (the pipeline CLI builds Indexed Jobs that run these entrypoints).