Asteroids! - #534
Open
mrosseel wants to merge 485 commits into
Open
Conversation
cache.pifinder.eu's chunk store moved from local disk to AWS S3 (bucket pifinder-nix-cache, us-west-1). The cutover used a fresh atticd DB, which regenerated both caches' NAR signing keypairs. Rotate the pinned dev key and wire in the now-provisioned pifinder-release key. pifinder: 8UU/...jBmE= -> Vkem...3gck= pifinder-release: (newly pinned) WG/F...KpoM= ATTIC_TOKEN is unchanged (RS256 server secret preserved), so CI keeps working once these land.
A push to main/nixos triggered the whole build pipeline (build-native → build-emulated → update-manifest → migration-tarball). By design mrosseel commits must build nothing; only labeled PRs (brickbots contributions) and pre-releases/releases (release.yml) should build. Drop the `push:` trigger and the event_name=='push' arms in build-native/native-wait; update-manifest still follows the build jobs, so it runs on labeled PRs.
…n fork PRs - software.py: read update-manifest.json from brickbots/PiFinder (canonical), not the mrosseel fork — devices were pointed at the wrong repo. - build.yml build-native: guard 'Push to Attic' on $ATTIC_TOKEN so fork PRs (no secrets) build verify-only instead of failing with AccessError, matching build-emulated. - build.yml update-manifest: drop 'ref: <head_ref>' checkout — a fork PR's head branch doesn't exist here, so it 404'd; default ref resolves for both.
A trusted build (secrets + write token) is the only thing that can push to the cache and write the nixos-manifest branch — a fork PR never can. Restore the push trigger but gate the build jobs on github.repository so it acts only in brickbots: a mrosseel push still builds nothing, while a merge to brickbots builds and publishes (attic push + update-manifest, whose fork-PR skip doesn't apply to pushes).
Adopt brickbots' canonical CI + the trusted testable-PR builder from brickbots#493 instead of the nixos branch's divergent workflows: + nixos-pr-build.yml (pull_request_target, label-gated, trusted) + nox.yml (brickbots standard lint/test) ~ web-integration-tests.yml (match brickbots main) - build.yml, lint.yml, release.yml (nixos-only; superseded) publish_manifest.sh / update_manifest.py already match brickbots#493.
…over rotation The cutover recreated the Attic cache with a fresh key (Vkem), but nothing deployed trusted it — the whole 8UU fleet got stranded. Attic can't dual-sign (serves only its own key), so the fix is to put the cache back on 8UU (done server-side) and revert the config to match. pifinder-release key kept (new cache, never previously trusted).
It holds always-on device config (avahi, hostname, substituters/keys, sudo), used long after the Debian→NixOS migration — the name was misleading. Update the flake import and the RELEASE.md reference. Pure rename, no content change.
…ostname nixos_upgrade.py: - Download progress now shows a size bar that moves *within* a path and names the package being copied, parsed live from nix's internal-json (resProgress byte events summed over copyPath activities; denominator = the dry-run 'unpacked' total, since Attic narinfos omit a compressed FileSize). All accounting is throttled and wrapped so a counter bug can never stall the stream or abort the upgrade; only a short log tail is kept (not the ~800k-line stream). Dropped the now-dead per-path path-info query, the size map, and write_sizes_file/UPGRADE_SIZES_FILE. - key-proof: fetch each cache's current signing key from its anonymous Attic cache-config endpoint and trust it for the pull (extra-trusted-public-keys; verification stays on) so a cache key rotation can't strand the fleet. networking.nix: own avahi here (the module shared by the running system and the migration build); fix the boot race (NM dispatcher re-scans avahi on connect) and make the PiFinder_data hostname stick — hostname-mode=none plus the dispatcher re-asserting hostname + avahi-set-host-name (NixOS bakes host-name=<static> into avahi's config, which a restart would otherwise revert). sys_utils.py / software.py: parse and show the package label under the bar.
…crashing
The external observing-lists feature added `list_descriptions` to
CompositeObject, but CACHE_VERSION was not bumped. Devices upgrading from the
prior release keep their existing composite_objects.pkl, whose unpickled
objects lack the new field (dataclass defaults are not applied on unpickle),
so opening any object's details crashes the whole app:
AttributeError: 'CompositeObject' object has no attribute 'list_descriptions'
The main process hosts the multiprocessing shared-state manager, so its death
cascades BrokenPipe/connection-reset into every worker — the symptom seen in
the logs; the real cause was masked because main()'s handler logs via the
multiprocess queue and then os._exit()s before the record is written.
- catalog_cache: bump CACHE_VERSION 1 -> 2 so pre-list_descriptions caches are
rebuilt on upgrade (the real fix for deployed devices).
- composite_object: getattr guard in composed_sections so a stale-cached object
degrades gracefully instead of taking down the process.
- main: print + flush the traceback before os._exit so a fatal exception in
main() lands in the journal instead of being lost to the log queue.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RjeCZ17KqhKzhKikWGwBDo
Camera switching only changed the persisted choice and ran switch-to-configuration boot, but the generic-extlinux builder always writes DEFAULT=nixos-default (the base camera) and device-tree overlays load only at boot. So a device set to imx477 kept booting the base imx462 DTB, the imx290 driver bound to absent hardware (Error writing reg 0x3038), and the camera never worked. Fix A keeps the one-image/no-rebuild specialisation design and just makes the chosen specialisation the boot default: - set-extlinux-default: fail-safe helper that repoints extlinux DEFAULT to the latest-generation nixos-<gen>-<camera> entry (base camera -> nixos-default); leaves a bootable DEFAULT untouched if the entry is missing. - pifinder-switch-camera: repoint DEFAULT to the chosen camera, then reboot (DT overlays are boot-only). - nixos_upgrade: re-apply the persisted camera's DEFAULT after activation, before the upgrade reboot (every rebuild resets DEFAULT to base). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RjeCZ17KqhKzhKikWGwBDo
The ADR said device update downloads ship only the genuinely-new chunks (~80 MB for a 1.5 GB closure). That conflates server-side storage / CI-upload dedup with the device download. Attic serves whole NARs over the standard binary-cache protocol: a device fetches the full compressed NAR of every changed store path, with no chunk-delta against the previous version. The only device-side saving is path-level (unchanged paths are not refetched). True client-side chunk-delta needs a casync/desync client with a local chunk store. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RjeCZ17KqhKzhKikWGwBDo
pifinder-src is rebuilt (new store hash) on every code change, and Attic ships whole NARs — so each change re-downloads ~43MB and rewrites ~70MB to the SD, even for a one-line edit. 56MB of that is stable: fonts (~31MB) and the pinned tetra3/cedar-solve solver (~25MB, ~15MB after trimming examples/tests/docs). Move both into their own derivations (like astro-data) and symlink them into pifinder-src. A routine code change now rewrites only the ~16MB code path; fonts/tetra3 are distributed once and shared across changes. tetra3 keeps cedar_detect_pb2 (ships in the repo) and is pre-compiled; it is symlinked after compileall so bytecode isn't written into the read-only store path. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RjeCZ17KqhKzhKikWGwBDo
Adds a 'Rollback' channel next to stable/beta/unstable that lists the on-disk system generations you can roll back to (all but the current one). - sys_utils.list_rollback_targets(): reads ONLY immutable generation data (the /nix/var/nix/profiles symlinks + store-path labels) — no sidecar JSON state to evolve or corrupt across up/downgrades. Entry = label + generation + date. - software.py: the Rollback channel is built locally, so it's available even when the manifest fetch fails — i.e. exactly when you're stranded on a bad build. Select -> confirm -> reuses update_software() (the entry's ref is the generation's store path, so it activates + reboots with no download). - nixos_upgrade.cleanup_old_generations: keep +3 (was +2) -> 2 rollback targets. Labels currently come from the store-path name; a follow-up can set system.nixos.label at build time for 'PR-379'-style names (and matching bootloader entries).
…able→beta→unstable)
Pin cedar-solve d8ff1d8 via uv2nix instead of the python/PiFinder/tetra3 git submodule. Removes the sys.path shims in solver/preview/utils, the pifinder-src tetra3 derivation, and .gitmodules. cedar-solve's stale numpy<2 / Pillow<9 caps are relaxed through uv dependency-metadata. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LMFdEnwncFyapQB38fiUHt
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LMFdEnwncFyapQB38fiUHt
…ive aarch64 runners)
…ration URL, artifact fix)
…orrect SSID encoding
…acing bash wifi-fallback
… generations, capture evidence, announce rollback
…not version string
…anifest terminology (ADRs 0004-0006)
…ions protect themselves
…DR 0004), Type=notify readiness - release.yml builds images.pifinder-migration and cuts the tarball from it - SD image seeds /var/lib/pifinder/current-build.json with its own store path; version labels resolve via the update manifest; pifinder-build.json deleted - pifinder.service Type=notify: app sends READY=1 once the UI is constructed and drawing (dev-safe no-op without NOTIFY_SOCKET) — the watchdog health check now means 'the screen is live'
Bring the per-frame optical-black pedestal (IMX290/462), its docs/ADR, and the runtime source-closure slimming onto the observable-asteroids branch. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015EWb2JET2JE1t4ya3HZPNL
…stal Gate BlackLevelTracker creation behind SQM_BLACK_LEVEL_TRACKER_ENABLED (default False) so the radiometer pedestal is OB when present, else the static profile bias_offset — no tracker in between. Lets us observe the optical-black pedestal's effect in isolation during rollout/validation. Reversible: flip the flag (or revert) to restore the tracker fallback. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015EWb2JET2JE1t4ya3HZPNL
All aperture/annulus/wing radii are defined for the ~1.0-scale Bayer-green photometry images (imx462 green: 490px vs the 512 solve image). The mono imx296 does photometry on its full-res 1088px frame (scale 2.125), where the fixed radii sit inside the star profile: the r=5 aperture holds only 83-94% of a star's flux (measured growth curves, 2026-07-18 sweeps), every local background annulus lands on the PSF itself, and the wing estimator's sky ring eats the wings so it measures f~1 and stays permanently inert. Convert all radii from solve-image pixels to the photometry image's pitch (_scaled_photometry_radii) and give WingEstimator a set_scale() that rescales its patch geometry and clears the rolling window on change. Both the production path (solver.update_sqm) and the calibration wizard's SQM preview use the shared helper. Effect by sensor: - imx462/imx290: radii round to (5, 10, 17) - essentially unchanged. - imx296: radii become (11, 21, 38); on the 2026-07-18 reference sweeps this moves the median error vs a hand-held meter from -0.20 to +0.00 and lets the estimator track real focus drift. - hq: also gets larger radii; its sqm_band_offset (like imx296's) should be refit against reference sweeps after this change. Validated: 121 tests pass across test_sqm, test_solver_sqm, test_sqm_calibration, test_sweep_frame_record, test_solver_shmem (9 new tests cover the scaling).
The sensor's OB clamp pins raw black to a target that moves with sensor state: on the 2026-07-18 imx296 reference sweeps the delivered level was 55.9-56.3 ADU while the device's wizard calibration (bias 58) and the profile constant (60) both over-subtracted. At dark-site signal levels a 2-4 ADU over-subtraction produces a strong SQM-vs-exposure slope (-0.9..-1.5 mag/decade) and kills short exposures outright (background - pedestal <= 0, 'background unresolved'). The BlackLevelTracker already measures the in-session intercept but was suppressed whenever a wizard calibration existed - precisely backwards: a stored bias is one moment's measurement of a moving level. Flip the priority in both SQM paths (radiometric and stellar): a confidently fitted tracked intercept wins for bias; the wizard's dark-current rate remains authoritative and is added on top (the intercept fit cannot separate dark from sky). The tracker's own stderr / deviation-band / lease gates bound how far it can stray, and an accepted fit >2 ADU from the static bias now logs at info. Replayed against the 2026-07-18 imx296 sweeps (radiometric feed, wizard anchor 58): tracker conditions on all four (stderr 0.26-0.32), fits 55.9-56.3, and the recomputed SQM exposure slopes collapse from -0.9..-1.5 to -0.01..+0.52 with the residual offset a tight constant (sd 0.04) for a later band-offset refit. Also: the published black_level_tracked flag now reflects the leased pedestal actually used, not the raw last fit. Validated: 136 tests pass across the SQM suites (4 updated/new).
Rich's 2026-07-18 calibration reports (imx296 + imx462, wizard run interleaved with the reference sweeps) show two capture defects: 1. The first ~3 frames of every stage report actual_exposure_us at the PREVIOUS exposure (req=1us, actual=999999us) yet pass the frame gate, because _capture_and_wait checks the exposure_time field, which echoes the committed setting rather than what the sensor delivered. Stale full-length frames land in the bias/dark stacks. 2. After the exposure change lands, the sensor's optical-black clamp re-settles to a new level over ~3 more frames (imx296: 56 -> 59 ADU). Frames in that window measure neither the old nor the new black level. (On the imx296 the settled clamp is genuinely exposure-dependent: ~56 ADU at 1s vs ~59 ADU at the 29us minimum - a min-exposure bias is only valid for its own regime, which is why the tracked in-session pedestal takes priority in operation.) Together these explain the wizard bias of 58 on a sensor whose 1s clamp level is 56: a median over stale-1s frames at 56, settling frames, and settled-29us frames at 59. Fix: _capture_and_wait gates on actual_exposure_us (exposure_time only as fallback), re-requests a capture when a stale frame arrives, and uses an exposure-proportional tolerance; bias and dark stages discard CLAMP_SETTLE_DISCARD_FRAMES frames after every exposure change (the dark ramp changes exposure per step). Adds ~30-40s to a wizard run. Evidence: dumps/logs/rich-imx296/sqm_calibration_report_imx296_*.json. Validated: 141 tests pass across the SQM suites (5 new).
feat(sqm): per-frame optical-black pedestal for IMX290/IMX462 (zero-touch)
mrosseel
added
testable
Telemetry menu under Settings > Telemetry: - Record: inline On/Off toggle (no submenu), same pattern as Test Mode. - Sections: one multi-select checklist with inline checkmarks (like the Catalogs filter) for what to record — IMU, SQM, Solves, Targets and Images. Toggling applies live to an in-progress recording. - Max Size: session cap (250 MB / 500 MB / 1 GB / 2 GB / Unlimited). - Load: unchanged. Images ships OFF by default, preserving the old telemetry_images=false: one 512x512 PNG is written per solve, so at ~1 solve/s it costs roughly 300 MB/hour against a few MB/hour for every other section combined. Session size cap. Frames are written by the camera process, so the recorder measures the session directory from its flush loop rather than counting its own writes. On reaching the cap, frame capture is suspended (the one unbounded consumer) while the event log keeps running, so a capped session stays useful instead of going dark. Default 1 GB. Also logs the SQM *ingredients*, not just the end products, so a session stays recomputable if the SQM/airglow maths changes in code later: - radio event gains the raw per-frame inputs — red/blue Bayer backgrounds, optical-black pedestal, photometry image size (green bg and exposure were already there). - the session header snapshots the FULL camera profile constants plus the airglow calibration and camera_type — the constants those ingredients were produced under. The derived published 'sqm' and applied 'floor' are still logged, but explicitly as a what-the-device-showed audit trail, not ground truth. The airglow import is optional: the model is part of the SQM stack only on branches that carry it (deepchart has the radiometer but not airglow), and there a session simply records no airglow constants.
Telemetry menu under Settings > Telemetry: - Record: inline On/Off toggle (no submenu), same pattern as Test Mode. - Sections: one multi-select checklist with inline checkmarks (like the Catalogs filter) for what to record — IMU, SQM, Solves, Targets and Images. Toggling applies live to an in-progress recording. - Max Size: session cap (250 MB / 500 MB / 1 GB / 2 GB / Unlimited). - Load: unchanged. Images ships OFF by default, preserving the old telemetry_images=false: one 512x512 PNG is written per solve, so at ~1 solve/s it costs roughly 300 MB/hour against a few MB/hour for every other section combined. Session size cap. Frames are written by the camera process, so the recorder measures the session directory from its flush loop rather than counting its own writes. On reaching the cap, frame capture is suspended (the one unbounded consumer) while the event log keeps running, so a capped session stays useful instead of going dark. Default 1 GB. Also logs the SQM *ingredients*, not just the end products, so a session stays recomputable if the SQM/airglow maths changes in code later: - radio event gains the raw per-frame inputs — red/blue Bayer backgrounds, optical-black pedestal, photometry image size (green bg and exposure were already there). - the session header snapshots the FULL camera profile constants plus the airglow calibration and camera_type — the constants those ingredients were produced under. The derived published 'sqm' and applied 'floor' are still logged, but explicitly as a what-the-device-showed audit trail, not ground truth. The airglow import is optional: the model is part of the SQM stack only on branches that carry it (deepchart has the radiometer but not airglow), and there a session simply records no airglow constants.
Telemetry menu under Settings > Telemetry: - Record: inline On/Off toggle (no submenu), same pattern as Test Mode. - Sections: one multi-select checklist with inline checkmarks (like the Catalogs filter) for what to record — IMU, SQM, Solves, Targets and Images. Toggling applies live to an in-progress recording. - Max Size: session cap (250 MB / 500 MB / 1 GB / 2 GB / Unlimited). - Load: unchanged. Images ships OFF by default, preserving the old telemetry_images=false: one 512x512 PNG is written per solve, so at ~1 solve/s it costs roughly 300 MB/hour against a few MB/hour for every other section combined. Session size cap. Frames are written by the camera process, so the recorder measures the session directory from its flush loop rather than counting its own writes. On reaching the cap, frame capture is suspended (the one unbounded consumer) while the event log keeps running, so a capped session stays useful instead of going dark. Default 1 GB. Also logs the SQM *ingredients*, not just the end products, so a session stays recomputable if the SQM/airglow maths changes in code later: - radio event gains the raw per-frame inputs — red/blue Bayer backgrounds, optical-black pedestal, photometry image size (green bg and exposure were already there). - the session header snapshots the FULL camera profile constants plus the airglow calibration and camera_type — the constants those ingredients were produced under. The derived published 'sqm' and applied 'floor' are still logged, but explicitly as a what-the-device-showed audit trail, not ground truth. The airglow import is optional: the model is part of the SQM stack only on branches that carry it (deepchart has the radiometer but not airglow), and there a session simply records no airglow constants.
pull_request_target defaults actions/checkout to the repository's default
branch. That branch carries this workflow so the event fires, but not
necessarily .github/scripts/, so update-manifest died with
bash: .github/scripts/publish_manifest.sh: No such file or directory
after a successful 19-minute build -- the image was pushed to Attic but no
device could ever see it, because the manifest was never written.
Check out the PR's base branch instead: it is the branch this workflow file
itself came from, so the scripts are always beside it, and it stays
maintainer-controlled. Checking out the PR head here is not an option -- this
job holds contents:write and the Attic token.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds an observable asteroids dynamic catalog, mirroring the existing comet/planet catalogs.
Soft00Bright.txt(bright numbered asteroids), fetched viadownload_utils.Note on scope/diff size: this branch is stacked on
deepchart→nixos→sqm, so the diff againstmainincludes that underlying work (same as the deepchart/nixos PRs). The asteroid-specific change is ~35 files (asteroids.py,asteroid_catalog.py, catalog wiring, tests).Opened to build a testable NixOS image and land the build in the
unstableupdate channel.🤖 Generated with Claude Code
https://claude.ai/code/session_01VaxurSrCzxrsfJrSjfA5bo