Harden 1.4 against frontend corruption + freeze/lock dependencies - #4094
Conversation
| sudo apt-get install --no-install-recommends --assume-yes shellcheck parallel | ||
| python -m pip install --upgrade pip | ||
| pip install poetry | ||
| pip install poetry==2.4.1 |
There was a problem hiding this comment.
Can we skip the python part since we don't have problems with and we are closer to a stable release ? I would limit the scope of this PR to just deal with the frontend.
There was a problem hiding this comment.
We can, but if we need to apply a backport in the future, there's no guarantee we won't introduce a bug.
I made sure we are pinning to the same dependency versions that are being used in the build today.
I'm far more comfortable pinning what we can before release.
There was a problem hiding this comment.
update: reran the dependency diff, got one drift, fixed now.
There was a problem hiding this comment.
The scope of this PR is quite big for a stable release.
It mixes lock of frontend and big changes in backend project management.
- Move projects to uv
- Move projects from setup.py to pyproject.
- Move projects to venv
- Remove yarn.lock ?? For what reason, even in master we have yarn.
Python projects are already dependency lock, if there is a place that we are not, we can just add such version.
There was a problem hiding this comment.
The non-split of those concerns into separate PRs is intentional. I want to address all before the stable release.
The diff can look scary, but almost all of it is just the lock files themselves, so it's not difficult to review carefully.
Python projects are already dependency lock, if there is a place that we are not, we can just add such version.
This is exactly what we are doing here: the Python dependencies are not pinned in practice (we "lock" packaging==20.4, but the image is shipping 26.2), and we have broken dependencies already, and nothing guarantees we won't have more while I'm writing this message:
(Yes, we don't use flask directly, it's brought transitively via connexion, but the point is: it is there and silently broken today. Which will be the next one?)
The practical effect on the built image is tight and we must work to make it under control, so we should spend effort in improving this rather than ignoring the problem.
You can run agents to inspect the differences before/after; we can adjust any detail if needed, but the dependency-locking before the release is a must alongside the anti-corruption measure.
Docker dive:
pip freezes:
pip-1.4-dev.txt
pip-branch.txt
thanks
e4346a0 to
8bf14cc
Compare
…ript 1.4.4-beta.14 shipped assets/VehicleSetupView.df0d3c8b.js containing UTF-16LE source map data instead of javascript, so /vehicle/setup never loaded on that release. Nothing in the build or CI noticed: gzip was valid and every other chunk was fine, the failure only showed up in the browser. Parse every .js and .js.gz in dist after compression, so a corrupt artifact aborts vite build and therefore both the frontend-tests job and the docker frontend-builder stage.
Align the frontend build image with a bun that can read the committed lockfile format; 1.0.3 ignored it and SIGSEGV'd on vite build.
Keep CI bun in sync with the Dockerfile frontend builder.
The committed lockfile predated three package.json changes, so it had no entry for @sentry/vite-plugin or vite-plugin-compression and still named the old gl-matrix fork owner. bun install therefore re-resolved and rewrote it on every build, and bun install --frozen-lockfile could not run at all. Resolves to the same 808 packages at the same versions that CI installs today, so this records current behaviour rather than changing it. Written by bun 1.3.14, which bun 1.0.3 cannot read, so this depends on the bun update.
Last updated in July 2024 and already listed in core/frontend/.gitignore. It has no entry for the plugins added since, still names the old gl-matrix fork owner, and nothing builds with yarn: CI and the Dockerfile both use bun. Anyone following it would install a different tree than the one that ships.
isort and mypy discover the root of each python project by listing setup.py, which every lib and service dropped when they moved to uv. Without this the hook still exits zero while mypy checks 2 directories instead of 18, so the strict typing of every core service silently stops running.
The pre-push hook and the CI python-tests job install the linters and test tooling with poetry, but the lock was ignored since f717dfe, so only the direct pins in pyproject.toml were fixed and every transitive dependency resolved fresh on each run. Tracking the lock finishes what 60c9e59 started, and makes poetry install refuse a lock that no longer matches pyproject.toml, so dependency changes have to be relocked and committed.
poetry.lock is written in lock-version 2.1, and an unpinned poetry is free to change both the resolver and the lock format under us, the same reason uv and bun are pinned for the image and frontend builds.
8bf14cc to
6f0072c
Compare
4 participants
This patch is an amalgamation of cherries from the following PRs, now targeting 1.4:
The other commits will be sent to
mastersoon.Closes #4091