feat(serverless): support forking checkpoints from a different W&B entity#741
feat(serverless): support forking checkpoints from a different W&B entity#741AnnaSuSu wants to merge 1 commit into
Conversation
|
any chance of a look here? adds support for forking checkpoints from a different W&B entity. it's mergeable and ready, glad to adjust if you'd want it done differently. |
…tity ServerlessBackend._experimental_fork_checkpoint built the source artifact path from the destination model's entity (model.entity or api.default_entity), so a checkpoint could only be forked within the same W&B entity. Forking e.g. willow-voice/willow_normal/kl-000-1 into wb-training/... failed because it looked for the artifact under wb-training. Add an optional from_entity parameter and resolve the source entity as from_entity -> model.entity -> api.default_entity via a small pure helper (_wandb_checkpoint_collection_path) that also raises a clear error when no entity is available. Re-implements the approach validated in OpenPipe#676. Closes OpenPipe#649
4effdff to
b351609
Compare
|
rebased onto current main (88fad83) — the conflict is resolved and the PR is mergeable again. on the conflict itself: #739 swapped the serverless backend's the rebase did require one real change. the test used to stub W&B by injecting a fake module into monkeypatch.setattr("art.serverless.backend.wandb_sdk.api", FakeApi)verified locally on the rebased branch:
@bradhilton the workflows on this head are sitting at |
What
ServerlessBackend._experimental_fork_checkpointbuilds the source W&B artifact path from the destination model's entity:So a checkpoint can only be forked within the same entity — forking e.g.
willow-voice/willow_normal/kl-000-1intowb-training/willow_normal/my-new-runfails because it looks for the artifact underwb-training. The only workaround is to download from the source entity and re-upload to the destination, doubling storage.Closes #649.
Fix
Add an optional
from_entityparameter to_experimental_fork_checkpointand resolve the source entity asfrom_entity → model.entity → api.default_entity. The resolution + path building move into a small pure helper,_wandb_checkpoint_collection_path, which also raises a clearValueErrorwhen no entity can be determined (previously this produced a"None/…"path).This re-implements the approach from #676 (validated there but voluntarily closed by its author); credit to @poofeth for the original.
Tests
New
tests/unit/test_serverless_fork_checkpoint.py— no GPU/backend deps, runs on the base install:_wandb_checkpoint_collection_pathresolution: explicitfrom_entitywins → falls back to model entity → falls back to default entity → raises when none._experimental_fork_checkpointthrough the W&B branch with a fakewandb.Apiand asserts the artifact query uses the explicit source entity (("lora", "src-entity/src-project/src-model")).