Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,13 @@ All notable changes to the [Nucleus Python Client](https://github.com/scaleapi/n
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## [0.21.0](https://github.com/scaleapi/nucleus-python-client/releases/tag/v0.21.0) - 2026-08-15

### Added
- **`NucleusClient.merge_model_runs()`.** Merges two or more model runs into one new run holding the union of their predictions, leaving the sources untouched. A benchmark evaluation names a single model run and a benchmark's items may span datasets, so a model uploaded as several runs previously had no single run covering the benchmark — every uncovered item scored as a false negative. Merge first, then pass the new run to `create_benchmark_evaluation_v2()`.

The merge is a full union: predictions are copied, never deduplicated. Colliding `annotation_id`s are rewritten rather than dropped, and the response reports `predictions_copied`, `predictions_ignored` and `annotation_ids_rewritten`.

## [0.20.0](https://github.com/scaleapi/nucleus-python-client/releases/tag/v0.20.0) - 2026-08-11

### Added
Expand Down
62 changes: 62 additions & 0 deletions nucleus/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -145,8 +145,10 @@
MESSAGE_KEY,
METADATA_KEY,
METRIC_TYPE_KEY,
MODEL_ID_KEY,
MODEL_IDS_KEY,
MODEL_RUN_ID_KEY,
MODEL_RUN_IDS_KEY,
MODEL_TAGS_KEY,
MODEL_TRAINED_SLICE_IDS_KEY,
NAME_KEY,
Expand Down Expand Up @@ -526,6 +528,66 @@ def delete_model_run(self, model_run_id: str):
{}, f"modelRun/{model_run_id}", requests.delete
)

def merge_model_runs(
self,
model_run_ids: List[str],
name: str,
*,
model_id: Optional[str] = None,
metadata: Optional[Dict[str, Any]] = None,
) -> Dict[str, Any]:
"""Merge several model runs into one new run holding all their predictions.

A benchmark evaluation names a single model run, and a benchmark's items may
span several datasets. A model whose predictions were uploaded as separate runs
— one per dataset, or one per inference batch — therefore has no single run
covering the benchmark, and every uncovered item scores as a false negative.
Merging the runs produces one run that does cover it, which you can then pass to
:meth:`create_benchmark_evaluation_v2`.

The merge is a full union: predictions are copied, never deduplicated. If two
source runs predict on the same item with the same ``annotation_id``, the
colliding id is rewritten rather than dropped, and the count of rewrites comes
back in ``annotation_ids_rewritten``.

The source runs are left untouched.

Parameters:
model_run_ids: Two or more model run ids (``run_*``) to merge.
name: Display name for the merged run.
model_id: Model the merged run belongs to (``prj_*``). Defaults to the
sources' shared model; required when they belong to different models.
metadata: Optional metadata for the merged run. The merge always records
``merged_from_model_run_ids`` alongside whatever you pass.

Returns:
Dict describing the merge::

{
"model_run_id": str, # the new run
"source_model_run_ids": List[str],
"dataset_ids": List[str], # datasets the new run spans
"predictions_copied": int,
"predictions_ignored": int, # already present in the target
"annotation_ids_rewritten": int,
"errors": List[str],
}
"""
if len(set(model_run_ids)) < 2:
raise ValueError(
"merge_model_runs needs at least two distinct model run ids, got "
f"{sorted(set(model_run_ids))}"
)
payload: Dict[str, Any] = {
MODEL_RUN_IDS_KEY: model_run_ids,
NAME_KEY: name,
}
if model_id is not None:
payload[MODEL_ID_KEY] = model_id
if metadata is not None:
payload[METADATA_KEY] = metadata
return self.make_request(payload, "modelRun/merge")

def create_dataset_from_project(
self,
project_id: str,
Expand Down
1 change: 1 addition & 0 deletions nucleus/constants.py
Original file line number Diff line number Diff line change
Expand Up @@ -111,6 +111,7 @@
MODEL_TRAINED_SLICE_IDS_KEY = "trained_slice_ids"
MODEL_ID_KEY = "model_id"
MODEL_RUN_ID_KEY = "model_run_id"
MODEL_RUN_IDS_KEY = "model_run_ids"
MODEL_PREDICTION_ID_KEY = "model_prediction_id"
MODEL_PREDICTION_LABEL_KEY = "model_prediction_label"
NAME_KEY = "name"
Expand Down
2 changes: 1 addition & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ ignore = ["E501", "E741", "E731", "F401"] # Easy ignore for getting it running

[tool.poetry]
name = "scale-nucleus"
version = "0.20.0"
version = "0.21.0"
description = "The official Python client library for Nucleus, the Data Platform for AI"
license = "MIT"
authors = ["Scale AI Nucleus Team <nucleusapi@scaleapi.com>"]
Expand Down