nucleus.benchmark

Benchmarks — cross-dataset ground-truth item sets for model evaluation.

A benchmark is a named collection of dataset items (with ground truth) that model runs are evaluated against. Benchmark evaluations score every benchmark item: items a model run has no predictions for count as false negatives, so leaderboard scores stay comparable across runs with different coverage.

Create and manage benchmarks via NucleusClient:

benchmark = client.create_benchmark("city-streets-v1", slice_id="slc_...")
evaluation = benchmark.create_evaluation_v2(
    model_id=model.id,
    rollup_groups=[RollupGroup("vehicle", ["car", "truck"])],
)
evaluation.wait_for_completion()

A finalized benchmark is immutable. To evolve one, create a new version downstream of it with parent_benchmark_id (see NucleusClient.create_benchmark()) — the child inherits the parent’s items, adds/removes on top, and takes a minor (default) or major version bump.

To assemble a benchmark incrementally, create a draft (draft=True): add items across many calls with Benchmark.add_items() / remove with Benchmark.remove_items(), then Benchmark.finalize() to freeze it into a "ready" benchmark. A draft cannot be evaluated until finalized.

Benchmark

A benchmark: a frozen set of ground-truth items models are scored against.
class nucleus.benchmark.Benchmark

A benchmark: a frozen set of ground-truth items models are scored against.

add_items(*, item_ids=None, items=None, slice_id=None, dataset_id=None, slice_ids=None, dataset_ids=None, scene_ids=None, wait_for_completion=True, verbose=True)

Add items to this draft benchmark.

Only valid while this benchmark is a draft (status == "draft"); a finalized benchmark is immutable. See NucleusClient.add_benchmark_items() for parameter details.

Returns:

self, refreshed.

Parameters:

  • item_ids (Optional[List[str]])

  • items (Optional[List[Dict[str, str]]])

  • slice_id (Optional[str])

  • dataset_id (Optional[str])

  • slice_ids (Optional[List[str]])

  • dataset_ids (Optional[List[str]])

  • scene_ids (Optional[List[str]])

  • wait_for_completion (bool)

  • verbose (bool)

Return type:

Benchmark

create_evaluation_v2(model_run_id=None, *, model_id=None, name=None, rollup_groups=None, exclusion_rules=None, preset=None)

Evaluate a model against this benchmark.

Anchor the evaluation on a model (model_id) for the run-free “model v2” flow. The legacy model_run_id anchor is deprecated. Provide exactly one. Uncovered benchmark members are scored as false negatives, so partial coverage still ranks comparably.

See NucleusClient.create_benchmark_evaluation_v2() for parameter details.

Returns:

The created evaluation.

Return type:

EvaluationV2

Parameters:
delete()

Delete this benchmark.

Return type:

None

finalize()

Finalize this draft benchmark, freezing it into a "ready" one.

After finalizing, the benchmark is immutable and can be evaluated. Fails if it is not a draft, is empty, or still has an add-items job in flight.

Returns:

self, with updated fields (status now "ready").

Return type:

Benchmark

items(*, limit=None, offset=None)

Return one page of this benchmark’s member item ids.

Parameters:

  • limit (Optional[int]) – Optional page size.

  • offset (Optional[int]) – Optional offset for pagination.

Returns:

The page of dataset item ids and the total member count.

Return type:

BenchmarkItemsPage

refresh()

Reload this benchmark from Nucleus.

Returns:

self, with updated fields.

Return type:

Benchmark

remove_items(item_ids)

Remove items from this draft benchmark.

Only valid while this benchmark is a draft. Unknown ids are ignored.

Parameters:

item_ids (List[str]) – Dataset item ids (di_*) to remove.

Returns:

self, refreshed.

Return type:

Benchmark

update(*, name=None, description=None, metadata=None, rollup_groups=None, allowed_label_matches_id=_UNSET, class_agnostic=None, exclusion_rules=None)

Update this benchmark’s name, description, metadata, draft taxonomy, or draft exclusion rules.

Only the arguments you pass are changed. Benchmark membership is frozen at creation and cannot be updated. The taxonomy fields (rollup_groups / allowed_label_matches_id / class_agnostic) and exclusion_rules are accepted only while the benchmark is a draft — see NucleusClient.update_benchmark(). Pass exclusion_rules=[] to clear a draft’s exclusions.

Returns:

self, with updated fields.

Parameters:

  • name (Optional[str])

  • description (Optional[str])

  • metadata (Optional[Dict[str, Any]])

  • rollup_groups (Optional[List[nucleus.evaluation_v2.RollupGroup]])

  • allowed_label_matches_id (Any)

  • class_agnostic (Optional[bool])

  • exclusion_rules (Optional[List[Union[nucleus.evaluation_v2_exclusions.EvaluationV2ExclusionRule, Dict[str, Any]]]])

Return type:

Benchmark