nucleus.benchmark

Benchmarks — frozen, cross-dataset ground-truth item sets for model evaluation.

A benchmark is a named collection of dataset items (with ground truth) that model runs are evaluated against. Benchmark evaluations score every benchmark item: items a model run has no predictions for count as false negatives, so leaderboard scores stay comparable across runs with different coverage.

Create and manage benchmarks via NucleusClient:

benchmark = client.create_benchmark("city-streets-v1", slice_id="slc_...")
evaluation = benchmark.create_evaluation_v2(
    model_run_id,
    rollup_groups=[RollupGroup("vehicle", ["car", "truck"])],
)
evaluation.wait_for_completion()

Benchmark

A benchmark: a frozen set of ground-truth items models are scored against.

class nucleus.benchmark.Benchmark

A benchmark: a frozen set of ground-truth items models are scored against.

create_evaluation_v2(model_run_id, *, name=None, rollup_groups=None, allowed_label_matches=None, allowed_label_matches_id=None, exclusion_rules=None, preset=None)

Evaluate a model run against this benchmark.

The run need not cover this benchmark’s datasets — uncovered members are scored as false negatives, so a partial run still ranks comparably. To give a run predictions across several datasets, use Dataset.upload_predictions_for_model_run().

See NucleusClient.create_benchmark_evaluation_v2() for parameter details.

Returns:

The created evaluation.

Return type:

EvaluationV2

Parameters:

• model_run_id (str)

• name (Optional[str])

• rollup_groups (Optional[List[nucleus.evaluation_v2.RollupGroup]])

• allowed_label_matches (Optional[List[nucleus.evaluation_v2.AllowedLabelMatch]])

• allowed_label_matches_id (Optional[str])

• exclusion_rules (Optional[List[Union[nucleus.evaluation_v2_exclusions.EvaluationV2ExclusionRule, Dict[str, Any]]]])

• preset (Optional[nucleus.evaluation_v2_preset.EvaluationV2Preset])

delete()

Delete this benchmark.

Return type:

None

items(*, limit=None, offset=None)

Return one page of this benchmark’s member item ids.

Parameters:

• limit (Optional[int]) – Optional page size.

• offset (Optional[int]) – Optional offset for pagination.

Returns:

The page of dataset item ids and the total member count.

Return type:

BenchmarkItemsPage

refresh()

Reload this benchmark from Nucleus.

Returns:

self, with updated fields.

Return type:

Benchmark

update(*, name=None, description=None, metadata=None)

Update this benchmark’s name, description, or metadata.

Only the arguments you pass are changed. Benchmark membership is frozen at creation and cannot be updated.

Returns:

self, with updated fields.

Parameters:

• name (Optional[str])

• description (Optional[str])

• metadata (Optional[Dict[str, Any]])

Return type:

Benchmark