nucleus.benchmark
=================

.. py:module:: nucleus.benchmark

.. autoapi-nested-parse::

   Benchmarks — cross-dataset ground-truth item sets for model evaluation.

   A benchmark is a named collection of dataset items (with ground truth) that
   model runs are evaluated against. Benchmark evaluations score every benchmark
   item: items a model run has no predictions for count as false negatives, so
   leaderboard scores stay comparable across runs with different coverage.

   Create and manage benchmarks via :class:`~nucleus.NucleusClient`::

       benchmark = client.create_benchmark("city-streets-v1", slice_id="slc_...")
       evaluation = benchmark.create_evaluation_v2(
           model_run_id,
           rollup_groups=[RollupGroup("vehicle", ["car", "truck"])],
       )
       evaluation.wait_for_completion()

   A finalized benchmark is immutable. To evolve one, create a **new version**
   downstream of it with ``parent_benchmark_id`` (see
   :meth:`NucleusClient.create_benchmark`) — the child inherits the parent's items,
   adds/removes on top, and takes a minor (default) or major version bump.

   To assemble a benchmark incrementally, create a **draft** (``draft=True``): add
   items across many calls with :meth:`Benchmark.add_items` / remove with
   :meth:`Benchmark.remove_items`, then :meth:`Benchmark.finalize` to freeze it into
   a ``"ready"`` benchmark. A draft cannot be evaluated until finalized.






.. autoapisummary::
   nucleus.benchmark.Benchmark


.. py:class:: Benchmark


   A benchmark: a frozen set of ground-truth items models are scored against.



   .. py:method:: add_items(*, item_ids = None, items = None, slice_id = None, dataset_id = None, slice_ids = None, dataset_ids = None, scene_ids = None, wait_for_completion = True, verbose = True)

      Add items to this **draft** benchmark.

      Only valid while this benchmark is a draft (``status == "draft"``); a
      finalized benchmark is immutable. See
      :meth:`NucleusClient.add_benchmark_items` for parameter details.

      :returns: self, refreshed.


   .. py:method:: create_evaluation_v2(model_run_id, *, name = None, rollup_groups = None, allowed_label_matches = None, allowed_label_matches_id = None, exclusion_rules = None, preset = None)

      Evaluate a model run against this benchmark.

      The run need not cover this benchmark's datasets — uncovered members are
      scored as false negatives, so a partial run still ranks comparably. To
      give a run predictions across several datasets, use
      :meth:`Dataset.upload_predictions_for_model_run`.

      See :meth:`NucleusClient.create_benchmark_evaluation_v2` for parameter
      details.

      :returns: The created evaluation.
      :rtype: :class:`~nucleus.evaluation_v2.EvaluationV2`


   .. py:method:: delete()

      Delete this benchmark.


   .. py:method:: finalize()

      Finalize this **draft** benchmark, freezing it into a ``"ready"`` one.

      After finalizing, the benchmark is immutable and can be evaluated. Fails
      if it is not a draft, is empty, or still has an add-items job in flight.

      :returns: self, with updated fields (``status`` now ``"ready"``).


   .. py:method:: items(*, limit = None, offset = None)

      Return one page of this benchmark's member item ids.

      :param limit: Optional page size.
      :param offset: Optional offset for pagination.

      :returns: The page of dataset item ids and the total member count.
      :rtype: :class:`~nucleus.data_transfer_object.evaluation_v2.BenchmarkItemsPage`


   .. py:method:: refresh()

      Reload this benchmark from Nucleus.

      :returns: self, with updated fields.


   .. py:method:: remove_items(item_ids)

      Remove items from this **draft** benchmark.

      Only valid while this benchmark is a draft. Unknown ids are ignored.

      :param item_ids: Dataset item ids (``di_*``) to remove.

      :returns: self, refreshed.


   .. py:method:: update(*, name = None, description = None, metadata = None)

      Update this benchmark's name, description, or metadata.

      Only the arguments you pass are changed. Benchmark membership is
      frozen at creation and cannot be updated.

      :returns: self, with updated fields.