nucleus.metrics.cuboid_metrics

CuboidIOU	Calculates the average IOU between cuboid annotations and predictions.
CuboidMetric	Abstract class for metrics of cuboids.
CuboidPrecision	Calculates the average precision between cuboid annotations and predictions.
CuboidRecall	Calculates the average recall between cuboid annotations and predictions.

class nucleus.metrics.cuboid_metrics.CuboidIOU(enforce_label_match=True, iou_threshold=0.0, confidence_threshold=0.0, iou_2d=False, annotation_filters=None, prediction_filters=None)

Calculates the average IOU between cuboid annotations and predictions.

Initializes CuboidIOU object.

Parameters:

• enforce_label_match (bool) – whether to enforce that annotation and prediction labels must match. Defaults to True

• iou_threshold (float) – IOU threshold to consider detection as valid. Must be in [0, 1]. Default 0.0

• iou_2d (bool) – whether to return the BEV 2D IOU if true, or the 3D IOU if false.

• confidence_threshold (float) – minimum confidence threshold for predictions. Must be in [0, 1]. Default 0.0

• annotation_filters (Optional[Union[nucleus.metrics.filtering.ListOfOrAndFilters, nucleus.metrics.filtering.ListOfAndFilters]]) – MetadataFilter predicates. Predicates are expressed in disjunctive normal form (DNF), like [[MetadataFilter(‘x’, ‘=’, 0), …], …]. DNF allows arbitrary boolean logical combinations of single field predicates. The innermost structures each describe a single column predicate. The list of inner predicates is interpreted as a conjunction (AND), forming a more selective and multiple column predicate. Finally, the most outer list combines these filters as a disjunction (OR).

• prediction_filters (Optional[Union[nucleus.metrics.filtering.ListOfOrAndFilters, nucleus.metrics.filtering.ListOfAndFilters]]) – MetadataFilter predicates. Predicates are expressed in disjunctive normal form (DNF), like [[MetadataFilter(‘x’, ‘=’, 0), …], …]. DNF allows arbitrary boolean logical combinations of single field predicates. The innermost structures each describe a single column predicate. The list of inner predicates is interpreted as a conjunction (AND), forming a more selective and multiple column predicate. Finally, the most outer list combines these filters as a disjunction (OR).

aggregate_score(results)

A metric must define how to aggregate results from single items to a single ScalarResult.

E.g. to calculate a R2 score with sklearn you could define a custom metric class

class R2Result(MetricResult):
    y_true: float
    y_pred: float

And then define an aggregate_score

def aggregate_score(self, results: List[MetricResult]) -> ScalarResult:
    y_trues = []
    y_preds = []
    for result in results:
        y_true.append(result.y_true)
        y_preds.append(result.y_pred)
    r2_score = sklearn.metrics.r2_score(y_trues, y_preds)
    return ScalarResult(r2_score)

Parameters:

results (List[nucleus.metrics.base.ScalarResult])

Return type:

nucleus.metrics.base.ScalarResult

call_metric(annotations, predictions)

A metric must override this method and return a metric result, given annotations and predictions.

Parameters:

• annotations (nucleus.annotation.AnnotationList)

• predictions (nucleus.prediction.PredictionList)

Return type:

nucleus.metrics.base.ScalarResult

class nucleus.metrics.cuboid_metrics.CuboidMetric(enforce_label_match=False, confidence_threshold=0.0, annotation_filters=None, prediction_filters=None)

Abstract class for metrics of cuboids.

The CuboidMetric class automatically filters incoming annotations and predictions for only cuboid annotations. It also filters predictions whose confidence is less than the provided confidence_threshold. Finally, it provides support for enforcing matching labels. If enforce_label_match is set to True, then annotations and predictions will only be matched if they have the same label.

To create a new concrete CuboidMetric, override the eval function with logic to define a metric between cuboid annotations and predictions.

Initializes CuboidMetric abstract object.

Parameters:

• enforce_label_match (bool) – whether to enforce that annotation and prediction labels must match. Default False

• confidence_threshold (float) – minimum confidence threshold for predictions. Must be in [0, 1]. Default 0.0

• annotation_filters (Optional[Union[nucleus.metrics.filtering.ListOfOrAndFilters, nucleus.metrics.filtering.ListOfAndFilters]]) –

  MetadataFilter predicates. Predicates are expressed in disjunctive normal form (DNF),

  like [[MetadataFilter(‘x’, ‘==’, 0), FieldFilter(‘label’, ‘==’, ‘pedestrian’)], …].

  DNF allows arbitrary boolean logical combinations of single field predicates. The innermost structures each describe a single field predicate. The list of inner predicates is interpreted as a conjunction (AND), forming a more selective and multiple column predicate. Finally, the most outer list combines these filters as a disjunction (OR).

• prediction_filters (Optional[Union[nucleus.metrics.filtering.ListOfOrAndFilters, nucleus.metrics.filtering.ListOfAndFilters]]) –

  MetadataFilter predicates. Predicates are expressed in disjunctive normal form (DNF),

  like [[MetadataFilter(‘x’, ‘==’, 0), FieldFilter(‘label’, ‘==’, ‘pedestrian’)], …].

  DNF allows arbitrary boolean logical combinations of single field predicates. The innermost structures each describe a single field predicate. The list of inner predicates is interpreted as a conjunction (AND), forming a more selective and multiple column predicate. Finally, the most outer list combines these filters as a disjunction (OR).

aggregate_score(results)

A metric must define how to aggregate results from single items to a single ScalarResult.

E.g. to calculate a R2 score with sklearn you could define a custom metric class

class R2Result(MetricResult):
    y_true: float
    y_pred: float

And then define an aggregate_score

def aggregate_score(self, results: List[MetricResult]) -> ScalarResult:
    y_trues = []
    y_preds = []
    for result in results:
        y_true.append(result.y_true)
        y_preds.append(result.y_pred)
    r2_score = sklearn.metrics.r2_score(y_trues, y_preds)
    return ScalarResult(r2_score)

Parameters:

results (List[nucleus.metrics.base.ScalarResult])

Return type:

nucleus.metrics.base.ScalarResult

call_metric(annotations, predictions)

A metric must override this method and return a metric result, given annotations and predictions.

Parameters:

• annotations (nucleus.annotation.AnnotationList)

• predictions (nucleus.prediction.PredictionList)

Return type:

nucleus.metrics.base.ScalarResult

class nucleus.metrics.cuboid_metrics.CuboidPrecision(enforce_label_match=True, iou_threshold=0.0, confidence_threshold=0.0, annotation_filters=None, prediction_filters=None)

Calculates the average precision between cuboid annotations and predictions.

Initializes CuboidIOU object.

Parameters:

• enforce_label_match (bool) – whether to enforce that annotation and prediction labels must match. Defaults to True

• iou_threshold (float) – IOU threshold to consider detection as valid. Must be in [0, 1]. Default 0.0

• confidence_threshold (float) – minimum confidence threshold for predictions. Must be in [0, 1]. Default 0.0

• annotation_filters (Optional[Union[nucleus.metrics.filtering.ListOfOrAndFilters, nucleus.metrics.filtering.ListOfAndFilters]]) – MetadataFilter predicates. Predicates are expressed in disjunctive normal form (DNF), like [[MetadataFilter(‘x’, ‘==’, 0), …], …]. DNF allows arbitrary boolean logical combinations of single field predicates. The innermost structures each describe a single column predicate. The list of inner predicates is interpreted as a conjunction (AND), forming a more selective and multiple column predicate. Finally, the most outer list combines these filters as a disjunction (OR).

• prediction_filters (Optional[Union[nucleus.metrics.filtering.ListOfOrAndFilters, nucleus.metrics.filtering.ListOfAndFilters]]) – MetadataFilter predicates. Predicates are expressed in disjunctive normal form (DNF), like [[MetadataFilter(‘x’, ‘==’, 0), …], …]. DNF allows arbitrary boolean logical combinations of single field predicates. The innermost structures each describe a single column predicate. The list of inner predicates is interpreted as a conjunction (AND), forming a more selective and multiple column predicate. Finally, the most outer list combines these filters as a disjunction (OR).

aggregate_score(results)

A metric must define how to aggregate results from single items to a single ScalarResult.

E.g. to calculate a R2 score with sklearn you could define a custom metric class

class R2Result(MetricResult):
    y_true: float
    y_pred: float

And then define an aggregate_score

def aggregate_score(self, results: List[MetricResult]) -> ScalarResult:
    y_trues = []
    y_preds = []
    for result in results:
        y_true.append(result.y_true)
        y_preds.append(result.y_pred)
    r2_score = sklearn.metrics.r2_score(y_trues, y_preds)
    return ScalarResult(r2_score)

Parameters:

results (List[nucleus.metrics.base.ScalarResult])

Return type:

nucleus.metrics.base.ScalarResult

call_metric(annotations, predictions)

A metric must override this method and return a metric result, given annotations and predictions.

Parameters:

• annotations (nucleus.annotation.AnnotationList)

• predictions (nucleus.prediction.PredictionList)

Return type:

nucleus.metrics.base.ScalarResult

class nucleus.metrics.cuboid_metrics.CuboidRecall(enforce_label_match=True, iou_threshold=0.0, confidence_threshold=0.0, annotation_filters=None, prediction_filters=None)

Calculates the average recall between cuboid annotations and predictions.

Initializes CuboidIOU object.

Parameters:

• enforce_label_match (bool) – whether to enforce that annotation and prediction labels must match. Defaults to True

• iou_threshold (float) – IOU threshold to consider detection as valid. Must be in [0, 1]. Default 0.0

• confidence_threshold (float) – minimum confidence threshold for predictions. Must be in [0, 1]. Default 0.0

• annotation_filters (Optional[Union[nucleus.metrics.filtering.ListOfOrAndFilters, nucleus.metrics.filtering.ListOfAndFilters]])

• prediction_filters (Optional[Union[nucleus.metrics.filtering.ListOfOrAndFilters, nucleus.metrics.filtering.ListOfAndFilters]])

aggregate_score(results)

A metric must define how to aggregate results from single items to a single ScalarResult.

E.g. to calculate a R2 score with sklearn you could define a custom metric class

class R2Result(MetricResult):
    y_true: float
    y_pred: float

And then define an aggregate_score

def aggregate_score(self, results: List[MetricResult]) -> ScalarResult:
    y_trues = []
    y_preds = []
    for result in results:
        y_true.append(result.y_true)
        y_preds.append(result.y_pred)
    r2_score = sklearn.metrics.r2_score(y_trues, y_preds)
    return ScalarResult(r2_score)

Parameters:

results (List[nucleus.metrics.base.ScalarResult])

Return type:

nucleus.metrics.base.ScalarResult

call_metric(annotations, predictions)

A metric must override this method and return a metric result, given annotations and predictions.

Parameters:

• annotations (nucleus.annotation.AnnotationList)

• predictions (nucleus.prediction.PredictionList)

Return type:

nucleus.metrics.base.ScalarResult