nucleus.prediction#

All of the prediction types supported. In general, prediction types are the same as annotation types, but come with additional, optional data that can be attached such as confidence or probability distributions.

`BoxPrediction`	Prediction of a bounding box.
`CategoryPrediction`	A prediction of a category.
`CuboidPrediction`	A prediction of 3D cuboid.
`KeypointsPrediction`	Prediction of keypoints.
`LinePrediction`	Prediction of a line.
`PolygonPrediction`	Prediction of a polygon.
`PredictionList`	Wrapper class separating a list of predictions by type.
`SceneCategoryPrediction`	A prediction of a category for a scene.
`SegmentationPrediction`	Predicted segmentation mask on a 2D image.

class nucleus.prediction.BoxPrediction(label, x, y, width, height, reference_id, confidence=None, annotation_id=None, metadata=None, class_pdf=None, embedding_vector=None, track_reference_id=None)#

Prediction of a bounding box.

Parameters:

label (str) – The label for this annotation (e.g. car, pedestrian, bicycle)
x (Union[float, int]) – The distance, in pixels, between the left border of the bounding box and the left border of the image.
y (Union[float, int]) – The distance, in pixels, between the top border of the bounding box and the top border of the image.
width (Union[float, int]) – The width in pixels of the annotation.
height (Union[float, int]) – The height in pixels of the annotation.
reference_id (str) – User-defined ID of the image to which to apply this annotation.
confidence (Optional[float]) – 0-1 indicating the confidence of the prediction.
annotation_id (Optional[str]) – The annotation ID that uniquely identifies this annotation within its target dataset item. Upon ingest, a matching annotation id will be ignored by default, and updated if update=True for dataset.annotate. If no annotation ID is passed, one will be automatically generated using the label, x, y, width, and height, so that you can make inserts idempotently and identical boxes will be ignored.
metadata (Optional[Dict]) –
Arbitrary key/value dictionary of info to attach to this annotation. Strings, floats and ints are supported best by querying and insights features within Nucleus. For more details see our metadata guide.

Coordinate metadata may be provided to enable the Map Chart in the Nucleus Dataset charts page. These values can be specified as { “lat”: 52.5, “lon”: 13.3, … }.
class_pdf (Optional[Dict]) – An optional complete class probability distribution on this annotation. Each value should be between 0 and 1 (inclusive), and sum up to 1 as a complete distribution. This can be useful for computing entropy to surface places where the model is most uncertain.
embedding_vector (Optional[List]) – Custom embedding vector for this object annotation. If any custom object embeddings have been uploaded previously to this dataset, this vector must match the dimensions of the previously ingested vectors.
track_reference_id (Optional[str]) – A unique string to identify the prediction as part of a group. For instance, multiple “car” predictions across several dataset items may have the same track_reference_id such as “red_car”.

classmethod from_json(payload)#

Instantiates annotation object from schematized JSON dict payload.

Parameters:: payload (dict) –

has_local_files_to_upload()#

Returns True if annotation has local files that need to be uploaded.

Nearly all subclasses have no local files, so we default this to just return false. If the subclass has local files, it should override this method (but that is not the only thing required to get local upload of files to work.)

Return type:: bool

to_json()#

Serializes annotation object to schematized JSON string.

Return type:: str

to_payload()#

Serializes annotation object to schematized JSON dict.

Return type:: dict

class nucleus.prediction.CategoryPrediction(label, reference_id, taxonomy_name=None, confidence=None, metadata=None, class_pdf=None, track_reference_id=None)#

A prediction of a category.

Parameters:

label (str) – The label for this annotation (e.g. car, pedestrian, bicycle).
reference_id (str) – The reference ID of the image you wish to apply this annotation to.
taxonomy_name (Optional[str]) – The name of the taxonomy this annotation conforms to. See Dataset.add_taxonomy().
confidence (Optional[float]) – 0-1 indicating the confidence of the prediction.
class_pdf (Optional[Dict]) – An optional complete class probability distribution on this prediction. Each value should be between 0 and 1 (inclusive), and sum up to 1 as a complete distribution. This can be useful for computing entropy to surface places where the model is most uncertain.
metadata (Optional[Dict]) –
Arbitrary key/value dictionary of info to attach to this annotation. Strings, floats and ints are supported best by querying and insights features within Nucleus. For more details see our metadata guide.
track_reference_id (Optional[str]) – A unique string to identify the prediction as part of a group. For instance, multiple “car” predictions across several dataset items may have the same track_reference_id such as “red_car”.

classmethod from_json(payload)#

Instantiates annotation object from schematized JSON dict payload.

Parameters:: payload (dict) –

has_local_files_to_upload()#

Returns True if annotation has local files that need to be uploaded.

Return type:: bool

to_json()#

Serializes annotation object to schematized JSON string.

Return type:: str

to_payload()#

Serializes annotation object to schematized JSON dict.

Return type:: dict

class nucleus.prediction.CuboidPrediction(label, position, dimensions, yaw, reference_id, confidence=None, annotation_id=None, metadata=None, class_pdf=None, track_reference_id=None)#

A prediction of 3D cuboid.

Parameters:

label (str) – The label for this annotation (e.g. car, pedestrian, bicycle)
position (Point3D) – The point at the center of the cuboid
dimensions (Point3D) – The length (x), width (y), and height (z) of the cuboid
yaw (float) – The rotation, in radians, about the Z axis of the cuboid
reference_id (str) – User-defined ID of the image to which to apply this annotation.
confidence (Optional[float]) – 0-1 indicating the confidence of the prediction.
annotation_id (Optional[str]) – The annotation ID that uniquely identifies this annotation within its target dataset item. Upon ingest, a matching annotation id will be ignored by default, and updated if update=True for dataset.annotate.
metadata (Optional[str]) –
Arbitrary key/value dictionary of info to attach to this annotation. Strings, floats and ints are supported best by querying and insights features within Nucleus. For more details see our metadata guide.
class_pdf (Optional[Dict]) – An optional complete class probability distribution on this annotation. Each value should be between 0 and 1 (inclusive), and sum up to 1 as a complete distribution. This can be useful for computing entropy to surface places where the model is most uncertain.
track_reference_id (Optional[str]) – A unique string to identify the prediction as part of a group. For instance, multiple “car” predictions across several dataset items may have the same track_reference_id such as “red_car”.

classmethod from_json(payload)#

Instantiates annotation object from schematized JSON dict payload.

Parameters:: payload (dict) –

has_local_files_to_upload()#

Returns True if annotation has local files that need to be uploaded.

Return type:: bool

to_json()#

Serializes annotation object to schematized JSON string.

Return type:: str

to_payload()#

Serializes annotation object to schematized JSON dict.

Return type:: dict

class nucleus.prediction.KeypointsPrediction(label, keypoints, names, skeleton, reference_id, confidence=None, annotation_id=None, metadata=None, class_pdf=None, track_reference_id=None)#

Prediction of keypoints.

Parameters:

label (str) – The label for this annotation (e.g. car, pedestrian, bicycle).
keypoints (List[Keypoint]) – The list of keypoints objects.
names (List[str]) – A list that corresponds to the names of each keypoint.
skeleton (List[List[int]]) – A list of 2-length lists indicating a beginning and ending index for each line segment in the skeleton of this keypoint label.
reference_id (str) – User-defined ID of the image to which to apply this annotation.
confidence (Optional[float]) – 0-1 indicating the confidence of the prediction.
annotation_id (Optional[str]) – The annotation ID that uniquely identifies this annotation within its target dataset item. Upon ingest, a matching annotation id will be ignored by default, and updated if update=True for dataset.annotate.
metadata (Optional[Dict]) –
Arbitrary key/value dictionary of info to attach to this annotation. Strings, floats and ints are supported best by querying and insights features within Nucleus. For more details see our metadata guide.
class_pdf (Optional[Dict]) – An optional complete class probability distribution on this annotation. Each value should be between 0 and 1 (inclusive), and sum up to 1 as a complete distribution. This can be useful for computing entropy to surface places where the model is most uncertain.
track_reference_id (Optional[str]) – A unique string to identify the prediction as part of a group. For instance, multiple “car” predictions across several dataset items may have the same track_reference_id such as “red_car”.

classmethod from_json(payload)#

Instantiates annotation object from schematized JSON dict payload.

Parameters:: payload (dict) –

has_local_files_to_upload()#

Returns True if annotation has local files that need to be uploaded.

Return type:: bool

to_json()#

Serializes annotation object to schematized JSON string.

Return type:: str

to_payload()#

Serializes annotation object to schematized JSON dict.

Return type:: dict

class nucleus.prediction.LinePrediction(label, vertices, reference_id, confidence=None, annotation_id=None, metadata=None, class_pdf=None, track_reference_id=None)#

Prediction of a line.

Parameters:

label (str) – The label for this prediction (e.g. car, pedestrian, bicycle).
vertices (List[Point]) – The list of points making up the line.
reference_id (str) – User-defined ID of the image to which to apply this annotation.
confidence (Optional[float]) – 0-1 indicating the confidence of the prediction.
annotation_id (Optional[str]) – The annotation ID that uniquely identifies this annotation within its target dataset item. Upon ingest, a matching annotation id will be ignored by default, and updated if update=True for dataset.annotate.
metadata (Optional[Dict]) –
Arbitrary key/value dictionary of info to attach to this prediction. Strings, floats and ints are supported best by querying and insights features within Nucleus. For more details see our metadata guide.
class_pdf (Optional[Dict]) – An optional complete class probability distribution on this annotation. Each value should be between 0 and 1 (inclusive), and sum up to 1 as a complete distribution. This can be useful for computing entropy to surface places where the model is most uncertain.
track_reference_id (Optional[str]) – A unique string to identify the prediction as part of a group. For instance, multiple “car” predictions across several dataset items may have the same track_reference_id such as “red_car”.

classmethod from_json(payload)#

Instantiates annotation object from schematized JSON dict payload.

Parameters:: payload (dict) –

has_local_files_to_upload()#

Returns True if annotation has local files that need to be uploaded.

Return type:: bool

to_json()#

Serializes annotation object to schematized JSON string.

Return type:: str

to_payload()#

Serializes annotation object to schematized JSON dict.

Return type:: dict

class nucleus.prediction.PolygonPrediction(label, vertices, reference_id, confidence=None, annotation_id=None, metadata=None, class_pdf=None, embedding_vector=None, track_reference_id=None)#

Prediction of a polygon.

Parameters:

label (str) – The label for this annotation (e.g. car, pedestrian, bicycle).
vertices (List[nucleus.annotation.Point]) –
reference_id (str) –
confidence (Optional[float]) –
annotation_id (Optional[str]) –
metadata (Optional[Dict]) –
class_pdf (Optional[Dict]) –
embedding_vector (Optional[list]) –
track_reference_id (Optional[str]) –

:param vertices List[Point]: The list of points making up the polygon. :param reference_id: User-defined ID of the image to which to apply this

annotation.

Parameters:

confidence (Optional[float]) – 0-1 indicating the confidence of the prediction.
annotation_id (Optional[str]) – The annotation ID that uniquely identifies this annotation within its target dataset item. Upon ingest, a matching annotation id will be ignored by default, and updated if update=True for dataset.annotate.
metadata (Optional[Dict]) –
Arbitrary key/value dictionary of info to attach to this annotation. Strings, floats and ints are supported best by querying and insights features within Nucleus. For more details see our metadata guide.
class_pdf (Optional[Dict]) – An optional complete class probability distribution on this annotation. Each value should be between 0 and 1 (inclusive), and sum up to 1 as a complete distribution. This can be useful for computing entropy to surface places where the model is most uncertain.
embedding_vector (Optional[list]) – Custom embedding vector for this object annotation. If any custom object embeddings have been uploaded previously to this dataset, this vector must match the dimensions of the previously ingested vectors.
track_reference_id (Optional[str]) – A unique string to identify the prediction as part of a group. For instance, multiple “car” predictions across several dataset items may have the same track_reference_id such as “red_car”.
label (str) –
vertices (List[nucleus.annotation.Point]) –
reference_id (str) –

classmethod from_json(payload)#

Instantiates annotation object from schematized JSON dict payload.

Parameters:: payload (dict) –

has_local_files_to_upload()#

Returns True if annotation has local files that need to be uploaded.

Return type:: bool

to_json()#

Serializes annotation object to schematized JSON string.

Return type:: str

to_payload()#

Serializes annotation object to schematized JSON dict.

Return type:: dict

class nucleus.prediction.PredictionList#: Wrapper class separating a list of predictions by type.

class nucleus.prediction.SceneCategoryPrediction(label, reference_id, taxonomy_name=None, confidence=None, metadata=None)#

A prediction of a category for a scene.

from nucleus import SceneCategoryPrediction

category = SceneCategoryPrediction(
    label="running",
    reference_id="scene_1",
    taxonomy_name="action",
    confidence=0.83,
    metadata={
        "weather": "clear",
    },
)

Parameters:

label (str) – The label for this annotation (e.g. action, subject, scenario).
reference_id (str) – The reference ID of the scene you wish to apply this annotation to.
taxonomy_name (Optional[str]) – The name of the taxonomy this annotation conforms to. See Dataset.add_taxonomy().
confidence (Optional[float]) – 0-1 indicating the confidence of the prediction.
metadata (Optional[Dict]) –
Arbitrary key/value dictionary of info to attach to this annotation. Strings, floats and ints are supported best by querying and insights features within Nucleus. For more details see our metadata guide.

classmethod from_json(payload)#

Instantiates annotation object from schematized JSON dict payload.

Parameters:: payload (dict) –

has_local_files_to_upload()#

Returns True if annotation has local files that need to be uploaded.

Return type:: bool

to_json()#

Serializes annotation object to schematized JSON string.

Return type:: str

to_payload()#

Serializes annotation object to schematized JSON dict.

Return type:: dict

class nucleus.prediction.SegmentationPrediction#

Predicted segmentation mask on a 2D image.

from nucleus import SegmentationPrediction

segmentation = SegmentationPrediction(
    mask_url="s3://your-bucket-name/pred-seg-masks/image_2_pred_mask_id_1.png",
    annotations=[
        Segment(label="grass", index="1"),
        Segment(label="road", index="2"),
        Segment(label="bus", index="3", metadata={"vehicle_color": "yellow"}),
        Segment(label="tree", index="4")
    ],
    reference_id="image_2",
    annotation_id="image_2_pred_mask_1",
)

Parameters:

mask_url (str) –
A URL pointing to the segmentation prediction mask which is accessible to Scale. This “URL” can also be a path to a local file. The mask is an HxW int8 array saved in PNG format, with each pixel value ranging from [0, N), where N is the number of possible classes (for semantic segmentation) or instances (for instance segmentation).

The height and width of the mask must be the same as the original image. One example for semantic segmentation: the mask is 0 for pixels where there is background, 1 where there is a car, and 2 where there is a pedestrian.

Another example for instance segmentation: the mask is 0 for one car, 1 for another car, 2 for a motorcycle and 3 for another motorcycle. The class name for each value in the mask is stored in the list of Segment objects passed for “annotations”
annotations (List[Segment]) – The list of mappings between the integer values contained in mask_url and string class labels. In the semantic segmentation example above these would map that 0 to background, 1 to car and 2 to pedestrian. In the instance segmentation example above, 0 and 1 would both be mapped to car, 2 and 3 would both be mapped to motorcycle
reference_id (str) – User-defined ID of the image to which to apply this annotation.
annotation_id (Optional[str]) – For segmentation predictions, this value is ignored because there can only be one segmentation prediction per dataset item. Therefore regardless of annotation ID, if there is an existing segmentation on a dataset item, it will be ignored unless update=True is passed to Dataset.annotate(), in which case it will be overwritten. Storing a custom ID here may be useful in order to tie this annotation to an external database, and its value will be returned for any export.

classmethod from_json(payload)#

Instantiates annotation object from schematized JSON dict payload.

Parameters:: payload (dict) –

has_local_files_to_upload()#

Check if the mask url is local and needs to be uploaded.

Return type:: bool

to_json()#

Serializes annotation object to schematized JSON string.

Return type:: str

to_payload()#

Serializes annotation object to schematized JSON dict.

Return type:: dict