SDK Fields#

Data Fields#

Data fields are a collection of classes that allow you to store different representations of data. In most cases data fields can act as both the input and target of a model. When selecting a data field, choose one that best approximates the use desired case.

Represents an image that can be attached to any other field and can act as an input or a target. Either the file_path or the data field needs to be passed to be a valid element.

Parameters:

file_path (str) – The location of the image to be loaded.
data (np.ndarray) – The numpy array with the loaded image of type uint8.
width (Optional[int]) – Width of the image (autofield if data is passed).
height (Optional[int]) – Height of the image (autofield if data is passed).
description (Optional[str]) – A description of the object or annotation information.
ref_field (Optional[List[object]]) – A list of objects the current BaseField is refering.
key_name (Optional[str]) – A string key, by which the field can be identified.
confidence (Optional[float]) – Confidence level of the field.
user_attributes (Optional[Dict[str, object]]) – Optional dictionary containing additional data regarding the field.

Returns:

Example:

import efemarai as ef
import numpy as np

image = np.zeros((100, 100), dtype=np.uint8)
image[10:30, 10:50] = 255

ef_image = ef.Image(data=image)
print(ef_image)

ef_image2 = ef.Image(file_path="./image.png")
print(ef_image2)

class efemarai.fields.data_fields.Mask(file_path: str | None = None, data: numpy.ndarray | None = None, width: int | None = None, height: int | None = None, confidence: float | None = None, id: bson.objectid.ObjectId | None = None, description: str | None = None, ref_field: list | None = None, key_name: str | None = None, user_attributes: Dict[str, object] | None = None)#: Used for Semantic Segmentation. Same as Image class, but whenever rescaled, Image.Nearest strategy is used.

Represents an instance mask that can be attached to any other field and can act as an input or a target.

Parameters:

file_path (str) – The location of the image to be loaded.
data (np.ndarray) – The numpy array with the loaded image.
width (Optional[int]) – Width of the image.
height (Optional[int]) – Height of the image.
instance_id (id) – Id of the instance within the dataset
label – ( Optional[AnnotationClass]): An AnnotationClass associated with the instance
description (str) – A description of the object or annotation information.
ref_field (List[object]) – A list of objects the current BaseField is refering to.
key_name (str) – A string key, by which the field can be identified.
confidence (float) – Confidence level of the field.
user_attributes (Dict[str, object]) – Optional dictionary containing additional data regarding the field.

Returns:

to_polygon()#

Converts efemarai.fields.data_fields.InstanceMask to efemarai.fields.annotation_fields.Polygon

Returns:: class: efemarai.fields.annotation_fields.Polygon

Example: See ef.Polygon for an example usage.

Annotation Fields#

Represents an annotation that can be attached to any other field and can act as a label. When returned by a model output, either name or id can be specified. When creating a dataset, both are required.

Parameters:

name (str) – Name of the class.
id (int) – Id of the class.
confidence (Optional[float]) – Confidence level. Prefer using the DataField confidence in general.
color (Optional[str]) – Color of the class in hex string. Used for visualization.
text_color (Optional[str]) – Color of the text of the annotation. Used for visualization.
category (Optional[str]) – Category of the class.
slug (Optional[str]) – Slugified name of the class that can be used for queries etc.

Returns:

class: efemarai.fields.annotation_fields.AnnotationClass: An AnnotationClass object.

Example:

When your code needs to return a target that has a label, you should create one with the name _or_ id that your model returns.

import efemarai as ef
output = {"class_id": 1, "name": "person", "value": 0.77}
label = ef.AnnotationClass(id=output["class_id"], name=output["name"])
label2 = ef.AnnotationClass(id=output["class_id"])
print(label, label2)

If you are appending ground truth data to a dataset, you need to resolve all of the information. You can do that through the dataset object to obtain the annotation.

label = dataset.get_annotation_class(id=output["class_id"])
label = dataset.get_annotation_class(name=output["name"])

class efemarai.fields.annotation_fields.Tag(label: AnnotationClass, probabilities: numpy.ndarray | None = None, confidence: float | None = None, id: bson.objectid.ObjectId | None = None, description: str | None = None, ref_field: list | None = None, key_name: str | None = None, user_attributes: Dict[str, object] | None = None)#

Represents an annotation that can be used for classification.

Parameters:

label (AnnotationClass) – An instance of AnnotationClass
probabilities (np.ndarray) – The softmax probabilities for a given prediction.
confidence (Optional[float]) – Confidence level of the field (autocalculated from probabilities).
id (Optional[ObjectID]) – Id of the field.
description (Optional[str]) – A description of the object or annotation information.
ref_field (List[object]) – A list of objects the current BaseField is refering to.
key_name (Optional[str]) – A string key, by which the field can be identified.
user_attributes (Optional[Dict[str, object]]) – Optional dictionary containing additional data regarding the field.

Returns:

Example:

import efemarai as ef
import numpy as np

outputs = [0.1, 0.2, 0.15, 0.55]

label = ef.AnnotationClass(id=np.argmax(outputs))
tag = ef.Tag(label=label, probabilities=outputs)
print(label, tag)

Represents an annotation that can be used for regression.

Parameters:

value (float) – A float to represent the annotation value
confidence (Optional[float]) – An optional confidence associated with the value
description (Optional[str]) – A description of the object or annotation information.
ref_field (List[object]) – A list of objects the current BaseField is refering to.
key_name (Optional[str]) – A string key, by which the field can be identified.
user_attributes (Optional[Dict[str, object]]) – Dictionary containing additional data regarding the field.

Returns:

Example:

import efemarai as ef
import numpy as np

output = 0.55

value = ef.Value(value=output)
print(value)

class efemarai.fields.annotation_fields.InstanceField(instance_id: int, label: AnnotationClass | None = None, confidence: float | None = None, id: bson.objectid.ObjectId | None = None, description: str | None = None, ref_field: list | None = None, key_name: str | None = None, user_attributes: Dict[str, object] | None = None)#

This is a base class. Represents an annotation of an instance that can be attached to any other field and can act as a label.

Parameters:

instance_id (id) – Id of the instance within the dataset.
label (AnnotationClass) – An AnnotationClass associated with the instance.
confidence (float) – Confidence level of the field.
description (Optional[str]) – A description of the object or annotation information.
ref_field (List[object]) – A list of objects the current BaseField is refering to.
key_name (Optional[str]) – A string key, by which the field can be identified.
user_attributes (Optional[Dict[str, object]]) – Dictionary containing additional data regarding the field.

Returns:

class efemarai.fields.annotation_fields.BoundingBox(xyxy: tuple, label: AnnotationClass, area: float | None = None, instance_id: int | None = None, description: str | None = None, confidence: float | None = None, id: bson.objectid.ObjectId | None = None, ref_field: list | None = None, key_name: str | None = None, user_attributes: Dict[str, object] | None = None)#

class efemarai.fields.annotation_fields.Polygon(vertices: List[List[Tuple[float, float]]], label: AnnotationClass, area: float | None = None, instance_id: int | None = None, confidence: float | None = None, id: bson.objectid.ObjectId | None = None, description: str | None = None, ref_field: list | None = None, key_name: str | None = None, user_attributes: Dict[str, object] | None = None)#

Represents a polygon annotation.

Parameters:

List[List[Tuple[float]]] (vertices) – Coordinates of the polygon. The values in are in format [[[point]…],[[point]…]] in image dimensions, where the first list contains all contours of the polygon, the internal ones - lists of single points of that contour.
label (AnnotationClass) – An AnnotationClass associated with the instance
area (Optional[float]) – Area of the polygon
instance_id (Optional[id]) – Id of the instance within the dataset
confidence (Optional[float]) – Confidence level of the field.
description (Optional[str]) – A description of the object or annotation information.
ref_field (List[object]) – A list of objects the current BaseField is refering to.
key_name (Optional[str]) – A string key, by which the field can be identified.
user_attributes (Optional[Dict[str, object]]) – Dictionary containing additional data regarding the field.

Returns:

You can construct a polygon from the output of a model that is a mask:

import efemarai as ef
import numpy as np

mask = np.zeros((100, 100), dtype=np.uint8)
mask[10:30, 10:50] = 255

label = ef.AnnotationClass(id=1)
polygon = ef.InstanceMask(label=label, data=mask).to_polygon()
print(label, polygon)

new_poly = ef.Polygon(label=label, vertices=polygon.vertices)
print(new_poly)

Represents a keypoint annotation.

Parameters:

x (float) – X coordinate of the keypoint in absolute value.
y (float) – Y coordinate of the keypoint in absolute value.
label (AnnotationClass) – An AnnotationClass associated with the instance
name (Optional[str]) – Name of the keypoint, typically provided within a Skeleton.
index (Optional[int]) – Index of the keypoint within a Skeleton.
occluded (Optional[bool]) – Whether the keypoint is occluded.
annotated (Optional[bool]) – In case the keypoint is occluded, if it is annotated or with default coords.
instance_id (Optional[id]) – Id of the instance within the dataset
confidence (Optional[float]) – Confidence level of the field.
description (Optional[str]) – A description of the object or annotation information.
ref_field (List[object]) – A list of objects the current BaseField is refering to.
key_name (Optional[str]) – A string key, by which the field can be identified.
user_attributes (Optional[Dict[str, object]]) – Dictionary containing additional data regarding the field.

Returns:

class efemarai.fields.annotation_fields.Skeleton(keypoints: List[Keypoint], label: AnnotationClass, edges: list | None = None, instance_id: int | None = None, confidence: float | None = None, id: bson.objectid.ObjectId | None = None, description: str | None = None, ref_field: list | None = None, key_name: str | None = None, user_attributes: Dict[str, object] | None = None)#

Represents a skeleton annotation. A sekeleton is a group of keypoints that represent an instance of an object.

Parameters:

keypoints (List[Keypoint]) – List of the Keypoints in the class
label (AnnotationClass) – An AnnotationClass associated with the instance
edges – (Optional[List[Tuple[int]]]) List of edges between keypoints. Each edge is a tuple (from_keypoint_index, to_keypoint_index) and is used for visualization.
instance_id (Optional[id]) – Id of the instance within the dataset
confidence (Optional[float]) – Confidence level of the field.
description (Optional[str]) – A description of the object or annotation information.
ref_field (List[object]) – A list of objects the current BaseField is refering to.
key_name (Optional[str]) – A string key, by which the field can be identified.
user_attributes (Optional[Dict[str, object]]) – Dictionary containing additional data regarding the field.

Returns:

Example of how to convert Mediapipe facemesh into the efemarai system:

import efemarai as ef
import cv2
import mediapipe as mp
from mediapipe.python.solutions.drawing_utils import _normalized_to_pixel_coordinates

model = mp.solutions.face_mesh
# >> The image is an ef.Image variable

results = model.process(cv2.cvtColor(image.data, cv2.COLOR_BGR2RGB))
sdk_outputs = []
label = ef.AnnotationClass(name="face")

# For each face in the image
for i, face_landmarks in enumerate(results.multi_face_landmarks):
    keypoints = []
    for j, idx in enumerate(face_landmarks.landmark):
        cord = _normalized_to_pixel_coordinates(idx.x, idx.y, image.width, image.height)
        keypoints.append(
            ef.Keypoint(
                x=cord[0],
                y=cord[1],
                name=model.name[j],
                index=j,
                annotated=True,
                occluded=False,
                instance_id=i,
                ref_field=image,
                label=label,
                confidence=1,
            )
        )

    sdk_outputs.append(
        ef.Skeleton(
            label=label,
            keypoints=keypoints,
            ref_field=image,
            instance_id=i,
            confidence=1,
        )
    )

Base Fields#

At the core of each data field, the BaseField creates a common representation of an element in Efemarai. It is used to store the id, description, reference field, name, confidence, and any user added attributes.

Parameters:

id (ObjectId) – A BaseField unique identifier.
description (str) – A description of the object or annotation information.
ref_field (List[object]) – A list of objects the current BaseField is refering.
key_name (str) – A string key, by which the field can be identified.
confidence (float) – Confidence level of the field.
user_attributes (Dict[str, object]) – Optional dictionary containing additional data regarding the field.

Returns:

A BaseField object.

Return type:

efemarai.fields.base_field.BaseField

Inputs and Targets#

Each dataset consists of some input data, which gets fed to a model, which produces outputs. The outputs are then compared against the annotations. We collect all input data under inputs and all annotations under targets. Inputs and targets are connceted through ref_field and the relation is many-to-many. For example, a single image (input) can have N different boxes, polygons, etc (targets) connected to it, and vice versa - one target field can be connected to N input fields. Inputs and targets are connected in Datapoints.

Datapoint#

A datapoint is the main element that holds the input/target pair. A datapoint is as big as a single input to the model. For example if your model takes a single image, and outputs bounding boxes, then your datapoint would contain a single input (ef.Image) and as many targets as you would like (ef.BoundingBox). However, if you have a multi input model, you would like a single datapoint to hold an arbitrary number of inputs and targets.

class efemarai.fields.datapoint.ModelOutput(outputs)#

static create_from(spec, datapoint, data)#

class efemarai.fields.datapoint.Datapoint(dataset, inputs=None, targets=None, id=None)#

static create_from(spec, data)#

static create_targets_from(spec, data)#

add_target(_target: BaseField | InstanceField)#

Used to add target to the datapoint.

Parameters:: _target – An instance of efemarai.fields.base_fields.BaseField.
Returns:: list[BaseField]
Return type:: targets

add_inputs(_input: BaseField | InstanceField)#

Used to add inputs to the datapoint.

Parameters:: _input – An instance of efemarai.fields.base_fields.BaseField.
Returns:: list[BaseField]
Return type:: inputs

upload(copy_files=True)#

Used to upload a datapoint to the system.

Parameters:: copy_files (bool) – Whether to copy the files or to upload them
Returns:: ObjectId of the uploaded datapoint
Return type:: datapoint_id

get_input(name)#

Returns an input with that particular key name.

Parameters:: name (str) – Key name of a specific field to be returned, or None if not found.
Returns:: An instance of BaseField

get_inputs_with_type(ef_type)#

A way to filter only inputs of a particular ef_type.

Parameters:: ef_type (BaseField) – An instance of efemarai.fields.base_fields.BaseField.
Returns:: list[BaseField] - a list of inputs of the ef_type

Example:

import efemarai as ef

images = datapoints[0].get_inputs_with_type(ef.Image)
# images is a list of all of the ef.Image elements in datapoints[0].
# each of those images can be accessed by datapoints[0].get_input(images[0].key_name)

get_targets_with_type(ef_type)#

A way to filter only targets of a particular ef_type.

Parameters:: ef_type (BaseField) – An instance of efemarai.fields.base_fields.BaseField.
Returns:: list[BaseField] - a list of inputs of the ef_type

get_input_refs(ids)#

Returns a list of field in input that refer to id.

Parameters:: ids (List[ObjectId]) – Ids when referenced to return object, or [] if not found.
Returns:: An instance of BaseField