SDK#

Session#

class efemarai.session.Session(token=None, url=None, config_file=None)#

Session related functionality. A session is the way to interact with the Efemarai endpoint. All commands use the active session to communicate and perform the needed actions.

load(filename, exists_ok=False, project_only=False) → dict#

Load a configuration file for creating projects, models, datasets and/or domains.

Parameters:

filename (str) – A configuration file.
exists_ok (bool, optional) – If any of the components defined in the configuration file exist, should they raise an exception or return the already existing element.
project_only (bool, optional) – Load only the specified project.

Riases:: ValueError: If any of the components exist and exists_ok is set to False.

Returns:: A dictionary of the potentially loaded projects, datasets, domains, models.
Return type:: dict

Example:

import efemarai as ef
result = ef.Session().load("efemarai.yaml")
print(result)

property projects#

Returns the list of projects.

Returns:: A list of the available projects.
Return type:: list [efemarai.project.Project]

project(name)#

Returns a project specified by the name.

Parameters:: name (str) – A project identifier. It can either be the name of the project or its id.
Returns:: A project object or None if it doesn’t exist.
Return type:: efemarai.project.Project

Example:

import efemarai as ef
project = ef.Session().project("Name")
print(project)

create_project(name, description=None, problem_type='Custom', private=False, exists_ok=False, **kwargs)#

Creates a project.

Parameters:

name (str) – Name of the project.
description (str, optional) – Description of the project.
problem_type (str, efemarai.problem_type.ProblemType) – Problem type of the project.
exists_ok (bool, optional) – If the current named project exist, should it be returned or an exception raised.

Raises:

ValueError – If the project exists and exists_ok is set to False.

Returns:

A project object.

Return type:

efemarai.project.Project

Project#

class efemarai.project.Project(session, id, name, description, problem_type)#

Provides project related functionality. A project is the main building block around which an ML model evaluation is performed.

A project is an association of a set of ML models, datasets and domains.

name#: (str) Name of the project.

description#: (str) Description of the project.

problem_type#: (efemarai.problem_type.ProblemType) Project problem type.

property models#: Returns a list of the models associated with the project.

model(model) → Model#

Returns the model specified by the name.

Parameters:: model (str) – A model identifier. It can either be the name of the model or its id.
Returns:: A model object or None if it doesn’t exist.
Return type:: efemarai.model.Model

create_model(name, description=None, repository=None, files=None, exists_ok=False, **kwargs) → Model#

Creates a model.

Parameters:

name (str) – Name of the model.
description (str, optional) – Free text description.
repository (dict, optional) – Model repository info. See ef.model.ModelRepository.
files (list[dict]) – Files needed to load the model e.g. weights or config files. See ef.model.ModelFile.
exists_ok (bool, optional) – If the current named model exist, should it be returned or an exception raised.

Raises:

ValueError – If the model exists and exists_ok is set to False.

Returns:

The created model.

Return type:

efemarai.model.Model

push_model(name, description=None, repository=None, files=None, **kwargs) → Model#

Pushes a new model version or creates a model, if there is no such model with this name.

Parameters:

name (str) – Name of the model.
description (str, optional) – Free text description.
repository (dict, optional) – Model repository info. See ef.model.ModelRepository.
files (list[dict]) – Files needed to load the model e.g. weights or config files. See ef.model.ModelFile.

Returns:

The created model.

Return type:

efemarai.model.Model

property datasets#

Returns a list of the datasets associated with the project.

Returns:: A list of available datasets.
Return type:: list[efemarai.dataset.Dataset]

dataset(dataset) → Dataset#

Returns the dataset specified by the name or id.

Parameters:: dataset (str) – A dataset identifier. It can either be the name of the dataset or its id.
Returns:: A dataset object or None if it doesn’t exist.
Return type:: efemarai.dataset.Dataset

create_dataset(name, stage, format=None, data_url=None, annotations_url=None, credentials=None, upload=False, num_datapoints=None, mask_generation=None, exists_ok=False, min_asset_area=15, **kwargs) → Dataset#

Creates a dataset.

Parameters:

name (str) – A dataset name.
format (str) – A format of the dataset. Possible formats are specified under ef.dataset.DatasetFormat.
stage (str) – Dataset stage - one of [train, validation, test].
data_url (str) – A url to fetch data. Can be local or remote.
annotations_url (str, optional) – A url that specifies the annotation file. Can be local or remote.
credentials (str, optional) – If the dataset if remote, Specify the access token or {username}:{password} to access.
upload (bool, optional) – Should the data be uploaded to Efemarai? This would make evaluation and testing much faster.
num_datapoints (int, optional) – Should a subset of the datapoints be loaded as a dataset? Specify the number of dataponts. Useful for quick testing. Use None to load all.
mask_generation (str) – What strategy should be used for mask generation.
exists_ok (bool, optional) – If the current named dataset exist, should it be returned or an exception raised.
min_asset_area (int, optional) – Assets with area smaller than this will be ignored. Default value of 15.

Raises:

ValueError – If the dataset exists and exists_ok is set to False.

Returns:

The created dataset.

Return type:

efemarai.dataset.Dataset

property domains#

Returns a list of the domains associated with the project.

Returns:: A list of available domains.
Return type:: list [efemarai.domain.Domain]

domain(domain) → Domain#

Returns the domain specified by the name or id.

Parameters:: domain (str) – A domain identifier. It can either be the name of the domain or its id.
Returns:: A domain object or None if it doesn’t exist.
Return type:: efemarai.domain.Domain

create_domain(name, transformations, exists_ok=False, **kwargs) → Domain#

Creates a domain. The recommended strategy is to use the UI to build up the domain visually. The domain then can be exported to a yaml configuration file for storage and loaded/reused through ef.Session().load(…).

Parameters:

name (str) – The name of the domain.
transformations (dict) – A dictionary of the transformations and their params.
exists_ok (bool, optional) – If the current named domain exist, should it be returned or an exception raised.

Raises:

ValueError – If the domain exists and exists_ok is set to False.

Returns:

The created domain.

Return type:

efemarai.domain.Domain

property baselines#

Returns a list of the baselines associated with the project.

Returns:: A list of available baselines.
Return type:: list [efemarai.baseline.Baseline]

baseline(baseline_run) → Baseline#

Returns the baseline specified by the name or id.

Parameters:: baseline_run (str) – A stress baseline_run identifier. It can either be the name of the stress test or its id.
Returns:: A baseline object or None if it doesn’t exist.
Return type:: efemarai.baseline.Baseline

create_baseline(name, model, dataset, **kwargs) → Baseline#

Creates a stress test.

Parameters:

name (str) – The name of the baseline
model (str, efemarai.model.Model) – The model to stress test.
domain (str, efemarai.domain.Domain) – The domain from which to generate samples.
dataset (str, efemarai.dataset.Dataset) – The dataset with which to generate samples.

Returns:

A baseline object.

Return type:

efemarai.baseline.Baseline

property stress_tests#

Returns a list of the stress tests associated with the project.

Returns:: A list of available stress tests.
Return type:: list [efemarai.stress_test.StressTest]

stress_test(test) → StressTest#

Returns the stress test specified by the name or id.

Parameters:: test (str) – A stress test identifier. It can either be the name of the stress test or its id.
Returns:: A stress test object or None if it doesn’t exist.
Return type:: efemarai.stress_test.StressTest

create_stress_test(name, model, domain, dataset, num_samples=25, num_runs=40, concurrent_runs=4, **kwargs) → StressTest#

Creates a stress test.

Parameters:

name (str) – The name of the stress test.
model (str, efemarai.model.Model) – The model to stress test.
domain (str, efemarai.domain.Domain) – The domain from which to generate samples.
dataset (str, efemarai.dataset.Dataset) – The dataset with which to generate samples.
num_samples (int, optional) – Sample count to generate.
num_runs (int, optional) – Number of searches to run.
concurrent_runs (int, optional) – Number of concurrent searches.

Returns:

A stress test object.

Return type:

efemarai.stress_test.StressTest

delete(delete_dependants=False)#: Deletes a project, including the domains, datasets, models, stress tests. Cannot be undone.

export_datasets(test_run_ids: list | None = None, min_score=0.0, include_dataset=False, path=None, unzip=True, ignore_cache=False, export_format=None)#

Returns the vulnerabilities dataset associated with the stress tests.

Parameters:

test_run_ids (list[str]) – Test runs ids for the datasets.
min_score (float, optional) – Minimum score to select samples.
include_dataset (bool, optional) – If the original dataset used for each of the stress tests should be included.
path (str, optional) – Path to the downloading location.
unzip (bool, optional) – If the zip file should be unzipped.
ignore_cache (bool, optional) – Force regeneration of the dataset by ignoring the cache. May lead to slower subsequent calls.
export_format (str) – The format of the output vulnerabilities dataset.

Returns:

The filename of the resulting object.

Return type:

str

download_joint_vulnerabilities_datasets(test_run_ids: list | None = None, min_score=0.0, include_dataset=False, path=None, unzip=True, ignore_cache=False, export_format=None)#

Returns a joint dataset of all specified stress tests.

Parameters:

test_run_ids (list[str]) – Ids of the test runs to download.
min_score (float, optional) – Minimum score to select samples.
include_dataset (bool, optional) – If the original dataset used for each stress test should be included.
path (str, optional) – Path to the downloading location.
unzip (bool, optional) – If the zip file should be unzipped.
ignore_cache (bool, optional) – Force regeneration of the dataset by ignoring the cache. May lead to slower subsequent calls.
export_format (str) – The format of the output vulnerabilities dataset.

Returns:

The filename of the resulting object.

Return type:

str

Dataset#

class efemarai.dataset.DatasetFormat(value)#

Possible dataset formats.

COCO = 'COCO'#

ImageRegression = 'ImageRegression'#

ImageNet = 'ImageNet'#

TFRecord = 'tfrecord'#

Custom = 'Custom'#

classmethod list()#

class efemarai.dataset.DatasetStage(value)#

An enumeration.

Train = 'train'#

Validation = 'validation'#

Test = 'test'#

classmethod list()#

class efemarai.dataset.Dataset(project, name, stage, id=None, format=None, data_url=None, annotations_url=None, state=None, state_message=None, classes=None, count=None)#

Provides dataset related functionality. It can be created through the efemarai.project.Project.create_dataset method.

Example:

import efemarai as ef
dataset = ef.Session().project("Name").create_dataset(...)
# do something else
dataset.reload()
if dataset.finished:
    print('Dataset has finished loading')

Example (2):

import efemarai as ef
project = ef.Session().project("Name")
dataset = project.dataset("Dataset Name")
print(f"Classes: {dataset.classes}")

stage#: (str) Stage of the dataset.

name#: (str) Name of the dataset.

classes#: (list [str]) List of names of the classes. None is used for those ids that don’t have a corresponding name.

format#: (str) Format of the dataset.

static deserialize_datapoint(data)#: Returns a datapoint from the data field.

property finished#

Returns true if the dataset loading has finished.

Return type:: bool

property failed#

Returns true if the dataset loading has failed.

Return type:: bool

property running#

Returns true if the is still being loaded - not failed or finished.

Return type:: bool

add_annotation_class(id, name, category=None, color=None)#

get_annotation_class(*, id=None, name=None)#

finalize(min_asset_area=15, mask_generation=None)#

delete(delete_dependants=False)#

Deletes the dataset.

You cannot delete an object that is used in a stress test or a baseline (delete those first). This cannot be undone.

reload()#

Reloads the dataset in place from the remote endpoint and return it.

Returns:: The updated dataset object.

download(num_samples=None, dataset_format=None, path=None, unzip=True, ignore_cache=False)#

Download the dataset locally.

Parameters:

num_samples (int) – Number of samples to download. Leave None for all.
dataset_format (str) – What format to download the dataset. Currently supported include COCO, YOLO, VOC, ImageNet, CSV.
path (str) – The path where to download the data.
unzip (bool) – Should the downloaded zip be unzipped.
ignore_cache (bool) – Force regeneration of the dataset by ignoring the cache. May lead to slower subsequent calls.

enhance(domain, samples_per_datapoint, new_name=None)#

Domain#

class efemarai.domain.Domain(project, id, name, transformations)#

Provides domain related functionality. The easiest way to interact with this object is through the UI. It can be created through the efemarai.project.Project.create_domain method.

name#: (str) Name of the domain

download(filename=None) → str#

Downalod the domain specifications to a local file.

Parameters:: filename (str, optional) – Specify the filename used to store the domain data.
Returns:: (str) The filename of the downloaded domain.

delete(delete_dependants=False)#: Delete the domain. You cannot delete an object that is used in a stress test or a baseline (delete those first). This cannot be undone.

Model#

class efemarai.model.Model(project, id, name, description, version, repository, files)#

Provides model related functionality.

It should be created through the efemarai.project.Project.create_model method.

Example:

import efemarai as ef
ef.Session().project("Name").create_model(...)

static push(project, name, description, repository, files, existing_model)#

Pushes a new model version.

You should use project.push_model() instead.

delete(delete_dependants=False)#

Deletes the model.

You cannot delete an object that is used in a stress test or a baseline (delete those first). Deletion cannot be undone.

versions()#: Returns all model versions.

Stress Test#

class efemarai.stress_test.StressTest(project, id, name, model, domain, dataset, state, state_message, reports)#

Provides stress test related functionality. It can be created through a efemarai.project.Project.create_stress_test method.

Example:

import efemarai as ef
test = ef.Session().project("Name").create_stress_test(...)
# do something else
test.reload()
if test.finished:
    print(f"Report: {test.report}")
    dataset_filename = test.vulnerabilities_dataset(min_score=0.1)

Example (2):

import efemarai as ef
project = ef.Session().project("Name")
test = project.stress_test("Test Name")
test.download_reports()

name#: (str) Name of the stress test.

property reports#: Returns the stress test reports. This can be slow as is potentially fetching a large object.

property model#: Returns the model associated with the stress test.

property domain#: Returns the domain associated with the stress test.

property dataset#: Returns the dataset associated with the stress test.

property finished#

Returns if the stress test has successfully finished.

Return type:: bool

property failed#

Returns if the stress test has failed.

Return type:: bool

property running#

Returns if the stress test is still running - not failed or finished.

Return type:: bool

delete()#: Deletes a stress test. This cannot be undone.

reload()#

Reloads the stress test in place from the remote endpoint and return it.

Rerturns:: The updated stress test object.

download_reports(filename=None)#

Download any generated reports.

Parameters:: filename (str) – Specify the filename used to store the report data.
Returns:: The filename of the downloaded report.
Return type:: str

vulnerabilities_dataset(min_score=0.0, include_dataset=False, path=None, unzip=True, ignore_cache=False, export_format=None)#

Returns the vulnerabilities dataset associated with the stress test.

Parameters:

min_score (float, optional) – Minimum score to select samples.
include_dataset (bool, optional) – If the original dataset used for the stress test should be included.
path (str, optional) – Path to the downloading location.
unzip (bool, optional) – If the zip file should be unzipped.
ignore_cache (bool, optional) – Force regeneration of the dataset by ignoring the cache. May lead to slower subsequent calls.
export_format (str) – The format of the output vulnerabilities dataset.

Returns:

The filename of the resulting object.

Return type:

str

Problem Type#

class efemarai.problem_type.ProblemType(value)#

Possible project problem types.

Classification = 'Classification'#

ObjectDetection = 'ObjectDetection'#

SemanticSegmentation = 'SemanticSegmentation'#

InstanceSegmentation = 'InstanceSegmentation'#

ImageRegression = 'ImageRegression'#

Keypoints = 'Keypoints'#

OutputImage = 'OutputImage'#

Custom = 'Custom'#

If you have any comments or recomentations, do send them over via email!