otx.api.entities.metrics#

This module implements the Metric entities.

Classes

`BarChartInfo`(name[, palette, visualization_type])	This represents a visualization using a bar chart.
`BarMetricsGroup`(metrics, visualization_info)	This class represent a bar or radial bar chart in the UI.
`ColorPalette`(value)	Enum class specifying the color palette to be used by the UI to display statistics.
`CountMetric`(name, value)	This metric represents an integer value.
`CurveMetric`(name, ys[, xs])	This metric represents a curve.
`DateMetric`(name[, date])	This metric represents a date time value.
`DurationMetric`(name, hour, minute, second)	This metric represents a duration metric, which include hour (int), minute (int), and second (float).
`InfoMetric`(name, value)	This metric represents a string value.
`LineChartInfo`(name[, x_axis_label, ...])	This represents a visualization using a line chart.
`LineMetricsGroup`(metrics, visualization_info)	This class represent a line chart in the UI.
`MatrixChartInfo`(name[, header, row_header, ...])	This represents a visualization using a matrix.
`MatrixMetric`(name, matrix_values[, ...])	This metric represents a matrix.
`MatrixMetricsGroup`(metrics, visualization_info)	This class represent a matrix chart in the UI.
`MetricEntity`()	This interface represents a metric, which is the smallest building block for the performance statistics.
`MetricsGroup`(metrics, visualization_info)	This class aggregates a list of metric entities and defines how this group will be visualized on the UI.
`MultiScorePerformance`([primary_score, ...])	This class can be used in tasks where performance is measured by multiple metrics.
`NullMetric`()	Represents 'Metric not found'.
`NullPerformance`()	This is used to represent 'Performance not found'.
`Performance`(score[, dashboard_metrics])	This performance class wraps the statistics of an entity (e.g., Model, Resultset).
`ScoreMetric`(name, value)	This metric represents a float value.
`TextChartInfo`(name)	This represents a visualization using text, which uses only a single string.
`TextMetricsGroup`(metrics, visualization_info)	This class represent a text chart in the UI.
`VisualizationInfo`(name, visualisation_type)	This represents the visualization info a metrics group.
`VisualizationType`(value)	This enum defines how the metrics will be visualized on the UI.

class otx.api.entities.metrics.BarChartInfo(name: str, palette: ColorPalette = ColorPalette.DEFAULT, visualization_type: VisualizationType = VisualizationType.BAR)[source]#

Bases: VisualizationInfo

This represents a visualization using a bar chart.

class otx.api.entities.metrics.BarMetricsGroup(metrics: Sequence[ScoreMetric | CountMetric], visualization_info: BarChartInfo)[source]#

Bases: MetricsGroup[Union[ScoreMetric, CountMetric], BarChartInfo]

This class represent a bar or radial bar chart in the UI.

Each metric in the metrics group represents the value of a single bar/radial bar in the chart.

class otx.api.entities.metrics.ColorPalette(value)[source]#

Bases: Enum

Enum class specifying the color palette to be used by the UI to display statistics.

If the statistics are per label, set to LABEL so the UI will use the label color palette. Otherwise, set to DEFAULT (allow the UI to choose a color palette)

class otx.api.entities.metrics.CountMetric(name: str, value: int)[source]#

Bases: MetricEntity

This metric represents an integer value.

Parameters:

name – The name of the metric
value – The value of the metric

Example

The count for number of images in a project

>>> count_metric = CountMetric(name="Number of images", value=20)

static type()[source]#: Returns the type of the MetricEntity.

class otx.api.entities.metrics.CurveMetric(name: str, ys: List[float], xs: List[float] | None = None)[source]#

Bases: MetricEntity

This metric represents a curve. The coordinates are represented as x and y lists.

Example

A line curve of: [(0,1), (1, 5), (2, 8)]

>>> CurveMetric("Line", xs=[0, 1, 2], ys=[1, 5, 8])
CurveMetric(name=`Line`, ys=(3 values), xs=(3 values))

A curve can also be defined only using the y values. For example, a loss curve of loss values: [0.5, 0.2, 0.1]. The x values will be automatically generated as a 1-based index (1, 2, 3, …)

>>> CurveMetric("Loss", ys=[0.5, 0.2, 0.1])
CurveMetric(name=`Loss`, ys=(3 values), xs=(None values))

Parameters:

name – The name of the curve
xs – the list of floats in x-axis
ys – the list of floats in y-axis

static type()[source]#: Returns the type of the MetricEntity.

property xs: List[float]#: Returns the list of floats on x-axis.

property ys: List[float]#: Returns the list of floats on y-axis.

class otx.api.entities.metrics.DateMetric(name: str, date: datetime | None = None)[source]#

Bases: MetricEntity

This metric represents a date time value.

Parameters:

name – The name of the date metric
date – The datetime value of the metric

Example

A DateMetric for model creation date (e.g., now).

>>> metric = DateMetric(name="Model creation", date=datetime.datetime.now(datetime.timezone.utc))

static type()[source]#: Returns the type of the MetricEntity.

class otx.api.entities.metrics.DurationMetric(name: str, hour: int, minute: int, second: float)[source]#

Bases: MetricEntity

This metric represents a duration metric, which include hour (int), minute (int), and second (float).

Parameters:

name – The name of the duration metric
hour – The hour value of the metric
minute – The minute value of the metric
second – The second value of the metric

Example

Creating a metric for training duration of 1 hour 5 minutes.

>>> duration_metric = DurationMetric(name="Training duration", hour=1, minute=5, second=0)

static from_seconds(name: str, seconds: float) → DurationMetric[source]#

Returns a duration metrics, with name and converted durations from seconds.

Example

Converting 70 seconds to duration metric.

>>> from otx.api.entities.metrics import DurationMetric
>>> dur_met = DurationMetric.from_seconds("test", 70)  # 1 hour 1 minute and 1.5 seconds
>>> dur_met.get_duration_string()
'1 minute 10.00 seconds'

Parameters:

name –
seconds –

Returns:

the duration metric with name and converted durations from seconds.

Return type:

DurationMetric

get_duration_string() → str[source]#

Returns the string representation of the duration.

Example

Duration string of 1 hour 1 minute and 1.50 seconds.

>>> from otx.api.entities.metrics import DurationMetric
>>> dur_met = DurationMetric("test", 1, 1, 1.5)  # 1 hour 1 minute and 1.5 seconds
>>> dur_met.get_duration_string()
'1 hour 1 minute 1.50 seconds'

Returns:: the string representation of the duration.

static type()[source]#: Returns the type of the MetricEntity.

class otx.api.entities.metrics.InfoMetric(name: str, value: str)[source]#

Bases: MetricEntity

This metric represents a string value.

Parameters:

name – The name of the info metric
value – The info of the metric

Example

An info metric of training from scratch

>>> info_metric = InfoMetric(name="Model info", value="This model is trained from scratch")

static type()[source]#: Returns the type of the MetricEntity.

class otx.api.entities.metrics.LineChartInfo(name: str, x_axis_label: str | None = None, y_axis_label: str | None = None, palette: ColorPalette = ColorPalette.DEFAULT)[source]#

Bases: VisualizationInfo

This represents a visualization using a line chart.

class otx.api.entities.metrics.LineMetricsGroup(metrics: Sequence[CurveMetric], visualization_info: LineChartInfo)[source]#

Bases: MetricsGroup[CurveMetric, LineChartInfo]

This class represent a line chart in the UI.

Multiple lines can be displayed in a single chart.

class otx.api.entities.metrics.MatrixChartInfo(name: str, header: str | None = None, row_header: str | None = None, column_header: str | None = None, palette: ColorPalette = ColorPalette.DEFAULT)[source]#

Bases: VisualizationInfo

This represents a visualization using a matrix.

class otx.api.entities.metrics.MatrixMetric(name: str, matrix_values: ndarray, row_labels: List[str] | None = None, column_labels: List[str] | None = None, normalize: bool = False)[source]#

Bases: MetricEntity

This metric represents a matrix. The cells are represented as a list of lists of integers.

In the case of a confusion matrix, the rows represent the ground truth items and the columns represent the predicted items.

Example

A matrix of: [[4,0,1], [0,3,2], [1,2,2]]

>>> MatrixMetric("Confusion Matrix", matrix_values=np.array([[4,0,1], [0,3,2], [1,2,2]]))
MatrixMetric(name=`Confusion Matrix`, matrix_values=(3x3) matrix, row labels=None, column labels=None)

Parameters:

name – The name of the matrix
matrix_values – the matrix data
row_labels – labels for the rows
column_labels – labels for the columns
normalize – set to True to normalize each row of the matrix

normalize()[source]#: Normalizes the confusion matrix by dividing by the sum of the rows.

static type()[source]#: Returns the type of the MetricEntity.

property column_labels: List[str] | None#: Returns the column labels.

property matrix_values: ndarray#: Returns the matrix data.

property row_labels: List[str] | None#: Returns the row labels.

class otx.api.entities.metrics.MatrixMetricsGroup(metrics: Sequence[MatrixMetric], visualization_info: MatrixChartInfo)[source]#

Bases: MetricsGroup[MatrixMetric, MatrixChartInfo]

This class represent a matrix chart in the UI.

Multiple matrices can be displayed in the same chart.

class otx.api.entities.metrics.MetricEntity[source]#

Bases: object

This interface represents a metric, which is the smallest building block for the performance statistics.

It only contains the name of the metric. See also MetricsGroup and Performance for the structure of performance statistics.

static type() → str[source]#: Returns the type of the MetricEntity, e.g. “curve.

property name#: Returns the name of the Metric Entity.

class otx.api.entities.metrics.MetricsGroup(metrics: Sequence[_Metric], visualization_info: _VisualizationInfo)[source]#

Bases: Generic[_Metric, _VisualizationInfo]

This class aggregates a list of metric entities and defines how this group will be visualized on the UI.

This class is the parent class to the different types of MetricsGroup that each represent a different type of chart in the UI.

Example

An accuracy as a metrics group

>>> acc = ScoreMetric("Accuracy", 0.5)
>>> visual_info = BarChartInfo("Accuracy", visualization_type=_VisualizationInfo.BAR)  # show as radial bar
>>> metrics_group = BarMetricsGroup([acc], visual_info)

Loss curves as a metrics group

>>> train_loss = CurveMetric("Train loss", xs=[0, 1, 2], ys=[5, 3, 1])
>>> val_loss = CurveMetric("Validation", xs=[0, 1, 2], ys=[6, 4, 2])
>>> visual_info = LineChartInfo("Loss curve", x_axis_label="# epoch", y_axis_label="Loss")
>>> metrics_group = LineMetricsGroup([train_loss, val_loss], visual_info)

class otx.api.entities.metrics.MultiScorePerformance(primary_score: ScoreMetric | None = None, additional_scores: List[ScoreMetric] | None = None, dashboard_metrics: List[MetricsGroup] | None = None)[source]#

Bases: Performance

This class can be used in tasks where performance is measured by multiple metrics.

Parameters:

primary_score – The main performance score.
additional_metrics – List of additional scores. When no primary score is provided, the first additional score takes priority as the main project score.
dashboard_metrics – (optional) additional statistics, containing charts, curves, and other additional info.

property additional_scores: List[ScoreMetric]#: Return the additional score metrics.

property primary_score: ScoreMetric | None#: Return the primary score metric.

class otx.api.entities.metrics.NullMetric[source]#

Bases: MetricEntity

Represents ‘Metric not found’.

static type()[source]#: Returns the type of the MetricEntity.

class otx.api.entities.metrics.NullPerformance[source]#

Bases: Performance

This is used to represent ‘Performance not found’.

class otx.api.entities.metrics.Performance(score: ScoreMetric, dashboard_metrics: List[MetricsGroup] | None = None)[source]#

Bases: object

This performance class wraps the statistics of an entity (e.g., Model, Resultset).

Parameters:

score – the performance score. This will be the point of comparison between two performances.
dashboard_metrics – (optional) additional statistics, containing charts, curves, and other additional info.

property score#: Return the score metric.

class otx.api.entities.metrics.ScoreMetric(name: str, value: float)[source]#

Bases: MetricEntity

This metric represents a float value.

This metric is typically used for storing performance metrics, such as accuracy, f-measure, dice score, etc.

Parameters:

name – The name of the score
value – The value of the score

Example

Accuracy of a model

>>> score_metric = ScoreMetric(name="Model accuracy", value=0.5)

static type()[source]#: Returns the type of the MetricEntity.

class otx.api.entities.metrics.TextChartInfo(name: str)[source]#

Bases: VisualizationInfo

This represents a visualization using text, which uses only a single string.

class otx.api.entities.metrics.TextMetricsGroup(metrics: Sequence[ScoreMetric | CountMetric | InfoMetric | DateMetric | DurationMetric], visualization_info: TextChartInfo)[source]#

Bases: MetricsGroup[Union[ScoreMetric, CountMetric, InfoMetric, DateMetric, DurationMetric], TextChartInfo]

This class represent a text chart in the UI.

Text charts contain only one metric, which can be of type CountMetric, ScoreMetric, DateMetric, DurationMetric or InfoMetric.

class otx.api.entities.metrics.VisualizationInfo(name: str, visualisation_type: VisualizationType, palette: ColorPalette = ColorPalette.DEFAULT)[source]#

Bases: object

This represents the visualization info a metrics group. See MetricsGroup.

property type: VisualizationType#: Returns the type of the visualization.

class otx.api.entities.metrics.VisualizationType(value)[source]#

Bases: Enum

This enum defines how the metrics will be visualized on the UI.