datumaro.components.merge.union_merge#

Classes

UnionMerge(**options)

Merge several datasets with "union" policy:

class datumaro.components.merge.union_merge.UnionMerge(**options)[source]#

Bases: Merger

Merge several datasets with “union” policy:

Label categories are merged according to the union of their label names.

For example, if Dataset-A has {“car”, “cat”, “dog”} and Dataset-B has {“car”, “bus”, “truck”} labels, the merged dataset will have {“bust”, “car”, “cat”, “dog”, “truck”} labels.

If there are two or more dataset items whose (id, subset) pairs match each other,

both are included in the merged dataset. At this time, since the same (id, subset) pair cannot be duplicated in the dataset, we add a suffix to the id of each source item. For example, if Dataset-A has DatasetItem(id=”magic”, subset=”train”) and Dataset-B has also DatasetItem(id=”magic”, subset=”train”), the merged dataset will have DatasetItem(id=”magic-0”, subset=”train”) and DatasetItem(id=”magic-1”, subset=”train”).

merge(sources: Sequence[IDataset]) → DatasetItemStorage[source]#

merge_categories(sources: Sequence[IDataset]) → Dict[source]#

class datumaro.components.merge.union_merge.AnnotationType(value)[source]#

Bases: IntEnum

An enumeration.

unknown = 0#

label = 1#

mask = 2#

points = 3#

polygon = 4#

polyline = 5#

bbox = 6#

caption = 7#

cuboid_3d = 8#

super_resolution_annotation = 9#

depth_annotation = 10#

ellipse = 11#

hash_key = 12#

feature_vector = 13#

tabular = 14#

rotated_bbox = 15#

cuboid_2d = 16#

Bases: object

id: str#

subset: str#

media: MediaElement | None#

annotations: Annotations#

attributes: Dict[str, Any]#

wrap(**kwargs)[source]#

media_as(t: Type[T]) → T[source]#

class datumaro.components.merge.union_merge.DatasetItemStorage[source]#

Bases: object

is_empty() → bool[source]#

put(item: DatasetItem) → bool[source]#

get(id: str | DatasetItem, subset: str | None = None, dummy: Any | None = None) → DatasetItem | None[source]#

remove(id: str | DatasetItem, subset: str | None = None) → bool[source]#

get_subset(name)[source]#

subsets()[source]#

get_annotated_items()[source]#

get_datasetitem_by_path(path)[source]#

get_annotations()[source]#

class datumaro.components.merge.union_merge.IDataset[source]#

Bases: object

subsets() → Dict[str, IDataset][source]#: Enumerates subsets in the dataset. Each subset can be a dataset itself.

get_subset(name) → IDataset[source]#

infos() → Dict[str, Any][source]#: Returns meta-info of dataset.

categories() → Dict[AnnotationType, Categories][source]#: Returns metainfo about dataset labels.

get(id: str, subset: str | None = None) → DatasetItem | None[source]#: Provides random access to dataset items.

media_type() → Type[MediaElement][source]#

Returns media type of the dataset items.

All the items are supposed to have the same media type. Supposed to be constant and known immediately after the object construction (i.e. doesn’t require dataset iteration).

ann_types() → List[AnnotationType][source]#: Returns available task type from dataset annotation types.

property is_stream: bool#

Boolean indicating whether the dataset is a stream

If the dataset is a stream, the dataset item is generated on demand from its iterator.

class datumaro.components.merge.union_merge.LabelCategories(items: List[str] = _Nothing.NOTHING, label_groups: List[LabelGroup] = _Nothing.NOTHING, *, attributes: Set[str] = _Nothing.NOTHING)[source]#

Bases: Categories

Method generated by attrs for class LabelCategories.

class Category(name, parent: str = '', attributes: Set[str] = _Nothing.NOTHING)[source]#

Bases: object

Method generated by attrs for class LabelCategories.Category.

name: str#

parent: str#

attributes: Set[str]#

class LabelGroup(name, labels: List[str] = [], group_type: GroupType = GroupType.EXCLUSIVE)[source]#

Bases: object

Method generated by attrs for class LabelCategories.LabelGroup.

name: str#

labels: List[str]#

group_type: GroupType#

items: List[str]#

label_groups: List[LabelGroup]#

classmethod from_iterable(iterable: Iterable[str | Tuple[str] | Tuple[str, str] | Tuple[str, str, List[str]]]) → LabelCategories[source]#

Creates a LabelCategories from iterable.

Parameters:

iterable –

This iterable object can be:

a list of str - will be interpreted as list of Category names
a list of positional arguments - will generate Categories with these arguments

Returns: a LabelCategories object

add(name: str, parent: str | None = None, attributes: Set[str] | None = None) → int[source]#

add_label_group(name: str, labels: List[str], group_type: GroupType) → int[source]#

find(name: str) → Tuple[int | None, Category | None][source]#

class datumaro.components.merge.union_merge.Merger(**options)[source]#

Bases: IMergerContext, CliPlugin

Merge multiple datasets into one dataset

static merge_infos(sources: Sequence[Dict[str, Any]]) → Dict[source]#: Merge several IDataset into one IDataset

static merge_categories(sources: Sequence[Dict[AnnotationType, Categories]]) → Dict[source]#

static merge_media_types(sources: Sequence[IDataset]) → Type[MediaElement] | None[source]#

static merge_ann_types(sources: Sequence[IDataset]) → Set[AnnotationType] | None[source]#

save_merge_report(path: str) → None[source]#

get_any_label_name(ann, label_id)[source]#

class datumaro.components.merge.union_merge.defaultdict#

Bases: dict

defaultdict(default_factory=None, /, […]) –> dict with default factory

The default factory is called without arguments to produce a new value when a key is not present, in __getitem__ only. A defaultdict compares equal to a dict with the same items. All remaining arguments are treated the same as if they were passed to the dict constructor, including keyword arguments.

copy() → a shallow copy of D.#

default_factory#: Factory for default value called by __missing__().