Validate#

Validate Dataset#

This command inspects annotations with respect to the task type and stores the results in JSON file.

The task types supported are classification, detection, segmentation and tabular (the -t/--task-type parameter).

The validation result contains

  • annotation statistics based on the task type

  • validation reports, such as

    • items not having annotations

    • items having undefined annotations

    • imbalanced distribution in class/attributes

    • too small or large values

  • summary

Usage:

datum validate [-h] -t TASK [-s SUBSET_NAME] [-p PROJECT_DIR]
               [target] [-- EXTRA_ARGS]

Parameters:

  • <target> (string) - Target dataset revpath. By default, validates the current project.

  • -t, --task-type (string) - Task type for validation

  • -s, --subset (string) - Dataset subset to be validated

  • -p, --project (string) - Directory of the project to operate on (default: current directory).

  • -h, --help - Print the help message and exit.

  • <extra args> - The list of extra validation parameters. Should be passed after the -- separator after the main command arguments:

    • -fs, --few-samples-thr (number) - The threshold for giving a warning for minimum number of samples per class

    • -ir, --imbalance-ratio-thr (number) - The threshold for giving imbalance data warning

    • -m, --far-from-mean-thr (number) - The threshold for giving a warning that data is far from mean

    • -dr, --dominance-ratio-thr (number) - The threshold for giving a warning bounding box imbalance

    • -k, --topk-bins (number) - The ratio of bins with the highest number of data to total bins in the histogram

Examples:

  • Validate a project’s subset as a classification dataset

    datum validate -t classification -s subset
    
  • Give warning when imbalance ratio of data with classification task over 40

    datum validate -p <path/to/project/> -t classification -- -ir 40
    

List of validation items (annomaly types)#

Anomaly Type

Description

Task Type

MissingLabelCategories

Metadata (ex. LabelCategories) should be defined

common

MissingAnnotation

No annotation found for an Item

common

MissingAttribute

An attribute key is missing for an Item

common

MultiLabelAnnotations

Item needs a single label

classification

UndefinedLabel

A label not defined in the metadata is found for an item

common

UndefinedAttribute

An attribute not defined in the metadata is found for an item

common

LabelDefinedButNotFound

A label is defined, but not found actually

common

AttributeDefinedButNotFound

An attribute is defined, but not found actually

common

OnlyOneLabel

The dataset consists of only label

common

OnlyOneAttributeValue

The dataset consists of only attribute value

common

FewSamplesInLabel

The number of samples in a label might be too low

common

FewSamplesInAttribute

The number of samples in an attribute might be too low

common

ImbalancedLabels

There is an imbalance in the label distribution

common

ImbalancedAttribute

There is an imbalance in the attribute distribution

common

ImbalancedDistInLabel

Values (ex. bbox width) are not evenly distributed for a label

detection, segmentation

ImbalancedDistInAttribute

Values (ex. bbox width) are not evenly distributed for an attribute

detection, segmentation

NegativeLength

The width or height of bounding box is negative

detection

InvalidValue

There’s invalid (ex. inf, nan) value for bounding box info.

detection

FarFromLabelMean

An annotation has an too small or large value than average for a label

detection, segmentation

FarFromAttrMean

An annotation has an too small or large value than average for an attribute

detection, segmentation

BrokenAnnotation

Some annotations are not defined for an item

tabular

EmptyLabel

A value of the label column is not defined for an item

tabular

EmptyCaption

A value of the caption column is not defined for an item

tabular

FewSamplesInCaption

The number of samples in a caption might be too low

tabular

RedundanciesInCaption

Redundancies of an caption for an item

tabular

ImbalancedCaptions

There is an imbalance in the caption distribution

tabular

ImbalancedDistInCaption

Values are not evenly distributed for a caption only if caption is number

tabular

FarFromCaptionMean

An annotation has an too small or large value than average for a caption only if caption is number

tabular

OutlierInCaption

An annotation has an outlier value based on Interquartile Range Method only if caption is number

tabular

Validation Result Format:

{
    'statistics': {
        ## common statistics
        'label_distribution': {
            'defined_labels': <dict>,   # <label:str>: <count:int>
            'undefined_labels': <dict>
            # <label:str>: {
            #     'count': <int>,
            #     'items_with_undefined_label': [<item_key>, ]
            # }
        },
        'attribute_distribution': {
            'defined_attributes': <dict>,
            # <label:str>: {
            #     <attribute:str>: {
            #         'distribution': {<attr_value:str>: <count:int>, },
            #         'items_missing_attribute': [<item_key>, ]
            #     }
            # }
            'undefined_attributes': <dict>
            # <label:str>: {
            #     <attribute:str>: {
            #         'distribution': {<attr_value:str>: <count:int>, },
            #         'items_with_undefined_attr': [<item_key>, ]
            #     }
            # }
        },
        'total_ann_count': <int>,
        'items_missing_annotation': <list>, # [<item_key>, ]

        ## statistics for classification task
        'items_with_multiple_labels': <list>, # [<item_key>, ]

        ## statistics for detection task
        'items_with_invalid_value': <dict>,
        # '<item_key>': {<ann_id:int>: [ <property:str>, ], }
        # - properties: 'x', 'y', 'width', 'height',
        #               'area(wxh)', 'ratio(w/h)', 'short', 'long'
        # - 'short' is min(w,h) and 'long' is max(w,h).
        'items_with_negative_length': <dict>,
        # '<item_key>': { <ann_id:int>: { <'width'|'height'>: <value>, }, }
        'bbox_distribution_in_label': <dict>, # <label:str>: <bbox_template>
        'bbox_distribution_in_attribute': <dict>,
        # <label:str>: {<attribute:str>: { <attr_value>: <bbox_template>, }, }
        'bbox_distribution_in_dataset_item': <dict>,
        # '<item_key>': <bbox count:int>

        ## statistics for segmentation task
        'items_with_invalid_value': <dict>,
        # '<item_key>': {<ann_id:int>: [ <property:str>, ], }
        # - properties: 'area', 'width', 'height'
        'mask_distribution_in_label': <dict>, # <label:str>: <mask_template>
        'mask_distribution_in_attribute': <dict>,
        # <label:str>: {
        #     <attribute:str>: { <attr_value>: <mask_template>, }
        # }
        'mask_distribution_in_dataset_item': <dict>,
        # '<item_key>': <mask/polygon count: int>

        ## statistics for tabular task
        'items_broken_annotation': <list>, # [<item_key>, ]
        'label_distribution': {
            'defined_labels': <dict>,   # <label:str>: <count:int>
            'empty_labels': <dict>
            # <label:str>: {
            #     'count': <int>,
            #     'items_with_empty_label': [<item_key>, ]
            # }
        },
        'caption_distribution': {
            'defined_captions': <dict>,   # <label:str>: <count:int>
            'empty_captions': <dict>
            # <label:str>: {
            #     'count': <int>,
            #     'items_with_empty_label': [<item_key>, ]
            # }
            'redundancies': <dict>
            # <label:str>: {
            #     'stopword': <dict>,
            #         'count': <int>,
            #         'items_with_redundancies': [<item_key>, ]
            #     'url': <dict>,
            #         'count': <int>,
            #         'items_with_redundancies': [<item_key>, ]
            #     }
            # }
        },

    },
    'validation_reports': <list>, # [ <validation_error_format>, ]
    # validation_error_format = {
    #     'anomaly_type': <str>,
    #     'description': <str>,
    #     'severity': <str>, # 'warning' or 'error'
    #     'item_id': <str>,  # optional, when it is related to a DatasetItem
    #     'subset': <str>,   # optional, when it is related to a DatasetItem
    # }
    'summary': {
        'errors': <count: int>,
        'warnings': <count: int>
    }
}

item_key is defined as,

item_key = (<DatasetItem.id:str>, <DatasetItem.subset:str>)

bbox_template and mask_template are defined as,

bbox_template = {
    'width': <numerical_stat_template>,
    'height': <numerical_stat_template>,
    'area(wxh)': <numerical_stat_template>,
    'ratio(w/h)': <numerical_stat_template>,
    'short': <numerical_stat_template>, # short = min(w, h)
    'long': <numerical_stat_template>   # long = max(w, h)
}
mask_template = {
    'area': <numerical_stat_template>,
    'width': <numerical_stat_template>,
    'height': <numerical_stat_template>
}

numerical_stat_template is defined as,

numerical_stat_template = {
    'items_far_from_mean': <dict>,
    # {'<item_key>': {<ann_id:int>: <value:float>, }, }
    'mean': <float>,
    'stddev': <float>,
    'min': <float>,
    'max': <float>,
    'median': <float>,
    'histogram': {
        'bins': <list>,   # [<float>, ]
        'counts': <list>, # [<int>, ]
    }
}