otx.algo.utils.weight_init#

This implementation replaces the functionality of mmengine.model.weight_init.

Functions

bias_init_with_prob(prior_prob)

Initialize conv/fc bias value according to a given probability value.

constant_init(module, val[, bias])

Initialize the weights and biases of a module with constant values.

initialize(module, init_cfg)

Initialize a module.

kaiming_init(module[, a, mode, ...])

Initialize the weights and biases of a module using the Kaiming initialization method.

normal_init(module[, mean, std, bias])

Initialize the weights and biases of a module using a normal distribution.

trunc_normal_(tensor[, mean, std, a, b])

Fills the input Tensor with values drawn from a truncated normal distribution.

trunc_normal_init(module[, mean, std, a, b, ...])

Initialize the weights and biases of a module using a truncated normal distribution.

uniform_init(module[, a, b, bias])

Initialize the weights and biases of a module using a uniform distribution.

update_init_info(module, init_info)

Update the _params_init_info in the module if the value of parameters are changed.

xavier_init(module[, gain, bias, distribution])

Initialize the weights and biases of a module using Xavier initialization.

Classes

BaseInit(*[, bias, bias_prob, layer])

Base Init class.

ConstantInit(val, **kwargs)

Initialize module parameters with constant values.

KaimingInit([a, mode, nonlinearity, ...])

Initialize module parameters with the values according to the method described in the paper below.

NormalInit([mean, std])

Initialize module parameters with the values drawn from the normal distribution.

PretrainedInit(checkpoint[, prefix, ...])

Initialize module by loading a pretrained model.

TruncNormalInit([mean, std, a, b])

Initialize module parameters with the values drawn from the normal distribution.

UniformInit([a, b])

Initialize module parameters with values drawn from the uniform distribution \(\mathcal{U}(a, b)\).

XavierInit([gain, distribution])

Initialize module parameters with values according to the method described in the paper below.

class otx.algo.utils.weight_init.BaseInit(*, bias: int | float = 0, bias_prob: float | None = None, layer: str | list | None = None)[source]#

Bases: object

Base Init class.

class otx.algo.utils.weight_init.ConstantInit(val: int | float, **kwargs)[source]#

Bases: BaseInit

Initialize module parameters with constant values.

Parameters:
  • val (int | float) – the value to fill the weights in the module with

  • bias (int | float) – the value to fill the bias. Defaults to 0.

  • bias_prob (float, optional) – the probability for bias initialization. Defaults to None.

  • layer (str | list[str], optional) – the layer will be initialized. Defaults to None.

__call__(module: Module) None[source]#

Initialize the module parameters.

Parameters:

module (nn.Module) – The module to initialize.

Returns:

None

class otx.algo.utils.weight_init.KaimingInit(a: int | float = 0, mode: str = 'fan_out', nonlinearity: str = 'relu', distribution: str = 'normal', **kwargs)[source]#

Bases: BaseInit

Initialize module parameters with the values according to the method described in the paper below.

Delving deep into rectifiers: Surpassing human-level performance on ImageNet classification - He, K. et al. (2015).

Parameters:
  • a (int | float) – the negative slope of the rectifier used after this layer (only used with 'leaky_relu'). Defaults to 0.

  • mode (str) – either 'fan_in' or 'fan_out'. Choosing 'fan_in' preserves the magnitude of the variance of the weights in the forward pass. Choosing 'fan_out' preserves the magnitudes in the backwards pass. Defaults to 'fan_out'.

  • nonlinearity (str) – the non-linear function (nn.functional name), recommended to use only with 'relu' or 'leaky_relu' . Defaults to ‘relu’.

  • bias (int | float) – the value to fill the bias. Defaults to 0.

  • bias_prob (float, optional) – the probability for bias initialization. Defaults to None.

  • distribution (str) – distribution either be 'normal' or 'uniform'. Defaults to 'normal'.

  • layer (str | list[str], optional) – the layer will be initialized. Defaults to None.

__call__(module: Module) None[source]#

Initialize the module parameters.

Parameters:

module (nn.Module) – The module to initialize.

class otx.algo.utils.weight_init.NormalInit(mean: int | float = 0, std: int | float = 1, **kwargs)[source]#

Bases: BaseInit

Initialize module parameters with the values drawn from the normal distribution.

\(\mathcal{N}(\text{mean}, \text{std}^2)\).

Parameters:
  • mean (int | float) – the mean of the normal distribution. Defaults to 0.

  • std (int | float) – the standard deviation of the normal distribution. Defaults to 1.

  • bias (int | float) – the value to fill the bias. Defaults to 0.

  • bias_prob (float, optional) – the probability for bias initialization. Defaults to None.

  • layer (str | list[str], optional) – the layer will be initialized. Defaults to None.

__call__(module: Module) None[source]#

Initialize the module parameters.

Parameters:

module (nn.Module) – The module to initialize.

Returns:

None

class otx.algo.utils.weight_init.PretrainedInit(checkpoint: str, prefix: str | None = None, map_location: str = 'cpu')[source]#

Bases: object

Initialize module by loading a pretrained model.

Parameters:
  • checkpoint (str) – the checkpoint file of the pretrained model should be load.

  • prefix (str, optional) – the prefix of a sub-module in the pretrained model. it is for loading a part of the pretrained model to initialize. For example, if we would like to only load the backbone of a detector model, we can set prefix='backbone.'. Defaults to None.

  • map_location (str) – map tensors into proper locations. Defaults to cpu.

__call__(module: Module) None[source]#

Initialize the module parameters by loading a pretrained model.

Parameters:

module (nn.Module) – The module to initialize.

class otx.algo.utils.weight_init.TruncNormalInit(mean: float = 0, std: float = 1, a: float = -2, b: float = 2, **kwargs)[source]#

Bases: BaseInit

Initialize module parameters with the values drawn from the normal distribution.

\(\mathcal{N}(\text{mean}, \text{std}^2)\) with values outside \([a, b]\).

Parameters:
  • mean (float) – the mean of the normal distribution. Defaults to 0.

  • std (float) – the standard deviation of the normal distribution. Defaults to 1.

  • a (float) – The minimum cutoff value.

  • b (float) – The maximum cutoff value.

  • bias (float) – the value to fill the bias. Defaults to 0.

  • bias_prob (float, optional) – the probability for bias initialization. Defaults to None.

  • layer (str | list[str], optional) – the layer will be initialized. Defaults to None.

__call__(module: Module) None[source]#

Apply weight initialization to the given module.

Parameters:

module (nn.Module) – The module to initialize.

class otx.algo.utils.weight_init.UniformInit(a: int | float = 0, b: int | float = 1, **kwargs)[source]#

Bases: BaseInit

Initialize module parameters with values drawn from the uniform distribution \(\mathcal{U}(a, b)\).

Parameters:
  • a (int | float) – the lower bound of the uniform distribution. Defaults to 0.

  • b (int | float) – the upper bound of the uniform distribution. Defaults to 1.

  • bias (int | float) – the value to fill the bias. Defaults to 0.

  • bias_prob (float, optional) – the probability for bias initialization. Defaults to None.

  • layer (str | list[str], optional) – the layer will be initialized. Defaults to None.

__call__(module: Module) None[source]#

Apply weight initialization to the given module.

Parameters:

module (nn.Module) – The module to initialize.

class otx.algo.utils.weight_init.XavierInit(gain: int | float = 1, distribution: str = 'normal', **kwargs)[source]#

Bases: BaseInit

Initialize module parameters with values according to the method described in the paper below.

Understanding the difficulty of training deep feedforward neural networks - Glorot, X. & Bengio, Y. (2010).

Parameters:
  • gain (int | float) – an optional scaling factor. Defaults to 1.

  • bias (int | float) – the value to fill the bias. Defaults to 0.

  • bias_prob (float, optional) – the probability for bias initialization. Defaults to None.

  • distribution (str) – distribution either be 'normal' or 'uniform'. Defaults to 'normal'.

  • layer (str | list[str], optional) – the layer will be initialized. Defaults to None.

__call__(module: Module) None[source]#

Initialize the module parameters.

Parameters:

module (nn.Module) – The module to initialize.

Returns:

None

otx.algo.utils.weight_init.bias_init_with_prob(prior_prob: float) float[source]#

Initialize conv/fc bias value according to a given probability value.

otx.algo.utils.weight_init.constant_init(module: Module, val: float, bias: float = 0) None[source]#

Initialize the weights and biases of a module with constant values.

Copied from mmengine.model.weight_init.constant_init

Parameters:
  • module (nn.Module) – The module to initialize.

  • val (float) – The constant value to initialize the weights with.

  • bias (float, optional) – The constant value to initialize the biases with. Defaults to 0.

otx.algo.utils.weight_init.initialize(module: Module, init_cfg: dict | list[dict]) None[source]#

Initialize a module.

Parameters:
  • module (torch.nn.Module) – the module will be initialized.

  • init_cfg (dict | list[dict]) – initialization configuration dict to define initializer. OpenMMLab has implemented 6 initializers including Constant, Xavier, Normal, Uniform, Kaiming, and Pretrained.

Example

>>> module = nn.Linear(2, 3, bias=True)
>>> init_cfg = dict(type='Constant', layer='Linear', val =1 , bias =2)
>>> initialize(module, init_cfg)
>>> module = nn.Sequential(nn.Conv1d(3, 1, 3), nn.Linear(1,2))
>>> # define key ``'layer'`` for initializing layer with different
>>> # configuration
>>> init_cfg = [dict(type='Constant', layer='Conv1d', val=1),
        dict(type='Constant', layer='Linear', val=2)]
>>> initialize(module, init_cfg)
>>> # define key``'override'`` to initialize some specific part in
>>> # module
>>> class FooNet(nn.Module):
>>>     def __init__(self):
>>>         super().__init__()
>>>         self.feat = nn.Conv2d(3, 16, 3)
>>>         self.reg = nn.Conv2d(16, 10, 3)
>>>         self.cls = nn.Conv2d(16, 5, 3)
>>> model = FooNet()
>>> init_cfg = dict(type='Constant', val=1, bias=2, layer='Conv2d',
>>>     override=dict(type='Constant', name='reg', val=3, bias=4))
>>> initialize(model, init_cfg)
>>> model = ResNet(depth=50)
>>> # Initialize weights with the pretrained model.
>>> init_cfg = dict(type='Pretrained',
        checkpoint='torchvision://resnet50')
>>> initialize(model, init_cfg)
>>> # Initialize weights of a sub-module with the specific part of
>>> # a pretrained model by using "prefix".
>>> url = 'http://download.openmmlab.com/mmdetection/v2.0/retinanet/'\
>>>     'retinanet_r50_fpn_1x_coco/'\
>>>     'retinanet_r50_fpn_1x_coco_20200130-c2398f9e.pth'
>>> init_cfg = dict(type='Pretrained',
        checkpoint=url, prefix='backbone.')
otx.algo.utils.weight_init.kaiming_init(module: Module, a: float = 0, mode: str = 'fan_out', nonlinearity: str = 'relu', bias: float = 0, distribution: str = 'normal') None[source]#

Initialize the weights and biases of a module using the Kaiming initialization method.

Copied from mmengine.model.weight_init.kaiming_init

Parameters:
  • module (nn.Module) – The module to initialize.

  • a (float) – The negative slope of the rectifier used after this layer (only used with ‘leaky_relu’ nonlinearity). Default is 0.

  • mode (str) – Either ‘fan_in’ (default) or ‘fan_out’. Choosing ‘fan_in’ preserves the magnitude of the variance of the weights in the forward pass. Choosing ‘fan_out’ preserves the magnitudes in the backward pass. Default is ‘fan_out’.

  • nonlinearity (str) – The non-linear function (nn.functional name), recommended to use ‘relu’ or ‘leaky_relu’. Default is ‘relu’.

  • bias (float) – The bias value. Default is 0.

  • distribution (str) – The type of distribution to use for weight initialization, either ‘normal’ (default) or ‘uniform’.

Returns:

The initialized tensor.

Return type:

Tensor

otx.algo.utils.weight_init.normal_init(module: Module, mean: float = 0, std: float = 1, bias: float = 0) None[source]#

Initialize the weights and biases of a module using a normal distribution.

Copied from mmengine.model.weight_init.normal_init

Parameters:
  • module (nn.Module) – The module to initialize.

  • mean (float) – The mean of the normal distribution. Default is 0.

  • std (float) – The standard deviation of the normal distribution. Default is 1.

  • bias (float) – The bias value. Default is 0.

otx.algo.utils.weight_init.trunc_normal_(tensor: Tensor, mean: float = 0.0, std: float = 1.0, a: float = -2.0, b: float = 2.0) Tensor[source]#

Fills the input Tensor with values drawn from a truncated normal distribution.

The values are effectively drawn from the normal distribution \(\mathcal{N}(\text{mean}, \text{std}^2)\) with values outside \([a, b]\) redrawn until they are within the bounds. The method used for generating the random values works best when \(a \leq \text{mean} \leq b\).

Modified from pytorch/pytorch

Parameters:
  • tensor (torch.Tensor) – an n-dimensional torch.Tensor.

  • mean (float) – the mean of the normal distribution.

  • std (float) – the standard deviation of the normal distribution.

  • a (float) – the minimum cutoff value.

  • b (float) – the maximum cutoff value.

otx.algo.utils.weight_init.trunc_normal_init(module: Module, mean: float = 0, std: float = 1, a: float = -2, b: float = 2, bias: float = 0) None[source]#

Initialize the weights and biases of a module using a truncated normal distribution.

Parameters:
  • module (nn.Module) – The module to initialize.

  • mean (float) – The mean of the truncated normal distribution. Default is 0.

  • std (float) – The standard deviation of the truncated normal distribution. Default is 1.

  • a (float) – The lower bound of the truncated normal distribution. Default is -2.

  • b (float) – The upper bound of the truncated normal distribution. Default is 2.

  • bias (float) – The bias value. Default is 0.

otx.algo.utils.weight_init.uniform_init(module: Module, a: int | float = 0, b: int | float = 1, bias: int | float = 0) None[source]#

Initialize the weights and biases of a module using a uniform distribution.

Parameters:
  • module (nn.Module) – The module to initialize.

  • a (int | float) – The lower bound of the uniform distribution. Default is 0.

  • b (int | float) – The upper bound of the uniform distribution. Default is 1.

  • bias (int | float) – The bias value. Default is 0.

otx.algo.utils.weight_init.update_init_info(module: Module, init_info: str) None[source]#

Update the _params_init_info in the module if the value of parameters are changed.

Parameters:
  • (obj (module) – nn.Module): The module of PyTorch with a user-defined attribute _params_init_info which records the initialization information.

  • init_info (str) – The string that describes the initialization.

otx.algo.utils.weight_init.xavier_init(module: Module, gain: int | float = 1, bias: int | float = 0, distribution: str = 'normal') None[source]#

Initialize the weights and biases of a module using Xavier initialization.

Parameters:
  • module (nn.Module) – The module to initialize.

  • gain (int | float) – The scaling factor for the weights. Default is 1.

  • bias (int | float) – The bias value. Default is 0.

  • distribution (str) – The distribution to use for weight initialization. Can be ‘uniform’ or ‘normal’. Default is ‘normal’.