otx.algo.utils.weight_init#
This implementation replaces the functionality of mmengine.model.weight_init.
Functions
|
Initialize conv/fc bias value according to a given probability value. |
|
Initialize the weights and biases of a module with constant values. |
|
Initialize a module. |
|
Initialize the weights and biases of a module using the Kaiming initialization method. |
|
Initialize the weights and biases of a module using a normal distribution. |
|
Fills the input Tensor with values drawn from a truncated normal distribution. |
|
Initialize the weights and biases of a module using a truncated normal distribution. |
|
Initialize the weights and biases of a module using a uniform distribution. |
|
Update the _params_init_info in the module if the value of parameters are changed. |
|
Initialize the weights and biases of a module using Xavier initialization. |
Classes
|
Base Init class. |
|
Initialize module parameters with constant values. |
|
Initialize module parameters with the values according to the method described in the paper below. |
|
Initialize module parameters with the values drawn from the normal distribution. |
|
Initialize module by loading a pretrained model. |
|
Initialize module parameters with the values drawn from the normal distribution. |
|
Initialize module parameters with values drawn from the uniform distribution \(\mathcal{U}(a, b)\). |
|
Initialize module parameters with values according to the method described in the paper below. |
- class otx.algo.utils.weight_init.BaseInit(*, bias: int | float = 0, bias_prob: float | None = None, layer: str | list | None = None)[source]#
Bases:
object
Base Init class.
- class otx.algo.utils.weight_init.ConstantInit(val: int | float, **kwargs)[source]#
Bases:
BaseInit
Initialize module parameters with constant values.
- Parameters:
val (int | float) – the value to fill the weights in the module with
bias (int | float) – the value to fill the bias. Defaults to 0.
bias_prob (float, optional) – the probability for bias initialization. Defaults to None.
layer (str | list[str], optional) – the layer will be initialized. Defaults to None.
- class otx.algo.utils.weight_init.KaimingInit(a: int | float = 0, mode: str = 'fan_out', nonlinearity: str = 'relu', distribution: str = 'normal', **kwargs)[source]#
Bases:
BaseInit
Initialize module parameters with the values according to the method described in the paper below.
- Parameters:
a (int | float) – the negative slope of the rectifier used after this layer (only used with
'leaky_relu'
). Defaults to 0.mode (str) – either
'fan_in'
or'fan_out'
. Choosing'fan_in'
preserves the magnitude of the variance of the weights in the forward pass. Choosing'fan_out'
preserves the magnitudes in the backwards pass. Defaults to'fan_out'
.nonlinearity (str) – the non-linear function (nn.functional name), recommended to use only with
'relu'
or'leaky_relu'
. Defaults to ‘relu’.bias (int | float) – the value to fill the bias. Defaults to 0.
bias_prob (float, optional) – the probability for bias initialization. Defaults to None.
distribution (str) – distribution either be
'normal'
or'uniform'
. Defaults to'normal'
.layer (str | list[str], optional) – the layer will be initialized. Defaults to None.
- class otx.algo.utils.weight_init.NormalInit(mean: int | float = 0, std: int | float = 1, **kwargs)[source]#
Bases:
BaseInit
Initialize module parameters with the values drawn from the normal distribution.
\(\mathcal{N}(\text{mean}, \text{std}^2)\).
- Parameters:
mean (int | float) – the mean of the normal distribution. Defaults to 0.
std (int | float) – the standard deviation of the normal distribution. Defaults to 1.
bias (int | float) – the value to fill the bias. Defaults to 0.
bias_prob (float, optional) – the probability for bias initialization. Defaults to None.
layer (str | list[str], optional) – the layer will be initialized. Defaults to None.
- class otx.algo.utils.weight_init.PretrainedInit(checkpoint: str, prefix: str | None = None, map_location: str = 'cpu')[source]#
Bases:
object
Initialize module by loading a pretrained model.
- Parameters:
checkpoint (str) – the checkpoint file of the pretrained model should be load.
prefix (str, optional) – the prefix of a sub-module in the pretrained model. it is for loading a part of the pretrained model to initialize. For example, if we would like to only load the backbone of a detector model, we can set
prefix='backbone.'
. Defaults to None.map_location (str) – map tensors into proper locations. Defaults to cpu.
- class otx.algo.utils.weight_init.TruncNormalInit(mean: float = 0, std: float = 1, a: float = -2, b: float = 2, **kwargs)[source]#
Bases:
BaseInit
Initialize module parameters with the values drawn from the normal distribution.
\(\mathcal{N}(\text{mean}, \text{std}^2)\) with values outside \([a, b]\).
- Parameters:
mean (float) – the mean of the normal distribution. Defaults to 0.
std (float) – the standard deviation of the normal distribution. Defaults to 1.
a (float) – The minimum cutoff value.
b (float) – The maximum cutoff value.
bias (float) – the value to fill the bias. Defaults to 0.
bias_prob (float, optional) – the probability for bias initialization. Defaults to None.
layer (str | list[str], optional) – the layer will be initialized. Defaults to None.
- class otx.algo.utils.weight_init.UniformInit(a: int | float = 0, b: int | float = 1, **kwargs)[source]#
Bases:
BaseInit
Initialize module parameters with values drawn from the uniform distribution \(\mathcal{U}(a, b)\).
- Parameters:
a (int | float) – the lower bound of the uniform distribution. Defaults to 0.
b (int | float) – the upper bound of the uniform distribution. Defaults to 1.
bias (int | float) – the value to fill the bias. Defaults to 0.
bias_prob (float, optional) – the probability for bias initialization. Defaults to None.
layer (str | list[str], optional) – the layer will be initialized. Defaults to None.
- class otx.algo.utils.weight_init.XavierInit(gain: int | float = 1, distribution: str = 'normal', **kwargs)[source]#
Bases:
BaseInit
Initialize module parameters with values according to the method described in the paper below.
- Parameters:
gain (int | float) – an optional scaling factor. Defaults to 1.
bias (int | float) – the value to fill the bias. Defaults to 0.
bias_prob (float, optional) – the probability for bias initialization. Defaults to None.
distribution (str) – distribution either be
'normal'
or'uniform'
. Defaults to'normal'
.layer (str | list[str], optional) – the layer will be initialized. Defaults to None.
- otx.algo.utils.weight_init.bias_init_with_prob(prior_prob: float) float [source]#
Initialize conv/fc bias value according to a given probability value.
- otx.algo.utils.weight_init.constant_init(module: Module, val: float, bias: float = 0) None [source]#
Initialize the weights and biases of a module with constant values.
Copied from mmengine.model.weight_init.constant_init
- otx.algo.utils.weight_init.initialize(module: Module, init_cfg: dict | list[dict]) None [source]#
Initialize a module.
- Parameters:
Example
>>> module = nn.Linear(2, 3, bias=True) >>> init_cfg = dict(type='Constant', layer='Linear', val =1 , bias =2) >>> initialize(module, init_cfg) >>> module = nn.Sequential(nn.Conv1d(3, 1, 3), nn.Linear(1,2)) >>> # define key ``'layer'`` for initializing layer with different >>> # configuration >>> init_cfg = [dict(type='Constant', layer='Conv1d', val=1), dict(type='Constant', layer='Linear', val=2)] >>> initialize(module, init_cfg) >>> # define key``'override'`` to initialize some specific part in >>> # module >>> class FooNet(nn.Module): >>> def __init__(self): >>> super().__init__() >>> self.feat = nn.Conv2d(3, 16, 3) >>> self.reg = nn.Conv2d(16, 10, 3) >>> self.cls = nn.Conv2d(16, 5, 3) >>> model = FooNet() >>> init_cfg = dict(type='Constant', val=1, bias=2, layer='Conv2d', >>> override=dict(type='Constant', name='reg', val=3, bias=4)) >>> initialize(model, init_cfg) >>> model = ResNet(depth=50) >>> # Initialize weights with the pretrained model. >>> init_cfg = dict(type='Pretrained', checkpoint='torchvision://resnet50') >>> initialize(model, init_cfg) >>> # Initialize weights of a sub-module with the specific part of >>> # a pretrained model by using "prefix". >>> url = 'http://download.openmmlab.com/mmdetection/v2.0/retinanet/'\ >>> 'retinanet_r50_fpn_1x_coco/'\ >>> 'retinanet_r50_fpn_1x_coco_20200130-c2398f9e.pth' >>> init_cfg = dict(type='Pretrained', checkpoint=url, prefix='backbone.')
- otx.algo.utils.weight_init.kaiming_init(module: Module, a: float = 0, mode: str = 'fan_out', nonlinearity: str = 'relu', bias: float = 0, distribution: str = 'normal') None [source]#
Initialize the weights and biases of a module using the Kaiming initialization method.
Copied from mmengine.model.weight_init.kaiming_init
- Parameters:
module (nn.Module) – The module to initialize.
a (float) – The negative slope of the rectifier used after this layer (only used with ‘leaky_relu’ nonlinearity). Default is 0.
mode (str) – Either ‘fan_in’ (default) or ‘fan_out’. Choosing ‘fan_in’ preserves the magnitude of the variance of the weights in the forward pass. Choosing ‘fan_out’ preserves the magnitudes in the backward pass. Default is ‘fan_out’.
nonlinearity (str) – The non-linear function (nn.functional name), recommended to use ‘relu’ or ‘leaky_relu’. Default is ‘relu’.
bias (float) – The bias value. Default is 0.
distribution (str) – The type of distribution to use for weight initialization, either ‘normal’ (default) or ‘uniform’.
- Returns:
The initialized tensor.
- Return type:
Tensor
- otx.algo.utils.weight_init.normal_init(module: Module, mean: float = 0, std: float = 1, bias: float = 0) None [source]#
Initialize the weights and biases of a module using a normal distribution.
Copied from mmengine.model.weight_init.normal_init
- otx.algo.utils.weight_init.trunc_normal_(tensor: Tensor, mean: float = 0.0, std: float = 1.0, a: float = -2.0, b: float = 2.0) Tensor [source]#
Fills the input Tensor with values drawn from a truncated normal distribution.
The values are effectively drawn from the normal distribution \(\mathcal{N}(\text{mean}, \text{std}^2)\) with values outside \([a, b]\) redrawn until they are within the bounds. The method used for generating the random values works best when \(a \leq \text{mean} \leq b\).
Modified from pytorch/pytorch
- otx.algo.utils.weight_init.trunc_normal_init(module: Module, mean: float = 0, std: float = 1, a: float = -2, b: float = 2, bias: float = 0) None [source]#
Initialize the weights and biases of a module using a truncated normal distribution.
- Parameters:
module (nn.Module) – The module to initialize.
mean (float) – The mean of the truncated normal distribution. Default is 0.
std (float) – The standard deviation of the truncated normal distribution. Default is 1.
a (float) – The lower bound of the truncated normal distribution. Default is -2.
b (float) – The upper bound of the truncated normal distribution. Default is 2.
bias (float) – The bias value. Default is 0.
- otx.algo.utils.weight_init.uniform_init(module: Module, a: int | float = 0, b: int | float = 1, bias: int | float = 0) None [source]#
Initialize the weights and biases of a module using a uniform distribution.
- otx.algo.utils.weight_init.update_init_info(module: Module, init_info: str) None [source]#
Update the _params_init_info in the module if the value of parameters are changed.
- Parameters:
(obj (module) – nn.Module): The module of PyTorch with a user-defined attribute _params_init_info which records the initialization information.
init_info (str) – The string that describes the initialization.