:py:mod:`nncf.quantization.advanced_parameters` =============================================== .. py:module:: nncf.quantization.advanced_parameters .. autoapi-nested-parse:: Structures and functions for passing advanced parameters to NNCF post-training quantization APIs. Classes ~~~~~~~ .. autoapisummary:: nncf.quantization.advanced_parameters.FP8Type nncf.quantization.advanced_parameters.GroupSizeFallbackMode nncf.quantization.advanced_parameters.QuantizationParameters nncf.quantization.advanced_parameters.FP8QuantizationParameters nncf.quantization.advanced_parameters.AdvancedBiasCorrectionParameters nncf.quantization.advanced_parameters.AdvancedSmoothQuantParameters nncf.quantization.advanced_parameters.AdvancedQuantizationParameters nncf.quantization.advanced_parameters.AdvancedAWQParameters nncf.quantization.advanced_parameters.AdvancedScaleEstimationParameters nncf.quantization.advanced_parameters.AdvancedGPTQParameters nncf.quantization.advanced_parameters.AdvancedLoraCorrectionParameters nncf.quantization.advanced_parameters.AdvancedAdaptiveCodebookParameters nncf.quantization.advanced_parameters.AdvancedCompressionParameters nncf.quantization.advanced_parameters.AdvancedAccuracyRestorerParameters .. py:class:: FP8Type Bases: :py:obj:`nncf.parameters.StrEnum` Defines FP8 special types (https://arxiv.org/pdf/2209.05433.pdf). :param E4M3: Mode with 4-bit exponent and 3-bit mantissa. :param E5M2: Mode with 5-bit exponent and 2-bit mantissa. .. py:class:: GroupSizeFallbackMode Bases: :py:obj:`nncf.parameters.StrEnum` Specifies how to handle nodes that do not support the given group size. :param ERROR: Raise an error if the given group size is not supported by a node. :param IGNORE: Skip nodes that cannot be compressed with the given group size. :param ADJUST: Automatically compute a suitable group size for unsupported nodes. When selected, each weight for which the channel size is not divisible by the general group size value will be compressed to a newly calculated group size. The new group size value is the maximal power of two (i.e., 2^k) such that: - channel size is divisible by it; - it is less than the originally specified group size value; - it is greater than or equal to `min_adjusted_group_size`. If it's not possible to find a value satisfying these requirements, such weight is compressed to the backup precision. If ratio < 1.0 and some weights have to be compressed to the backup precision because of group size issues, then these weights won't contribute to the ratio of backup mode group. .. py:class:: QuantizationParameters Contains quantization parameters for weights or activations. :param num_bits: The number of bits to use for quantization. :type num_bits: Optional[int] :param mode: The quantization mode to use, such as 'symmetric', 'asymmetric', etc. :type mode: nncf.common.quantization.structs.QuantizationMode :param signedness_to_force: Whether to force the weights or activations to be signed (True), unsigned (False) :type signedness_to_force: Optional[bool] :param per_channel: True if per-channel quantization is used, and False if per-tensor quantization is used. :type per_channel: Optional[bool] :param narrow_range: Whether to use a narrow quantization range. If False, then the input will be quantized into quantization range * [0; 2^num_bits - 1] for unsigned quantization and * [-2^(num_bits - 1); 2^(num_bits - 1) - 1] for signed quantization If True, then the ranges would be: * [0; 2^num_bits - 2] for unsigned quantization and * [-2^(num_bits - 1) + 1; 2^(num_bits - 1) - 1] for signed quantization :type narrow_range: Optional[bool] .. py:class:: FP8QuantizationParameters Contains convert parameters for weights or activations. :param destination_type: Currently contains E4M3 or E5M2 for FP8 precision. :type destination_type: FP8Type .. py:class:: AdvancedBiasCorrectionParameters Contains advanced parameters for fine-tuning bias correction algorithm. :param apply_for_all_nodes: Whether to apply the correction to all nodes in the model, or only to nodes that have a bias. :type apply_for_all_nodes: bool :param threshold: The threshold value determines the maximum bias correction value. The bias correction are skipped If the value is higher than threshold. :type threshold: Optional[float] .. py:class:: AdvancedSmoothQuantParameters Contains advanced alpha parameters for SmoothQuant algorithm. It regulates the calculation of the smooth scale for different node types. A negative value switches off the algorithm for current node type. In case of inaccurate results, this parameter may be adjusted in the range from 0 to 1 or set -1 to disable SmoothQuant algorithm. :param convolution: Whether to apply smoothing for Convolution layers. :type convolution: float :param matmul: Whether to apply smoothing for MatMul layers. :type matmul: float .. py:class:: AdvancedQuantizationParameters Contains advanced parameters for fine-tuning quantization algorithm. :param overflow_fix: This option controls whether to apply the overflow issue fix for the 8-bit quantization. :type overflow_fix: Optional[nncf.quantization.advanced_parameters.OverflowFix] :param quantize_outputs: Whether to insert additional quantizers right before each of the model outputs. :type quantize_outputs: bool :param inplace_statistics: Defines whether to calculate quantizers statistics by backend graph operations or by default Python implementation, defaults to True. :type inplace_statistics: bool :param disable_channel_alignment: Whether to disable the channel alignment. :type disable_channel_alignment: bool :param disable_bias_correction: Whether to disable the bias correction. :type disable_bias_correction: bool :param batchwise_statistics: Determines whether quantizer statistics should be calculated for each item of the batch or for the entire batch, default is None. "None" means that if torch.DataLoader was passed as a data source for the calibration dataset, then in case batch_size of the data source > 1 batchwise_statistics sets to True, otherwise sets to False. :type batchwise_statistics: Optional[bool] :param quantizer_propagation_rule: An instance of the `QuantizerPropagationRule` enum that specifies how quantizers should be propagated and merged across branching nodes in the model's computational graph. The strategies are as follows: - DO_NOT_MERGE_BRANCHES: No merging of quantization parameters across branches. - MERGE_IF_ALL_BRANCHES_SAME : Merge only if all branch quantization configurations are identical. - MERGE_WITH_POTENTIAL_REQUANTIZATION: Merge common configurations and allow for requantization on branches with additional options. - MERGE_ALL_IN_ONE: Attempt to merge into a single global quantization configuration if possible given hardware constraints. MERGE_ALL_IN_ONE is a default value. :type quantizer_propagation_rule: QuantizerPropagationRule :param activations_quantization_params: Quantization parameters for activations. :type activations_quantization_params: nncf.quantization.advanced_parameters.QuantizationParameters :param weights_quantization_params: Quantization parameters for weights. :type weights_quantization_params: nncf.quantization.advanced_parameters.QuantizationParameters :param activations_range_estimator_params: Range estimator parameters for activations. :type activations_range_estimator_params: nncf.quantization.range_estimator.RangeEstimatorParameters :param weights_range_estimator_params: Range estimator parameters for weights. :type weights_range_estimator_params: nncf.quantization.range_estimator.RangeEstimatorParameters :param bias_correction_params: Advanced bias correction parameters. :type bias_correction_params: nncf.quantization.advanced_parameters.AdvancedBiasCorrectionParameters :param smooth_quant_alphas: SmoothQuant-related parameters mapping. It regulates the calculation of the smooth scale. The default value stored in AdvancedSmoothQuantParameters. A negative value for each field switches off type smoothing. In case of inaccurate results, fields may be adjusted in the range from 0 to 1 or set -1 to disable smoothing for type. :type smooth_quant_alphas: nncf.quantization.advanced_parameters.AdvancedSmoothQuantParameters :param smooth_quant_alpha: Deprecated SmoothQuant-related parameter. :type smooth_quant_alpha: float :param backend_params: Backend-specific parameters. :type backend_params: dict[str, Any] .. py:class:: AdvancedAWQParameters Contains advanced parameters for AWQ algorithm. :param subset_size: The number of samples for AWQ. :type subset_size: int :param percent_to_apply: The percent of outliers for correction. :type percent_to_apply: float :param alpha_min: Minimum value of smoothness parameter for grid search. :type alpha_min: float :param alpha_max: Maximal value of smoothness parameter for grid search. :type alpha_max: float :param steps: The number of the steps in grid search. :type steps: int :param prefer_data_aware_scaling: Determines whether to use activations to calculate scales if activations are presented. :type prefer_data_aware_scaling: bool .. py:class:: AdvancedScaleEstimationParameters Contains advanced parameters for scale estimation algorithm. :param subset_size: The number of samples for scale estimation. :type subset_size: int :param initial_steps: The number of the steps for absmax scale rectification. :type initial_steps: int :param scale_steps: The number of the steps for grid search scale rectification from 1.0 to 1.0 - 0.05 * scale_step. :type scale_steps: int :param weight_penalty: coefficient for penalty between fp and compressed weights. If -1 then doesn't apply. :type weight_penalty: float .. py:class:: AdvancedGPTQParameters Contains advanced parameters for GPTQ algorithm. :param damp_percent: The percent of the average Hessian diagonal to use for dampening, recommended value is 0.1. :type damp_percent: float :param block_size: The size of the blocks used during quantization. Defaults to 128. :type block_size: int :param subset_size: Number of data samples to calculate Hessian. Defaults to 128. :type subset_size: int .. py:class:: AdvancedLoraCorrectionParameters Contains advanced parameters for lora correction algorithm. :param adapter_rank: rank of lora adapters. Defaults to 16. :type adapter_rank: int :param num_iterations: number of correction iterations. Defaults to 3. :type num_iterations: int :param apply_regularization: Whether to add a regularization during the correction process. Defaults to True. Helpful for big rank values to avoid overfitting. :type apply_regularization: bool :param subset_size: Number of data samples for lora correction algorithm. Defaults to 128. :type subset_size: int :param use_int8_adapters: Whether to 8-bit quantize lora adapters, otherwise they kept in the original weights precision. Defaults to True. :type use_int8_adapters: bool .. py:class:: AdvancedAdaptiveCodebookParameters Contains advanced parameters for adaptive codebook estimation. :param value_type: The target tensor data type for the codebook. Defaults to f8e4m3. :type value_type: TensorDataType :param across_blocks: Whether to use across-block codebooks (e.g., all down_proj has the same codeboook). Defaults to False. :type across_blocks: bool :param num_elements: The number of elements in each codebook entry. Defaults to 16. :type num_elements: int .. py:class:: AdvancedCompressionParameters Contains advanced parameters for compression algorithms. :param statistics_path: Directory path to dump statistics. :type statistics_path: str :param lora_adapter_rank: Rank of lora adapters for FQ_LORA format. Defaults to 256. :type lora_adapter_rank: int :param group_size_fallback_mode: Specifies how to handle nodes that do not support the given group size. :type group_size_fallback_mode: GroupSizeFallbackMode :param min_adjusted_group_size: Minimum group size for adjustable group size searching. Defaults to 32. The reason behind this argument is to avoid too small group size values, which may lead to performance issues. :type min_adjusted_group_size: int :param awq_params: Advanced parameters for AWQ algorithm. :type awq_params: AdvancedAWQParameters :param scale_estimation_params: Advanced parameters for Scale Estimation algorithm. :type scale_estimation_params: AdvancedScaleEstimationParameters :param gptq_params: Advanced parameters for GPTQ algorithm. :type gptq_params: AdvancedGPTQParameters :param lora_correction_params: Advanced parameters for Lora Correction algorithm. :type lora_correction_params: AdvancedLoraCorrectionParameters :param backend_params: Backend-specific parameters. :type backend_params: dict[str, Any] :param codebook: The codebook (LUT) for the weight compression. Applicable for vector quantization. Must be a numpy array or ov Tensor. :type codebook: TTensor :param adaptive_codebook_params: Advanced parameters for adaptive codebook estimation. :type adaptive_codebook_params: AdvancedAdaptiveCodebookParameters .. py:class:: AdvancedAccuracyRestorerParameters Contains advanced parameters for fine-tuning the accuracy restorer algorithm. :param max_num_iterations: The maximum number of iterations of the algorithm. In other words, the maximum number of layers that may be reverted back to floating-point precision. By default, it is limited by the overall number of quantized layers. :type max_num_iterations: int :param tune_hyperparams: Whether to tune of quantization parameters as a preliminary step before reverting layers back to the floating-point precision. It can bring an additional boost in performance and accuracy, at the cost of increased overall quantization time. The default value is `False`. :type tune_hyperparams: int :param ranking_subset_size: Size of a subset that is used to rank layers by their contribution to the accuracy drop. :type ranking_subset_size: Optional[int] :param num_ranking_workers: The number of parallel workers that are used to rank quantization operations. :type num_ranking_workers: Optional[int] :param intermediate_model_dir: Path to the folder where the model, which was fully quantized with initial parameters, should be saved. :type intermediate_model_dir: Optional[str] :param restore_mode: Specifies how to revert operations to their original precision. :type restore_mode: RestoreMode