class nbla::BatchNormalization

template<typename T>
class BatchNormalization : public nbla::BaseFunction<const vector<int>&, float, float, bool, bool, bool>

Batch normalization at training time defined as.

\[\begin{split} \begin{array}{lcl} \mu &=& \frac{1}{M} \sum x_i\\ \sigma^2 &=& \frac{1}{M} \left(\sum x_i - \mu\right)^2\\ \hat{x}_i &=& \frac{x_i - \mu}{\sqrt{\sigma^2 + \epsilon}} \\ y_i &=& \hat{x}_i \gamma + \beta. \end{array} \end{split}\]
In testing, mean and variance computed by moving average calculated during training are used.

Inputs:

  • N-D array of input.

  • N-D array of beta which is learned.

  • N-D array of gamma which is learned.

  • N-D array of running mean (modified during forward execution).

  • N-D array of running variance (modified during forward execution).

Outputs (1 or 3):

  • N-D array.

  • (Optional) N-D array of batch mean.

  • (Optional) N-D array of batch variance.

See also

Ioffe and Szegedy, Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. https://arxiv.org/abs/1502.03167

Template Parameters:

T – Data type for computation.

Param axes:

Axes mean and variance are taken.

Param decay_rate:

Decay rate of running mean and variance.

Param eps:

Tiny value to avoid zero division by std.

Param batch_stat:

Use mini-batch statistics rather than running ones.

Subclassed by nbla::SyncBatchNormalization< T >

Public Functions

inline virtual shared_ptr<Function> copy() const

Copy another instance of Function with the same context.

inline virtual vector<dtypes> in_types()

Get input dtypes.

Last in_type will be used repeatedly if size of in_types is smaller than size of inputs

inline virtual vector<dtypes> out_types()

Get output dtypes.

Last out_type will be used repeatedly if size of out_types is smaller than size of outputs

inline virtual int min_inputs()

Get minimum number of inputs.

This is meant to be used in setup function with in_types which is used to get maximum number of inputs.

inline virtual int min_outputs()

Get minimum number of outputs.

This is meant to be used in setup function with out_types which is used to get max number of outputs.

inline virtual string name()

Get function name in string.

inline virtual vector<string> allowed_array_classes()

Get array classes that are allowed to be specified by Context.

inline virtual bool grad_depends_output_data(int i, int o) const

Dependency flag for checking if in-grad depends on out-data.

Checking if i-th input’ gradient computation requires o-th output’s data or not.

Note

If any of inputs requires an output variable data when computing its gradient, this function must be overridden to return appropriate boolean value. Otherwise, backward computation will be incorrect.

Parameters:

  • i – [in] Input variable index.

  • o – [in] Output variable index.