Modules

The nnabla.core.module.Module class represents a construction block of neural network.

Module

class nnabla.core.module.Module[source]

Module is a construction block of a computation model. Modules normally are constructed by lower level operators or other Modules, thus, nesting them in a tree-like structure may construct a more complex computation model.

Example

User may construct his model by derived from this class. Like:

import nnabla as nn
import nnabla.parametric_functions as PF
import nnabla.functions as F

class ConvBn(nn.Module):
    def __init__(self, outmaps, kernel=1, stride=1, act=None):
        self.outmaps = outmaps
        self.kernel = kernel
        self.stride = stride
        self.act = act

    def call(self, x, training=True):
        kernel = (self.kernel, self.kernel)
        pad = (self.kernel // 2, self.kernel // 2)
        stride = (self.stride, self.stride)
        h = PF.convolution(x, self.outmaps, kernel,
                           pad, stride, with_bias=False)
        h = PF.batch_normalization(h, batch_stat=training)
        if self.act is None:
            return h
        return self.act(h)


class ResUnit(nn.Module):
    def __init__(self, channels, stride=1, skip_by_conv=True):
        self.conv1 = ConvBn(channels // 4, 1, 1,
                            act=lambda x: F.relu(x, inplace=True))
        self.conv2 = ConvBn(channels // 4, 3, stride,
                            act=lambda x: F.relu(x, inplace=True))
        self.conv3 = ConvBn(channels, 1)
        self.skip_by_conv = skip_by_conv
        self.skip = ConvBn(channels, 1, stride)

    def call(self, x, training=True):
        h = self.conv1(x)
        h = self.conv2(h)
        h = self.conv3(h)

        s = x
        if self.skip_by_conv:
            s = self.skip(s)
        h = F.relu(F.add2(h, s, inplace=True), inplace=True)
        return h

To use this model, user may do like the following code:

res_unit = ResUnit(1024)
x = nn.Variable((64, 3, 32, 32))
x.d = np.random.random(x.shape)
y = res_unit(x)
y.forward(clear_buffer=True)

For working with dynamic network, user may do like the following:

res_unit = ResUnit(1024)
with nn.auto_forward():
    x = nn.Variable.from_numpy_array(np.random.random((1, 3, 32, 32)))
    y = res_unit(x)
    print(y.d)

For training, please set the parameters in module scope to optimizer. For example,

import nnabla.solvers as S

resnet = ResNet(18)
loss = resnet(x, y_)

solver = S.Sgd(lr=1e-3)
solver.set_parameters(resnet.get_parameters())

for _ in range(max_iter):
    x.d, y_.d = data.next()
    loss.forward()
    solver.zero_grad()
    loss.backward()
    solver.weight_decay(1e-5)
    solver.update()

In this example, we supposed ResNet is a derived class of Module, x, y_ is Variable, data is an instance of DataIterator, supposed it has already been attached to a DataSet.

Note:

From this example, we knew that model parameters are owned by model. Here it is variable resnet. These parameters will be referred when network is forward or backward or solve.update(). Hence, it is necessary to keep this module instance from being unexpectedly released, to ensure forward() or backward() can refer to these variables.

call(*args, **kwargs)[source]

User needs implement this function to construct their neural network. In the implementation, user may instantiate existing predefined Modules as its members, then use it. For example:

class AModule(nn.Module):
   def __init__(...):
      ...
      self.cnb = ConvBN(128) # A submodule is instantiated here.

   def call(...):
      h = self.cnb(x) # Using beforehand instantiated submodule.

or directly use parametric functions or functions:

class AModule(nn.Module):
    ...
    def call(...):
        ...
        h = PF.convolution(x, self.outmaps, ...)
        return h

Note

The following usage is currently not supported, it might be supported in future version:

class AModule(nn.Module):
   def __init__(...):
      ...
      self.cnb = [ConvBN(k) for k in [8, 16, 32]] # using an array to hold module instances.
      self.cnb = {f'name_{k}': ConvBN(k) for k in [8, 16, 32]} # using a dict to hold module instances.

Note

The following method to temporarily instantiate a module is also not allowed:

class AModule(nn.Module):
   def call(...):
      ...
      cnb = ConvBN(k) # Instantiate a temporary instance of Module is not allowed
      y = cnb(x)
      return y

Because when leave this scope, the parameters registered to cnb module will be released, which cause unexpected result.

get_parameters(recursive=True, grad_only=False, memo=None)[source]

Obtain an OrderedDict object of all parameters in current Module.

For example,

x = nn.Variable.from_numpy_array((np.random.random((8, 32, 256, 256))))
conv_bn = ConvBn(2)
y = conv_bn(x)

params = conv_bn.get_parameters()
for parameter_name, parameter_value in params.items():
    print("{}:{}".format(parameter_name, parameter_value.shape))

The output looks like:

conv/W:(2, 32, 1, 1)
bn/beta:(1, 2, 1, 1)
bn/gamma:(1, 2, 1, 1)
bn/mean:(1, 2, 1, 1)
bn/var:(1, 2, 1, 1)

Notice that the parameter name looks like a filepath, with splash separated nested scope name. In addition, module name default is used with a prefix @.

Parameters
  • recursive (bool, optional, default=True) – Whether obtain the parameters of current module’s submodules. Default is True.

  • grad_only (bool, optional, default=False) – Whether only obtain the grad. Default is False.

Returns

Flattened parameter’s name-value pairs of current Module.

Return type

OrderedDict

load_parameters(path, extension='.h5', raise_if_missing=True)[source]

Load parameters from a file into this module.

Parameters

path – str or file-like object

property parameter_scope

A module has its owned parameter_scope, which can avoid to pollute global parameter name space. User may obtain the parameter_scope of a module by this property.

Returns

The parameter scope of current Module.

Return type

OrderedDict

save_parameters(path, extension='.h5')[source]

Save parameters of this module to a file.

Parameters

path – str or file-like object

property training

Return a bool value which indicates whether current Module is in training state or not. A module may be set to training state or not, so that the computation graph created from this module can be changed according to this state. For example,

class ConvBN(Module):
    ...
    def call(self, x):
        h = self.conv1(x)
        if self.training:
            h = self.drop_out(h)
        h = F.relu(h, inplace=True)
        return h

conv_bn = ConvBN()
conv_bn.training = True
train_y = conv_bn(x)

conv_bn.training = False
eval_y = conv_bn(x)
Returns

which indicates whether current Module is in training state.

Return type

bool

zero_grad()[source]

Clear the gradient of the parameters in this module to 0.