Graph Definition¶

In NNabla, Graph Definition represents a kind of representation of a computation graph which is special designed for storage optimization and format converter.

A computation graph can be defined by the call of NNabla functions. Such computation graph has instantiated the input and output variables of the functions, inherent topology has been established for forward or backward computation. But for persistence of such graph, another abstract representation, so-called protobuf graph(or network), as abbreviation - proto graph is used normally. In this graph, only the information being necessary for persistence are kept, the information only used for computation will be dropped.

Graph Definition provides a group of functions and classes, tends to facilitate user creates protobuf network from their computation graph, and saving and restoring their neural network from a persistent protobuf network representation.

ProtoGraph¶

class nnabla.graph_def.ProtoGraph(networks=None, parameter_scope=None)¶

This class represents a group of proto networks. It normally corresponds to a .nnp file. In a .nnp file, there might be one or multiple networks, for example, there might be a network for doing directly inferring, another network with similar network structure, sharing same parameters, using for training. This class works as a container of proto networks, providing a group of functions for accessing proto network by its name. Especially, when there is only one network in it, also some short-cut functions are provided for directly operating with this network. For example,

import nnabla as nn

g = nn.graph_def.load("my_model.nnp") # Suppose there is only one network in this file.
x1 = nn.Variable(input_shape)
x1.d = ... # load data here.
y1 = g.networks['executor_net'](x1)  #<== (1)
y1.forward()
print(y1.d)

x2 = nn.Variable(input_shape)
x2.d = ... # load data here.
y2 = g(x2) #<== (2)
# y2 = g.default_graph()(x2) #<== (3)
y2.forward()
print(y2.d)

The computation graph y1 and y2 are exactly same. (2) and (3) are equal. If there are multiple networks in a graph, the first network being loaded acted as its default network. Please not use default_graph() when there are multiple networks in graph, since the default heavily depends on concrete implementation.

If you know the name of each network, you may access proto network in this graph by its member name. For example,

g = nn.graph_def.load("my_model.nnp") # Suppose there is only one network in this file.
x = nn.Variable(input_shape)
x.d = ... # load data here.
y = g.executor_net(x) # here, we knew there is a network named "executor_net" existed.
y.forward()
print(y.d)

as_proto(include_parameter=False, only_parameter=False, networks=None, variable_batch_size=True)¶

This function exports a protobuf data structure, which can be manipulated by google protobuf APIs.

Parameters

include_parameter (bool, optional, default=False) – Whether exports the parameters to protobuf data structure.
only_parameter (bool, optional, default=False) – Whether only exports the parameters to protobuf data structure.
networks (array of proto networks, optional, default=None) – User may provides their networks to export a protobuf data structure.
variable_batch_size (bool, optional, default=True) – Replace batch size of current network with an abstract placeholder, so that batch size can be replaced with other value in use time.

property current_context¶: Current backend context of this proto network.

default_graph()¶: This function returns default proto network in this graph. Which network is default graph depends on its loading sequence. Hence, it is safe to be used when there is only one network.

expand_loop_control()¶: This function expands loop control statements for all networks in this graph.

static from_proto(proto, batch_size=None, param_scope=None, rng=None)¶

This function create a proto graph object from a protobuf data structure.

Parameters

proto (protobuf object) – A protobuf data structure.
batch_size (int, optional, default=None) – The batch size will be applied to this graph. If it is None, it is pending to apply a the batch size value.
param_scope (OrderedDict, optional, default=None) – User may provide an owned parameter scope.
rng (np.random.RandomState, optional, default=None) – A random seed, which is used in parameter initialization.

get_parameters(grad_only=False)¶: Get parameters in current module name scope.

ProtoNetwork¶

class nnabla.graph_def.ProtoNetwork(owner, name=None, batch_size=None)¶

This class represents a protobuf network, which comes from a corresponding computation graph or restored from a saved protobuf network(e.g. .nnp file).

This class describes a neural network by the following members:

functions: An OrderedDict of name-value pairs, the value is ProtoFunction object.

variables: An OrderedDict of name-value pairs, the value is ProtoVariable object.

parameters: An OrderedDict of name-value pairs, the value is ProtoVariable object.

inputs: A string list, which contains the name of input variables of this network.

outputs: A string list, which contains the name of output variables of this network.

variables represents activations in networks, parameters mainly includes weights and all learnable parameters. functions represents functions in networks, the sequence of functions might not equal forward sequence. Please use forward_sequence to obtain exactly forward function sequence.

__call__(*args, **kwargs)¶

Generate a computation graph of this protonetwork.

Parameters

args (tuple of nn.Variables or None) –

The inputs of network, which can be different from the inputs of original computation graph as long as the network allows.

For example,
import nnabla as nn
import numpy as np

resnet = nn.graph_def.load("resnet.nnp")
x.d = np.random.random(input_shape)
y = resnet(x)
The variable y corresponding to a computation graph, user may perform forward like:
y.forward()
If user does not provide inputs for this function, because proto network has the memory of network inputs, this function will create corresponding nn.Variable objects as the inputs of this network. This input variables actually are placeholder of input, hence, users need to find these input variables and fill actual value to these placeholders, so that this computation graph is ready for forward or backward.

For example,
g = nn.graph_def.load("resnet.nnp")
y = g() # Not provide input variables
To feed training or evaluation data to this network, user needs to locate input variable, for example:
input = g.networks[network_name].variables[input_name].variable_instance
input.d = np.random.random(input_shape)

batch_size (int, optional, default=None):

If provided, batch_size will be applied for newly created computation graph. For example,
g = nn.graph_def.load("my_model.nnp")
y = g(batch_size=32)

In this sample, batch_size will be used to create a computation graph with specified batch size. Supposed x is the input of network, its original shape is (1, 3, 32, 32), then the actual computation graph will be (32, 3, 32, 32).

as_proto(**kwargs)¶

This function returns a protobuf data structure, which can be directly accessed by the functions in nnabla.utils.nnabla_pb2. Thus, it allows user further manipulates this protobuf representation, for example, performing format converting, or network structure optimization.

Parameters: variable_batch_size (bool, optional) – If true, the batch size of the network will be replaced with an abstract representation, so that it can be replaced with other value in restoring computation graph.
Returns: A protobuf object.
Return type: protobuf

execute_on_proto(execute)¶

This function performs a virtual forward, following the sequence from inputs to output. This function does not use recursive call to perform graph-travel, instead, a non-recursive algorithm is used to graph-travel. For each function, execute is called when meet a function, a ProtoFunction object is passed in for further operation with this function.

Parameters

execute (callable) –

A callback function (or callable object), which is called when each ProtoFunction is met in travelling graph.

execute should look like:

def execute(pf: ProtoFunction):
    # Do what you want to do with pf
    pass

Or:

class MyCallback:
    def __call__(pf: ProtoFunction):
        # Do what you want to do with pf
        pass

expand_loop_control()¶

This function expand loop control statement and generate a new proto network object without loop control statement. loop control statement cannot be created by python code, it can be only created by interactive neural network design tool. The following briefly introduce its specification:

As for variable,

In nntxt, if the variable includes a field repeat_id, it means that this variable is in surround with a loop control structure. A renaming rule is applied if expanding this network. The variable name will be added a postfix, like:

For old style, e.g.:

BatchNormalization_6/bn/mean --> BatchNormalization_6/bn/mean_RepeatStart[0]
                                                                 ^        ^  repeat_time
                                                              repeat_id[index]

original_name --> original_name + << _%repeat_id%[%repeat_time%],  for each in repeat_id >>

For new style, e.g.:

BatchNormalization_6{RepeatStart}/bn/mean --> BatchNormalization_6[0]/bn/mean_RepeatStart
                                                                   ^
                                                              repeat_time
original_name --> original_name + << [%repeat_time%],  for each in repeat_id >>

As for RepeatStart, RepeatEnd
The functions or variables nodes between these 2 layers will be repeated. Expanding will create times of functions and variables, and connected them each other.
As for RecurrentInput,
Axis of RecurrentParam points out which axis will be split-ed. And each branch will duplicated the functions and variables with this repeat_id. This layer works like a split function.
As for RecurrentOutput,
RecurrentOutput merge multiple branches into one output, looks like a stack function.
As for Delay
First time, the output is its input[1], after that, the output is its input[0]

forward_sequence()¶

This function creates an iteratable for iterating functions in network with the sequence of actually forward.

For example,

for pf in proto_network.forward_sequence():
    print(pf.name)

promote(callback)¶

User may manipulate a proto network by a callback, like NnpNetworkPass.

Parameters: callback (NnpNetworkPass,) – Currently, only NnpNetworkPass object is supported as a network promotion callback.

Developers may manipulate a proto network by a modifier, acts as a callback. nnabla.utils.nnp_graph.NnpNetworkPass is a kind of modifier. The following gives a simple example to illustrate this usage:

Example

from nnabla as nn
from nnabla.utils import nnp_graph

verbose = 1
callback = nnp_graph.NnpNetworkPass(verbose)

@callback.on_generate_function_by_name('Convolution')
def change_convolution_param(f):
    print('{}'.format(f.proto.convolution_param.pad.dim[:]))
    f.proto.convolution_param.pad.dim[:] = [1, 1]
    return f

g = nn.graph_def.load("my_model.nnp")
n = g.default_graph().promote(callback)
x = nn.Variable(input_shape)
y = n(x)
y.forward()

In this example, a callback is defined to change pad of a Convolution function, locating this target function by the name of function, here, only the function with the name 'Convolution' is located and operated.

save(filename, include_parameter=False, variable_batch_size=True)¶

This function saves current proto network to a file, which is specified by filename, normally, e.g. a .nnp file.

Parameters

filename (str) – string filename, its extension name is used to determine the file format. The extension name normally is .nnp.
include_parameter (bool, optional, default=False) – Whether saving parameters to protobuf tree.
variable_batch_size (bool, optional, default=True) – Whether replace current network’s batch size dimension with an abstract representation. If it is true, it is possible to use another batch size when this network is reused.

ProtoVariable¶

class nnabla.graph_def.ProtoVariable(shape, name=None, need_grad=False, var_type='Buffer')¶: This class represents a variable, so-called proto variable. Passing this variable to network definition, a proto network will be generated in a proto graph scope. If this procedure is done under a with statement as g, a proto network will be generated in g. Otherwise, a global graph scope is used, a proto network will be generated in global graph scope.

ProtoFunction¶

class nnabla.graph_def.ProtoFunction(func, f_type, args, name=None, owner=None)¶

This class represent a function that is used to define a proto network.

There are the following properties to describe a proto function:

name: The name of this function.
type: The type of this function, e.g. ReLU.
inputs: An array of string name, which represents the proto variables of inputs.
outputs: An array of string name, which represents the proto variables of outputs.

graph_call(**kwargs)¶: This function create function instance for generating computation graph.

load¶

nnabla.graph_def.load(filename, batch_size=None, exclude_parameter=False, parameter_only=False, extension='.nntxt', parameter_scope=None, rng=None)¶

Load network from files

Parameters

filename (str or list or file-like object) – Filename string ,list of filenames or file-like object.
batch_size (int) – The batch size expected to be set.
exclude_parameter (bool) – If True, only load model, not load parameters of this model. Default is False.
parameter_only (bool) – If True, only load model parameters. Default is False.
extension (str) – This parameter is needed when filename is a file-like object. Default is .nntxt.
parameter_scope (OrderedDict) – User may provide a user owned parameter scope. If this parameter is not provided, loaded parameters will be created in created proto_graph’s parameter_scope. This parameter_scope is default initialized with empty dictionary.
rng (random state) – User may specify random state, which cause parameters are initialized by determined random seed.

Returns

A ProtoGraph object, in which, there are one or multiple ProtoNetwork objects.

Return type

ProtoGraph

Example

The following example loads a model and generate the output variable through this model:

import nnabla as nn
import nnabla.functions as F
import nnabla.parametric_functions as PF

def fusion_net(x):
    def unit(i, prefix):
        c1 = PF.convolution(i, 4, (3, 3), pad=(1, 1), name=prefix + '-c1')
        c2 = PF.convolution(F.relu(c1), 4,
                            (3, 3), pad=(1, 1), name=prefix + '-c2')
        c = F.add2(c2, c1, inplace=True)
        return c
    c = unit(x, 'c1')
    c2 = unit(c, 'c2')
    y = PF.affine(c2, 5, name='fc')
    return y

x = nn.ProtoVariable((64, 3, 32, 32))
y = fusion_net(x)
g = nn.graph_def.get_default_graph()  # Get generated graph_def
g.save("fusion_net.nnp")
...
g = nn.graph_def.load("fusion_net.nnp")
x = nn.Variable((64, 3, 32, 32))
x.d = ... # user provided input data for this graph
y = g(x) # create computation graph by passing in nn.Variable()
y.forward() # calculate output by this graph
...

# You may use your special context(e.g. cuda context)
with context_scope(ctx):
   y = g(x) # create computation graph representation with specified backend context.
   y.forward() # forward using specified backend

save¶

nnabla.graph_def.save(filename, content, include_parameters=False, variable_batch_size=True, extension='.nnp')¶

Save network

Parameters

filename (str or file object) –
Filename to store information. The file extension is used to determine the saving file format. .nnp: (Recommended) Creating a zip archive with nntxt (network definition etc.) and h5 (parameters). .nntxt: Protobuf in text format. .protobuf: Protobuf in binary format (unsafe in terms of

backward compatibility).
content (list) – Currently only ProtoGraph or PhotoNetwork objects are supported.
include_parameters (bool) – Includes parameter into single file. This is ignored when the extension of filename is nnp.
variable_batch_size (bool) – Whether or not convert batch size of computation graph to a special value, so that user may use any other batch_size value when using it.

Example

The following example creates a two inputs and two outputs MLP, and save the network structure and the initialized parameters.

import nnabla as nn
import nnabla.functions as F
import nnabla.parametric_functions as PF

def mlp_module(x0, x1):
    h1_0 = PF.affine(x0, 100, name='affine1_0')
    h1_1 = PF.affine(x1, 100, name='affine1_0')
    h1 = F.tanh(h1_0 + h1_1)
    h2 = F.tanh(PF.affine(h1, 50, name='affine2'))
    y0 = PF.affine(h2, 10, name='affiney_0')
    y1 = PF.affine(h2, 10, name='affiney_1')
    return y0, y1

with nn.graph_def.graph() as g:
    x0 = nn.ProtoVariable((64, 100))
    x1 = nn.ProtoVariable((64, 100))
    y0, y1 = mlp_module(x0, x1)

nn.graph_def.save("mlp_net.nnp", [g])

Create Protobuf Representation from Computation Graph¶

create_graph_from_variable¶

nnabla.graph_def.create_graph_from_variable(name, variables, names=None, parameter_scope=None)¶

Create a Proto Graph from one or multiple outputs.

If developers have a computation graph, it means that they have a nn.Variable() object, it might be loss of a network or an output variable of an executor network, this variable inherently corresponds to a computation network. From these variables, we can create corresponding proto network by this function.

Parameters

name (str) – The name of generated proto_network.
variables (nn.Variables) – One or multiple variables, if multiple variables, this function adds a sink function to reduce these multiple outputs to one.
names (dict, optional, default=None) – A name to nn.Variable mapping table. This function default names all activation variables and parameters with internal naming rule. But sometimes, developers expects specially name some of variables so that these variable can be accessed conveniently. In generating proto network, when the variable occurs in this mapping table, the corresponding name of that variable will be used to name the variable in proto network.
parameter_scope (OrderedDict, optional, default=None) – Developers may provide a parameter scope, thus, when create proto networks, the name will be replaced if corresponding variable is found in specified parameter_scope, which make the name of weights or some parameters meaningful.

Example

import nnabla as nn

x = nn.Variable((1, 3, 32, 32))
y = my_model(x)
g = nn.graph_def.create_graph_from_variable("proto_network_name", y)
g.save("my_model.nnp")

get_default_graph¶

nnabla.graph_def.get_default_graph(*args, **kwargs)¶

This function obtain current default graph_def.

If user does not create their proto network in a with statement scope, proto network will default be created in a global scope. User may retrieve this proto graph by this function.

Example

import nnabla as nn
from nnabla.core.modules import ResUnit

resunit = ResUnit(16)
input = nn.ProtoVariable((64, 3, 32, 32))
y = resunit(input)
graph_def = nn.graph_def.get_graph_graph()

Note

If user does not ensure whether there is any previous existing proto graph remained in global graph scope. It is better to call reset_default_graph(). If user uses with statement like with nn.graph_def.graph() as g, this point is no need to care about.

Returns: A proto graph is returned
Return type: ProtoGraph

get_default_graph_by_variable¶

nnabla.graph_def.get_default_graph_by_variable(proto_variable)¶

This function obtain a specify network by its outputs.

User may retrieve one of the networks in default proto graph scope, if this network has the specified outputs. Let us image that there is a global proto graph, when user passed a ProtoVariable to model, during the procedure that create output variables, proto network is generated in this global proto graph. By this function, user may retrieve this generated proto network, saving it or do any other operations.

Note

This proto network will become invalid after reset_default_graph(). For example,

proto_variable_inputs = [nn.ProtoVariable(v.d.shape) for v in inputs]
outputs = module(*proto_variable_inputs)
net = nn.graph_def.get_default_graph_by_variable(outputs[0])
...
nn.graph_def.reset_default_graph()
y = net(x) # Cannot access net anymore, become invalid at this point

graph¶

nnabla.graph_def.graph(**kwargs)¶

This function is only used in with statement.

Parameters

name (str, optional, default=None) – User may specify a name for the generated proto network. This name is useful for saving to .nnp.
parameter_scope (OrderedDict, optional, default=None) – User may specify a parameter scope, thus, the parameters are created during creating model will be placed into this parameter scope.

For example,

import nnabla as nn

proto_variable_inputs = [nn.ProtoVariable(v.d.shape) for v in inputs]
with nn.graph_def.graph() as g:
    outputs = module(*proto_variable_inputs)

g.save("my_model.nnp")
Here, inputs is an array of input nn.Variables. Modules is a module object instantiated from a Module definition.

reset_default_graph¶

nnabla.graph_def.reset_default_graph()¶: This function clear all information in global graph scope.