Modules

Since expressions are just objects, you can put them inside lists, dicts, etc., and you can also pass them to and return them from functions. By taking advantage of this, you can define very complex networks without very much clutter.

For example, Layer class simplifies definitions of feedforward neural networks. It takes an input size and output size and returns a function. This function, in turn, creates a fully-connected layer \(y = f(Wx+b)\), including properly-initialized parameters \(W\) and \(b\). The same example above using Layer would be:

h = Layer(2, nh)(i)
o = Layer(nh, 1)(h)
e = distance2(o, c)

The parameters of the two layers are there, but hidden from our view. The reason for the two-step usage is to control parameter sharing. If you create two Layer objects, you get two different layers with different parameters. But if you call it once and use the result twice, you get two layers with shared parameters.

A possible definition of Layer would be:

class MyLayer(object):
    def __init__(self, ni, no):
        self.W = parameter(numpy.random.uniform(-1., 1., (no, ni)))
        self.b = parameter(numpy.zeros((no,)))
    def __call__(self, i):
        return tanh(dot(self.W, i) + self.b)

The real Layer has some extra bells and whistles.

• it can take a sequence of input sizes instead of a single input size
• a negative input size gives you an input that takes an integer i and selects the ith row of weights
• the bias option lets you specify the initial bias; None means no bias
• the f option lets you set the activation function (default is tanh)