Neural network operations

Penne also provides functions specific to neural networks: sigmoid, tanh, hardtanh, softmax, logsoftmax, rectify (ReLU), crossentropy (log loss), distance2 (mean-squared loss).

penne.sigmoid(x)

Logistic sigmoid function, :math:` rac{1}{1+exp -x}`.

penne.tanh(x)

penne.hardtanh(x)

Hard tanh function, equivalent to clip(x, -1, 1).

penne.softmax(arg, axis=-1)

Softmax function, \(y_i = \exp x_i / \sum_{i'} \exp x_{i'}\).

Parameters:axis – along which to perform the softmax (default is last).

penne.logsoftmax(arg, axis=-1)

Log-softmax function, \(y_i = \log \left(\exp x_i / \sum_{i'} \exp x_{i'} ight)\).

Use this instead of log(softmax(x)) for better numerical stability.

param axis:along which to perform the softmax (default is last).

penne.rectify(x)

Rectified linear unit, \(\max(0, x)\).

penne.crossentropy(logp, correct)

Cross-entropy, a.k.a. log-loss.

Parameters:

• logp – vector of log-probabilities
• correct – observed probabilities

penne.distance2(x, y)

Squared Euclidean distance, a.k.a. mean-squared loss.

class penne.Dropout(p=0.5)

Factory for dropouts.

Example usage:

d = Dropout(0.5)
y = d(x)

The reason for the extra level of indirection is so that all the dropouts can be enabled or disabled together.

Parameters:p (float) – probability of dropout Returns:dropout function Return type:Expression -> Expression

class penne.Layer(insize, outsize, f=<class 'penne.expr.tanh'>, gain=None, bias=0.0, model=None)

Fully-connected layer.

Parameters:

• insize –

  Input size or sequence of input sizes.

  • If an input size is n > 0, then that input will expect an n-dimensional vector.
  • If an input size is n < 0, then that input will expect an integer in [0, n), which you can either think of as a one-hot vector or as an index into a lookup table.
  • If an input size is “diag”, then that input will have a diagonal weight matrix.
• outsize – Output size.
• f – Activation function (default tanh).
• bias – Initial bias, or None for no bias.