layers module¶

class layers.BatchNormalization(momentum, epsilon)¶

Bases: object

backward(g_in, **params)¶: g_in.shape = (batch_size, out_dim) in_dim = out_dim of batch norm

forward(x, **params)¶: x.shape = (batch_size, in_dim)

get_beta()¶

Returns the beta parameters.

Parameters: None –
Returns: The beta parameters.
Return type: numpy.ndarray

Notes

None

get_dbeta()¶

Returns the gradients of beta parameters.

Parameters: None –
Returns: ret – The gradients of beta parameters, or None if does not exist yet.
Return type: None or numpy.ndarray

Notes

None

get_dgamma()¶

Returns the gradients of gamma parameters.

Parameters: None –
Returns: ret – The gradients of gamma parameters, or None if does not exist yet.
Return type: None or numpy.ndarray

Notes

None

get_gamma()¶

Returns the gamma parameters.

Parameters: None –
Returns: The gamma parameters.
Return type: numpy.ndarray

Notes

None

get_learnable_params()¶

get_learnable_params_grads()¶

get_reg_loss()¶

Returns the regularization loss of the weight parameters.

Parameters: None –
Returns: The regularization loss of the weight parameters
Return type: float

Notes

None

if_has_learnable_params()¶

Returns if the layer has learnable params. Dense layer does have learnable params.

Parameters: None –
Returns: True if the layer has learnable params.
Return type: has_learnable_params

Notes

None

set_beta(beta)¶

Sets the beta parameters.

Parameters: beta (numpy.ndarray) – The beta parameters.
Returns
Return type: None

Notes

None

set_gamma(gamma)¶

Sets the gamma parameters.

Parameters: gamma (numpy.ndarray) – The gamma parameters.
Returns
Return type: None

Notes

None

set_learnable_params(**learnable_params)¶

class layers.Dense(in_dim, out_dim, kernel_initializer, bias_initializer, kernel_regularizer, activation)¶

Bases: object

Dense (fully-connected) layer class.

in_dim¶

Input dimension.

Type: int

out_dim¶

Output dimension.

Type: int

kernel_initializer¶

The weight parameter initializer.

Type: Initializer

bias_initializer¶

The bias parameter initializer.

Type: Initializer

kernel_regularizer¶

The weight parameter regularizer.

Type: Regularizer

activation¶

Layer activation.

Type: Activation

w¶

The weight parameters, of shape (in_dim, out_dim)

Type: numpy.ndarray

b¶

The bias parameters, of shape (1, out_dim)

Type: numpy.ndarray

cache¶

The run-time cache for storing activations, etc.

Type: dict

grads¶

The run-time cache for storing gradients.

Type: dict

__init__(in_dim, out_dim, kernel_initializer, bias_initializer, kernel_regularizer, activation)¶: Constructor.

get_w()¶: Returns the weight parameters.

get_b()¶: Returns the bias parameters.

set_w()¶: Sets the weight parameters.

set_b()¶: Sets the bias parameters.

get_dw()¶: Returns the gradients of weight parameters.

get_db()¶: Returns the gradients bias parameters.

get_reg_loss_w()¶: Returns the regularization loss of the weight parameters.

get_reg_grad_w()¶: Returns the regularization gradient of the weight parameters.

forward(x)¶: Forward-propagates signals through the layer and its activation.

backward(g_in)¶: Back-propagates gradients through the the activation of the layer and then the layer.

get_learnable_params()¶: Get all learnable params.

set_learnable_params(**learnable_params)¶: Set all learnable params.

get_learnable_params_grads()¶: Get the gradients of the learnable params.

__repr__()¶: Returns the string representation of class.

backward(g_in, **params)¶

Back-propagates gradients through the the activation of the layer and then the layer.

Parameters

g_in (numpy.ndarray) – Incoming (from later layers or losses) gradients, of shape (batch_size, out_dim).
params (dict) – Dict of params for forward pass such as train or test mode, seed, etc. Unused in Dense layer.

Returns

g_out – Outgoing (to previous layers, or input data) gradients, of shape (batch_size, in_dim).

Return type

numpy.ndarray

Notes

g_in.shape = (batch_size, out_dim) self.cache[“x”].shape = (batch_size, in_dim) self.w.shape=(in_dim, out_dim) self.b.shape=(1, out_dim) dw.shape=(in_dim, out_dim) db.shape=(1, out_dim) g_out.shape = (batch_size, in_dim)

forward(x, **params)¶

Forward-propagates signals through the layer and its activation.

Parameters

x (numpy.ndarray) – Input data to layer of shape (batch_size, in_dim).
params (dict) – Dict of params for forward pass such as train or test mode, seed, etc. Unused in Dense layer.

Returns

a – Activation of linear transformation, of shape (batch_size, out_dim).

Return type

numpy.ndarray

Notes

x.shape = (batch_size, in_dim) self.w.shape=(in_dim, out_dim) self.b.shape=(1, out_dim) z.shape = (batch_size, out_dim) a.shape = (batch_size, out_dim)

get_b()¶

Returns the bias parameters.

Parameters: None –
Returns: The bias parameters.
Return type: numpy.ndarray

Notes

None

get_db()¶

Returns the gradients of bias parameters.

Parameters: None –
Returns: ret – The gradients of bias parameters, or None if does not exist yet.
Return type: None or numpy.ndarray

Notes

None

get_dw()¶

Returns the gradients of weight parameters.

Parameters: None –
Returns: ret – The gradients of weight parameters, or None if does not exist yet.
Return type: None or numpy.ndarray

Notes

None

get_learnable_params()¶

Get all learnable params.

Parameters: None –
Returns: Dict of learanble params.
Return type: dict

Notes

None

get_learnable_params_grads()¶

Get the gradients of the learnable params.

Parameters: None –
Returns: Dict of grads of learanble params.
Return type: dict

Notes

None

get_reg_grad_w()¶

Returns the regularization gradient of the weight parameters.

Parameters: None –
Returns: Returns the regularization gradient of the weight parameters. Float 0.0 if does not exist yet - since added later doesn’t matter if 0.0 is float or matrix.
Return type: float or numpy.ndarray

Notes

None

get_reg_loss()¶

Returns the regularization loss of the weight parameters.

Parameters: None –
Returns: The regularization loss of the weight parameters
Return type: float

Notes

None

get_w()¶

Returns the weight parameters.

Parameters: None –
Returns: The weight parameters.
Return type: numpy.ndarray

Notes

None

if_has_learnable_params()¶

Returns if the layer has learnable params. Dense layer does have learnable params.

Parameters: None –
Returns: True if the layer has learnable params.
Return type: has_learnable_params

Notes

None

set_b(b)¶

Sets the bias parameters.

Parameters: b (numpy.ndarray) – The bias parameters.
Returns
Return type: None

Notes

None

set_learnable_params(**learnable_params)¶

Set all learnable params.

Parameters: learnable_params (dict) – Dict of learnable params.
Returns
Return type: None

Notes

None

set_w(w)¶

Sets the weight parameters.

Parameters: w (numpy.ndarray) – The weight parameters.
Returns
Return type: None

Notes

None

class layers.Dropout(p)¶

Bases: object

Inv dropout - scaling at train time

backward(g_in, **params)¶

forward(x, **params)¶

if_has_learnable_params()¶

class layers.RNN(in_dim, out_dim, hidden_dim, kernel_h_initializer, bias_h_initializer, kernel_o_initializer, bias_o_initializer, kernel_regularizer, activation_h, activation_o)¶

Bases: object

Many-to-many RNN layer for character-to-character sequence modelling.

in_dim¶

Input dimension.

Type: int

out_dim¶

Output dimension.

Type: int

hidden_dim¶

Hidden dimension.

Type: int

kernel_h_initializer¶

The weight parameter initializer of the hidden neurons.

Type: Initializer

bias_h_initializer¶

The bias parameter initializer of the hidden neurons.

Type: Initializer

kernel_o_initializer¶

The weight parameter initializer of the output neurons.

Type: Initializer

bias_o_initializer¶

The bias parameter initializer of the output neurons.

Type: Initializer

kernel_regularizer¶

The weight parameter regularizer for all parameters. Separate for h and o neurons. Not used yet.

Type: Regularizer

activation_h¶

Layer activation of hidden neurons.

Type: Activation

activation_o¶

Layer activation of output neurons.

Type: Activation

u¶

The weight parameters dotted with the input vector, of shape (in_dim, hidden_dim)

Type: numpy.ndarray

w¶

The weight parameters dotted with the pre-activation hidden vector, of shape (hidden_dim, hidden_dim)

Type: numpy.ndarray

b¶

The bias parameters added to the input-previous hidden vector linear combination, of shape (1, hidden_dim)

Type: numpy.ndarray

v¶

The weight parameters dotted with the activated hidden vector, of shape (hidden_dim, out_dim)

Type: numpy.ndarray

c¶

The bias parameters added to the dotted activated hidden vector, of shape (1, out_dim)

Type: numpy.ndarray

cache¶

The run-time cache for storing activations, etc.

Type: dict

grads¶

The run-time cache for storing gradients.

Type: dict

h_shape¶

Hidden vector shape.

Type: tuple

has_learnable_params¶

If layer has learnable/trainable params.

Type: bool

__init__(in_dim, out_dim, kernel_initializer, bias_initializer, kernel_regularizer, activation)¶: Constructor.

get_u()¶: Returns the u parameters.

get_w()¶: Returns the w parameters.

get_b()¶: Returns the b parameters.

get_v()¶: Returns the v parameters.

get_c()¶: Returns the c parameters.

set_u()¶: Sets the u parameters.

set_w()¶: Sets the w parameters.

set_b()¶: Sets the b parameters.

set_v()¶: Sets the v parameters.

set_c()¶: Sets the c parameters.

get_du()¶: Returns the gradients of u parameters.

get_dw()¶: Returns the gradients of w parameters.

get_db()¶: Returns the gradients b parameters.

get_dv()¶: Returns the gradients of v parameters.

get_dc()¶: Returns the gradients c parameters.

get_learnable_params()¶: Get all learnable params.

set_learnable_params(**learnable_params)¶: Set all learnable params.

get_learnable_params_grads()¶: Get the gradients of the learnable params.

get_reg_loss()¶: Returns the regularization loss of the weight parameters.

if_has_learnable_params()¶: Returns if layer has learnable params.

forward(x, \*\*params)¶: Forward-propagates signals through the layer and its activation.

backward(g_in, \*\*params)¶: Back-propagates gradients through the the activation of the layer and then the layer. Note that the RNN layer implements backpropagation through time (BPTT).

__repr__()¶: Returns the string representation of class.

backward(g_in, **params)¶

Back-propagates gradients through the the activation of the layer and then the layer. Note that the RNN layer implements backpropagation through time (BPTT).

Parameters

g_in (numpy.ndarray) – Incoming (from later layers or losses) gradients, of shape (batch_size, out_dim).
params (dict) – Dict of params for forward pass such as train or test mode, seed, etc. Unused in Dense layer.

Returns

g_out – Outgoing (to previous layers, or input data) gradients, of shape (batch_size, in_dim). Not implemented yet!

Return type

numpy.ndarray

Notes

Shapes are commented below.

forward(x, **params)¶

Forward-propagates signals through the layer and its activation.

Parameters

x (numpy.ndarray) – Input data to layer of shape (batch_size, in_dim).
params (dict) – Dict of params for forward pass such as train or test mode, seed, etc. Unused in RNN layer.

Returns

p – Activation of the RNN layer output neurons, of shape (batch_size, out_dim).

Return type

numpy.ndarray

Notes

Shapes are commented below.

get_b()¶

Returns the b parameters.

Parameters: None –
Returns: The b parameters.
Return type: numpy.ndarray

Notes

None

get_c()¶

Returns the c parameters.

Parameters: None –
Returns: The c parameters.
Return type: numpy.ndarray

Notes

None

get_db()¶

Returns the gradients of b parameters.

Parameters: None –
Returns: ret – The gradients of b parameters, or None if does not exist yet.
Return type: None or numpy.ndarray

Notes

None

get_dc()¶

Returns the gradients of c parameters.

Parameters: None –
Returns: ret – The gradients of c parameters, or None if does not exist yet.
Return type: None or numpy.ndarray

Notes

None

get_du()¶

Returns the gradients of u parameters.

Parameters: None –
Returns: ret – The gradients of u parameters, or None if does not exist yet.
Return type: None or numpy.ndarray

Notes

None

get_dv()¶

Returns the gradients of v parameters.

Parameters: None –
Returns: ret – The gradients of v parameters, or None if does not exist yet.
Return type: None or numpy.ndarray

Notes

None

get_dw()¶

Returns the gradients of w parameters.

Parameters: None –
Returns: ret – The gradients of w parameters, or None if does not exist yet.
Return type: None or numpy.ndarray

Notes

None

get_learnable_params()¶

Get all learnable params.

Parameters: None –
Returns: Dict of learanble params.
Return type: dict

Notes

None

get_learnable_params_grads()¶

Get the gradients of the learnable params.

Parameters: None –
Returns: Dict of grads of learanble params.
Return type: dict

Notes

None

get_reg_loss()¶

get_u()¶

Returns the u parameters.

Parameters: None –
Returns: The u parameters.
Return type: numpy.ndarray

Notes

None

get_v()¶

Returns the v parameters.

Parameters: None –
Returns: The v parameters.
Return type: numpy.ndarray

Notes

None

get_w()¶

Returns the w parameters.

Parameters: None –
Returns: The w parameters.
Return type: numpy.ndarray

Notes

None

if_has_learnable_params()¶

Returns if the layer has learnable params. Dense layer does have learnable params.

Parameters: None –
Returns: True if the layer has learnable params.
Return type: has_learnable_params

Notes

None

reset_hidden_state()¶

set_b(b)¶

Sets the b parameters.

Parameters: b (numpy.ndarray) – The b parameters.
Returns
Return type: None

Notes

None

set_c(c)¶

Sets the c parameters.

Parameters: c (numpy.ndarray) – The c parameters.
Returns
Return type: None

Notes

None

set_learnable_params(**learnable_params)¶

Set all learnable params.

Parameters: learnable_params (dict) – Dict of learnable params.
Returns
Return type: None

Notes

None

set_u(u)¶

Sets the u parameters.

Parameters: u (numpy.ndarray) – The u parameters.
Returns
Return type: None

Notes

None

set_v(v)¶

Sets the v parameters.

Parameters: v (numpy.ndarray) – The v parameters.
Returns
Return type: None

Notes

None

set_w(w)¶

Sets the w parameters.

Parameters: w (numpy.ndarray) – The w parameters.
Returns
Return type: None

Notes

None