Download Package `FCNN4R`

Package ‘FCNN4R’
March 9, 2016
Type Package
Title Fast Compressed Neural Networks for R
Version 0.6.2
Date 2016-03-08
Author Grzegorz Klima <[email protected]>
Maintainer Grzegorz Klima <[email protected]>
Description Provides an interface to kernel routines from the FCNN C++ library.
FCNN is based on a completely new Artificial Neural Network representation that
offers unmatched efficiency, modularity, and extensibility. FCNN4R provides
standard teaching (backpropagation, Rprop, simulated annealing, stochastic
gradient) and pruning algorithms (minimum magnitude, Optimal Brain Surgeon),
but it is first and foremost an efficient computational engine. Users can
easily implement their algorithms by taking advantage of fast gradient computing
routines, as well as network reconstruction functionality (removing weights
and redundant neurons, reordering inputs, merging networks). Networks can be
exported to C functions in order to integrate them into virtually any software
solution.
Depends R (>= 3.0), stats, graphics, methods, Rcpp
License GPL (>= 2)
NeedsCompilation yes
Repository CRAN
Date/Publication 2016-03-09 00:57:57
R topics documented:
FCNN4R-package . . . . . . . . . . .
is.mlp_net . . . . . . . . . . . . . . .
mlp_eval . . . . . . . . . . . . . . . .
mlp_export_C . . . . . . . . . . . . .
mlp_net . . . . . . . . . . . . . . . .
mlp_net-absolute-weight-indices . . .
mlp_net-accessing-individual-weights
.
.
.
.
.
.
.
.
.
.
.
.
.
.
1
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
2
3
4
4
5
6
7
2
FCNN4R-package
mlp_net-class . . . . . . . . . . . . .
mlp_net-combining-two-networks . .
mlp_net-display . . . . . . . . . . . .
mlp_net-export-import . . . . . . . .
mlp_net-general-information . . . . .
mlp_net-manipulating-network-inputs
mlp_net-MSE-gradients . . . . . . . .
mlp_net-names . . . . . . . . . . . .
mlp_net-weights-access . . . . . . . .
mlp_plot . . . . . . . . . . . . . . . .
mlp_prune_mag . . . . . . . . . . . .
mlp_prune_obs . . . . . . . . . . . .
mlp_rm_neurons . . . . . . . . . . .
mlp_rnd_weights . . . . . . . . . . .
mlp_set_activation . . . . . . . . . .
mlp_teach_bp . . . . . . . . . . . . .
mlp_teach_grprop . . . . . . . . . . .
mlp_teach_rprop . . . . . . . . . . .
mlp_teach_sa . . . . . . . . . . . . .
mlp_teach_sgd . . . . . . . . . . . .
read-write-fcnndataset . . . . . . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Index
FCNN4R-package
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
8
9
10
10
11
12
13
14
15
16
16
17
18
19
19
20
21
23
24
25
26
28
Fast Compressed Neural Networks for R
Description
Provides an interface to kernel routines from the FCNN C++ library. FCNN is based on a completely new Artificial Neural Network representation that offers unmatched efficiency, modularity,
and extensibility. FCNN4R provides standard teaching (backpropagation, Rprop, simulated annealing, stochastic gradient) and pruning algorithms (minimum magnitude, Optimal Brain Surgeon),
but it is first and foremost an efficient computational engine. Users can easily implement their
algorithms by taking advantage of fast gradient computing routines, as well as network reconstruction functionality (removing weights and redundant neurons, reordering inputs, merging networks).
Networks can be exported to C functions in order to integrate them into virtually any software
solution.
Author(s)
Grzegorz Klima <[email protected]>
References
G. Klima. A new approach towards implementing artificial neural networks. Technical Report,
http://fcnn.sourceforge.net/, 2013.
is.mlp_net
3
Examples
# set up the XOR problem inputs and outputs
inp <- c(0, 0, 1, 1, 0, 1, 0, 1)
dim(inp) <- c(4, 2)
outp <- c(0, 1, 1, 0)
dim(outp) <- c(4, 1)
# create a 2-6-1 network
net <- mlp_net(c(2, 6, 1))
# set activation function in all layers
net <- mlp_set_activation(net, layer = "a", "sigmoid")
# randomise weights
net <- mlp_rnd_weights(net)
# tolerance level
tol <- 0.5e-4
# teach using Rprop, assign trained network and plot learning history
netmse <- mlp_teach_rprop(net, inp, outp, tol_level = tol,
max_epochs = 500, report_freq = 10)
net <- netmse$net
plot(netmse$mse, type = 'l')
# plot network with weights
mlp_plot(net, TRUE)
# if the algorithm had converged, prune using Optimal Brain Surgeon and plot
if (mlp_mse(net, inp, outp) <= tol) {
net <- mlp_prune_obs(net, inp, outp, tol_level = tol,
max_reteach_epochs = 500, report = TRUE)[[1]]
mlp_plot(net, TRUE)
}
# check network output
round(mlp_eval(net, inp), digits = 3)
is.mlp_net
Is it an object of mlp_net class?
Description
This function checks whether argument is an object of mlp_net class.
Usage
is.mlp_net(x)
Arguments
x
Value
Logical value.
an object to be checked
4
mlp_export_C
mlp_eval
Evaluation
Description
Evaluate network output.
Usage
mlp_eval(net, input)
Arguments
net
an object of mlp_net class
input
numeric matrix, each row corresponds to one input vector, the number of columns
must be equal to the number of neurons in the network input layer
Value
Numeric matrix with rows representing network outputs corresponding to input rows.
mlp_export_C
Export multilayer perceptron network to a C function
Description
This function exports multilayer perceptron network to a C function with optional affine input and
output transformations: Ax+b for inputs and Cx+d for outputs.
Usage
mlp_export_C(fname, net, with_bp = FALSE, A = NULL, b = NULL, C = NULL,
d = NULL)
Arguments
fname
character string with the filename
net
an object of mlp_net class
with_bp
logical, should backpropagation code for online learning be exported?
A
numeric matrix (optional), input linear transformation
b
numeric vector (optional), input translation
C
numeric matrix (optional), output linear transformation
d
numeric vector (optional), output translation
mlp_net
5
Value
Logical value, TRUE if export was successful, FALSE otherwise.
Examples
# create a 2-3-1 network
net <- mlp_net(c(2, 3, 1))
# randomise weights
net <- mlp_rnd_weights(net)
# show the network
show(net)
# export network to a C function
mlp_export_C("test.c", net)
# show the output file
file.show("test.c")
mlp_net
Create objects of mlp_net class
Description
Function used for creating multilayer perceptron networks.
Usage
mlp_net(layers, name = NULL)
Arguments
layers
vector providing numbers of neurons in each layer
name
character string, network name (optional)
Value
Returns an object of mlp_net class.
See Also
mlp_net for details.
6
mlp_net-absolute-weight-indices
Examples
# create a 2-3-1 network
net <- mlp_net(c(2, 3, 1))
# randomise weights
net <- mlp_rnd_weights(net)
# show basic information about the network
show(net)
mlp_net-absolute-weight-indices
Retrieving absolute weight index
Description
In some situations absolute weight index (i.e. index within all weights including inactive ones)
needs to be computed based on information about connected neurons’ indices or weight index
within actives ones. The latter functionality is especially useful in implementation of pruning algorithms.
Usage
mlp_get_w_idx(net, layer, nidx, nplidx)
mlp_get_w_abs_idx(net, idx)
Arguments
net
an object of mlp_net class
layer
integer value (vector), layer index
nidx
integer value (vector), neuron index
nplidx
integer value (vector), index of the neuron in the previous layer determining
connection from neuron nidx in layer, 0 denotes bias of neuron nidx in layer
idx
integer value (vector), weight index (indices) within active ones
Value
Absolute weight index.
mlp_net-accessing-individual-weights
7
mlp_net-accessing-individual-weights
Setting and retrieving status (on/off) and value of individual weight(s)
Description
The following functions can be used to access individual weight(s), i.e. set or retrieve status(es)
(on/off) and value(s).
Usage
mlp_set_w_st(net, on, idx = NULL, layer = NULL, nidx = NULL,
nplidx = NULL)
mlp_set_w(net, val, idx = NULL, layer = NULL, nidx = NULL,
nplidx = NULL)
mlp_get_w_st(net, idx = NULL, layer = NULL, nidx = NULL, nplidx = NULL)
mlp_get_w(net, idx = NULL, layer = NULL, nidx = NULL, nplidx = NULL)
Arguments
net
an object of mlp_net class
on
logical value (vector), should the weight be set on or off?
idx
integer value (vector), weight absolute index
layer
integer value (vector), layer index
nidx
integer value (vector), neuron index
nplidx
integer value (vector), index of the neuron in the previous layer determining
connection from neuron nidx in layer, 0 denotes bias of neuron nidx in layer
val
numeric value (vector), connection (or bias) value to be set
Value
mlp_set_w_st returns network (an object of mlp_net class) with state(s) (on/off) of selected weight(s)
set.
mlp_set_w returns network (an object of mlp_net class) with value(s) of selected weight(s) set.
mlp_get_w_st returns logical value (vector), TRUE if connection/bias is active, FALSE otherwise.
mlp_get_w returns numeric value (vector), selected weight value(s).
8
mlp_net-class
mlp_net-class
An S4 class representing Multilayer Perception Network.
Description
The mlp_net class represents the Multilayer Perception Network employing the so-called compressed representation, which was inspired by the Compressed Column Storage familiar from sparse
matrix algebra. Although the representation and algorithms working with it are somewhat complicated, the user is provided with a simple and intuitive interface that completely hides the internal
workings of the package, which in its large part is written in C++.
Slots
m_name character string, network name
m_layers integer vector, stores the numbers of neurons in layers
m_n_pointers integer vector, stores the so-called ’pointers’ to neurons
m_n_prev integer vector, stores the number of connected neurons in the previous layer
m_n_next integer vector, stores the number of connected neurons in the next layer
m_w_pointers integer vector, stores the so-called ’pointers’ to weights
m_w_values numeric vector, values of connection weights and biases
m_w_flags logical vector, states (active/inactive) of weights and biases
m_w_on integer value, the number of active weights
m_af integer vector, activation functions’ indices
m_af_p numeric vector, activation functions’ slope parameters
References
G. Klima. A new approach towards implementing artificial neural networks. Technical Report,
http://fcnn.sourceforge.net/, 2013.
See Also
mlp_net for creating objects of this class.
mlp_net-combining-two-networks
9
mlp_net-combining-two-networks
Combining two networks into one
Description
These functions construct new network by merging two networks (they must have the same number
of layers) or by connecting one network outputs to another network inputs (the numbers of output
and input neurons must agree). These functions may be used in constructing deep learning networks
or constructing networks with some special topologies.
Usage
mlp_merge(net1, net2, same_inputs = FALSE)
mlp_stack(net1, net2)
Arguments
net1
an object of mlp_net class
net2
an object of mlp_net class
same_inputs
logical, if TRUE both merged networks are assumed to take the same inputs
(they share the input layer), default is FALSE
Value
Both functions return an object of mlp_net class.
Examples
# create two 2-2-2 networks with random weights and plot them
net1 <- mlp_net(c(2, 2, 2))
net1 <- mlp_rnd_weights(net1)
mlp_plot(net1, TRUE)
net2 <- mlp_net(c(2, 2, 2))
net2 <- mlp_rnd_weights(net2)
mlp_plot(net2, TRUE)
# create a 4-3-2 network with random weights and plot it
net3 <- mlp_net(c(4, 3, 2))
net3 <- mlp_rnd_weights(net3)
mlp_plot(net3, TRUE)
# construct new network using net1, net2, and net3 and plot it
net4 <- mlp_stack(mlp_merge(net1, net2), net3)
mlp_plot(net4, TRUE)
10
mlp_net-export-import
mlp_net-display
Displaying networks (objects of mlp_net class)
Description
These methods can be used to display objects of mlp_net class. show and print provide short
information about network structure and activation functions, summary gives detailed information
about all network connections.
Usage
## S4 method for signature 'mlp_net'
show(object)
## S4 method for signature 'mlp_net'
print(x)
## S4 method for signature 'mlp_net'
summary(object)
Arguments
object
an object of mlp_net class
x
an object of mlp_net class
mlp_net-export-import Export and import multilayer perceptron network to/from a text file in
FCNN format
Description
These functions can be used to export and import multilayer perceptron network to/from a text file
in FCNN format.
Usage
mlp_export_fcnn(fname, net)
mlp_import_fcnn(fname)
Arguments
fname
character string with the filename
net
an object of mlp_net class
mlp_net-general-information
11
Details
Files are organised as follows:
• the first comment (beginning with #) is treated as network information (name) string,
• all other comments are ignored,
• network structure is represented by five block of numbers:
–
–
–
–
the first line determines numbers of neurons in consecutive layers,
the second block of 0’s and 1’s determines which weights are turned off/on,
the third block contains active weights’ values,
the last block determines hidden and output layers’ activation functions and their slope
parameters - each line contains 2 numbers: the function index and its slope parameter.
Value
mlp_export_fcnn returns logical value, TRUE if export was successful, FALSE otherwise.
mlp_import_fcnn returns an object of mlp_net class or NULL, if import failed.
See Also
mlp_net for network representation details.
Examples
# create a 2-3-1 network
net <- mlp_net(c(2, 3, 1))
# randomise weights
net <- mlp_rnd_weights(net)
# show the network
show(net)
# export network
mlp_export_fcnn("test.net", net)
# show the output file
file.show("test.net")
# import network
net2 <- mlp_import_fcnn("test.net")
# show the imported network
show(net2)
mlp_net-general-information
General information about network
Description
The following functions return basic information about the network.
12
mlp_net-manipulating-network-inputs
Usage
mlp_get_layers(net)
mlp_get_no_active_w(net)
mlp_get_no_w(net)
Arguments
net
an object of mlp_net class
Value
mlp_get_layers returns an integer vector with numbers of neurons in consecutive layers.
mlp_get_no_active_w returns the number of active weights (connections and biases).
mlp_get_no_w returns the total number (including inactive) of weights (connections and biases).
See Also
mlp_net-class for details on internal network representation.
mlp_net-manipulating-network-inputs
Manipulating network inputs
Description
These functions construct new network by removing redundant (i.e. not connected to the next layer)
inputs or reordering / expanding network inputs.
Usage
mlp_rm_input_neurons(net, report = FALSE)
mlp_expand_reorder_inputs(net, newnoinputs, inputsmap)
Arguments
net
an object of mlp_net class
report
logical value, if TRUE, information about removed neurons will be printed on
the console (FALSE by default)
newnoinputs
integer value, determines the number of inputs in the new network
inputsmap
integer vector, determines the mapping of old inputs into new ones - the ith value
of this vector will be the new index of ith input
mlp_net-MSE-gradients
13
Value
mlp_rm_input_neurons returns a two-element list. The first element (net) is the network (an
object of mlp_net class) with all redundant input neurons removed, the second (ind) - the indices
of input neurons that were not removed.
mlp_expand_reorder_inputs returns an object of mlp_net class.
Examples
# construct a 2-4-3 network, plot result
nn <- mlp_net(c(2, 4, 3))
nn <- mlp_rnd_weights(nn)
mlp_plot(nn, TRUE)
# expand inputs, the new no. of inputs will be 5, with the first input
# becoming the 3rd and the second retaining its position, plot result
nn <- mlp_expand_reorder_inputs(nn, 5, c(3, 2))
mlp_plot(nn, TRUE)
# remove redundant neurons (i.e. 1, 4, 5) and plot result
nn <- mlp_rm_input_neurons(nn, TRUE)$net
mlp_plot(nn, TRUE)
mlp_net-MSE-gradients Computing mean squared error, its gradient, and output derivatives
Description
The functions use fast FCNN kernel routines and are intended for implementing teaching and pruning algorithms.
Usage
mlp_mse(net, input, output)
mlp_grad(net, input, output)
mlp_gradi(net, input, output, i)
mlp_gradij(net, input, i)
mlp_jacob(net, input, i)
Arguments
net
an object of mlp_net class
input
numeric matrix, each row corresponds to one input vector, the number of columns
must be equal to the number of neurons in the network input layer
14
mlp_net-names
output
numeric matrix with rows corresponding to expected outputs, the number of
columns must be equal to the number of neurons in the network output layer,
the number of rows must be equal to the number of input rows
i
data row index
Details
mlp_mse returns the mean squared error (MSE). MSE is understood as half of the squared error
averaged over all outputs and data records.
mlp_grad computes the gradient of MSE w.r.t. network weights. This function is useful when
implementing batch teaching algorithms.
mlp_gradi computes the gradient of MSE w.r.t. network weights at the ith data record. This is
normalised by the number of outputs only, the average over all rows (all i) returns the same as
grad(input, output). This function is useful for implementing on-line teaching algorithms.
mlp_gradij computes gradients of network outputs, i.e the derivatives of outputs w.r.t. active
weights, at given data row. The derivatives of outputs are placed in subsequent columns of the returned matrix. Scaled by the output errors and averaged they give the same as gradi(input, output, i).
This function is useful in implementing teaching algorithms using second order corrections and Optimal Brain Surgeon pruning algorithm.
mlp_jacob computes the Jacobian of network outputs, i.e the derivatives of outputs w.r.t. inputs, at
given data row. The derivatives of outputs are placed in subsequent columns of the returned matrix.
Value
mlp_mse returns mean squared error (numeric value).
mlp_grad returns two-element lists with the first field (grad) containing numeric vector with gradient and the second (mse) - the mean squared error.
mlp_gradi returns numeric vector with gradient.
mlp_gradij returns numeric matrix with gradients of outputs in consecutive columns.
mlp_jacob returns numeric matrix with derivatives of outputs in consecutive columns.
mlp_net-names
Get and set network names
Description
The following functions can be used for retrieving and setting network names.
Usage
mlp_get_name(net)
mlp_set_name(net, name)
mlp_net-weights-access
15
Arguments
net
an object of mlp_net class
name
character string with network name
Value
mlp_get_name returns character string with network name.
mlp_set_name returns network (an object of mlp_net class) with name set to new value.
mlp_net-weights-access
Set and retrieve (active) weights’ values
Description
One of FCNN’s design objectives (and main advantages) is the complete separation of teaching (and
pruning) algorithms from internal network structure workings. This goal is achieved through fast
access to (active) weights vector facilitated by FCNN’s ‘compressed’ network representation. The
following two functions allow users to efficiently retrieve and set network (active) weights vector.
Usage
mlp_set_weights(net, weights)
mlp_get_weights(net)
Arguments
net
an object of mlp_net class
weights
numeric vector of new active weights’ values
Value
mlp_set_weights returns network (an object of mlp_net class) with active weights set to given
values.
mlp_set_weights returns numeric vector of active weights’ values.
16
mlp_prune_mag
mlp_plot
Plotting multilayer perceptron network
Description
This function plots a multilayer perceptron network’s structure. Optionally, weights’ values are
displayed on graph.
Usage
mlp_plot(net, show_weights = FALSE, show_neuron_idx = TRUE)
Arguments
net
an object of mlp_net class
show_weights
logical, should weights’ values be displayed? (FALSE by default)
show_neuron_idx
logical, should neurons’ indices be displayed? (TRUE by default)
Value
This function does not return value.
mlp_prune_mag
Minimum magnitude pruning
Description
Minimum magnitude pruning is a brute force, easy-to-implement pruning algorithm in which in
each step the weight with the smallest absolute value is turned off. This algorithm requires reteaching network in almost every step and yields suboptimal results.
Usage
mlp_prune_mag(net, input, output, tol_level, max_reteach_epochs, report,
plots = FALSE)
Arguments
net
an object of mlp_net class
input
numeric matrix, each row corresponds to one input vector, the number of columns
must be equal to the number of neurons in the network input layer
output
numeric matrix with rows corresponding to expected outputs, the number of
columns must be equal to the number of neurons in the network output layer,
the number of rows must be equal to the number of input rows
mlp_prune_obs
17
tol_level
numeric value, error (MSE) tolerance level
max_reteach_epochs
integer value, maximal number of epochs (iterations) allowed when reteaching
network
report
logical value, if TRUE, information about the pruning process will be printed on
the console (FALSE by default)
plots
logical value, if TRUE, the initial network is plotted and then replotted every
time neuron is removed and at the end of pruning (FALSE by default)
Value
Three-element list, the first field (net) contains the pruned network, the second (wcount) - the
number of connections removed (inactivated), the third (ncount) - the number of neurons removed.
mlp_prune_obs
Optimal Brain Surgeon pruning
Description
The Optimal Brain Surgeon algorithm is a robust (yet computationally demanding) pruning algorithm in which candidate weight to be turned off is determined based on information about the
inverse of (approximate) Hessian matrix of the MSE.
Usage
mlp_prune_obs(net, input, output, tol_level, max_reteach_epochs, report,
plots = FALSE, alpha = 1e-05)
Arguments
net
an object of mlp_net class
input
numeric matrix, each row corresponds to one input vector, the number of columns
must be equal to the number of neurons in the network input layer
output
numeric matrix with rows corresponding to expected outputs, the number of
columns must be equal to the number of neurons in the network output layer,
the number of rows must be equal to the number of input rows
tol_level
numeric value, error (MSE) tolerance level
max_reteach_epochs
integer value, maximal number of epochs (iterations) allowed when reteaching
network
report
logical value, if TRUE, information about the pruning process will be printed on
the console (FALSE by default)
plots
logical value, if TRUE, the initial network is plotted and then replotted every
time neuron is removed and at the end of pruning (FALSE by default)
alpha
numeric value, scaling factor used for initial Hessian approximation
18
mlp_rm_neurons
Value
Three-element list, the first field (net) contains the pruned network, the second (wcount) - the
number of connections removed (inactivated), the third (ncount) - the number of neurons removed.
References
B. Hassibi, D. G. Stork, and G. J. Wolff. Optimal Brain Surgeon and General Network Pruning.
Technical Report CRC-TR-9235, RICOH California Research Centre, 1992.
mlp_rm_neurons
Remove redundant neurons in a multilayer perceptron network
Description
This function removes redundant neurons from the network, i.e. hidden layers’ neurons that are
not connected to neurons in the previous layer or the next layer. If a neuron is not connected to
neurons in the previous layer but is connected to neurons in the next layer (effectively acts as an
additional bias), biases of neurons in the next layer are properly adjusted, therefore, the resulting
network behaves just like the initial one.
Usage
mlp_rm_neurons(net, report = FALSE)
Arguments
net
an object of mlp_net class
report
logical value, if TRUE, information about removed neurons will be printed on
the console (FALSE by default)
Value
Three-element list. The first element (net) is the network (an object of mlp_net class) with all
redundant neurons removed, the second (ncount) - the number of neurons removed, the third
(wcount) - the number of weights removed.
mlp_rnd_weights
19
mlp_rnd_weights
This function sets network weights to random values drawn from uniform distribution.
Description
This function sets network weights to random values drawn from uniform distribution.
Usage
mlp_rnd_weights(net, a = 0.2)
Arguments
net
a
an object of mlp_net class
numeric value, values will be drawn from uniform distribution on [-a, a] (by
default a = 0.2)
Value
Network (an object of mlp_net class) with randomised weights.
mlp_set_activation
Set network activation functions
Description
This function sets activation function (and its slope parameter) for neurons in the hidden layers and
in the output layer.
Usage
mlp_set_activation(net, layer, activation = c("threshold", "sym_threshold",
"linear", "sigmoid", "sym_sigmoid", "tanh", "sigmoid_approx",
"sym_sigmoid_approx"), slope = 0)
Arguments
net
layer
activation
slope
an object of mlp_net class
integer vector or character value, index (indices) of layer(s) whose activation
function will be changed or character: "a" denotes all layers, "h" - hidden
layer(s), "o" - the output layer
character string, activation function name, admissible options are: "threshold",
"sym_threshold", "linear", "sigmoid", "sym_sigmoid" (and "tanh"), "sigmoid_approx",
and "sym_sigmoid_approx"
numeric value, activation function slope parameter, if 0 the default parameter
value is chosen for each activation function
20
mlp_teach_bp
Value
This function returns network (an object of mlp_net class) with activation function set.
mlp_teach_bp
Backpropagation (batch) teaching
Description
Backpropagation (a teaching algorithm) is a simple steepest descent algorithm for MSE minimisation, in which weights are updated according to (scaled) gradient of MSE.
Usage
mlp_teach_bp(net, input, output, tol_level, max_epochs, learn_rate = 0.7,
l2reg = 0, report_freq = 0)
Arguments
net
an object of mlp_net class
input
numeric matrix, each row corresponds to one input vector, the number of columns
must be equal to the number of neurons in the network input layer
output
numeric matrix with rows corresponding to expected outputs, the number of
columns must be equal to the number of neurons in the network output layer,
the number of rows must be equal to the number of input rows
tol_level
numeric value, error (MSE) tolerance level
max_epochs
integer value, maximal number of epochs (iterations)
learn_rate
numeric value, learning rate in the backpropagation algorithm (default 0.7)
l2reg
numeric value, L2 regularization parameter (default 0)
report_freq
integer value, progress report frequency, if set to 0 no information is printed on
the console (this is the default)
Value
Two-element list, the first field (net) contains the trained network, the second (mse) - the learning
history (MSE in consecutive epochs).
Note
The name ‘backpropagation’ is commonly used in two contexts, which sometimes causes confusion.
Firstly, backpropagation can be understood as an efficient algorithm for MSE gradient computation
that was first described by Bryson and Ho in the ’60s of 20th century and reinvented in the ’80s.
Secondly, the name backpropagation is (more often) used to refer to the steepest descent method
that uses gradient of MSE computed efficiently by means of the aforementioned algorithm. This
ambiguity is probably caused by the fact that in practically all neural network implementations, the
derivatives of MSE and weight updates are computed simultaneously in one backward pass (from
output layer to input layer).
mlp_teach_grprop
21
References
A.E. Bryson and Y.C. Ho. Applied optimal control: optimization, estimation, and control. Blaisdell
book in the pure and applied sciences. Blaisdell Pub. Co., 1969.
David E. Rumelhart, Geoffrey E. Hinton, and Ronald J. Williams. Learning representations by
back-propagating errors. Nature, 323(6088):533-536, October 1986.
mlp_teach_grprop
Rprop teaching - minimising arbitrary objective function
Description
This implementation (‘generalisation’) of the Rprop algorithm allows users to teach network to
minimise arbitrary objective function provided that functions evaluating objective and computing
gradient are provided.
Usage
mlp_teach_grprop(net, obj_func, gradient, epochs, stop = NULL,
report_freq = 0, report_action = NULL, u = 1.2, d = 0.5, gmax = 50,
gmin = 1e-06)
Arguments
net
an object of mlp_net class
obj_func
function taking an object of mlp_class class as a single argument returning
objective to be minimised
gradient
function taking an object of mlp_class class as a single argument returning
gradient of the objective
epochs
integer value, number of epochs (iterations)
stop
function (or NULL), a function taking objective history to date and returning
Boolean value (if TRUE is returned, algorithm stops) (the default is not to stop
until all iterations are performed)
report_freq
integer value, progress report frequency, if set to 0 no information is printed on
the console (this is the default)
report_action
function (or NULL), additional action to be taken while printing progress reports, this should be a function taking network as a single argument (default
NULL)
u
numeric value, Rprop algorithm parameter (default 1.2)
d
numeric value, Rprop algorithm parameter (default 0.5)
gmax
numeric value, Rprop algorithm parameter (default 50)
gmin
numeric value, Rprop algorithm parameter (default 1e-6)
22
mlp_teach_grprop
Value
Two-element list, the first field (net) contains the trained network, the second (obj) - the learning
history (value of the objective function in consecutive epochs).
References
M. Riedmiller. Rprop - Description and Implementation Details: Technical Report. Inst. f. Logik,
Komplexitat u. Deduktionssysteme, 1994.
Examples
## Not run:
# set up XOR problem
inp <- c(0, 0, 1, 1, 0, 1, 0, 1)
dim(inp) <- c(4, 2)
outp <- c(0, 1, 1, 0)
dim(outp) <- c(4, 1)
# objective
obj <- function(net)
{
return(mlp_mse(net, inp, outp))
}
# gradient
grad <- function(net)
{
return(mlp_grad(net, inp, outp)$grad)
}
# stopping citerion
tol <- function(oh) {
if (oh[length(oh)] <= 5e-5) { return(TRUE); }
return(FALSE)
}
# create a 2-6-1 network
net <- mlp_net(c(2, 6, 1))
# set activation function in all layers
net <- mlp_set_activation(net, layer = "a", "sigmoid")
# randomise weights
net <- mlp_rnd_weights(net)
# teach
netobj <- mlp_teach_grprop(net, obj, grad, epochs = 500,
stop = tol,
report_freq = 1)
# plot learning history
plot(netobj$obj, type = 'l')
## End(Not run)
mlp_teach_rprop
mlp_teach_rprop
23
Rprop teaching
Description
Rprop is a fast and robust adaptive step method based on backpropagation. For details, please refer
to the original paper given in References section.
Usage
mlp_teach_rprop(net, input, output, tol_level, max_epochs, l2reg = 0,
u = 1.2, d = 0.5, gmax = 50, gmin = 1e-06, report_freq = 0)
Arguments
net
an object of mlp_net class
input
numeric matrix, each row corresponds to one input vector, the number of columns
must be equal to the number of neurons in the network input layer
output
numeric matrix with rows corresponding to expected outputs, the number of
columns must be equal to the number of neurons in the network output layer,
the number of rows must be equal to the number of input rows
tol_level
numeric value, error (MSE) tolerance level
max_epochs
integer value, maximal number of epochs (iterations)
l2reg
numeric value, L2 regularization parameter (default 0)
u
numeric value, Rprop algorithm parameter (default 1.2)
d
numeric value, Rprop algorithm parameter (default 0.5)
gmax
numeric value, Rprop algorithm parameter (default 50)
gmin
numeric value, Rprop algorithm parameter (default 1e-6)
report_freq
integer value, progress report frequency, if set to 0 no information is printed on
the console (this is the default)
Value
Two-element list, the first field (net) contains the trained network, the second (mse) - the learning
history (MSE in consecutive epochs).
References
M. Riedmiller. Rprop - Description and Implementation Details: Technical Report. Inst. f. Logik,
Komplexitat u. Deduktionssysteme, 1994.
24
mlp_teach_sa
mlp_teach_sa
Teaching networks using Simulated Annealing
Description
This function can be used to teach an ANN to minimise arbitrary objective function.
Usage
mlp_teach_sa(net, obj_func, Tinit = 1, epochs = 1000, report_freq = 0,
report_action = NULL)
Arguments
net
an object of mlp_net class
obj_func
function taking an object of mlp_class class as a single argument returning
objective to be minimised
Tinit
numeric value, initial temperature (default is 1)
epochs
integer value, number of epochs (iterations) (default is 1000)
report_freq
integer value, progress report frequency, if set to 0 no information is printed on
the console (this is the default)
report_action
function (or NULL), additional action to be taken while printing progress reports, this should be a function taking network as a single argument (default
NULL)
Value
Two-element list, the first field (net) contains the trained network, the second (obj) - the learning
history (value of the objective function in consecutive epochs).
Examples
## Not run:
# set up XOR problem
inp <- c(0, 0, 1, 1, 0, 1, 0, 1)
dim(inp) <- c(4, 2)
outp <- c(0, 1, 1, 0)
dim(outp) <- c(4, 1)
# objective
obj <- function(net)
{
return(mlp_mse(net, inp, outp))
}
# create a 2-6-1 network
net <- mlp_net(c(2, 6, 1))
# set activation function in all layers
net <- mlp_set_activation(net, layer = "a", "sigmoid")
mlp_teach_sgd
25
# teach
netobj <- mlp_teach_sa(net, obj, Tinit = 1, epochs = 1000,
report_freq = 1)
# plot learning history
plot(netobj$obj, type = 'l')
## End(Not run)
mlp_teach_sgd
Stochastic gradient descent with (optional) RMS weights scaling,
weight decay, and momentum
Description
This function implements the stochastic gradient descent method with optional modifications: L2
regularization, root mean square gradient scaling, weight decay, and momentum.
Usage
mlp_teach_sgd(net, input, output, tol_level, max_epochs, learn_rate,
l2reg = 0, minibatchsz = 100, lambda = 0, gamma = 0, momentum = 0,
report_freq = 0)
Arguments
net
an object of mlp_net class
input
numeric matrix, each row corresponds to one input vector number of columns
must be equal to the number of neurons in the network input layer
output
numeric matrix with rows corresponding to expected outputs, number of columns
must be equal to the number of neurons in the network output layer, number of
rows must be equal to the number of input rows
tol_level
numeric value, error (MSE) tolerance level
max_epochs
integer value, maximal number of epochs (iterations)
learn_rate
numeric value, (initial) learning rate, depending on the problem at hand, learning
rates of 0.001 or 0.01 should give satisfactory convergence
l2reg
numeric value, L2 regularization parameter (default 0)
minibatchsz
integer value, the size of the mini batch (default 100)
lambda
numeric value, rmsprop parameter controlling the update of mean squared gradient, reasonable value is 0.1 (default 0)
gamma
numeric value, weight decay parameter (default 0)
momentum
numeric value, momentum parameter, reasonable values are between 0.5 and
0.9 (default 0)
report_freq
integer value, progress report frequency, if set to 0 no information is printed on
the console (this is the default)
26
read-write-fcnndataset
Value
Two-element list, the first field (net) contains the trained network, the second (mse) - the learning
history (MSE in consecutive epochs).
read-write-fcnndataset
Reading and writing datasets in the FCNN format
Description
These functions can be used to read and write datasets from/to a text file in the FCNN format.
Datasets in the similar FANN format (comments are not supported by FANN) can also be read by
read.fcnndataset.
Usage
read.fcnndataset(fname)
write.fcnndataset(fname, input, output)
Arguments
fname
character string with the filename
input
numeric matrix, each row corresponds to one input vector
output
numeric matrix with rows corresponding to expected outputs, the number of
rows must be equal to the number of input rows
Details
Files are organised as follows:
• The first comment (beginning with #) is the dataset information (ignored on read),
• three numbers determine: number of records, no. of inputs and no. of outputs,
• each data record has two or three lines:
– (optional) record information in a comment (beginning with #),
– line with input values,
– line with output values.
Value
read.fcnndataset returns a dataframe.
write.fcnndataset does not return.
read-write-fcnndataset
Examples
# set up the XOR problem inputs and outputs
inp <- c(0, 0, 1, 1, 0, 1, 0, 1)
dim(inp) <- c(4, 2)
outp <- c(0, 1, 1, 0)
dim(outp) <- c(4, 1)
# write dataset
write.fcnndataset("xor.dat", inp, outp)
# show the output file
file.show("xor.dat")
# read dataset
xordf <- read.fcnndataset("xor.dat")
# show the imported dataset
show(xordf)
27
Index
mlp_get_w_abs_idx
(mlp_net-absolute-weight-indices),
6
mlp_get_w_idx
(mlp_net-absolute-weight-indices),
6
mlp_get_w_st
(mlp_net-accessing-individual-weights),
7
mlp_get_weights
(mlp_net-weights-access), 15
mlp_grad (mlp_net-MSE-gradients), 13
mlp_gradi (mlp_net-MSE-gradients), 13
mlp_gradij (mlp_net-MSE-gradients), 13
mlp_import_fcnn
(mlp_net-export-import), 10
FCNN4R-package, 2
mlp_jacob (mlp_net-MSE-gradients), 13
mlp_merge
is.mlp_net, 3
(mlp_net-combining-two-networks),
9
mlp_eval, 4
mlp_mse (mlp_net-MSE-gradients), 13
mlp_expand_reorder_inputs
mlp_net, 5, 5, 8, 11
(mlp_net-manipulating-network-inputs),
mlp_net-absolute-weight-indices, 6
12
mlp_net-accessing-individual-weights,
mlp_export_C, 4
7
mlp_export_fcnn
mlp_net-class,
8
(mlp_net-export-import), 10
mlp_net-combining-two-networks, 9
mlp_get_layers
mlp_net-display, 10
(mlp_net-general-information),
mlp_net-export-import, 10
11
mlp_net-general-information, 11
mlp_get_name (mlp_net-names), 14
mlp_net-manipulating-network-inputs,
mlp_get_no_active_w
12
(mlp_net-general-information),
mlp_net-method
(mlp_net-class), 8
11
mlp_net-MSE-gradients,
13
mlp_get_no_w
mlp_net-names,
14
(mlp_net-general-information),
mlp_net-weights-access, 15
11
mlp_plot, 16
mlp_get_w
mlp_prune_mag, 16
(mlp_net-accessing-individual-weights),
7
mlp_prune_obs, 17
∗Topic classes
is.mlp_net, 3
mlp_net, 5
mlp_net-class, 8
∗Topic package
FCNN4R-package, 2
∗Topic pruning
mlp_prune_mag, 16
mlp_prune_obs, 17
∗Topic teaching
mlp_teach_bp, 20
mlp_teach_grprop, 21
mlp_teach_rprop, 23
mlp_teach_sa, 24
mlp_teach_sgd, 25
28
INDEX
mlp_rm_input_neurons
(mlp_net-manipulating-network-inputs),
12
mlp_rm_neurons, 18
mlp_rnd_weights, 19
mlp_set_activation, 19
mlp_set_name (mlp_net-names), 14
mlp_set_w
(mlp_net-accessing-individual-weights),
7
mlp_set_w_st
(mlp_net-accessing-individual-weights),
7
mlp_set_weights
(mlp_net-weights-access), 15
mlp_stack
(mlp_net-combining-two-networks),
9
mlp_teach_bp, 20
mlp_teach_grprop, 21
mlp_teach_rprop, 23
mlp_teach_sa, 24
mlp_teach_sgd, 25
print,mlp_net-method (mlp_net-display),
10
read-write-fcnndataset, 26
read.fcnndataset
(read-write-fcnndataset), 26
show,mlp_net-method (mlp_net-display),
10
summary,mlp_net-method
(mlp_net-display), 10
write.fcnndataset
(read-write-fcnndataset), 26
29