Download Training neural networks II

CS 6501: Deep Learning for
Computer Graphics
Training Neural Networks II
Connelly Barnes
Overview
• Preprocessing
• Initialization
• Vanishing/exploding gradients problem
• Batch normalization
• Dropout
• Additional neuron types:
• Softmax
Preprocessing
• Common: zero-center, can normalize variance.
Slide from Stanford CS231n
Preprocessing
• Can also decorrelate the data by using PCA, or whiten data
Slide from Stanford CS231n
Preprocessing for Images
• Center the data only
• Compute a mean image (examples of mean faces)
• Either grayscale or compute separate mean for channels (RGB)
• Subtract the mean from your dataset
Overview
• Preprocessing
• Initialization
• Vanishing/exploding gradients problem
• Batch normalization
• Dropout
• Additional neuron types:
• Softmax
Initialization
• Need to start gradient descent at an initial guess
• What happens if we initialize all weights to zero?
Slide from Stanford CS231n
Initialization
• Idea: random numbers (e.g. normal distribution)
𝑤𝑖𝑗 = 𝒩(𝜇, 𝜎)
𝜇=0, 𝜎 = const
• OK for shallow networks, but what about deep networks?
Initialization, 𝜎 = 0.01
• Simulation: multilayer perceptron, 10 fully-connected hidden layers
• Tanh() activation function
Hidden layer activation function statistics:
Are there any problems with this?
Hidden Layer 1
Hidden Layer 10
Slide from Stanford CS231n
Initialization, 𝜎 = 1
• Simulation: multilayer perceptron, 10 fully-connected hidden layers
• Tanh() activation function
Hidden layer activation function statistics:
Are there any problems with this?
Hidden Layer 1
Hidden Layer 10
Slide from Stanford CS231n
Xavier Initialization
𝜎=
1
𝑛in
𝑛in : Number of neurons feeding into given neuron
(actually, Xavier used a uniform distribution)
Hidden layer activation function statistics:
Reasonable initialization for tanh() activation function.
But what happens with ReLU?
Hidden Layer 1
Hidden Layer 10
Slide from Stanford CS231n
Xavier Initialization, ReLU
𝜎=
1
𝑛in
𝑛in : Number of neurons feeding into given neuron
Hidden layer activation function statistics:
Hidden Layer 1
Hidden Layer 10
Slide from Stanford CS231n
He et al. 2015 Initialization, ReLU
𝜎=
2
𝑛in
𝑛in : Number of neurons feeding into given neuron
Hidden layer activation function statistics:
Hidden Layer 1
Hidden Layer 10
Slide from Stanford CS231n
Other Ways to Initialize?
• Start with an existing pre-trained neural network’s weights, fine tune
its weights by re-running gradient descent
• This is really transfer learning, since it also transfers knowledge
from the previously trained network
• Previously, people used unsupervised pre-training with autoencoders
• But we have better initializations now
Overview
• Preprocessing
• Initialization
• Vanishing/exploding gradients problem
• Batch normalization
• Dropout
• Additional neuron types:
• Softmax
Vanishing/exploding gradient problem
• Recall from the backpropagation algorithm (last class slides):
𝜕𝐸
= 𝛿𝑗 𝑜𝑖
𝜕𝑤𝑖𝑗
𝜕𝐸 𝜕𝑜𝑗
𝛿𝑗 =
= 𝜑′(𝑜𝑗 )
𝜕𝑜𝑗 𝜕net𝑗
(𝑜𝑗 −𝑡𝑗 )
𝑙∈𝐿
𝛿𝑙 𝑤𝑗𝑙
• Take 𝛅 over all neurons in a layer.
• We can call this a “learning speed.”
if 𝑗 is an output neuron
if 𝑗 is an interior neuron
Vanishing/exploding gradient problem
• Vanishing gradients problem: neurons in earlier layers learn more
slowly than in latter layers.
Image from Nielson 2015
Vanishing/exploding gradient problem
• Vanishing gradients problem: neurons in earlier layers learn more
slowly than in latter layers.
• Exploding gradients problem: gradients are significantly larger in
earlier layers than latter layers.
• How to avoid?
• Use a good initialization
• Do not use sigmoid for deep networks
• Use momentum with carefully tuned schedules, e.g.:
Image from Nielson 2015
Overview
• Preprocessing
• Initialization
• Vanishing/exploding gradients problem
• Batch normalization
• Dropout
• Additional neuron types:
• Softmax
Batch normalization
• It would be great if we could just whiten the inputs to all neurons in a
layer: i.e. zero mean, variance of 1.
• Avoid vanishing gradients problem, improve learning rates!
• For each input k to the next layer:
• Slight problem: this reduces representation ability of network
• Why?
Batch normalization
• It would be great if we could just whiten the inputs to all neurons in a
layer: i.e. zero mean, variance of 1.
• Avoid vanishing gradients problem, improve learning rates!
• For each input k to the next layer:
• Slight problem: this reduces representation ability of network
• Why?
Get stuck in this part of the activation function
Batch normalization
• First whiten each input k independently, using statistics from the
mini-batch:
• Then introduce parameters to scale and shift each input:
• These parameters are learned by the optimization.
Batch normalization
Dropout: regularization
• Randomly zero outputs of p fraction of the neurons during training
• Can we learn representations that are robust to loss of neurons?
Intuition: learn and remember useful information
even if there are some errors in the computation
(biological connection?)
Slide from Stanford CS231n
Dropout
• Another interpretation: we are learning a large ensemble of models
that share weights.
Slide from Stanford CS231n
Dropout
• Another interpretation: we are learning a large ensemble of models
that share weights.
• What can we do during testing to correct for the dropout process?
• Multiply all neurons outputs by p.
• Or equivalently (inverse dropout) simply divide all neurons
outputs by p during training.
Slide from Stanford CS231n
Overview
• Preprocessing
• Initialization
• Vanishing/exploding gradients problem
• Batch normalization
• Dropout
• Additional neuron types:
• Softmax
Softmax
• Often used in final output layer to convert neuron outputs into a class
probability scores that sum to 1.
• For example, might want to convert the final network output to:
• P(dog) = 0.2
(Probabilities in range [0, 1])
• P(cat) = 0.8
• (Sum of all probabilities is 1).
Softmax
• Softmax takes a vector z and outputs a vector of the same length.