Practical aspects of Deep Learning

Practical aspects of Deep Learning >> Improving Deep Neural Networks: Hyperparameter Tuning, Regularization and Optimization

98% train . 1% dev . 1% test

33% train . 33% dev . 33% test

60% train . 20% dev . 20% test

Make the Neural Network deeper

Get more test data

Get more training data

Add regularization

Increase the number of units in each hidden layer

Increase the regularization parameter lambda

Decrease the regularization parameter lambda

Get more training data

Use a bigger neural network

The process of gradually decreasing the learning rate during training.

Gradual corruption of the weights in the neural network if it is trained on noisy data.

A regularization technique (such as L2 regularization) that results in gradient descent shrinking the weights on every iteration.

A technique to avoid vanishing gradient by imposing a ceiling on the values of the weights.

Weights are pushed toward becoming smaller (closer to 0)

Doubling lambda should roughly result in doubling the weights

Weights are pushed toward becoming bigger (further from 0)

Gradient descent taking bigger steps with each iteration (proportional to lambda)

You do not apply dropout (do not randomly eliminate units) and do not keep the 1/keep_prob factor in the calculations used in training

You apply dropout (randomly eliminating units) but keep the 1/keep_prob factor in the calculations used in training.

You apply dropout (randomly eliminating units) and do not keep the 1/keep_prob factor in the calculations used in training

You do not apply dropout (do not randomly eliminate units), but keep the 1/keep_prob factor in the calculations used in training.

Increasing the regularization effect

Reducing the regularization effect

Causing the neural network to end up with a higher training set error

Causing the neural network to end up with a lower training set error

It makes the cost function faster to optimize

It makes the parameter initialization faster

It makes it easier to visualize the data

Normalization is another word for regularization–It helps to reduce variance

Practical aspects of Deep Learning >> Improving Deep Neural Networks: Hyperparameter Tuning, Regularization and Optimization