Recording the derivation of BackPropagation & Levenberg–Marquardt & Bayesian Regularization Algorithm, to have a deeper understanding of the Neural Networks.
In fact, there are two types of differential methods in neural networks computing: forward-mode differentiation and backward-mode differentiation(Automatic Differentiation).
Consider the computational graph below, we need to compute the gradient of the output with respect to the input, namely , where can be multi-dimensional vectors.
- For forward-mode differentiation:
Each step of the iteration starts from the input cacluate upward, for example we consider the step of , thus we need to calculate derivative by chain rule: