The SciPy Neural Network Hack

Added: 2026-05-16

[00:00:00]We'll use the BFGS implementation built into the SciPy optimize package, specifically within the minimize function.

[00:00:08]To use BFGS, [music] the minimize function requires us to pass in an objective function that accepts a vector of parameters, >> [music] >> input and output data, and returns both the cost and gradients.

[00:00:21]Our neural network implementation doesn't quite follow these semantics, [music] so we'll use a wrapper function to give it this behavior.

[00:00:28]We'll also pass in initial parameters, [music] set the Jacobian parameter to true since we're computing the gradient within our neural network class, set the method to BFGS, pass in our input and output data, and some [music] options.

[00:00:41]Finally, we'll implement a callback function that allows us to track the cost function value as we train [music] the network.

[00:00:48]Once the network is trained, we'll replace the original random parameters with [music] the trained parameters.

[00:00:54]If we plot the cost against the number of iteration through training, we see a nice monotonically decreasing function.

[00:01:00]Further, we see that the number of function evaluations required to find a solution is less than 100, and far less than the 10 to the 27th function evaluations >> [music] >> that would have been required to find a solution by brute force as shown in part three.

[00:01:15]Finally, we can evaluate our gradient at our solution [music] and see very small values. This makes sense as our minimum should be quite flat.