Neural Networks I

Yesterday was my first computational neuroscience class, a field that is totally foreign (and interesting) to me. We started off talking about neural networks, a field which has its roots in systems theory and the work of Nicholas Rashevsky, “a Russian émigré theoretical physicist who developed a program in ‘mathematical biophysics’ at the University of Chicago during the 1930s”. For example, he was one of the first scientists to analyze traffic mathematically (which confuses me a bit, since I’m not sure how much of an issue traffic was in the ’20’s or ’30’s).

At any rate, work building on that of Rashevsky and others plodded along for a while, until the seminal work of Warren McCulloch and Walter Pitts at MIT (A Logical Calculus of Ideas Immanent in Nervous Activity, 1943, Bulletin of Mathematical Biophysics 5:115-133) while considering the question: what kind of things can be computed mathematically? Today the subject of their paper is often referred to as a McCulloch-Pitts neuron.

A McCulloch-Pitts neuron is quite simple: essentially, it is a simplified tunable linear threshold unit (the Wikipedia article I’ve linked to does a particularly good job at explaining it), and what McCulloch and Pitts showed was that with a network of these, one can compute any logical expression, or any combination of logical predicates. The device is tunable in the sense that each input can be weighted (and a tunable ‘bias’ is often applied, too), and by varying the different weights with respect to a given threshold, the operation being computed can either be AND, OR, NOT, etc. Clearly this is incredibly simplified – no probabilities, no further considerations in the mechanism, nothing but the bare bones. The fact that a network of these can be so powerful was a big deal in the ’40’s, and people began wondering whether such ‘neural networks’ could be made to act like a brain. Clearly the jump from simple logical operations using a network of McCulloch-Pitts neuron to advanced concepts (like pattern recognition) is huge.

Twenty years later, Frank Rosenblatt – an engineer at Cornell – came up with what he called the Perceptron. Essentially, Rosenblatt was looking for a way to come up with an image recognition machine, equipped with some programmatic behavior to make it better as time went on. This resembles neural networks to a large extent – here’s a picture from Wolfram research:

As in McCulloch-Pitts neural networks, the weighted inputs (and bias, which in the image is 1) are summed, and assessed against a given threshold. Each input comes from a set of ‘feelers’ on the screen where the image is displayed e.g. photoreceptors. Things get a teensy bit complicated, and I’m trying to keep this equation-free, so I’ll jump to the punchline: Rosenblatt was able to show that for a linearly separable problem (that is, trying to distinguish between two different things that form different ‘clumps’ in an n-dimensional parameter space that can be separated by an n-1 dimensional hyperplane), the Perceptron only takes a finite number of examples for it to learn successfully. This Perceptron Convergence Theorem was again a Very Big Deal, and led to a good deal of research in the field for a little under 10 years…

Until it all stopped. Why? Because of a monograph (Perceptrons) – published by Minsky and Papert (at MIT as well) – that put an end to it all, and gave rise to the New and Improved (i.e. ‘hot’) field of Artificial Intelligence. For many years after that, the field of neural networks was completely and utterly dead.

Until next time…