Intro  Learning  Patterns  Real Life  Genetics  Classifiers  Biology  Neural Nets  Connectionism  Life  AI

Essays on Complex Systems Part 8 - Connectionism

Connectionism and Networks

Wolfram Classes, Energy Flows, Turbulence

I've noticed in Adapt.cpp, Neural.cpp, and Classify.cpp that the system of inputs and drains of energy is critical to how the system acts, and that the possible behaviors fit Wolfram descriptions.

Numerous other situations in which Wolfram classes have proven to be useful also have in common flows of something. And it is flow that produces turbulence.

Do Wolfram classes provide a general description of the process of diffusion, how energy (or anything) dissipates into a system, how a system goes toward equilibrium?

Is turbulence produced not just by flow, but by flow against an opposing resistive force (i.e. viscosity, either physical or metaphorical), so that the absorption can't occur instantaneously? Viscosity (friction) is nonlinear, which is what makes the system nonlinear? And the net system behavior is determined by the interaction of the input and the resistance. Is turbulence the most efficient means of equalizing an imbalance? In a liquid, it should be, since the doubling whorls maximize the contact area between the two regions.

If so, there's lots of interesting implications and applications.

In an economy, the flow is of money. Class I and II would be depression and recession, class III runaway inflation or other chaotic instability, and class IV the region of extended useful transients. Tuning it would imply keeping some measure of monetary flow within the class IV range.

Some of the "Wolfram" behaviors in places where they apply seem intuitively obvious. In Life.cpp, if the probability of a cell being on is very low, it makes sense that the number of cells on will quickly dwindle to nothing. And if probability is about even, then you'll get perpetually changing patterns without either death or runaway growth, etc. Not quite that simple, but that's the idea.

Misc.

2 to 8 inputs per node produce usable outputs. Too few is too stable, dead. Too many, and the system never settles down, chaotic.

Under what conditions is this true, however? It's obviously not the case in a brain. Neurons may have 200,000 inputs. If there are ten billion neurons in the brain, each neuron receives input from .002% of the other neurons.

Would an input-limiting rule apply to designing complex systems such as governments or companies? I.e. too-dispersed decision making produces erratic output? Or too many inputs to a single decision maker makes decision making impossible?

Remember my plan for a highly interconnected system of nodes outlined in c:\ai\expert\lists.doc. It involved data structure nodes that had multiple branch points. And see the "Requirements for Adaptation and Learning". If each of the nodes contained a member function that executed a tiny piece of code, then the pointers that pointed to other nodes could be used as railroad-type switching relays, directing in which order the tiny pieces of code would be executed. The pointers could be modified, or their associated probabilities (used as strengths) adjusted, to modify the program flow. The structure would thus be similar to that of a neural net, while the function would actually be much different, since instead of paying attention only to a few "output" nodes after the percolating process was finished, bits of code would be getting executed WHILE the percolation was happening. This also simulates specialized subassemblies in the brain. That is, output doesn't occur at some predetermined endpoint of the circuit. Activation of any node might result in some output. Then the signal might continue on down the line.

That should be fairly easy to add to an existing ANN program: each node contains a pointer to a function. When it comes time to calculate that node's inputs and outputs, execute that function. 

Comparisons between classifiers and neural nets:

The two seem awfully similar, except that the "rules" in a classifier continually scan a communal bulletin board (i.e. are in touch with a centralized status report on the state of the whole), while a neural net node effectively "scans" only the state of the nodes that input to it (i.e. is only in touch with the state of its local environment).

This makes the classifier system look more "wholistic" and possibly better, but it also seems like a large-scale classifier system would be extremely time-inefficient.

A strength of the classifier system is that one posted message (event) can trigger multiple subsequent events (multiple threads). It is also possible that a rule that is invoked by a particular message might accumulate so much strength that it crowds out all the other rules that responded to that message. When that happens, it means that rule A, when posted, will ALWAYS trigger rule B, transforming them (conceptually) from two distinct rules into one -- they have combined into a larger subunit. A long chain of such strong connections could develop. When you combine the chaining effect with the multiple thread effect, you have an extremely powerful system: A message such as "your pot is boiling over" can trigger 2 strong messages, "take it off the stove" and "turn off the heat". "Take it off the stove" can trigger its own subchain, "and put it down somewhere else", while "turn off the heat" simultaneously triggers, "and don't touch the burner". And yet, better still, there is always the possibility that the chain will get broken up or modified with insertions or deletions if a better combination appears in the system.

(System could be implemented such that each message for posting corresponds to one of the system's elemental instructions in its set? If each rule contains a pointer to the instruction, each instruction only needs to occur once in the data scheme. Furthermore, a system set up to use these pointers instead of specific messages could be a "general purpose" system, which could be tailored to a particular application by just changing the data pointed to.)

I don't think a standard neural net setup has anything corresponding to this, since each node does not correspond to an action. My plan a few paragraphs above might be able to change this, but the model is still hampered by the fact that each net node is only aware of its immediate neighborhood, while the classifiers constantly monitor the global system's state.

On the other hand, you should be able to "tap" into a network at any location(s) you want, and the states of those locations could be used to trigger any events you want. (There's no requirement to consider any particular nodes as "output" nodes. Just insert probes into the net, and consider those the outputs. Furthermore, you can indeed backpropagate from any point.)

Filtering is the key similarity of nets and classifiers, and the key requirement for the ability to make analogies. (discussion of filtering). Filtering is the exclusion of raw data that is (currently) irrelevant. It's the essence of "abstraction". The system learns, in any given situation, what information is trivial or irrelevant. That's where the bidding system comes in in a classifier system. A given rule's strength (bid) is a measure of its relevance. Interestingly, it's the RULE that is declared relevant or not, however, not a particular piece of information. That is, a bulletin board message isn't declared relevant or not, but the RULE that says "if THAT message exists, then THIS would be a good thing to do" is declared relevant or not.

A rule in a classifier might correspond to an entire PATH in a neural net. That is, the rule says, "If this is the situation, then do that..." and rules compete based on their strengths. A network PATH basically says the same thing: "If this is the input, then THAT will be the output."

The difference is that the net paths compete with each other by INTERFERING with each other. A path will be followed unless some other path introduces an input that interferes with the flow, blocking or diverting it. This could be analogous to the bidding system, except that an additional input doesn't actually have to bid "higher" -- it just has to be strong enough to block or divert the flow at that point. A classifier has to have a multitude of rules to cover any conceivable situation, whereas a network must have a multitude of connections so that interference can occur at any conceivable point along the path.

Consider a network as a system of conduits carrying water with the strength-connections being valves. See where that leads. Also try thinking of it as an "ocean" of waves creating interference patterns with each other.

It would be very nice to conclude that networks are hard to work with and impossible to conceptualize because they're just "black boxes", discount them, and just move on to work with classifiers.

Unfortunately, all biological brains use networks, so you can't help but wonder what they've got going for them. (Physical compactness seems a primary advantage.) On the other hand, it could be that a network is the only thing you can implement with a biological system. The fact that it occurs in nature doesn't by itself necessarily mean it's the best that can be done.