Backpropagation artificial neural network program in Borland C 4.0 for MSDOS, BGI graphics

dneural.c

/*	dneural.c	7-22-95   experiment with neural nets
	Copyright (C)1995 Steven Whitney.
	Published under GNU GPL (General Public License) Version 3, with ABSOLUTELY NO WARRANTY.
	Initially published by http://25yearsofprogramming.com.

related notes have been moved into complex.doc

11-23-95 This is the last version of neural.c before the conversion to C++.
SAVE for reference until (if) C++ version makes it completely obsolete.
It uses the Borland BGI DOS graphics functions.

1-9-04: I retrieved (but didn't delete) this from floppy backup.  I think it's
safe to say this is completely obsolete, but with its old linked list method
and page switching, it is sort of interesting to keep.  The notes may also
be interesting; this was a very difficult program to get working properly at all,
since I hadn't a clue how a real neural net program was supposed to work.

-------------------

EXPERIMENTAL - there are several untested sections in here.
OK	Currently uses (-1,1) not (0,1) - but .TRS files can use 0,1:
	all zeroes are translated to -1 on input.

TO DO:

(Think more about the implications before doing either of these:)
(And back up this version into neural.1 before making large scale changes.)

-turn struct node into a C++ class; move createnode() into a constructor;
 probably move processnode() and adjuststrength() into member functions.

-create a container of the nodes instead of the current linked list method.

----------------------------

-try to think up a way to use bucket brigade instead of backprop;
it would better implement the idea of reinforcement rather than
"error correction", which seems artificial.  Could a node simply transfer
some of its weight back to the node that led to it each time that path
is taken?  Then, if weight is depleted, it could somehow affect whether
the path is taken at all?  Or goes negative?

-a node doesn't deplete anything when it "fires into" another node. It would
be a more "economic" (more bucket-brigade-like) model if it did.  It would
also be somewhat analogous to a "refractory period."  (this is the same
idea as above - but added months later!).

	OUTPUT = (SOME FORMULA) * INPUT
	should correspond to a neuron's firing rate
	(and whether it's to exert a stimulating or inhibiting influence).

	currently it doesn't simulate firing rate at all.
	should prohibit output from being 0 (weight could still be 0).

	if this is negative, it means the neuron's basic output is inhibiting -
	to ALL its outputs (unless weight turns it positive, which weight can only
	do if IT can be + or -).

	WEIGHT = (set by backprop)
	and corresponds to the effect the firing neuron has on its recipient,
	its "access".  low weight means the firing neuron isn't very important
	to the recipient.
	It would make the most sense if weight could only be positive, a modifier to output.
		But if it's only positive, sign change MUST be in OUTPUT, and if that's
		the case, then a neuron can only output either ALL positive or
		ALL negative numbers.

	(OUTPUT * WEIGHT) must be capable of both + and -.
------------------------------------
make output continuously variable (see book);
This will translate an input from -MAXINT to +MAXINT into an output value
also from -MAXINT to +MAXINT, using a sine translation that approximates
the graph in CUC page 349, for "continuously variable" output:
->output = 32767 * SIN(->input / 65534 * 3.1415926)
	(cast all to doubles to prevent overflow)
	What it does in effect is cause mid-range input values to cause greater
	incremental change in the output value.  Changes to input at either
	extreme have little effect.  But when ->input is 0, output is still 0.
	(see neural.xls).
	The changes would be made in processnode().

-	might be better to have output between -10000 and +10000, then USE
	it as a 4-digit double from -1 to +1.
	However, this would make ->output the maximum possible output.
	Leaving ->strength as it is would allow scaling rather than just + or - computations.

SUCCESSFUL:
-Made input and output -1,1 instead of 0,1.  The only
 programming changes are in processnode(), and all .TRS files must be in
 1,-1 format.  Because of <=0 test in processnode(),
 it also generalizes so that 0,1 format can be used after training.
 And it trains much more quickly than it did with (0,1).
 Remember, if it's trained on (-1,1), you either have to use those as inputs,
 or translate the inputs to that before any "thinking".
-------------------- 
-6th test: duplicate an input CHARacter at the output
-7th test:  try simple math on 1 input (x 10, etc.)
			see what it does with inputs it's not trained on.
-------------------- 
LATER:
-see complex.doc .
-instead of random new connections, connect 2 nodes that are "working hard", or
 "highly active" at the time, according to some measure: the idea is that the
 measure should be an indicator that these two nodes are somewhere in the
 best current path that might lead to a solution, but they haven't been able
 to get it.
-see note in classify.cpp about "progressive rewards".  It may already be
 implemented here in that the backpropped values are scaled to the
 magnitude of the error (are they?), but it might be possible to do it better.

NOTES:
-if a node is OFF (transmits NO output) if input <= 0,
a zero or negative INPUT (even at a sensory node) cannot
produce any output.  A zero input cannot produce a positive value.
This is why, when (0=F,1=T) is used, an input can't be inverted:
if(input <= 0)		(output)0 x -100(weight) = 0 = off
if(input <= 0)		(output)0 x 100(weight) = 0 = off
if(input > 0)       (output)1 x -100(weight) = -100 = off.
if(input > 0)       (output)1 x 100(weight) = 100 = on.
Solved by using (-1,1).
-seems as though no matter what input->output translation is used, there will
 always be one set of inputs that will produce a dead net.
-see complex.doc.

===
*/
#include <stdio.h>
#include <stddef.h>
#include <dos.h>
#include <stdlib.h>
#include <conio.h>
#include <string.h>
#include <time.h>
#include <ctype.h>
#include <graphics.h>
#include <stdarg.h>
#include "my.h"
// since mylib.cpp is now C++ only, it will be easier just to copy and paste any fns this needs
#include "mylib.cpp"

// neuron types
#define	PROCESSOR	0	
#define	SENSORY		1
#define	OUTPUT		2
						// video pages to use (numbering starts at 0)
#define	TEXT		0
#define	GRAPHICS	1
						// setup (and maximum) values for global variables.
                        // if training set file values exceed any of them
                        // (except pcount), program aborts with error.
#define TSCOUNT		4
#define SCOUNT		2
#define PCOUNT		10
#define OCOUNT		1

// ----------------------------------------------------------------------
// global variables
// ----------------------------------------------------------------------
struct INLIST
{
	struct NODE *node;		// pointer to a donor node
	struct INLIST *next;	// next in list: 0 = end of list
};
struct OUTLIST				// list of recipient nodes to receive output
{
	struct NODE *node;		// address of recipient node
	int strength;			// output TO the recipient node = output * strength
	struct OUTLIST *next;	// next recipient node in list: 0 = end of list.
};
struct NODE						// a single neuron, 19 bytes
{
	char type;					// type of node; used as a short int
	int input;					// sum of all inputs
	int output;					// output sent when fired
	int error;					// backpropagated error at this node
	int x;						// screen coordinates for display
	int y;
	struct OUTLIST *outlist;	// head of list of recipient nodes
	struct INLIST *inlist;		// head of list of donor nodes
	struct NODE *previous;		// pointer to previous node in list
	struct NODE *next;			// pointer to next node in sequential list
} *nodehead = 0, *nodetail = 0;	// head and tail of master list
int nodecount = 0;				// count of total nodes (all types)

int connectcount = 0;			// count of total connections
int densitylimit = 8;			// maximum allowed average connections/node
int graphicmode;				// mode: VGAHI, etc.
int maxx, maxy;					// maximum screen coordinates
char memerr[] = "Out of memory.";
int oldpage = TEXT;

// ----------------------------------------------------------------------
// more global variables -- specific to the net's task
// iset[number of training sets][# of sensory nodes to fill each set].
// oset[number of training sets][# of output nodes to test].
// Training sets now loaded from file, and all assignments here except pcount
// and iset[], oset[] dimensions are overridden in loadtrainingsets().
// ----------------------------------------------------------------------
int scount = SCOUNT;		// number of sensory nodes to create
int pcount = PCOUNT;		// number of processor nodes to create
int ocount = OCOUNT;		// number of output nodes to create
int tscount = TSCOUNT;		// how many training sets there are
int tsindex;				// which training set is being run
int iset[TSCOUNT][SCOUNT];	// inputs for training sets
int oset[TSCOUNT][OCOUNT];	// correct answers for training sets

// ----------------------------------------------------------------------
// loadtrainingsets()	read all training set info from disk
// returns TRUE if successful, aborts program if not
// file format for sample file "AND.TRS":
// 4			number of training sets
// 2			s-node count (# needed)
// 1			o-node count (# needed)
// Inputs:		4 sets x 2 inputs each	("Inputs:" is in file, but unused,)
// 0 0									(for ease in editing file)
// 1 0
// 0 1
// 1 1
// Outputs:		4 sets x 1 output each
// 0
// 0
// 0
// 1
// Lines after these can be used to document the training set's task.
// ----------------------------------------------------------------------
int loadtrainingsets(const char *filename)
{
int i, j, k;
char buf[256], *temp;
FILE *infile;

if(!(infile = fopen(filename,"r")))
	aborts("Training set file not found.");
if(fscanf(infile,"%d\n%d\n%d\n",&tscount,&scount,&ocount) != 3)
	aborts("Training set file corrupt.");
									// maximum array sizes exceeded?
if((tscount > TSCOUNT) || (scount > SCOUNT) || (ocount > OCOUNT))
{
	printf("Program needs larger array dimensions.  Recompile for:\n");
	printf("TSCOUNT = %d, SCOUNT = %d, OCOUNT = %d\n",tscount,scount,ocount);
	exit(1);
}
for(k = 0 ; k < 2 ; k++)
{
	fgets(buf,256,infile);					// discard "Inputs:" or "Outputs:"
	for(i = 0 ; i < tscount ; i++)		    // for each training set,
	{
		if(!fgets(buf,256,infile))			// read line
			aborts("Training set file corrupt.");
		buf[strlen(buf)-1] = '\0';			// strip trailing c/r
		j = 0;
		temp = strtok(buf," ");     		// tokenize line, and
		if(k == 0)                      	// 1st pass reads inputs
		{
			iset[i][j] = atoi(temp);		// assign first value
			if(iset[i][j] == 0)
				iset[i][j] = -1;
			j++;
			while(temp = strtok(NULL," "))	// assign remaining iset[][] values
			{
				iset[i][j] = atoi(temp);
				if(iset[i][j] == 0)         // translate 0 to -1
					iset[i][j] = -1;		// makes it easier to maintain
				j++;						// the files, but -1 works better.
			}
		}
		else								// 2nd pass reads correct outputs
		{
			oset[i][j] = atoi(temp);		// assign first value
			if(oset[i][j] == 0)
				oset[i][j] = -1;
			j++;
			while(temp = strtok(NULL," "))	// assign remaining iset[][] values
			{
				oset[i][j] = atoi(temp);
				if(oset[i][j] == 0)
					oset[i][j] = -1;
				j++;
			}
		}
	}
}
fclose(infile);
return(TRUE);
}
// ----------------------------------------------------------------------
// changepage()	 set both output and display to graphics or text page
// and change display mode accordingly
// don't use - mode is now VGAHI, only 1 page.
// ----------------------------------------------------------------------
void changepage(int page)
{
return;
if(page != oldpage)
{
	setvisualpage(page);
	setactivepage(page);
	if(page == TEXT)
		restorecrtmode();
	else
		setgraphmode(graphicmode);	// clears screen and resets everything
	oldpage = page;
}
return;
}				// end changepage()
// ----------------------------------------------------------------------
// createnode()  initialize a new neuron and add to master list.
// usually appends to end of master list, but if the new node is a
// p-node, and o-nodes already exist, that means it's an insertion,
// and the new node is randomly inserted into the p-node section.
// returns TRUE if successful, FALSE if not (out of memory)
// ----------------------------------------------------------------------
int createnode(const char nodetype)	// type of node to create
{
int i, offset;
struct NODE *newnode, *listptr;
									// create and initialize the node
if((newnode = (struct NODE *)malloc(sizeof(struct NODE))) == NULL)	
{
	puts(memerr);
	return(FALSE);
}
newnode->type = nodetype;
newnode->input = 0;
newnode->output = 0;
newnode->error = 0;
newnode->outlist = 0;
newnode->inlist = 0;
newnode->previous = 0;			// don't delete: first node's previous must be 0
newnode->next = 0;
switch(nodetype)				// position across screen by node type
{
	case SENSORY: 	newnode->x = random(maxx/5); break;
	case PROCESSOR: newnode->x = random(maxx*3/5)+maxx/5; break;
	case OUTPUT: 	newnode->x = random(maxx/5)+maxx*4/5; break;
}
newnode->y = random(maxy);
								// insert node into master list.
// (if a p-node, nodetail certainly has a value)
// (if last node is an o-node, we know this is a post-createnetwork() addition
// and therefore has to be an insertion)
if((nodetype == PROCESSOR) && (nodetail->type == OUTPUT))
{
 	listptr = nodehead;						// start at first node
    while(listptr->next->type == SENSORY)	// find last s-node
		listptr = listptr->next;
	offset = random(++pcount);				// select insert point (& increment pcount)
	for(i = 0 ; i < offset ; i++)			// move to insert point
		listptr = listptr->next;
	newnode->next = listptr->next;			// point new node at next node
	newnode->previous = listptr;			// point new node back at previous
	newnode->next->previous = newnode;		// point next node at the new p-node
	listptr->next = newnode;				// point previous node at new p-node
}
else										// append the node to the master list
{											// this section always used during createnetwork()
	if(!nodehead)							// first node?
		nodehead = nodetail = newnode;
	else
	{
		listptr = nodehead;
		while(listptr->next)
			listptr = listptr->next;
		listptr->next = newnode;			// append to list
		newnode->previous = listptr;		// backwards-connect
		nodetail = newnode;					// new end-of-master-list
	}
}
nodecount++;
return(TRUE);
}				// end createnode()
// ----------------------------------------------------------------------
// selectatrandom()  select a node from the pool at random.
// returns the address of the chosen node, or 0 if it overruns (shouldn't).
// ----------------------------------------------------------------------
struct NODE *selectatrandom(void)
{
int i, n;
struct NODE *nodeptr;

n = random(nodecount);			// calculate a random offset from nodehead
nodeptr = nodehead;				// start at beginning
for(i = 0 ; i < n ; i++)		// find node at (offset)
	nodeptr = nodeptr->next;
return(nodeptr);	
}					// end selectatrandom()
// ----------------------------------------------------------------------
// connected()  tests whether two nodes are already connected
// ----------------------------------------------------------------------
int connected(struct NODE *fromnode, struct NODE *tonode)
{
struct OUTLIST *outptr;

outptr = fromnode->outlist;			// set to start of fromnode's outlist
while(outptr)
{
	if(outptr->node == tonode)		// already in list
		return(TRUE);
	outptr = outptr->next;
}
return(FALSE);	
}
// ----------------------------------------------------------------------
// connect()   connect two (specified) already existing neurons
// returns TRUE if successful, FALSE if not (out of memory)
// doesn't check whether connection between the two already exists,
// so check before calling.
// ----------------------------------------------------------------------
int connect(struct NODE *fromnode, struct NODE *tonode)
{
struct OUTLIST *outptr, *outtemp;
struct INLIST *inptr, *intemp;
								// create an outlist node
if((outtemp = (struct OUTLIST *)malloc(sizeof(struct OUTLIST))) == NULL)
{
	puts(memerr);
	return(FALSE);
}
outtemp->node = tonode;			// initialize its data
								// start w/random weights, CUC p.343
outtemp->strength = random(20001)-10000;	// +10000 to -10000
outtemp->next = 0;
								// append it to donor node's output list
outptr = fromnode->outlist;		// set to head of outlist list
if(!outptr)						// if no head,
	fromnode->outlist = outtemp;// this becomes it.
else
{
	while(outptr->next)			// find end of list
		outptr = outptr->next;
	outptr->next = outtemp;		// append the new connection to the list
}
								// create an inlist node
if((intemp = (struct INLIST *)malloc(sizeof(struct INLIST))) == NULL)
{
	puts(memerr);
	return(FALSE);
}
intemp->node = fromnode;		// points at the donor (fromnode),
intemp->next = 0;				// (not to the connection itself)

inptr = tonode->inlist;			// set to head of inlist list
if(!inptr)
	tonode->inlist = intemp;
else
{
	while(inptr->next)
		inptr = inptr->next;	// find end of list
	inptr->next = intemp;		// append 		
}
connectcount++;
return(TRUE);
}				// end connect()
// ----------------------------------------------------------------------
// randomconnect()	establish a new connection from a (specified) node
// to another node that it doesn't already output to.
// only allows forward connections.
// automatically ensures only legal connections between node types.
// returns TRUE if successful, FALSE if not
// ----------------------------------------------------------------------
int randomconnect(struct NODE *fromnode)
{
int found, passcount, before;
struct NODE *tonode, *nodeptr;

passcount = 0;
do
{
	if(++passcount == 10000)		// if this node too hard to find a
	{								// connection for, get a different fromnode
		putchar(7);
//		createnode(PROCESSOR);		// starts with no connections
		return(FALSE);				// another possibility
//		passcount = 0;
	}
	found = FALSE;
	tonode = selectatrandom();		// choose any node
	if(tonode == fromnode)			// same as fromnode?
		continue;
	before = FALSE;         		// does it appear ahead of fromnode in the list?
	nodeptr = nodehead;
    while(nodeptr)
    {
		if(nodeptr == tonode)		// whichever it hits first will
        {							// cause it to exit with status
        	before = TRUE;
			break;
        }
		if(nodeptr == fromnode)		
        	break;
		nodeptr = nodeptr->next;
	}
	if(before)						// yes, it's before fromnode, try again...
    	continue;
	switch(fromnode->type)			// a type fromnode can't connect to?
	{
		case SENSORY:			
		case PROCESSOR:
			if(tonode->type == SENSORY)
				continue;
			break;
		case OUTPUT:				// output node can't connect to anything
			continue;
	}
									// make sure it isn't already in fromnode's outlist
	if(connected(fromnode,tonode))
		continue;
	found = TRUE;					// finally, it passed all tests
}
while(!found);
return(connect(fromnode,tonode));	// could still fail for lack of memory!
}				// end randomconnect()
// ----------------------------------------------------------------------
// randomnewconnection()
// establish new connection between 2 randomly selected nodes.
// alternatively, could run through entire network, and select fromnode
// based on some sort of eligibility or desirability test(s).
// ----------------------------------------------------------------------
int randomnewconnection(void)
{
struct NODE *fromnode;

// test for net becoming too densely connected.  could be done elsewhere.
if((connectcount / nodecount) > densitylimit)
	createnode(PROCESSOR);
do
{
	fromnode = selectatrandom();
}
while((fromnode->type == OUTPUT)	// output nodes can't be donors.
|| (!randomconnect(fromnode)));		// couldn't connect, try another.

return(TRUE);
}					// end randomnewconnection()
// ----------------------------------------------------------------------
// processnode()  determine a node's state and update all its outputs --
// returns TRUE if an output was sent, FALSE if none were.
// so far, the return isn't used.  something else may be more useful.
// ----------------------------------------------------------------------
int processnode(struct NODE *node)
{
int sentoutput = FALSE;		// true if any recipient node's input was changed
int change;
long test;
struct OUTLIST *outptr;

if(node->type == OUTPUT)	// don't delete this block (tho same as below),
{							// o-nodes may be treated differently
// 	if(node->input <= 0)	// translate numeric to on/off
//     	node->output = -1;
//     else
//     	node->output = 1;
	node->output = node->input;	// just copy input to output for testing
	return(FALSE);				// an o-node has no outputs to process
}

// entirely removing test below prohibits learning - no filtering
// but book says binary threshold, as below, is hard to train
						// this test could be much more complicated
						// and could alter node->output in a better way.
if(node->input <= 0)	// below firing threshold
	node->output = -1;
else
	node->output = 1;
								// now send signals out through all dendrites
if(node->output)
{
	outptr = node->outlist;		 // set to head of recipient list
	while(outptr)				
	{
		change = node->output * outptr->strength;
		if(change)
		{						// prevent overflow
			test = (long)(outptr->node->input) + (long)change;
			if((test <= 32767L) && (test >= -32767L))
               	outptr->node->input = (int)test;
			else				// now only need to know if < or > 0
            	outptr->node->input = (test > 0L)? 32767 : -32767;
			sentoutput = TRUE;
		}
		outptr = outptr->next;
	}
}
return(sentoutput); 	
}						// end processnode()
// ----------------------------------------------------------------------
// testforsolution()  determine whether output nodes contain correct answer
// also sets the error value of each o-node as start point for backprop()
// error = desired output - actual output
// ----------------------------------------------------------------------
int testforsolution(void)
{
int solutionfound = TRUE;		// any node with an error turns it FALSE
int sd = 0;
struct NODE *nodeptr = nodehead;

while(nodeptr)
{
	if(nodeptr->type == OUTPUT)
	{					// nodeptr->output was calculated in processnode()
		nodeptr->error = oset[tsindex][sd++] - nodeptr->output;
		if(nodeptr->error)
			solutionfound = FALSE;
	}
	nodeptr = nodeptr->next;
}
return(solutionfound);
}						// end testforsolution()
// ----------------------------------------------------------------------
// think() make ONE computational pass through the network,
// Traverses nodes in sequential order, the order in which they were born.
// returns TRUE if any node changed another's input value, FALSE if not
// ----------------------------------------------------------------------
int think(void)
{
int sentoutput = FALSE;			// a node changed another's input value?
struct NODE *nodeptr;

nodeptr = nodehead;			// reset all input values to 0 before pass
while(nodeptr)
{
	if(nodeptr->type != SENSORY)   // don't reset s-nodes!
		nodeptr->input = 0;
	nodeptr = nodeptr->next;
}
nodeptr = nodehead;
while(nodeptr)
{
	if(processnode(nodeptr))		// determine off/on & send outputs
		sentoutput = TRUE;
	nodeptr = nodeptr->next;
}						// end while(nodeptr)
return(sentoutput);
}							// end think()
// ----------------------------------------------------------------------
// adjuststrength()   for one (specified) node, calculate the errors
// at the nodes inputting to it, and adjust the connection strengths
// accordingly.
// returns TRUE if any connection strength was changed, FALSE if not
// procedure is: look up the donor node in recipient->inlist,
// then search the donor's->outlist to find pointer to the connection
// for changing the strength.
// ----------------------------------------------------------------------
int adjuststrength(struct NODE *nodeptr)	// recipient node
{
int change;
int strengthchanged = FALSE;
long test;
struct INLIST *inptr;
struct OUTLIST *outptr;

inptr = nodeptr->inlist;	// start of recipient's list of donor nodes
while(inptr)
{
	outptr = inptr->node->outlist;	// start a donor's recipient node list
	while(outptr)
	{
		if(outptr->node == nodeptr)		// found the actual connection
		{
										// calculate error at donor node
			if(!(inptr->node->error))	// (if it hasn't already been set)
			{
				test = (long)(nodeptr->error) * (long)(outptr->strength);
				if((test <= 32767L) && (test >= -32767L))
					inptr->node->error = (int)test;
				else				// now only need to know if < or > 0
					inptr->node->error = (test > 0L)? 32767 : -32767;
			}
										// change connection strength
			change = nodeptr->error * inptr->node->output;
			if(change)
			{			// not same as above:  \/ note +, not *
				test = (long)(outptr->strength) + (long)change;
				if((test <= 32767L) && (test >= -32767L))
    	           	outptr->strength = (int)test;
				else				// now only need to know if < or > 0
	            	outptr->strength = (test > 0L)? 32767 : -32767;
				strengthchanged = TRUE;
			}
			break;					// found it, so quit looking
		}
		outptr = outptr->next;
	}						// end while(outptr)
	inptr = inptr->next;
}							// end while(inptr)
return(strengthchanged);	
}					// end adjuststrength()
// ----------------------------------------------------------------------
// backprop()   make ONE pass backwards through the net,
// adjusting strength values to correct for output errors.
// returns TRUE if any connection strength was changed, FALSE if not
// ----------------------------------------------------------------------
int backprop(void)
{
int strengthchanged = FALSE;	
struct NODE *nodeptr;

nodeptr = nodehead;			// reset all error values to 0 before pass
while(nodeptr)
{
	if(nodeptr->type != OUTPUT)	// keep o-node errors from testforsolution()
		nodeptr->error = 0;
 	nodeptr = nodeptr->next;
}
nodeptr = nodetail;				// start at end and traverse backwards
while(nodeptr)
{
	if(adjuststrength(nodeptr))		// propagate node's error to its inputs
		strengthchanged = TRUE;
	nodeptr = nodeptr->previous;	
}								// end while(nodeptr)
return(strengthchanged);	
}							// end backprop()
// ----------------------------------------------------------------------
// createnetwork()	create the entire neural network:
// create all the neurons and establish their connections to each other
// returns TRUE if successful, FALSE if not
// ----------------------------------------------------------------------
int createnetwork(void)
{
int i, j, n;
struct NODE *nodeptr, *donornode;
struct INLIST *inptr;
	

						// create the neurons
						// order created determines calculation order, too 
						// and several routines *depend* on 
                        // them having been created in this order:

for(i = 0 ; i < scount ; i++)	if(!createnode(SENSORY))	return(FALSE);
for(i = 0 ; i < pcount ; i++)	if(!createnode(PROCESSOR))	return(FALSE);
for(i = 0 ; i < ocount ; i++)	if(!createnode(OUTPUT))		return(FALSE);

// connect each node to designated number of other nodes.  the way this 
// is done, s- and p-nodes will have at least n outputs, but might have
// more if any were added to correct an under-connected o-node.
// s- and p-nodes can have any number of input nodes.
// o-nodes will have at least n inputs.

j = 0;
nodeptr = nodehead;
while(nodeptr)
{
//	printf("Connecting node %d\n",j++);
	switch(nodeptr->type)	// # and type of connections depends on node type
	{	
    	case SENSORY: 			
	        n = 2;              // number of output connections to create
	        break;
		case PROCESSOR: 
	        n = 1; 				// output connections
	        break;
		case OUTPUT:   			// backwards: o-node must be recipient,
                                // so determine number of existing 
								// input connections (donors).
   			n = 2;				// (minimum acceptable donor count)
            i = 0;				
			inptr = nodeptr->inlist;	// set to start of inlist
            while(inptr)
            {
             	i++;					// count donors
                inptr = inptr->next;
            }
            if(i < n)
            {
				n = n - i;				// number still needed for minimum
				for(i = 0 ; i < n ; i++)
			    {
					do				// randomly select a s- or p-node donor
					{				// that it's not already connected to
						donornode = selectatrandom();
					}
					while((donornode->type == OUTPUT) || 
						(connected(donornode,nodeptr)));
					if(!connect(donornode,nodeptr))
						return(FALSE);
			    }	// end for(i = 0 to n)
			}		// end if(more connections needed)
            break;		
		default: puts("Illegal node type encountered"); return(FALSE);
	}				// end switch
	if(nodeptr->type != OUTPUT)			// o-node already taken care of 
		for(i = 0 ; i < n ; i++)
			if(!randomconnect(nodeptr))	
				return(FALSE);
	nodeptr = nodeptr->next;
}							// end while(nodeptr) - connect all nodes
return(TRUE);
}				// end createnetwork()

// ----------------------------------------------------------------------
// drawnodes()	display the entire network graphically
// ----------------------------------------------------------------------
void drawnodes(int showconnect)	// whether to show connections
{								// FALSE if just updating node fills
int color;
struct NODE *nodeptr;
struct OUTLIST *outptr;

nodeptr = nodehead;
while(nodeptr)
{
	switch(nodeptr->type)
	{
		case SENSORY: color = RED; break;
		case PROCESSOR: color = GREEN; break;
		case OUTPUT: color = WHITE; break;
	}
	setcolor(color);
	circle(nodeptr->x,nodeptr->y,5);
	if(nodeptr->input > 0)					// above threshold, so fill it in
    {
		setfillstyle(SOLID_FILL,color);     	
		floodfill(nodeptr->x,nodeptr->y,color); 
    }
	nodeptr = nodeptr->next;
}
if(!showconnect)		
	return;
nodeptr = nodehead;				// draw the connections
while(nodeptr)
{
	outptr = nodeptr->outlist;
	while(outptr)
	{
		if(nodeptr->input > 0)
			setcolor(LIGHTGRAY);
		else         
        	setcolor(BLUE);
		line(nodeptr->x,nodeptr->y,outptr->node->x,outptr->node->y);
		outptr = outptr->next;
	}
	nodeptr = nodeptr->next;
}
return;	
}			// end drawnodes()

// ----------------------------------------------------------------------
// initsensory()  set up initial state of sensory nodes
// ----------------------------------------------------------------------
void initsensory(int print)
{
struct NODE *nodeptr;
int sd = 0;					// index, which s-node is being filled

if(print)
{
	gotoxy(1,1);
	printf("Input set %d:\n",tsindex);
}
nodeptr = nodehead;
while(nodeptr)
{
	if(nodeptr->type == SENSORY)
	{
		nodeptr->input = iset[tsindex][sd++];
		if(print)
			printf("%d ",nodeptr->input);
	}
	nodeptr = nodeptr->next;
}	
if(print)
	printf("\n");
return;
}				// end initsensory()

// ----------------------------------------------------------------------
//  displayoutput() display the results from the output cells.
// ----------------------------------------------------------------------
void displayoutput(void)
{
struct NODE *nodeptr;

printf("Result: ");
nodeptr = nodehead;
while(nodeptr)
{
	if(nodeptr->type == OUTPUT)
	{
		printf("%d ",nodeptr->output);
	}
	nodeptr = nodeptr->next;
}	
printf("\n");
return;	
}				// end displayoutput()

// ----------------------------------------------------------------------
// nodereport()	report various node data for every node in network
// ----------------------------------------------------------------------
void nodereport(void)
{
int i, pause = FALSE;
char ch;
struct NODE *nodeptr;
struct OUTLIST *outptr;

i = 0;
nodeptr = nodehead;
while(nodeptr)
{
	switch(nodeptr->type)
    {
     	case SENSORY:	ch = 'S'; break;
        case PROCESSOR:	ch = 'P'; break;
        case OUTPUT:	ch = 'O'; break;
    }
	printf("%d-%c: In = %d.  Weights: ",i++,ch,nodeptr->input);
    outptr = nodeptr->outlist;				// report on connection strengths
    while(outptr)
    {
		printf("%d ",outptr->strength);
     	outptr = outptr->next;
    }
	printf("\n");
	if(pause)					// could pause listing every 10 nodes
    	if(i && (i % 10))		// (note pause is always FALSE, above)
	    	getch();
	nodeptr = nodeptr->next;
}	
printf("There are %d connections.\n",connectcount);
return;	
}					// end nodereport()
    

// ----------------------------------------------------------------------
// work()	get sensory input from user; calculate and display output
// permits random-sequence testing as to whether net is really trained.
// and experimenting with inputs *similar* to those in the training sets.
// input goes into the iset[0] array
// returns FALSE if user wants to quit, TRUE otherwise
// ----------------------------------------------------------------------
int work(void)
{
int i;
char buf[80];
int snode;			// which s-node is being filled
	            	// get exactly enough inputs (s-node count) from user
printf("\nNeed inputs for %d sensory nodes...\n",scount);
for(snode = 0 ; snode < scount ; snode++)
{
	printf("Enter input[%d], <CR> to quit: ",snode);
	gets(buf);
	if(!strlen(buf))
		return(FALSE);
	sscanf(buf,"%d",&i);
	iset[0][snode] = i;
}
tsindex = 0;				// "training set" 0 will hold the input data
initsensory(FALSE);			// transfer data to sensory nodes
think();					// calculate result
displayoutput();			// display result
return(TRUE); 	
}						// end work()

// ----------------------------------------------------------------------
// train()	train network on all input sets
// returns TRUE if successfully trained, FALSE if not
// ----------------------------------------------------------------------
int train(void)
{
int trained, graphicon = TRUE;
int passcount, passlimit = 20000;		// passes through inner loop
int opasscount, opasslimit = 600;		// passes through outer loop

opasscount = 0;
do
{
	trained = TRUE;
	if(++opasscount == opasslimit)		// too many failed passes,
	{									
		randomnewconnection();          // so add new connection to net.
		putchar(7);
		trained = FALSE;				// ensure outermost do will restart
		opasscount = 0;
		continue;						// restart outermost do
	}
	for(tsindex = 0 ; tsindex < tscount ; tsindex++)	// each training set
	{
		initsensory(graphicon);		// fill sensory nodes
		think();                    // forward-process
		if(graphicon)				// graphic display of nodes
        	drawnodes(TRUE);
		if(!testforsolution())	// also calculates error at each o-node		
		{
			trained = FALSE;
			passcount = 0;
			do				// exits when trained, user aborts, or rewired
			{
				if(++passcount == passlimit)	// frustration: rewire net,
				{								// and break from do() to
					randomnewconnection();		// stop backprop runs.
					putchar(7);
					tsindex = 0;		// same as restarting for() loop
					break;	// trained is false, so will finish for(), but
				}			// then break to outer do to retest all sets.
				backprop();
				think();
				if(kbhit())						// test for user input
				{
					switch(tolower(getch()))
					{				
						case 27:				// quit
							return(FALSE);
						case 'g':			// toggle graphic node display
                        	graphicon = !graphicon; break;
						default:
							break;
					}
				}		// end if(kbhit)
			}while(!testforsolution());
		}								// end if(!testforsolution)
	}									// end for(each input set)
}while(!trained);						
drawnodes(TRUE);	// display after trained
return(TRUE); 	
}					// end train()

// ----------------------------------------------------------------------
// main()
// ----------------------------------------------------------------------
void main(int argc, char **argv)
{
int ch, created, trained;
char filename[30];

randomize();

puts("Neural net demonstration program.");
puts("Usage: NEURAL trainset.TRS");
puts("     Default training set file is AND.TRS.\n");

strcpy(filename,"AND.TRS");				// set up a default training set
if(argc > 1)							// user-specified training set?
 	strcpy(filename,argv[1]);
loadtrainingsets(filename);			
    	

puts("If program hangs up, <ESC> will quit.");
printf("\nAny key to continue...");
getch();

graphicmode = initgraf();
maxx = getmaxx(); 
maxy = getmaxy();

if(!(created = createnetwork()))
	puts("Unable to create network.");
else
{
	if(!(trained = train()))
		puts("Network not trained.");
	else
		puts("Trained on all input sets.");
}
getch();
nodereport();						// show some node data
if(created && trained)
{

}
// 2006: I think these had been in the block above, but were moved out to allow
// testing even when the creation or training failed (?).
	getch();
	while(work());			// get inputs from user & calculate results

closegraph();
exit(0);
}				// end main()

	Search the web Search this site
Copyright ©2011 Steven Whitney. Last modified Tue 05/24/2011 07:31:36 -0700.