Does this neural network model exist?

Question

I'm looking for a neural network model with specific characteristics. This model may not exist...

I need a network which doesn't use "layers" as traditional artificial neural networks do. Instead, I want [what I believe to be] a more biological model.

This model will house a large cluster of interconnected neurons, like the image below. A few neurons (at bottom of diagram) will receive input signals, and a cascade effect will cause successive, connected neurons to possibly fire depending on signal strength and connection weight. This is nothing new, but, there are no explicit layers...just more and more distant, indirect connections.

As you can see, I also have the network divided into sections (circles). Each circle represents a semantic domain (a linguistics concept) which is the core information surrounding a concept; essentially a semantic domain is a concept.

Connections between nodes within a section have higher weights than connections between nodes of different sections. So the nodes for "car" are more connected to one another than nodes connecting "English" to "car". Thus, when a neuron in a single section fires (is activated), it is likely that the entire (or most of) the section will also be activated.

All in all, I need output patterns to be used as input for further output, and so on. A cascade effect is what I am after.

I hope this makes sense. Please ask for clarification where needed.

Are there any suitable models in existence that model what I've described, already?

Answer 1

your network also mimics this http://nn.cs.utexas.edu/?fullmer:evolving

but doesn't really allow the network to learn, but be replaced.

which may be covered here

http://www.alanturing.net/turing_archive/pages/reference%20articles/connectionism/Turing%27s%20neural%20networks.html

Answer 2

The answers involving genetic algorithms sound fine (especially the one citing Darrell Whitley's work).

Another alternative would be to simply randomly connect nodes? This is done, more or less, with recurrent neural networks.

You could also take a look at LeCun's highly successful convolutional neural networks for an example of an ANN with a lot of layers that is somewhat like what you've described here that was designed for a specific purpose.

Answer 3

Your neural network resembles a neural network which is created using Evolutionary Algorithms for example genetic algorithm.

See following articles for details.

• Han - Evolutionary neural networks for anomaly detection based on the behavior of a program
• WHITLEY - Genetic Algorithms and Neural Networks

For a summary in this type of neural network. Neurons and their connections are created using evolutionary techniques. Therefore they do not have strict layer approach. Hans uses following technique:

"Genetic Operations:

The crossover operator produces a new descendant by exchanging partial sections between two neural networks. It selects two distinct neural networks randomly and chooses one hidden node as the pivot point.Then, they exchange the connection links and the corresponding weight based on the selected pivot point.

The mutation operator changes a connection link and the corresponding weight of a randomly selected neural network. It performs one of two operations: addition of a new connection or deletion of an existing connection.

The mutation operator selects two nodes of a neural network randomly. If there is no connection between them, it connects two nodes with random weights.
Otherwise, it removes the connection link and weight information. "

Following figure from Whitley's article.

@ARTICLE{Han2005Evolutionary,
  author = {Sang-Jun Han and Sung-Bae Cho},
  title = {Evolutionary neural networks for anomaly detection based on the behavior
of a program},
  journal = {Systems, Man, and Cybernetics, Part B: Cybernetics, IEEE Transactions
on},
  year = {2005},
  volume = {36},
  pages = {559 -570},
  number = {3},
  month = {june },

}

@article{whitley1995genetic,
  title={Genetic algorithms and neural networks},
  author={Whitley, D.},
  journal={Genetic algorithms in engineering and computer science},
  pages={203--216},
  year={1995},
  publisher={Citeseer}
}

Answer 4

All in all, I need output patterns to be used as input for further output, and so on. A cascade effect is what I am after.

That sounds like a feed-forward net with multiple hidden layers. Don't be scared of the word "layer" here, with multiple ones it would be just like you have drawn there.. something like a 5-5-7-6-7-6-6-5-6-5 -structured net (5 inputs, 8 hidden layers with varying amount of nodes in each and 5 outputs).

You can connect the nodes to each other any way you like from layer to another. You can leave some unconnected by simple using constant zero as a weight between them, or if object oriented programming is used, simply leave unwanted connections out of connection phase. Skipping the layers might be harder with a standard NN-model, but one way could be using a dummy node for each layer a weight needs to cross. Just copying the original output*weight -value from node to dummy would be same as skipping a layer and this would also keep the standard NN-model intact.

If you want the net just to output some 1's and 0's, a simple step-function can be used as an activation function in each node: 1 for values more than 0.5, 0 otherwise.

I'm not sure if this is want you want, but this way you should be able to build a net you described. However, I have no idea how are you planning to teach your net to produce some semantic domains. Why not just let the net learn its own weights? This can be achieved with simple input-output -examples and a backpropagation -algorithm. If you use standard model to do build your net, also the mathematics of the learning wouldn't be any different from any other feed-forward net. Last but not least, you can probably find a library that is suitable for this task with only minor or with no change at all to the code.