Neural Network / Machine Learning memory storage

Question

I am currently trying to set up an Neural Network for information extraction and I am pretty fluent with the (basic) concepts of Neural Networks, except for one which seem to puzzle me. It is probably pretty obvious but I can't seem to found information about it.

Where/How do Neural Networks store their memory? ( / Machine Learning)

There is quite a bit of information available online about Neural Networks and Machine Learning but they all seem to skip over memory storage. For example after restarting the program, where does it find its memory to continue learning/predicting? Many examples online don't seem to 'retain' memory but I can't imagine this being 'safe' for real/big-scale deployment.

I have a difficult time wording my question, so please let me know if I need to elaborate a bit more. Thanks,

---

EDIT: - To follow up on the answers below

Every Neural Network will have edge weights associated with them. These edge weights are adjusted during the training session of a Neural Network.

This is exactly where I am struggling, how do/should I vision this secondary memory? Is this like RAM? that doesn't seem logical.. The reason I ask because I haven't encountered an example online that defines or specifies this secondary memory (for example in something more concrete such as an XML file, or maybe even a huge array).

Answer 1

This may be answered in two steps:

1. What is "memory" in a Neural Network (referred to as NN)?

As a neural network (NN) is trained, it builds a mathematical model that tells the NN what to give as output for a particular input. Think of what happens when you train someone to speak a new language. The human brain creates a model of the language. Similarly, a NN creates mathematical model of what you are trying to teach it. It represents the mapping from input to output as a series of functions. This math model is the memory. This math model is the weights of different edges in the network. Often, a NN is trained and these weights/connections are written to the hard disk (XML, Yaml, CSV etc). Whenever a NN needs to be used, these values are read back and the network is recreated.

2. How can you make a network forget its memory?

Think of someone who has been taught two languages. Let us say the individual never speaks one of these languages for 15-20 years, but uses the other one every day. It is very likely that several new words will be learnt each day and many words of the less frequent language forgotten. The critical part here is that a human being is "learning" every day. In a NN, a similar phenomena can be observed by training the network using new data. If the old data were not included in the new training samples, then the underlying math model will change so much that the old training data will no longer be represented in the model. It is possible to prevent a NN from "forgetting" the old model by changing the training process. However, this has the side effect that such a NN cannot learn completely new data samples.

Answer 2

I would say your approach is wrong. Neural Networks are not dumps of memory as we see on the computer. There are no addresses where a particular chunk of memory resides. All the neurons together make sure that a given input leads to a particular output.

Lets compare it with your brain. When you taste sugar, your tongue's taste buds are the input nodes which read chemical signals and transmit electric signals to brain. The brain then determines the taste using the various combinations of electric signals.

There are no lookup tables. There is no primary and secondary memories, only short and long term memory.

Answer 3

Memory storage is implementation-specific and not part of the algorithm per se. It is probably more useful to think about what you need to store rather than how to store it.

Consider a 3-layer multi-layer perceptron (fully connected) that has 3, 8, and 5 nodes in the input, hidden, and output layers, respectively (for this discussion, we can ignore bias inputs). Then a reasonable (and efficient) way to represent the needed weights is by two matrices: a 3x8 matrix for weights between the input and hidden layers and an 8x5 matrix for the weights between the hidden and output layers.

For this example, you need to store the weights and the network shape (number of nodes per layer). There are many ways you could store this information. It could be in an XML file or a user-defined binary file. If you were using python, you could save both matrices to a binary .npy file and encode the network shape in the file name. If you implemented the algorithm, it is up to you how to store the persistent data. If, on the other hand, you are using an existing machine learning software package, it probably has its own I/O functions for storing and loading a trained network.

Answer 4

Every Neural Network will have edge weights associated with them. These edge weights are adjusted during the training session of a Neural Network. I suppose your doubt is about storing these edge weights. Well, these values are stored separately in a secondary memory so that they can be retained for future use in the Neural Network.

Answer 5

I would expect discussion of the design of the model (neural network) would be kept separate from the discussion of the implementation, where data requirements like durability are addressed.

A particular library or framework might have a specific answer about durable storage, but if you're rolling your own from scratch, then it's up to you.

For example, why not just write the trained weights and topology in a file? Something like YAML or XML could serve as a format.

Also, while we're talking about state/storage and neural networks, you might be interested in investigating associative memory.