CODEWITHSUNDEEP
c#
user3859922
user3859922

Reputation:

Location of a referenced object in an array?

Background::::: I am working with a massive (2-dimensional) array of a custom neuron type for a Neural Network that seems to work well. When working with the smaller Neuron array sizes e.g. Neuron[,] NodeList = new Neuron[800,600]; it will serialize fine.

However it gets stuck in some object tree flattening loop(According to some forum in my history) here and runs out of memory Neuron[,] NodeList = new Neuron[10000,9000] After a little research on the Google I found that I will have to use another serializer or implement my own way of saving the data. I opted for a simple data writer as it would be easier

Problem(Skip here if you want :p)::: For my custom data writer i'm writing all the data needed to a file. In my Neuron type it stores a reference to another Neuron. All Neurons stored are all in the NodeList array as mentioned (including linked ones)

class Neuron
{
    //Some Old Neuron Info
    float IThreshold = 1;
    float IFiringValue = 1;

    //Link
    float IWeight = 1;
    Neuron NextNeuron;   //This Line HERE
    ...
}

For my data writer I am going through each neuron in the array and storing its info... like location in the array, weighting etc I also want to store the location where its linked neuron is in the array

I would like to do something like

int NewX = NodeList[x,y].NextNeuron.ParentArrayIndex(0);
int NewX = NodeList[x,y].NextNeuron.ParentArrayIndex(1);

Where the 0 & 1 merely represent the array dimension Is any of this possible or have a different way of getting it without searching through and checking they equal?

Any help appreciated,... and thank you :)

Upvotes: 1

Views: 100

Answers (1)

dbc
dbc

Reputation: 116795

You are going to need to introduce some sort of persistent identifier for your nodes, for instance:

struct NeuronId
{
    ushort Idx0 { get; }
    ushort Idx1 { get; }
}

You are also going to need to remember this inside your Neuron class. Having done this, you can have utilities like:

NeuronTable.GetNeuron(NeuronId id);

NeuronTable.SetNeuron(Neuron neuron, NeuronId id);

The option of keeping a back-reference dictionary Dictionary<Neuron, NeuronId> is unlikely to work in your case, since internally the c# dictionary stores its keys & values in a single dynamic 1-d array of Entry structures -- which will likely grow so large as to run out of memory or have more than Int32.MaxInt entries!

(I suspect that, when you were serializing your data with the old serializer, it actually cloned the NextNeuron nodes. Microsoft's XML serializer does not automatically convert cross-references to persistent indirect handles.)

Having done that, since you are defining the file format yourself rather than using a standard XML writer, you could write your data in two document sections:

  1. The Neuron database. This serializes the 2d array of neurons including all data except for the NextNeuron connections.

  2. The cross-reference database. This serializes all the neurons cross-references as a KeyValuePair<NeuronId, NeuronId> list, that you stream out by indexing through the neuron table.

Then on deserialization:

  1. Read the nodes into the huge table.

  2. Stream in the list of pairs and set up the cross-references.

By the way, is your array sparse? If so, you might consider using some sort of sparse matrix representation.

Also, if only a tiny, tiny percent of neurons have a non-null NextNeuron, you could avoid storing the NeuronId inside each Neuron and instead build up a Dictionary<Neuron, NeuronId> of just the cross-referenced neurons to use while writing.

Upvotes: 2

Related Questions