CODEWITHSUNDEEP
algorithmmatlabneural-networkpca
A.J.
A.J.

Reputation: 1027

Why am I getting a weight matrix full of NaNs?

(Hebbian learning)

I was given the task of programming the Oja's learning rule and Sanger's learning rule in Matlab, to train a neural network. This NN has 6 inputs and 4 outputs, and my training set comes from a multivariate uniform distribution, such as Xi ~ U(-ai,ai) and ai≠aj, for all i≠j

These are the most relevant files (Most comments and oja.m were not included)


main.m

TS = generarVectoresUnif(6, [1, 4, 9, 36, 25, 16], 512);
TS = TS';
W = unifrnd(0,1,[4,6]);
% it not very fast. That's why I put 500 iterations
W_sanger = sanger(W,TS,500, 0.05)


generarVectoresUnif.m

function [ TS ] = generarVectoresUnif( dim, rangos, n )
dimensiones = int8(dim);
tamanio = int32(n);
TS = [];

for i = 1:dimensiones
   TS = [TS, unifrnd(-rangos(i), rangos(i), [tamanio, 1]) ];
end


sanger.m

( NOTE: W is a 4 x 6 size matrix. Wi is the weight vector for the i-th output. Wij = (Wi)j. In the example, TS is a 6 x 512 size matrix )

function [ W ] = sanger( W_init, trainingset, iteraciones , eta)

W = W_init;

% obtiene los tamaños desde los parametros de entrada
size_input = size(W,2);
size_output = size(W,1);
n_patterns = size(trainingset, 2);


% one-tenth part
diezmo = iteraciones/10;


for it = 1:iteraciones

   if 0 == mod(it, diezmo)
      disp(horzcat('Iteracion numero ', num2str(it), ' de ',num2str(iteraciones)));
   end

   % for each pattern
   for u = 1:n_patrones

      DeltaW = zeros(size(W));

      % Vi = sum{j=1...N} Wij * Xj
      V = W * trainingset(:,u);

      % sumatorias(i,j) is going to replace sum{k=1..i} Vk*Wkj
      sumatorias = zeros(size_output,size_input);
      for j = 1:size_input
         for k = 1:size_output
             % sumar de 1 hasta i, sin hacer otro ciclo
             sumatorias(k,j) = (V' .* [ones(1,k), zeros(1,size_output-k)]) * W(:,j);
         end
      end

       % calcula la variacion
       for i = 1:size_output
          for j=1:size_input
             % Delta Wij = eta * Vi * ( xj - sum{k=1..i} Vk*Wkj )
              DeltaW(i,j) = eta * V(i,1) * (trainingset(j,u) - sumatorias(i,j));
          end
       end

       W = W + DeltaW;      
       %W = 1/norm(W) * W; %<---is it necessary? [Hertz] doesn't mention it

  end

end

Could you tell me please what I am doing wrong? Values of the matrix grow really fast. I have the same problem with oja.m

I've tried:

Upvotes: 2

Views: 980

Answers (2)

mbatchkarov
mbatchkarov

Reputation: 16049

You need small values of eta. Consider your update rule:

DeltaW(i,j) = eta * V(i,1) * (trainingset(j,u) - sumatorias(i,j));

If eta is large, DeltaW is likely to have a large absolute value (i.e. very big, e.g. 100000, or very small, e.g. -111111). The next time around the loop sumatorias(i,j) will be quite large, because it is a function of the weights. The more iterations you have, the larger your weights will become, eventually leading to an overflow.

Upvotes: 1

A.J.
A.J.

Reputation: 1027

Ok. After several tries, I made it work.

I choose a relatively small value of eta: 0.00001

W_sanger = sanger(W,TS,1000, 0.00001)

It's still slow, because of not taking advantage of matrix multiplication, which is optimized by Matlab.

I hope it helps someone else to not repeat the same mistake.

Regards!

Upvotes: 0

Related Questions