Reputation: 2936
If I want to add new nodes to on of my tensorflow layers on the fly, how can I do that?
For example if I want to change the amount of hidden nodes from 10 to 11 after the model has been training for a while. Also, assume I know what value I want the weights coming in and out of this node/neuron to be.
I can create a whole new graph, but is there a different/better way?
Upvotes: 4
Views: 3120
Reputation: 2335
Let us assume that res
is output taking into account there are 10 hidden layers and let us call 10th hidden layer layer10
. So what you do is
res = f(layer10)
where f
is some function which operates on layer10
and returns resulting tensor.
Pre-create layer11
as some operation on layer10
and res2
as output from layer11
.
Now running you just evaluate res
which will use on 10 layers. When you want to use 11 layers use res2
. If you know the weights then you can assign it like tensor.assign(val).eval()
Upvotes: 1
Reputation: 4244
TensorFlow has many advantadges, but dynamic graph modification isn't one of them.
If you truly want to be able to change graphs on the fly, I recommend you PyTorch (http://pytorch.org/). Although it's much more recent than TensorFlow, so the documentation isn't as complete.
Upvotes: 5
Reputation: 5206
Instead of creating a whole new graph you might be better off creating a graph which has initially more neurons than you need and mask it off by multiplying by a non-trainable variable which has ones and zeros. You can then change the value of this mask variable to allow effectively new neurons to act for the first time.
Upvotes: 6