CODEWITHSUNDEEP
pythonkeras
nairouz mrabah
nairouz mrabah

Reputation: 1217

Keras dynamic graphs

It is known that Keras is based on static graphs. However, it seems to me that it is not really the case since I can make my graph implementation dynamic. Here is a simple example proving my claim:

import keras.backend as K
import numpy as np
from keras.models import Model
from keras.layers import Dense, Input, Dropout
from keras.losses import mean_squared_error

def get_mnist():
     np.random.seed(1234) # set seed for deterministic ordering
     (x_train, y_train), (x_test, y_test) = mnist.load_data()
     x_all = np.concatenate((x_train, x_test), axis = 0)
     Y = np.concatenate((y_train, y_test), axis = 0)
     X = x_all.reshape(-1,x_all.shape[1]*x_all.shape[2])
     p = np.random.permutation(X.shape[0])
     X = X[p].astype(np.float32)*0.02
     Y = Y[p]
     return X, Y


X, Y  = get_mnist()
drop = K.variable(0.2)
input = Input(shape=(784,))
x = Dropout(rate=drop.value)(input)
x = Dense(128, activation="relu", name="encoder_layer")(x)
decoder = Dense(784, activation="relu", name="decoder_layer")(x)
autoencoder = Model(inputs=input, outputs=decoder)
autoencoder.compile(optimizer='adadelta', loss= mean_squared_error)
autoencoder.fit(X, X, batch_size=256, epochs=300)
K.set_value(drop, 0.5)
autoencoder.fit(X, X, batch_size=256, epochs=300)

It is obvious that we can change the value of drop at any time, even after compiling the model.
If it is a Static Graph, how should I be able to do so?
Am I missing the point?
In case I do, what is the real interpretation of Dynamic Graphs?

Upvotes: 0

Views: 1214

Answers (2)

Subham Tiwari
Subham Tiwari

Reputation: 460

Certainly not because the dropout layer doesn't really affect the network architecture.

Upvotes: 0

Ryan A.
Ryan A.

Reputation: 86

It's true that you can change drop at any time, but that doesn't mean that Keras supports a dynamic graph. You are most likely used to seeing a neural network node described as a linear function with an activation function. By stacking these nodes you get a neural network. Then by reasoning what dropout does is take out nodes dynamically. However, this is not the Keras implementation of dropout. Dropout will set the nodes output to zero. This can be done by setting all of the weights in the respective node to zero. These operations are equivalent since the premise is that the next layer receives no output from the dropped node. However, the first method requires a dynamic graph, and the second can be done with both a dynamic and static graph, which is the implementation that Keras uses. Thus, there is no need for a dynamic graph for this operation.

To understand what a dynamic graph is, it's the model itself changing. The dropout operation just doesn't change the graph architecture (number of nodes, number of layers, ect. ) after the initial construction of the graph.

Upvotes: 3

Related Questions