CODEWITHSUNDEEP
tensorflowtensorflow-datasetstensorflow-estimator
Mahtab
Mahtab

Reputation: 11

Using tf.set_random_seed with tf.estimator.Estimator

I am using the tf.estimator.Estimator to manage training and testing part of my code. I am tuning some hyperparameters so I need to make sure that weights are initialized with the same random seed. Is there anyway to set_random_seed for a session created by tf.estimator?

Upvotes: 1

Views: 1191

Answers (1)

Olivier Moindrot
Olivier Moindrot

Reputation: 28198

You should define the random seed in the configuration passed to the estimator:

seed = 2018
config = tf.estimator.RunConfig(model_dir=model_dir, tf_random_seed=seed)

estimator = tf.estimator.Estimator(model_fn, config=config, params=params)

Here is the documentation for RunConfig.

One thing to be careful about is that each time you run estimator.train(train_input_fn), a new graph is created to train the model (by calling train_input_fn to create the input pipeline and calling model_fn on the output of train_input_fn).

One issue is that this new graph will also be set with the same random seed each time.

Example

Let me explain with an example. Suppose you perform data augmentation in your input pipeline, and you evaluate your model every epoch. This would give you something like that:

def train_input_fn():
    features = tf.random_uniform([])
    labels = tf.random_uniform([])
    dataset = tf.data.Dataset.from_tensors((features, labels))
    return dataset


def model_fn(features, labels, mode, params):
    loss = features
    global_step = tf.train.get_global_step()
    train_op = global_step.assign_add(1)
    return tf.estimator.EstimatorSpec(mode, loss=loss, train_op=train_op)


seed = 2018
config = tf.estimator.RunConfig(model_dir="test", tf_random_seed=seed)
estimator = tf.estimator.Estimator(model_fn, config=config)

num_epochs = 10
for epoch in range(num_epochs):
    estimator.train(train_input_fn, steps=1)
    estimator.evaluate(train_input_fn, steps=1)

The input function creates random features (and labels). What happens is that the features created are going to be exactly the same at each epoch. The output will be like:

INFO:tensorflow:loss = 0.17983198, step = 1
INFO:tensorflow:Saving dict for global step 1: global_step = 1, loss = 0.006007552
INFO:tensorflow:loss = 0.17983198, step = 2
INFO:tensorflow:Saving dict for global step 2: global_step = 2, loss = 0.006007552
INFO:tensorflow:loss = 0.17983198, step = 3
INFO:tensorflow:Saving dict for global step 3: global_step = 3, loss = 0.006007552
...

You can see that the loss (equal to the input features) is the same at each epoch, which means the same random seed is used at each epoch.

This is an issue if you want to evaluate at each epoch and perform data augmentation, because you will end up with the same data augmentation at each epoch.

Solution

One quick solution is to remove the random seed. However this prevents you from running reproducible experiments.

Another better solution is to create a new estimator at each epoch with the same model_fn but a different random seed:

seed = 2018

num_epochs = 10
for epoch in range(num_epochs):
    config = tf.estimator.RunConfig(model_dir="test", tf_random_seed=seed + epoch)
    estimator = tf.estimator.Estimator(model_fn, config=config)

    estimator.train(train_input_fn, steps=1)
    estimator.evaluate(train_input_fn, steps=1)

The features will change correctly at each epoch:

INFO:tensorflow:loss = 0.17983198, step = 1
INFO:tensorflow:Saving dict for global step 1: global_step = 1, loss = 0.006007552
INFO:tensorflow:loss = 0.22154999, step = 2
INFO:tensorflow:Saving dict for global step 2: global_step = 2, loss = 0.70446754
INFO:tensorflow:loss = 0.48594844, step = 3

Upvotes: 6

Related Questions