CODEWITHSUNDEEP
tensorflowdeep-learning
Chen Chieh Yu
Chen Chieh Yu

Reputation: 25

How to reset all variable in Adam Optimizer (Tensorflow), then I can retrain model with new optimizer

I am doing something related to curriculum learning, therefore I want to reset my optimizer for each training stage. However in Adam, there are some slots maintaining the first moment and second moment, also the power value of beta1 and beta2. I have try to define the tensorflow saver without watching those Variable as below:

def define_saver(exclude='(r'.*Adam.*', r'.*beta.*')'):
"""Create a saver for the variables we want to checkpoint.

Args:
  exclude: List of regexes to match variable names to exclude.

Returns:
  Saver object.
"""
variables = []
exclude = exclude or []
exclude = [re.compile(regex) for regex in exclude]
for variable in tf.global_variables():
    if any(regex.match(variable.name) for regex in exclude):
        continue
    variables.append(variable)
saver = tf.train.Saver(variables, keep_checkpoint_every_n_hours=5)
return saver

I am not sure am I doing right or not, because I have no much idea how Adam work in details, hope someone can share his/her ideas. thanks.

Upvotes: 1

Views: 1244

Answers (2)

tu_curious
tu_curious

Reputation: 407

Another way to do this (for any optimizer) could be:

#Get initial states
init_states = [var.value() for var in optimizer.variables()]

#Do the optimization
...

#reset optimizer state to init_state
for val,var in zip(init_states,optimizer.variables()): var.assign(val)

(You can use tf.print(optimizer.variables()) in your optimization loop to verify)

Upvotes: 0

P-Gn
P-Gn

Reputation: 24651

You can access the state of an Optimizer through its variables member function. If you group all their initializer, you end up with a reset op for the optimizer.

opt = tf.train.AdamOptimizer()
...
opt_reset = tf.group([v.initializer for v in opt.variables()])

Upvotes: 3

Related Questions