What is the correct keyword for the Proximal AdaGrad optimizer on Tensorflow?

Question

I was experimenting with the Proximal AdaGrad for science fair and I was not able to use it because it counts it as it not existing.

My code:

import tensorflow as tf
from tensorflow import keras
import matplotlib.pyplot as plt
import numpy as np
import time

start_time = time.time()


data = tf.keras.datasets.fashion_mnist

(train_images, train_labels), (test_images, test_labels) = data.load_data()

class_names = ['T-shirt', 'Trouser', 'Pullover', 'Dress', 'Coat', 'Sandal', 'Shirt', 'Sneaker', 'Bag', 'Ankle Boot']

train_images = train_images/255.0

test_images = test_images/255.0

model = keras.Sequential([
                           keras.layers.Flatten(input_shape=(28, 28)),
                           keras.layers.Dense(100, activation="relu"),
                           keras.layers.Dense(10, activation="softmax")
])

model.compile(optimizer="Proximal AdaGrad", loss="sparse_categorical_crossentropy", metrics=["accuracy"])

model.fit(train_images, train_labels, epochs=200)

test_loss, test_acc = model.evaluate(test_images, test_labels)

print("Test acc is:", test_acc)
print("--- %s seconds ---" % (time.time() - start_time))

The Error:

ValueError                                Traceback (most recent call last)
<ipython-input-2-2d12844ae498> in <module>()
     24 ])
     25 
---> 26 model.compile(optimizer="Proximal AdaGrad", loss="sparse_categorical_crossentropy", metrics=["accuracy"])
     27 
     28 model.fit(train_images, train_labels, epochs=200)

6 frames
/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/training/tracking/base.py in _method_wrapper(self, *args, **kwargs)
    455     self._self_setattr_tracking = False  # pylint: disable=protected-access
    456     try:
--> 457       result = method(self, *args, **kwargs)
    458     finally:
    459       self._self_setattr_tracking = previous_value  # pylint: disable=protected-access

/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/training.py in compile(self, optimizer, loss, metrics, loss_weights, sample_weight_mode, weighted_metrics, target_tensors, distribute, **kwargs)
    250         'experimental_run_tf_function', True)
    251 
--> 252     self._set_optimizer(optimizer)
    253     is_any_optimizer_v1 = any(isinstance(opt, optimizers.Optimizer)
    254                               for opt in nest.flatten(self.optimizer))

/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/training.py in _set_optimizer(self, optimizer)
   1451       self.optimizer = [optimizers.get(opt) for opt in optimizer]
   1452     else:
-> 1453       self.optimizer = optimizers.get(optimizer)
   1454 
   1455     if (self._dtype_policy.loss_scale is not None and

/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/optimizers.py in get(identifier)
    844   elif isinstance(identifier, six.string_types):
    845     config = {'class_name': str(identifier), 'config': {}}
--> 846     return deserialize(config)
    847   else:
    848     raise ValueError('Could not interpret optimizer identifier:', identifier)

/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/optimizers.py in deserialize(config, custom_objects)
    813       module_objects=all_classes,
    814       custom_objects=custom_objects,
--> 815       printable_module_name='optimizer')
    816 
    817 

/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/utils/generic_utils.py in deserialize_keras_object(identifier, module_objects, custom_objects, printable_module_name)
    178     config = identifier
    179     (cls, cls_config) = class_and_config_for_serialized_keras_object(
--> 180         config, module_objects, custom_objects, printable_module_name)
    181 
    182     if hasattr(cls, 'from_config'):

/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/utils/generic_utils.py in class_and_config_for_serialized_keras_object(config, module_objects, custom_objects, printable_module_name)
    163     cls = module_objects.get(class_name)
    164     if cls is None:
--> 165       raise ValueError('Unknown ' + printable_module_name + ': ' + class_name)
    166   return (cls, config['config'])
    167 

ValueError: Unknown optimizer: Proximal AdaGrad

I tried naming it other things such as "ProximalAdaGrad" and "ProximalGrad" but non of those worked. The Activation Function looks to have no issues but the Optimizer itself seems to have a bug. I searched for a post on GitHub but did not find anyone posting an issue about this.

Answer 1

There is an open issue about this. The TensorFlow implementation exists (even in TensorFlow 2.x, as tf.compat.v1.train.ProximalAdagradOptimizer), but there is no corresponding Keras implementation at the moment. However, the Keras API is able to wrap an existing TensorFlow optimizer, so you should be able to do the following:

# This works both in recent 1.x and 2.0
optimizer = tf.compat.v1.train.ProximalAdagradOptimizer(0.001)
model.compile(optimizer=optimizer,
              loss="sparse_categorical_crossentropy",
              metrics=["accuracy"])