CODEWITHSUNDEEP
tensorflowgradienttape
Ulf Wållgren
Ulf Wållgren

Reputation: 171

Why is GradientTape returning None when I use numpy math

Why is GradientTape returning None when I use numpy math

I am trying to understand tensorflow GradientTape calculation for RL loss function. When I call a function using np.math the GradientTape returns None. If I use tf.math in the function it works fine. I have looked at tf-agents like ppo and sac and they are doing exactly(?) what I am trying to do (I have tried at last 50 other versions). What's wrong in the code below? What am I missing?

window 10, python 3.6.8, tensorflow 2.0.0 ref:https://github.com/chagmgang/tf2.0_reinforcement_learning/blob/master/policy/ppo.py

import numpy as np
import tensorflow as tf
import matplotlib.pyplot as plt

def my_loss1(x):
    y=tf.sin(x)
    y=tf.abs(y)
    return y

def my_loss2(x):
    y=np.sin(x)
    y=np.abs(y)
    return y

def main(ver):    
    x = np.linspace(0,10,25)
    dsin_dx=np.cos(x)    
    xx = tf.constant(x)
    with tf.GradientTape() as tape:
        tape.watch(xx)
        if ver==0:
            # my_loss1 with tf math 
            loss1=my_loss1(xx)
        if ver==1:
            #my loss with numpy math
            loss1=my_loss2(np.array(xx))    
            loss1 = tf.convert_to_tensor(loss1, dtype=tf.float64)
        print(loss1)
        loss=tf.reduce_sum(loss1)
        print('loss=',loss)
    grads = tape.gradient(loss, xx)

    fig, ax = plt.subplots(2)
    ax[0].plot(x,loss1,'r')
    print('grads', grads)

    if not grads is None:
        ax[1].plot(x, grads)
        ax[1].plot(x,dsin_dx)
    plt.show()

if __name__ == '__main__':
    main(ver=0)  # This works ok
    main(ver=1)  # This returns grads = None

Upvotes: 1

Views: 1285

Answers (2)

Jochen Sauter
Jochen Sauter

Reputation: 51

It can be even more malicious: I had the case, that if there is any numpy array somewhere in the code under the gradient tape, there are gradients computed that look quite reasonable, but are actually wrong. No warnings whatsoever were issued by tensorflow. Replacing the numpy array by tf variables fixed it.

Upvotes: 0

Ulf Wållgren
Ulf Wållgren

Reputation: 171

The problem is that the Gradient tape only records tensors. Numpy variables are not recorded why the gradient can't be calqulated in case ver=1. Loss1 in ver1 looks identical to loss1 in ver=0 but the dependentsy to xx is broken by numpy.
My ref. has this error when calculation get_gaes() and the calculation of the grads is incorrect.

Upvotes: 1

Related Questions