CODEWITHSUNDEEP
tensorflowlinear-regressiongradient-descent
anirudh
anirudh

Reputation: 423

Loss Increasing for model only for bigger data

I implemented a simple Linear Regression model using Tensor Flow. However it only works for around 10-15 data points. Any more than that and the the loss function starts drastically increasing until it reaches infinity. The data is correct because I have synthetically generated it. The sklearn Linear Regression model works perfectly for the same data.

size = 8
x = np.float32(np.arange(size).reshape((size,1)))
y = x*8

class Linear_Model():
    def __init__(self,input_dim,lr=0.01):
        self.w = tf.Variable(tf.ones(shape=(input_dim,1)))
        self.b= tf.Variable(tf.zeros(shape=(input_dim)))
        self.lr = lr
    def predict(self,x):
        return tf.matmul(x,self.w) + self.b 

    def compute_loss(self,label,predictions):
        return tf.reduce_mean(tf.square(label-predictions))

    def train(self,x,y,epochs=12,batch_size=64):
        dataset = tf.data.Dataset.from_tensor_slices((x,y))
        dataset = dataset.shuffle(buffer_size=1024).batch(batch_size)
        for i in range(epochs):
            start = time.time()
            for step,(x,y) in enumerate(dataset):
                with tf.GradientTape() as tape:
                    preds = self.predict(x)
                    loss = self.compute_loss(y,preds)
                    dw,db = tape.gradient(loss,[self.w,self.b])
                    self.w.assign_sub(self.lr*dw)
                    self.b.assign_sub(self.lr*db)
                    print("Epoch %d : loss = %.4f time taken = %.3f s"% (i,float(loss),time.time()-start))
model = Linear_Model(1,0.01)
model.train(x,y,epochs=15)

Edit - From playing around with the learning rate I saw that a learning rate of 0.01 is too big. However this is not a problem for all the implementations I've seen around the web. What's happening here?

Upvotes: 0

Views: 38

Answers (1)

Amir
Amir

Reputation: 16587

The reason that your loss explodes is that your data is not normalized. As you increased the number of data-points, the magnitude of the input data becomes larger.

How can I fix it?

Normalized your data before feeding into the model:

x = (x - x.min()) / (x.max() - x.min())
y = (y - y.min()) / (y.max() - y.min())

Upvotes: 3

Related Questions