What does this peculiar Tensorboard histogram mean?

Question

I am trying to implement a regression Sequential model with Keras and I am getting very weird results. My code is below. I am using a tf.data dataset as my input dataset. My loss goes down initially but then starts oscillating. Would there be a reason this is happening?

    #initialize model
    model = keras.models.Sequential()
    
    model.add(Dense(1+(len(feature_names)-1), input_shape=((len(feature_names)-1),)))
    model.add(LeakyReLU())
    model.add(Dropout(0.5))
    
    model.add(Dense(10))
    model.add(LeakyReLU())
    
    #output layer, 1 unit
    model.add(Dense(1))
    
    loss = 'mean_squared_error'
    optimizer = tf.keras.optimizers.Nadam(learning_rate=0.0001)
    
    #compile the model
    m = tf.keras.metrics.RootMeanSquaredError()
    model.compile(optimizer=optimizer, loss=loss, metrics=[m,'mae'])
    
#    early_stop = keras.callbacks.EarlyStopping(monitor='val_loss', patience=4)
    tb = TensorBoard(histogram_freq=1)
    
    ##try learning rates, different noise, dropout, using only nasadem
    
    #run the model
    history = model.fit(x=final_train_dataset,epochs=count,steps_per_epoch=steps_per_epoch,validation_data=final_valid_dataset,callbacks=[tb])

5468/5468 [==============================] - 19s 4ms/step - loss: 19.8461 - root_mean_squared_error: 4.4549 - mae: 3.1814 - val_loss: 13.2963 - val_root_mean_squared_error: 3.6464 - val_mae: 3.0513
Epoch 2/25
   1/5468 [..............................] - ETA: 0s - loss: 12.9226 - root_mean_squared_error: 3.5948 - mae: 2.63732020-09-03 17:02:51.641109: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:184] Filling up shuffle buffer (this may take a while): 216632 of 1000000
2020-09-03 17:03:01.641093: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:184] Filling up shuffle buffer (this may take a while): 432841 of 1000000
2020-09-03 17:03:11.641090: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:184] Filling up shuffle buffer (this may take a while): 644957 of 1000000
2020-09-03 17:03:14.202555: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:233] Shuffle buffer filled.
5468/5468 [==============================] - 52s 9ms/step - loss: 4.5396 - root_mean_squared_error: 2.1306 - mae: 1.5395 - val_loss: 9.2400 - val_root_mean_squared_error: 3.0397 - val_mae: 2.3317
Epoch 3/25
   1/5468 [..............................] - ETA: 0s - loss: 2.1542 - root_mean_squared_error: 1.4677 - mae: 1.14152020-09-03 17:03:43.456237: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:184] Filling up shuffle buffer (this may take a while): 209810 of 1000000
2020-09-03 17:03:53.456238: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:184] Filling up shuffle buffer (this may take a while): 430811 of 1000000
2020-09-03 17:04:03.456243: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:184] Filling up shuffle buffer (this may take a while): 649754 of 1000000
2020-09-03 17:04:05.717317: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:233] Shuffle buffer filled.
5468/5468 [==============================] - 51s 9ms/step - loss: 3.5548 - root_mean_squared_error: 1.8854 - mae: 1.3483 - val_loss: 12.9111 - val_root_mean_squared_error: 3.5932 - val_mae: 3.1018
Epoch 4/25
   1/5468 [..............................] - ETA: 0s - loss: 3.0977 - root_mean_squared_error: 1.7600 - mae: 1.38832020-09-03 17:04:34.262132: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:184] Filling up shuffle buffer (this may take a while): 220010 of 1000000
2020-09-03 17:04:44.262123: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:184] Filling up shuffle buffer (this may take a while): 440231 of 1000000
2020-09-03 17:04:54.262103: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:184] Filling up shuffle buffer (this may take a while): 658897 of 1000000
2020-09-03 17:04:56.431462: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:233] Shuffle buffer filled.
5468/5468 [==============================] - 52s 9ms/step - loss: 1.9917 - root_mean_squared_error: 1.4113 - mae: 1.0305 - val_loss: 18.9676 - val_root_mean_squared_error: 4.3552 - val_mae: 3.4140
Epoch 5/25
   1/5468 [..............................] - ETA: 0s - loss: 2.0044 - root_mean_squared_error: 1.4158 - mae: 0.98262020-09-03 17:05:25.768521: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:184] Filling up shuffle buffer (this may take a while): 209530 of 1000000
2020-09-03 17:05:35.768491: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:184] Filling up shuffle buffer (this may take a while): 415149 of 1000000
2020-09-03 17:05:45.768498: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:184] Filling up shuffle buffer (this may take a while): 619178 of 1000000
2020-09-03 17:05:49.626056: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:233] Shuffle buffer filled.
5468/5468 [==============================] - 52s 10ms/step - loss: 3.1466 - root_mean_squared_error: 1.7739 - mae: 1.2621 - val_loss: 9.5821 - val_root_mean_squared_error: 3.0955 - val_mae: 2.3618
Epoch 6/25
   1/5468 [..............................] - ETA: 0s - loss: 2.2696 - root_mean_squared_error: 1.5065 - mae: 1.13252020-09-03 17:06:18.050735: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:184] Filling up shuffle buffer (this may take a while): 216085 of 1000000
2020-09-03 17:06:28.050765: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:184] Filling up shuffle buffer (this may take a while): 433756 of 1000000
2020-09-03 17:06:38.050744: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:184] Filling up shuffle buffer (this may take a while): 647129 of 1000000
2020-09-03 17:06:40.516620: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:233] Shuffle buffer filled.
5468/5468 [==============================] - 51s 9ms/step - loss: 6.5886 - root_mean_squared_error: 2.5668 - mae: 1.9638 - val_loss: 12.8445 - val_root_mean_squared_error: 3.5839 - val_mae: 2.6729
Epoch 7/25
   1/5468 [..............................] - ETA: 0s - loss: 6.2392 - root_mean_squared_error: 2.4978 - mae: 1.95852020-09-03 17:07:08.786525: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:184] Filling up shuffle buffer (this may take a while): 211271 of 1000000
2020-09-03 17:07:18.786501: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:184] Filling up shuffle buffer (this may take a while): 413948 of 1000000
2020-09-03 17:07:28.786525: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:184] Filling up shuffle buffer (this may take a while): 628456 of 1000000
2020-09-03 17:07:32.091402: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:233] Shuffle buffer filled.
5468/5468 [==============================] - 51s 9ms/step - loss: 8.6716 - root_mean_squared_error: 2.9448 - mae: 2.2906 - val_loss: 8.4462 - val_root_mean_squared_error: 2.9062 - val_mae: 2.0556
Epoch 8/25
   1/5468 [..............................] - ETA: 0s - loss: 7.3663 - root_mean_squared_error: 2.7141 - mae: 2.09252020-09-03 17:07:59.862839: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:184] Filling up shuffle buffer (this may take a while): 196030 of 1000000
2020-09-03 17:08:09.862769: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:184] Filling up shuffle buffer (this may take a while): 396211 of 1000000
2020-09-03 17:08:19.862786: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:184] Filling up shuffle buffer (this may take a while): 596521 of 1000000
2020-09-03 17:08:24.550654: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:233] Shuffle buffer filled.
5468/5468 [==============================] - 53s 10ms/step - loss: 4.6825 - root_mean_squared_error: 2.1639 - mae: 1.5158 - val_loss: 3.3543 - val_root_mean_squared_error: 1.8315 - val_mae: 1.4609
Epoch 9/25
   1/5468 [..............................] - ETA: 0s - loss: 3.3096 - root_mean_squared_error: 1.8192 - mae: 1.34092020-09-03 17:08:52.847230: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:184] Filling up shuffle buffer (this may take a while): 213665 of 1000000
2020-09-03 17:09:02.847221: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:184] Filling up shuffle buffer (this may take a while): 420633 of 1000000
2020-09-03 17:09:12.847262: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:184] Filling up shuffle buffer (this may take a while): 625401 of 1000000
2020-09-03 17:09:16.325973: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:233] Shuffle buffer filled.
5468/5468 [==============================] - 53s 10ms/step - loss: 7.0318 - root_mean_squared_error: 2.6518 - mae: 2.0913 - val_loss: 5.9001 - val_root_mean_squared_error: 2.4290 - val_mae: 1.7496
Epoch 10/25
   1/5468 [..............................] - ETA: 0s - loss: 5.4777 - root_mean_squared_error: 2.3405 - mae: 1.84602020-09-03 17:09:45.541587: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:184] Filling up shuffle buffer (this may take a while): 211860 of 1000000
2020-09-03 17:09:55.541592: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:184] Filling up shuffle buffer (this may take a while): 425901 of 1000000
2020-09-03 17:10:05.541639: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:184] Filling up shuffle buffer (this may take a while): 638506 of 1000000
2020-09-03 17:10:08.444475: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:233] Shuffle buffer filled.
5468/5468 [==============================] - 51s 9ms/step - loss: 5.5155 - root_mean_squared_error: 2.3485 - mae: 1.6887 - val_loss: 7.0059 - val_root_mean_squared_error: 2.6469 - val_mae: 1.9491
Epoch 11/25
   1/5468 [..............................] - ETA: 0s - loss: 4.7595 - root_mean_squared_error: 2.1816 - mae: 1.60422020-09-03 17:10:36.978236: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:184] Filling up shuffle buffer (this may take a while): 220628 of 1000000
2020-09-03 17:10:46.978223: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:184] Filling up shuffle buffer (this may take a while): 439429 of 1000000
2020-09-03 17:10:56.978219: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:184] Filling up shuffle buffer (this may take a while): 658497 of 1000000
2020-09-03 17:10:58.887525: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:233] Shuffle buffer filled.
5468/5468 [==============================] - 50s 9ms/step - loss: 7.7528 - root_mean_squared_error: 2.7844 - mae: 2.1798 - val_loss: 7.4304 - val_root_mean_squared_error: 2.7259 - val_mae: 2.1477
Epoch 12/25
   1/5468 [..............................] - ETA: 0s - loss: 5.9227 - root_mean_squared_error: 2.4337 - mae: 1.86982020-09-03 17:11:26.748628: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:184] Filling up shuffle buffer (this may take a while): 215455 of 1000000
2020-09-03 17:11:36.748645: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:184] Filling up shuffle buffer (this may take a while): 432008 of 1000000
2020-09-03 17:11:46.748652: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:184] Filling up shuffle buffer (this may take a while): 648981 of 1000000
2020-09-03 17:11:49.102683: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:233] Shuffle buffer filled.
5468/5468 [==============================] - 52s 9ms/step - loss: 5.3337 - root_mean_squared_error: 2.3095 - mae: 1.7406 - val_loss: 16.7965 - val_root_mean_squared_error: 4.0984 - val_mae: 2.9269
Epoch 13/25
   1/5468 [..............................] - ETA: 0s - loss: 3.7040 - root_mean_squared_error: 1.9246 - mae: 1.49812020-09-03 17:12:18.282279: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:184] Filling up shuffle buffer (this may take a while): 220812 of 1000000
2020-09-03 17:12:28.282279: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:184] Filling up shuffle buffer (this may take a while): 446392 of 1000000
2020-09-03 17:12:38.282271: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:184] Filling up shuffle buffer (this may take a while): 665649 of 1000000
2020-09-03 17:12:39.861825: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:233] Shuffle buffer filled.
5468/5468 [==============================] - 52s 9ms/step - loss: 8.0562 - root_mean_squared_error: 2.8384 - mae: 2.1867 - val_loss: 16.5570 - val_root_mean_squared_error: 4.0690 - val_mae: 3.2326
Epoch 14/25
   1/5468 [..............................] - ETA: 0s - loss: 7.7569 - root_mean_squared_error: 2.7851 - mae: 2.19592020-09-03 17:13:10.059481: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:184] Filling up shuffle buffer (this may take a while): 218698 of 1000000
2020-09-03 17:13:20.059483: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:184] Filling up shuffle buffer (this may take a while): 442427 of 1000000
2020-09-03 17:13:30.059488: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:184] Filling up shuffle buffer (this may take a while): 667466 of 1000000
2020-09-03 17:13:31.590296: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:233] Shuffle buffer filled.
5468/5468 [==============================] - 50s 9ms/step - loss: 5.5334 - root_mean_squared_error: 2.3523 - mae: 1.7780 - val_loss: 2.7601 - val_root_mean_squared_error: 1.6614 - val_mae: 1.3854
Epoch 15/25
   1/5468 [..............................] - ETA: 0s - loss: 5.4189 - root_mean_squared_error: 2.3279 - mae: 1.71692020-09-03 17:14:00.199430: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:184] Filling up shuffle buffer (this may take a while): 221723 of 1000000
2020-09-03 17:14:10.199432: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:184] Filling up shuffle buffer (this may take a while): 439283 of 1000000
2020-09-03 17:14:20.199441: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:184] Filling up shuffle buffer (this may take a while): 656318 of 1000000
2020-09-03 17:14:22.153380: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:233] Shuffle buffer filled.
5468/5468 [==============================] - 50s 9ms/step - loss: 7.9082 - root_mean_squared_error: 2.8121 - mae: 2.2185 - val_loss: 5.2582 - val_root_mean_squared_error: 2.2931 - val_mae: 1.8866
Epoch 16/25
   1/5468 [..............................] - ETA: 0s - loss: 8.4431 - root_mean_squared_error: 2.9057 - mae: 2.26912020-09-03 17:14:49.720142: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:184] Filling up shuffle buffer (this may take a while): 206675 of 1000000
2020-09-03 17:14:59.720147: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:184] Filling up shuffle buffer (this may take a while): 421484 of 1000000
2020-09-03 17:15:09.720163: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:184] Filling up shuffle buffer (this may take a while): 621983 of 1000000
2020-09-03 17:15:13.378677: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:233] Shuffle buffer filled.
5468/5468 [==============================] - 53s 10ms/step - loss: 8.1283 - root_mean_squared_error: 2.8510 - mae: 2.2215 - val_loss: 5.8047 - val_root_mean_squared_error: 2.4093 - val_mae: 1.7678
Epoch 17/25
   1/5468 [..............................] - ETA: 0s - loss: 6.9673 - root_mean_squared_error: 2.6396 - mae: 2.06012020-09-03 17:15:42.974799: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:184] Filling up shuffle buffer (this may take a while): 216994 of 1000000
2020-09-03 17:15:52.974823: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:184] Filling up shuffle buffer (this may take a while): 431703 of 1000000
2020-09-03 17:16:02.974787: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:184] Filling up shuffle buffer (this may take a while): 647963 of 1000000
2020-09-03 17:16:05.286592: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:233] Shuffle buffer filled.
5468/5468 [==============================] - 52s 10ms/step - loss: 5.6694 - root_mean_squared_error: 2.3811 - mae: 1.8346 - val_loss: 4.0211 - val_root_mean_squared_error: 2.0053 - val_mae: 1.4926
Epoch 18/25
   1/5468 [..............................] - ETA: 0s - loss: 4.3882 - root_mean_squared_error: 2.0948 - mae: 1.69472020-09-03 17:16:35.245666: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:184] Filling up shuffle buffer (this may take a while): 219541 of 1000000
2020-09-03 17:16:45.245695: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:184] Filling up shuffle buffer (this may take a while): 434849 of 1000000
2020-09-03 17:16:55.245697: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:184] Filling up shuffle buffer (this may take a while): 657326 of 1000000
2020-09-03 17:16:57.154813: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:233] Shuffle buffer filled.
5468/5468 [==============================] - 51s 9ms/step - loss: 7.6286 - root_mean_squared_error: 2.7620 - mae: 2.1647 - val_loss: 8.8990 - val_root_mean_squared_error: 2.9831 - val_mae: 2.1807
Epoch 19/25
   1/5468 [..............................] - ETA: 0s - loss: 6.3365 - root_mean_squared_error: 2.5172 - mae: 1.98792020-09-03 17:17:25.767055: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:184] Filling up shuffle buffer (this may take a while): 210541 of 1000000
2020-09-03 17:17:35.767069: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:184] Filling up shuffle buffer (this may take a while): 428168 of 1000000
2020-09-03 17:17:45.767050: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:184] Filling up shuffle buffer (this may take a while): 648394 of 1000000
2020-09-03 17:17:48.163076: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:233] Shuffle buffer filled.
5468/5468 [==============================] - 52s 9ms/step - loss: 5.6925 - root_mean_squared_error: 2.3859 - mae: 1.8557 - val_loss: 7.0036 - val_root_mean_squared_error: 2.6464 - val_mae: 2.0192
Epoch 20/25
   1/5468 [..............................] - ETA: 0s - loss: 5.9111 - root_mean_squared_error: 2.4313 - mae: 1.86942020-09-03 17:18:17.564046: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:184] Filling up shuffle buffer (this may take a while): 218969 of 1000000
2020-09-03 17:18:27.564032: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:184] Filling up shuffle buffer (this may take a while): 438295 of 1000000
2020-09-03 17:18:37.564054: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:184] Filling up shuffle buffer (this may take a while): 655413 of 1000000
2020-09-03 17:18:39.599428: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:233] Shuffle buffer filled.
5468/5468 [==============================] - 52s 10ms/step - loss: 6.8373 - root_mean_squared_error: 2.6148 - mae: 1.9742 - val_loss: 4.4934 - val_root_mean_squared_error: 2.1198 - val_mae: 1.5544
Epoch 21/25
  20/5468 [..............................] - ETA: 16s - loss: 6.7229 - root_mean_squared_error: 2.5929 - mae: 1.95182020-09-03 17:19:09.851911: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:184] Filling up shuffle buffer (this may take a while): 210717 of 1000000
2020-09-03 17:19:19.851916: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:184] Filling up shuffle buffer (this may take a while): 414573 of 1000000
2020-09-03 17:19:29.851935: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:184] Filling up shuffle buffer (this may take a while): 631533 of 1000000
2020-09-03 17:19:33.007503: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:233] Shuffle buffer filled.
5468/5468 [==============================] - 52s 9ms/step - loss: 5.7252 - root_mean_squared_error: 2.3927 - mae: 1.7921 - val_loss: 19.9770 - val_root_mean_squared_error: 4.4696 - val_mae: 3.9483
Epoch 22/25
  20/5468 [..............................] - ETA: 14s - loss: 5.5166 - root_mean_squared_error: 2.3488 - mae: 1.75092020-09-03 17:20:01.793350: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:184] Filling up shuffle buffer (this may take a while): 206965 of 1000000
2020-09-03 17:20:11.793374: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:184] Filling up shuffle buffer (this may take a while): 417262 of 1000000
2020-09-03 17:20:21.793379: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:184] Filling up shuffle buffer (this may take a while): 618303 of 1000000
2020-09-03 17:20:25.882237: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:233] Shuffle buffer filled.
5468/5468 [==============================] - 53s 10ms/step - loss: 7.3727 - root_mean_squared_error: 2.7153 - mae: 2.1631 - val_loss: 3.7680 - val_root_mean_squared_error: 1.9411 - val_mae: 1.4858
Epoch 23/25
  22/5468 [..............................] - ETA: 13s - loss: 6.6480 - root_mean_squared_error: 2.5784 - mae: 2.03622020-09-03 17:20:55.131738: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:184] Filling up shuffle buffer (this may take a while): 197145 of 1000000
2020-09-03 17:21:05.131721: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:184] Filling up shuffle buffer (this may take a while): 411294 of 1000000
2020-09-03 17:21:15.131712: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:184] Filling up shuffle buffer (this may take a while): 624435 of 1000000
2020-09-03 17:21:18.642404: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:233] Shuffle buffer filled.
5468/5468 [==============================] - 52s 9ms/step - loss: 5.5347 - root_mean_squared_error: 2.3526 - mae: 1.7962 - val_loss: 8.6975 - val_root_mean_squared_error: 2.9492 - val_mae: 2.3858
Epoch 24/25
  23/5468 [..............................] - ETA: 13s - loss: 5.2042 - root_mean_squared_error: 2.2813 - mae: 1.72872020-09-03 17:21:46.952287: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:184] Filling up shuffle buffer (this may take a while): 217458 of 1000000
2020-09-03 17:21:56.952289: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:184] Filling up shuffle buffer (this may take a while): 421999 of 1000000
2020-09-03 17:22:06.952280: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:184] Filling up shuffle buffer (this may take a while): 614871 of 1000000
2020-09-03 17:22:11.265809: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:233] Shuffle buffer filled.
5468/5468 [==============================] - 53s 10ms/step - loss: 2.9234 - root_mean_squared_error: 1.7098 - mae: 1.2129 - val_loss: 8.1747 - val_root_mean_squared_error: 2.8591 - val_mae: 2.3386
Epoch 25/25
  23/5468 [..............................] - ETA: 12s - loss: 2.7464 - root_mean_squared_error: 1.6572 - mae: 1.18782020-09-03 17:22:39.998529: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:184] Filling up shuffle buffer (this may take a while): 210233 of 1000000
2020-09-03 17:22:49.998541: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:184] Filling up shuffle buffer (this may take a while): 420717 of 1000000
2020-09-03 17:22:59.998540: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:184] Filling up shuffle buffer (this may take a while): 631855 of 1000000
2020-09-03 17:23:03.105398: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:233] Shuffle buffer filled.
5468/5468 [==============================] - 51s 9ms/step - loss: 3.2256 - root_mean_squared_error: 1.7960 - mae: 1.2697 - val_loss: 8.8438 - val_root_mean_squared_error: 2.9738 - val_mae: 2.2013

I took a look at the Tensorboard histograms and I got a weird histogram for one of the layers but I don't know how to interpret it.

Any help deciphering what is going on would be greatly appreciated!

Answer 1

The histograms you see are weight distributions of two dense layers.
Every dense layer has weights(w1, w2, .., wn) and a bias(b). The weights are called kernels in Tensorflow.

The histograms represent kernels and biases of two Dense layers.

How to interpret them?

As this is 3d histogram first it contains three axis.

x-axis - The actual value the kernels and biases take. ie value of b,w1,w2 etc
y-axis - The epoch number.
z-axis - The number of weights

We will take dense_1/bias_0 for our example. In this
x-axis : -0.8 to 1.6
Interpretation - All the values in the bias vector of the dense_1 layer are between -0.8 and 1.6.

y-axis : 0 - 22
Interpretation - The histogram shows values for each epoch from 0 to 22.

z-axis :
Interpretation - It shows how many biases have the value represented by x-axis. ie. If more biases have a value equal to zero you will get the highest peak (highest value of z) at x = 0.

So from the histogram for dense_1/bias_0 you can interpret that.
At epoch = 0 , all the biases had value 0. Since you get a peak at x = 0 and no z value for other values of x.

At epoch = 22, your bias vector does not contain any value between 0.0 and 0.4.

Why are these histograms used?
These histograms are used to see whether the model is actually learning or not. If the histogram remains the same for every epoch, then your model is not training properly and there is some error.

Useful links:

1. Tensorboard Tutorial
2. Using Tensorboard for debugging