Early Stopping doesn't work properly --- choose wrong epoch

Question

I don't understand what the problem is. Yet, part of code which belong to early stopping, has problem to work properly. Let me share details.

early_stopping = EarlyStopping(monitor='val_loss', 
                               min_delta=0.001, 
                               patience=50, 
                               verbose=1, 
                               mode='min', 
                               restore_best_weights=True)

This is my early stopping part of code. I want to add that I tried 20 for patience, to see changes, yet problem still went on.

Epoch 43/200
24/24 ━━━━━━━━━━━━━━━━━━━━ 0s 791us/step - loss: 0.5384 - val_loss: 0.0752
Epoch 44/200
24/24 ━━━━━━━━━━━━━━━━━━━━ 0s 788us/step - loss: 0.4744 - val_loss: 0.0836
Epoch 45/200
24/24 ━━━━━━━━━━━━━━━━━━━━ 0s 797us/step - loss: 0.4310 - val_loss: 0.0992
Epoch 46/200
24/24 ━━━━━━━━━━━━━━━━━━━━ 0s 788us/step - loss: 0.2604 - val_loss: 0.0942
Epoch 47/200
24/24 ━━━━━━━━━━━━━━━━━━━━ 0s 802us/step - loss: 0.4305 - val_loss: 0.1033
Epoch 48/200
24/24 ━━━━━━━━━━━━━━━━━━━━ 0s 782us/step - loss: 0.4294 - val_loss: 0.0960
Epoch 49/200
24/24 ━━━━━━━━━━━━━━━━━━━━ 0s 799us/step - loss: 0.4992 - val_loss: 0.0994
Epoch 50/200
24/24 ━━━━━━━━━━━━━━━━━━━━ 0s 797us/step - loss: 0.5215 - val_loss: 0.0966
Epoch 50: early stopping
Restoring model weights from the end of the best epoch: 1.

Here is the first epoch result:

Epoch 1/200
24/24 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.4989 - val_loss: 0.1611

Even though there is a better val_loss, the code automatically assumes the best epoch value is 1.

Normally my code does not work on a single result. The code works on 30 different targets. For the 1st target early stopping works correctly, but for the next ones there is a problem. But here is the strange thing; for example, it works correctly for the 1st target, the 2nd and 3rd ones are wrong, then it works correctly for the 4th target. How should I fix it?

I changed patience number. Also, I tried to add checkpoint to save manually. After that, I tried to upload again best epoch number to fix it. Yet, I did not succeeded it.( but this is another soultion type that I tried, it is not the main issue what I have)