Pyplot not ploting some elements of data-array

Question

In one of my projects I am training Neural Networks to play a game. To see their process I make a graph with pyplot/matplotlib showing their best and average fitnessScore. The majority of this works, but for some reason it wont plot some of the data from its data array. How can I get it to plot all of the data?

The graph doesn't even appear first time the print statement fires, although there is one element in the scores array. The second print statement is followed by a graph only showing the first element of the array, the third statement by a graph showing the first and second etc..

Below is my code, for the sake of simplicity I removed the Network part and just use random integers to simulate fitnessscores:

import numpy as np
import json
import matplotlib.pyplot as plt
import random

def graphFitnessScores(fileName):
    with open(fileName, "r") as file:
       alldata = [json.loads(line) for line in file.readlines()]
       plt.pause(0.001)
       plt.clf()
       bestFitnesPerGen = [alldata[i]["bestFitnesses"][0] for i in range(len(alldata))]
       print("generation %d, amountOfLinesInFile:%d, scores:%s" % (alldata[-1]["generationNumber"],len(alldata), bestFitnesPerGen))
       best, = plt.plot(bestFitnesPerGen, '-r.')
       averageBestPerGen = [sum(alldata[i]["bestFitnesses"])/len(alldata[i]["bestFitnesses"]) for i in range(len(alldata))]
       average, = plt.plot(averageBestPerGen, '-b.')

       plt.xlabel("number of the generation")
       plt.xticks(range(len(alldata)+1))
       plt.ylabel("Fitnessscore")
       plt.legend([best,average], ["Best of the generation", "Average of the generation"])
       plt.axis([0, len(alldata), 0, np.amax(bestFitnesPerGen)+10]) #[xMin,xMax,yMin,yMax]

def writeBestFitnessesToFile(fitnessScores, generationNumber):
    with open(str("mytestFile.txt"), 'a+') as file:
       dataAsString = '{"generationNumber" : %d, "bestFitnesses" : %s}\n' % (generationNumber, fitnessScores)
       file.write(dataAsString)



def startTraining(numberOfGenerations, generationSize):
   #.other code but irrelevant for question.
   testFitnesses = [random.randint(2,100) for i in range(generationSize)]
   scoredFitnesses = sorted(testFitnesses,reverse=True)
   writeBestFitnessesToFile(scoredFitnesses,0)
   graphFitnessScores(str("mytestFile.txt"))

   for i in range(1,numberOfGenerations+1):
      nextGeneration = initGeneration(i, generationSize)
   plt.show()


def initGeneration(generationNumber, generationSize):
   testFitnesses = [random.randint(2,100) for i in range(generationSize)]
   scoredNewGeneration = sorted(testFitnesses,reverse=True)
   writeBestFitnessesToFile(scoredNewGeneration,generationNumber)
   graphFitnessScores(str("mytestFile.txt"))

startTraining(3,100)

I am using Python 3.6.4

To make this code work properly, please delete the content of 'myTestFile.txt' before each execution.

the print statement in graphFitnessScore prints:

generation 0, amountOfLinesInFile:1, scores:[99]
generation 1, amountOfLinesInFile:2, scores:[99, 100]
generation 2, amountOfLinesInFile:3, scores:[99, 100, 100]
generation 3, amountOfLinesInFile:4, scores:[99, 100, 100, 98]

And the file 'mytestFile.txt' looks like this after the final print statement:

{"generationNumber" : 0, "bestFitnesses" : [99, 97, 96, 96, 94, 93, 93, 92, 92, 91, 91, 90, 89, 89, 88, 88, 86, 86, 85, 84, 84, 83, 83, 81, 80, 78, 78, 77, 77, 74, 72, 71, 70, 70, 67, 67, 67, 66, 65, 65, 64, 63, 63, 61, 59, 58, 58, 56, 56, 56, 56, 55, 55, 54, 52, 51, 50, 49, 48, 48, 46, 46, 46, 45, 43, 41, 41, 41, 41, 41, 40, 39, 39, 38, 38, 33, 32, 30, 27, 26, 24, 24, 20, 18, 17, 17, 15, 14, 13, 13, 10, 10, 10, 9, 9, 5, 5, 5, 2, 2]}
{"generationNumber" : 1, "bestFitnesses" : [100, 100, 100, 99, 99, 96, 95, 91, 90, 90, 90, 90, 90, 89, 89, 88, 88, 88, 87, 86, 86, 85, 85, 85, 85, 84, 84, 84, 83, 82, 82, 78, 78, 77, 74, 73, 73, 71, 68, 66, 65, 64, 63, 62, 62, 61, 60, 59, 59, 58, 57, 56, 55, 55, 54, 54, 53, 50, 47, 47, 47, 45, 43, 42, 41, 41, 41, 40, 39, 37, 36, 36, 35, 34, 32, 32, 32, 31, 30, 29, 28, 26, 23, 23, 21, 20, 20, 20, 18, 17, 17, 16, 14, 14, 14, 12, 12, 11, 5, 5]}
{"generationNumber" : 2, "bestFitnesses" : [100, 99, 97, 97, 97, 95, 94, 93, 90, 90, 90, 89, 89, 89, 89, 87, 86, 86, 83, 83, 82, 82, 79, 75, 75, 73, 72, 72, 71, 71, 70, 69, 69, 65, 64, 63, 62, 62, 61, 60, 59, 59, 58, 58, 58, 55, 53, 53, 52, 51, 49, 46, 45, 45, 45, 45, 45, 44, 44, 41, 41, 41, 40, 40, 39, 37, 36, 36, 36, 33, 32, 31, 28, 27, 26, 25, 22, 22, 22, 22, 19, 18, 17, 16, 16, 15, 13, 13, 12, 11, 8, 7, 7, 7, 5, 5, 3, 3, 2, 2]}
{"generationNumber" : 3, "bestFitnesses" : [98, 98, 97, 95, 95, 95, 92, 91, 90, 90, 90, 89, 89, 88, 88, 87, 86, 86, 85, 85, 84, 84, 83, 81, 81, 81, 80, 80, 78, 75, 73, 73, 72, 71, 71, 70, 69, 68, 66, 66, 65, 64, 64, 63, 63, 63, 63, 62, 61, 60, 59, 57, 57, 55, 55, 53, 51, 50, 49, 47, 46, 45, 44, 40, 40, 39, 39, 38, 35, 35, 34, 32, 28, 26, 25, 24, 24, 23, 22, 22, 19, 19, 19, 17, 15, 14, 14, 10, 9, 8, 7, 6, 6, 5, 4, 3, 3, 2, 2, 2]}

But the graph never plots the last score of the array. So for the last iteration it shows:

Answer 1

The reason why your graph is incomplete is because you're showing immediately after the last function call (inside startTraining -> initGeneration -> graphFitnessScores).

So the solution is to add a plt.pause(1) before plt.show() so that all points get a chance to be plotted before the (last) figure is shown. The amount of time inside pause would probably need to be larger the larger your data set (number of generations) is.