CODEWITHSUNDEEP
pythontimercroncron-task
Fasty
Fasty

Reputation: 804

How to repeatedly stop a program after x minutes and re run it changing one variable in python?

I have a program written in python which goes as follows:

for i in range(4954):
    ........
    save("/Downloads/pos_" + str(i) + ".h5")

The fact is the program running time increases exponentially(we observed it using time.time())

So what I need is run the program for 10 min and then re run the program. But I just need to change the i in for loop to the number at which it is stopped. I can do a cron job, but what should I do to change i value?

Upvotes: 0

Views: 290

Answers (4)

Carlo Zanocco
Carlo Zanocco

Reputation: 2019

Here my solution:

import os
import time
import threading

class test_class(object):
    def __init__(self):
        self.iter_number = 0
        self.new_iter_number = 0

        self.iter_file_dir = "/tmp_file.txt" # tmp file where sto the new iteration number remaining

        self.end_iteration = False # Flag to stop the iteration

        self.delay = 1 # Seconds that means 10 minutes

        with open(self.iter_file_dir, "r") as file:
            self.iter_number = int(file.readline())

        self.run()

    def for_function(self):
        for i in range(self.iter_number):
            save("/Downloads/pos_" + str(i) + ".h5")
            if(self.end_iteration):
                self.new_iter_number = self.iter_number - i
                break

    def run(self):
        threading.Thread(name="Iteration_Thread", target=self.for_function).start()

        time_remaining = self.delay-time.time()%self.delay
        time.sleep(time_remaining)

        self.end_iteration = True

        # Save the file with the new value
        with open(self.iter_file_dir, 'w') as f:
            f.write(str(self.new_iter_number))
test_class()

The idea behind this code is simple:

  1. Create a tmp_file.txt where you want, and write in them the number of iteration that you need in the for.
  2. Create a timer that sleep for X minutes and after that set a flag that terminate the for loop.
  3. After that save in the tmp_file.txt the new calculated iteration number

Each time that the cron run the program it will load the new iteration value. Obviously when the iteration are finished in the file the 0 is written, so no more iteration were performed.

Upvotes: 0

r.ook
r.ook

Reputation: 13878

Consider using a generator:

import time

def gen(n):
    for i in range(n):
        yield n

g = gen(4954)

time_limit = 600

while some_condition:
# whatever condition required for you to continually restart this section

    start = time.time()
    while time.time() - start < time_limit:
        ... some code...
        try: 
            save("/Downloads/pos_" + str(next(g)) + ".h5")
        except StopIteration:
            print('end of iterations')

Where g will remember the iteration it's on and continue the next time you call next(g).

Upvotes: 1

gdlmx
gdlmx

Reputation: 6789

If each iteration (each i value) in the loop is independent with each other, you can parallelize it with ProcessPoolExecutor:

import concurrent.futures

def downloader(i):
    ...
    save("/Downloads/pos_" + str(i) + ".h5")

def main():
    i_range = list(range(4954))
    with concurrent.futures.ProcessPoolExecutor(10) as executor: # create 10 workers (subprocesses)
        for number, result in zip(i_range , executor.map(downloader, i_range)):
            print('File %d is scheduled to download!' % (number))

if __name__ == '__main__':
    main()

This requires Python3.

Upvotes: 1

claasz
claasz

Reputation: 2134

You could use marshal.dump(https://docs.python.org/3/library/marshal.html) to save the value when stopping the program, and then loading it via marshal.load when starting.

Honestly, I think it would be a much better approach to fix the root cause of the problem, i.e. solving the exponential run time.

Upvotes: 1

Related Questions