CODEWITHSUNDEEP
pythonloopsfor-looplong-running-processespause
user1610950
user1610950

Reputation: 1917

How can I pause a long running loop?

I have a question about pausing / resuming a long running program. I'll use Python in this example, but it could be for any programming language really.

Let's say I want to sum up all the numbers to a billion for example

results = []

for i in range(1000000000):
    results.append(i + i*2)

with open('./sum.txt', 'w') as outfile:
    for r in results:
        output.write(r+'\n')
outfile.close()

That's a very simple program and in the end I would like to write the values out to a file.

Let's say I start this script and it's taking hours and I want to shut down my computer. Now I could wait but what are some ways that I could stop this process while keeping the data in tact?

I've though about opening the file before the loop and inside the loop, instead of appending to the results list, I could just write [append] to the opened file but that seems like still if I quit the program, the file never closes and the data could get corrupted. Even if I just write to the file in the loop then directly close the file after the write operation, couldn't that still cause data corruption or something?

What are some ways that I could tackle a problem like this. Where I'd like to be able to stop a program and later, restart the program and it could continue from where it left off?

If you have a long running script that does most of the work on the inner part of a loop.

@MisterMiyagi Can you please specify what your actual question is? Creating safe-points inside the loop is the way to go if you want to preserve the loop's state. Whether file I/O is safe against corruption doesn't depend on your application so much as operating/file system. In Linux, it's safe to write to a temporary file, then rename it; files are also closed automatically when a program exits.

I was more interested in a safe place inside the loop. What are some recommended ways to do that in a program that's structured like the one above.

Upvotes: 0

Views: 4200

Answers (2)

Jacob Vlijm
Jacob Vlijm

Reputation: 3159

Create a condition for the loop to either continue or hold

A possible solution would be in the script, periodically checking for a "trigger" file to exist. In the example below, the loop checks once in hundred cycles, but you could make it 500 for example.

If and as long as the file exists, the loop will hold, checking again every two seconds.
While this works fine, you will have to accept the fact that it slightly slows down the loop, depending on n; larger n will decrease the effect. ANY solution inside the loop will however have at least some effect on the loop.

Although this is on python, the concept should be possible in any language, and on any os.

The print functions are only to show that it works of course:)

The edited loop:

import os
import time

results = []
n = 100

for i in range(1000000000):
    results.append(i + i*2)
    print(i)
    if i%n == 0:
        while os.path.exists("test123"):
            time.sleep(2)
            print("wait")   
        else:
            print("go on")
        
with open('./sum.txt', 'w') as outfile:
    for r in results:
        output.write(r+'\n')
outfile.close()

The result, if I create the trigger file and remove it again:

4096
4097
4098
4099
wait
wait
wait
wait
wait
wait
go on
4100
4101
4102

Upvotes: 1

Ceasar
Ceasar

Reputation: 23083

Ctrl+C won't corrupt the file. I'm not sure how sophisticated you need to be, but a simple solution would be to be able to give the program an input of where to resume from:

def main(start=0):
    with open('./sum.txt', 'w') as outfile:
        for i in range(start, 10000000):
            outfile.write(str(i) +'\n')


if __name__ == "__main__":
    import sys
    try:
        start = int(sys.argv[1])
    except IndexError:
        start = 0
    main()

Then run this like to resume from 1000:

python foo.py 1000

Upvotes: 1

Related Questions