CODEWITHSUNDEEP
pythonscrapy
DatCra
DatCra

Reputation: 373

Terminate scrapy and keep output file

Made a coding mistake, and my spdier is likely running in a loop. How can I terminate the spider and save the output json file. My experience was forced termination will lead to no output file being saved. Any suggestion?

Can pkill save the output file? https://stackoverflow.com/a/46727707/7870384

Upvotes: 1

Views: 320

Answers (1)

Granitosaurus
Granitosaurus

Reputation: 21406

Just send one SIGTERM signal and scrapy will terminate gracefully once downloader queue is empty.
If you send two SIGTERM signals scrapy will terminate forcefully and ignore downloader queue or anything else that is waiting in Twisted engine.
If you send SIGKILL the process will be dumped and closed without giving scrapy a chance to do anything

In other words just send kill <pid> (uses SIGTERM signal by default) if you send kill -9 <pid> it will use SIGKILL signal

Upvotes: 4

Related Questions