CODEWITHSUNDEEP
python-2.7subprocesspython-multithreading
zorro
zorro

Reputation: 94

Why is this Daemon thread blocking?

Why does the following code block on cc.start()? The crawler.py contains code similar to http://doc.scrapy.org/en/latest/topics/practices.html#run-from-script

import scrapy
import threading
from subprocess import Popen, PIPE

def worker():
    crawler = Popen('python crawler.py', stdout=PIPE, stderr=PIPE, shell=True)
    while True:
        line = crawler.stderr.readline()
        print(line.strip())

cc = threading.Thread(target=worker())
cc.setDaemon(True)
cc.start()
print "Here" # This is not printed
# Do more stuff

crawler.py contains the following code:

from scrapy.crawler import CrawlerProcess
import scrapy

class MySpider(scrapy.Spider):
    name = 'stackoverflow'
    start_urls = ['http://stackoverflow.com/questions?sort=votes']

def parse(self, response):
    for href in response.css('.question-summary h3 a::attr(href)'):
        full_url = response.urljoin(href.extract())
        yield scrapy.Request(full_url, callback=self.parse_question)

    process = CrawlerProcess({
    'USER_AGENT': 'Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)'
    })

process.crawl(MySpider)
process.start() # the script will block here until the crawling is finished

Upvotes: 0

Views: 1012

Answers (1)

Sanju
Sanju

Reputation: 2194

threading.Thread takes callable object as argument (ex function name), you are actually calling the function when you are creating a thread instance

cc = threading.Thread(target=worker())

what you need to do is just pass the function to be called with thread

cc = threading.Thread(target=worker)

Upvotes: 2

Related Questions