CODEWITHSUNDEEP
pythonbeanstalkdbeanstalkc
Srikanth
Srikanth

Reputation: 21

How to use beanstalkc in Python to queue URLs and perform jobs

I have a function named spider which takes seed as an argument. seed is the name of the URL I send to the spider function. Now my question is how do I use beanstalkc in Python to queue the URLs and perform the jobs.

Upvotes: 1

Views: 2309

Answers (1)

Damian
Damian

Reputation: 449

According to the tutorial you would need:

  1. beanstalkd server is running.

  2. Connect:

    import beanstalkc
beanstalk = beanstalkc.Connection(host='localhost', port=14711)

  3. Add jobs using:

    beanstalk.put('seed url')

  4. Get job via:

    job = beanstalk.reserve()
spider(job.body)

  5. Mark job as completed:

    job.delete()

Upvotes: 1

Related Questions