CODEWITHSUNDEEP
pythondjangoscrapy
Murat Demir
Murat Demir

Reputation: 716

How can I execute Scrapy in Django?

I have a task that I have to scrape some websites with Scrapy. I also have to delete and edit some URLs with Django. I want to use Django 3.1 and Scrapy 2.3.0.

I already try Django-dynamic-scraper and Djangoitem. Because of some version problems, I couldn't use them.

There was a class to execute Django command's called call_command

call_command('Scrapy crawl ../first_bot/first_bot/spider.py', stdout=out)

But it is just for the Django command.

Is there any way to execute scrapy module when I click on a button on Django?

Upvotes: 0

Views: 1011

Answers (1)

hi1
hi1

Reputation: 36

Paste code in python file start.py and Use this code in scrapy directory.

from scrapy.crawler import CrawlerProcess
from scrapy.utils.project import get_project_settings

process = CrawlerProcess(get_project_settings())

# var = name of spider

process.crawl(var)
process.start()

In django run start.py using this code here ever you like

import platform
import os

if plt == "Windows":
    cwd = os.path.join("<path to file>", "start.py")
    os.system('{} {}'.format('python', cwd))

elif plt == "Linux":
    cwd = os.path.join("<path to file>", "start.py")
    os.system('{} {}'.format('python3', cwd))

Upvotes: 1

Related Questions