CODEWITHSUNDEEP
pythondjangoscrapy
Jin
Jin

Reputation: 55

Import Django model class in Scrapy project

I have a django project and a scrapy project

and I want to import django model from to scrapy project.

This is my spider:

import scrapy
from scrapy.spiders import CrawlSpider, Rule
from scrapy.linkextractors import LinkExtractor

import sys
sys.path.append('/home/ubuntu/venv/dict')
from dmmactress.models import EnActress

class JpnNameSpider(scrapy.Spider):

name = jp_name
allowed_domains = ['enjoyjapan.co.kr']
rx = EnActress.objects.values_list('name', flat=True)
rxs = rx.reverse()
start_urls = ['http://enjoyjapan.co.kr/how_to_read_japanese_name.php?keyword=%s' % jp for jp in rxs]

def parse(self, response):
    for sel in response('//*[@id="contents"]/div/div[1]/div/div[1]'):
        item = JapanessItem()
        item['koname'] = sel.xpath('div[4]/div[1]()/text()').extract()
        item['jpname'] = sel.xpath('div[2]/div[1]()/text()').extract()
        yield item

    next_page = response.css('#contents > div > div:nth-child(4) > div > a::attr(href)').extract_first()
    if naxt_page is not None:
        next_page = response.urljoin(next_page)
        yield scrapy.Request(next_page, self.parse)

and I got error when I ran spider

django.core.exceptions.ImproperlyConfigured: Requested setting DEFAULT_INDEX_TABLESPACE, but settings are not configure.

-- Can someone help me to see what I am doing wrong?

Thanks in advance!

Upvotes: 0

Views: 359

Answers (1)

Jens Astrup
Jens Astrup

Reputation: 2454

You need to initialize Django before using models outside the context of Django apps:

import django 
django.setup()

Upvotes: 2

Related Questions