CODEWITHSUNDEEP
pythonmacosscrapy
chrisHG
chrisHG

Reputation: 80

Scrapy no module named ' '

[Im working on OSX]

I'll post the relevant portions of my program down below:

Spider:

# -*- coding: utf-8 -*-
import scrapy
#import pandas as pd
from ..items import Homedepotv2Item
from scrapy.http import Request


class HomedepotspiderSpider(scrapy.Spider):
    name = 'homeDepotSpider'
    allowed_domains = ['homedepot.com']


    pathName = '/Users/user/Desktop/homeDepotv2Helpers/homeDepotInfo.csv'
    export = pd.read_csv(pathName, skiprows = [0], header = None)
        #pathName: Find the correct path for the file
        #skiprows: The first row is occupied for the title, we dont need that
    omsList = export.values.T[1].tolist() #Transpose the matrix + get second path



    start_urls = ['https://www.homedepot.com/p/{omsID}'].format(omsID = omsID)
        for omsID in omsList]

    def parse(self, response):

    #call home depot function
        for item in self.parseHomeDepot(response):
            yield item

        pass

Settings:

BOT_NAME = 'homeDepotv2'

SPIDER_MODULES = ['homeDepotv2.spiders']
NEWSPIDER_MODULE = 'homeDepotv2.spiders'

When I try running my spider by using the command: scrapy crawl homeDepotSpider

I get this error ModuleNotFoundError: No module named 'homeDepotv2'

Initially I thought I was having a directory error so instead of using cd to find my directory I copied in the pathname for the directory of the spider which was

/Users/userName/homeDepotv2_Spider/build/lib/homeDepotv2

However that still returned the same error.

Not too sure what's wrong here, so any help would be appreciated!

and here is the fire hierarchy: enter image description here enter image description here

Upvotes: 0

Views: 770

Answers (1)

Dr Pi
Dr Pi

Reputation: 417

Check this video, Path append | how to fix "Module not found" with Scrapy items.py

I had the same problem, the solution is to use:

from sys import path
path.append(/Users/userName/homeDepotv2_Spider)

You may need to check/modify the path, as Scrapy makes 2 directories with same name.

Upvotes: 1

Related Questions