CODEWITHSUNDEEP
pythonelasticsearchscrapy
Slyper
Slyper

Reputation: 906

Scrapy-elasticsearch plugin issue

I have recently installed this plugin which is working great ...

Now my issue is that when I repopulate the ES 'index' with new data, I want to delete the existing 'index' first in ES. This is to delete old data in ES.

The above mentioned plugin contains this file scrapyelasticsearch.py where I think I can add this code 

es.delete(index='my-index', doc_type='test')

to delete the index before repopulating.

The plugin will automatically recreate the index before inserting data.

Question: I couldn't find where this file (scrapyelasticsearch.py) is located ? I am using Ubuntu 16.04, with ES and Scrapy also installed.

I tried this command to find this package 

dpkg -l scrapyelasticsearch

but received this error 

dpkg-query: no packages found matching scrapyelasticsearch

If anyone has used this plugin/package, please help me find this file scrapyelasticsearch.py

Any help is very appreciated. Thanks

Upvotes: 1

Views: 380

Answers (1)

Granitosaurus
Granitosaurus

Reputation: 21436

The file is located in your site-packages directory of your python installation. So if you're running on system's python (not a virtual environment) it would be something like: 

/usr/lib/python3.5/site-packages/

However, you should not modify site-package data!

What you should do is clone or fork the project on github, make your changes to it, and install this fork on your system. 

git clone https://github.com/knockrentals/scrapy-elasticsearch.git
cd scrapy-elasticsearch
your_editing_program 'scrapyelasticsearch/scrapyelasticsearch.py'
# make changes
pip uninstall scrapy-elasticsearch  # uninstall old original package
pip install .  # install your package, you can also add -e flag for real time modifications

Upvotes: 1

Related Questions