CODEWITHSUNDEEP
amazon-web-servicesflaskamazon-ec2scrapyaws-lambda
Dude
Dude

Reputation: 91

Scrapy and flask on ec2

So basically im new into this, so bear with me, please.

I have 3 python spiders that uses: scrappy,scrappy-user-agent,pandas,MongoDB. they scrape around 150-200 pages every 12 hours and store the data locally into MongoDB collections. And I have a flask app that connects the API endpoints with the collections and returns the data as response.

Would it possible to deploy both to same ec2 instance, or would flask and response be slowed down for users while the scrapping is done in parallel in same machine?

Upvotes: 0

Views: 321

Answers (1)

Jundiaius
Jundiaius

Reputation: 7630

It is possible to deploy them both in the same instance. However, you need to know how much memory and CPU your both applications use, and choose your instance type accordingly.

Given the low frequency of your web scraping, it is very possible that it does not take much memory and CPU, but it may be the case if you are doing some heavy processing of the scrapped data.

To know about the memory and CPU configurations of each instance type: https://aws.amazon.com/ec2/instance-types/

Upvotes: 1

Related Questions