CODEWITHSUNDEEP
pythonscrapy
Mario Jaya
Mario Jaya

Reputation: 61

Scrapy error when executing fetch(url) command

I have this error when executing the fetch(url) command using Scrapy. I'm using Scrapy 2.7.1, Python 3.10.6 with miniconda on Ubuntu 22.04

Here are the steps that I have done:

scrapy startproject worldometers

cd worldometers

scrap shell

fetch("https://www.worldometers.info/world-population/population-by-country/")

This is the error messages that i got:

File "/home/miniconda3/envs/main/lib/python3.10/site-packages/twisted/internet/defer.py", line 857, in _runCallbacks
    current.result = callback(  # type: ignore[misc]
  File "/home/miniconda3/envs/main/lib/python3.10/site-packages/scrapy/utils/defer.py", line 285, in f
    return deferred_from_coro(coro_f(*coro_args, **coro_kwargs))
  File "/home/miniconda3/envs/main/lib/python3.10/site-packages/scrapy/utils/defer.py", line 272, in deferred_from_coro
    event_loop = get_asyncio_event_loop_policy().get_event_loop()
  File "/home/miniconda3/envs/main/lib/python3.10/asyncio/events.py", line 656, in get_event_loop
    raise RuntimeError('There is no current event loop in thread %r.'
RuntimeError: There is no current event loop in thread 'Thread-1 (start)'.
2022-11-12 21:08:55 [py.warnings] WARNING: /home/miniconda3/envs/main/lib/python3.10/site-packages/twisted/internet/defer.py:857: RuntimeWarning: coroutine 'SpiderMiddlewareManager.scrape_response.<locals>.process_callback_output' was never awaited
  current.result = callback(  # type: ignore[misc]

Any suggestion to fix the error?

Currently stuck, I tried searching the internet but still got no solution

Upvotes: 1

Views: 597

Answers (1)

Mario Jaya
Mario Jaya

Reputation: 61

solved it by commenting this line of code in file settings.py of the project:

# Set settings whose default value is deprecated to a future-proof value
REQUEST_FINGERPRINTER_IMPLEMENTATION = '2.7'
#TWISTED_REACTOR = 'twisted.internet.asyncioreactor.AsyncioSelectorReactor'

Upvotes: 4

Related Questions