CODEWITHSUNDEEP
pythonweb-scrapingtwitter
Monish K
Monish K

Reputation: 109

An error occured during an HTTP request: HTTP Error 404: Not Found GetOldTweets3

When I try scraping tweets based on username it shows me "An error occurred during an HTTP request: HTTP Error 404: Not Found. Try to open in browser: https://twitter.com/search?q=%20from%3A3mindia%20since%3A2020-04-01%20until%3A2020-04-30&src=typd "

The link given is directing me to website which is not a 404.

    An error occured during an HTTP request: HTTP Error 404: Not Found
    Try to open in browser: https://twitter.com/search?q=%20from%3A3mindia%20since%3A2020-04 01%20until%3A2020-04-30&src=typd
Traceback (most recent call last):
  File "C:\Users\\anaconda3\lib\site-packages\GetOldTweets3\manager\TweetManager.py", line 343, in getJsonResponse
    response = opener.open(url)
  File "C:\Users\\anaconda3\lib\urllib\request.py", line 531, in open
    response = meth(req, response)
  File "C:\Users\\anaconda3\lib\urllib\request.py", line 641, in http_response
    'http', request, response, code, msg, hdrs)
  File "C:\Users\\anaconda3\lib\urllib\request.py", line 569, in error
    return self._call_chain(*args)
  File "C:\Users\\anaconda3\lib\urllib\request.py", line 503, in _call_chain
    result = func(*args)
  File "C:\Users\\anaconda3\lib\urllib\request.py", line 649, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 404: Not Found

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\\anaconda3\lib\site-packages\IPython\core\interactiveshell.py", line 3331, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-3-41ed802d4e96>", line 1, in <module>
    CleanTweets("3mindia", "2020-04-01", "2020-04-30", "3M India")
  File "<ipython-input-2-1fa226d02b36>", line 3, in CleanTweets
    tweets = got.manager.TweetManager.getTweets(tweetCriteria)
  File "C:\Users\\anaconda3\lib\site-packages\GetOldTweets3\manager\TweetManager.py", line 65, in getTweets
    json = TweetManager.getJsonResponse(tweetCriteria, refreshCursor, cookieJar, proxy, user_agent, debug=debug)
  File "C:\Users\\anaconda3\lib\site-packages\GetOldTweets3\manager\TweetManager.py", line 348, in getJsonResponse
    sys.exit()
SystemExit

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\\anaconda3\lib\site-packages\IPython\core\ultratb.py", line 1151, in get_records
    return _fixed_getinnerframes(etb, number_of_lines_of_context, tb_offset)
  File "C:\Users\Monis\anaconda3\lib\site-packages\IPython\core\ultratb.py", line 319, in wrapped
    return f(*args, **kwargs)
  File "C:\Users\\anaconda3\lib\site-packages\IPython\core\ultratb.py", line 353, in _fixed_getinnerframes
    records = fix_frame_records_filenames(inspect.getinnerframes(etb, context))
  File "C:\Users\\anaconda3\lib\inspect.py", line 1502, in getinnerframes
    frameinfo = (tb.tb_frame,) + getframeinfo(tb, context)
AttributeError: 'tuple' object has no attribute 'tb_frame'
An exception has occurred, use %tb to see the full traceback.

---------------------------------------------------------------------------

During handling of the above exception, another exception occurred:

SystemExit

Upvotes: 6

Views: 5284

Answers (2)

Md Tahmid Rashid
Md Tahmid Rashid

Reputation: 26

One user seemed to have changed the code from the GOT3 library and managed to workaround the issue. The only problem is that the resulting HTML is unformatted. So someone needs to work on that. Specifically the changes they mentioned are:

updated user_agents (updated with the ones used by TWINT);

updated endpoint (/search?)

some updates to the URL structure:

url = "https://twitter.com/search?"

    

url += ("q=%%20%s&src=typd%s"
        "&include_available_features=1&include_entities=1&max_position=%s"
        "&reset_error_state=false")

if not tweetCriteria.topTweets:
    url += "&f=live"`

Upvotes: 0

Zico
Zico

Reputation: 185

Refer to this for details on error:

https://github.com/Mottl/GetOldTweets3/issues/98

twitter may have removed the endpoint the GetOldTweets3 - https://twitter.com/i/search/timeline?.

Refer to this post also: Why getoldtweets3 library provides 404 error?

Upvotes: 2

Related Questions