CODEWITHSUNDEEP
web-scrapingxpathscrapyempty-list
Borgiven
Borgiven

Reputation: 21

Scrapy shell returning empty list

i'm trying to scrape from this site https://gol.gg/game/stats/43023/page-game/ using the scrapy shell but it always return empty list or nothing. i have tried using: response.css('').get() and response.xpath('').extract() with SelectotGadjet or the source code but i just can't scrape anything. i this site unscrapable or i'm i doing something really wrong, i'm a novice at web scraping.

Upvotes: 1

Views: 284

Answers (1)

Alexander
Alexander

Reputation: 17355

response.xpath('').extract() should fail as an invalid xpath expression.

It worked fine for me, steps to reproduce:

scrapy shell https://gol.gg/game/stats/43023/page-game/

response.xpath('//*')

Output:

[<Selector xpath='//*' data='<html lang="en">\n<head>\n             ...'>,
 <Selector xpath='//*' data='<head>\n                              ...'>,
 <Selector xpath='//*' data='<script>var gameid=43023;</script>'>,
 <Selector xpath='//*' data='<title>BLG vs OMG game 1 - LPL Summer...'>,
 <Selector xpath='//*' data='<base href="https://gol.gg/game/">'>,
 <Selector xpath='//*' data='<meta name="description" content="Vie...'>,
 <Selector xpath='//*' data='<meta name="keywords" content="lol, l...'>,
 <Selector xpath='//*' data='<meta name="twitter:card" content="su...'>,
 <Selector xpath='//*' data='<meta name="twitter:site" content="@b...'>,
 <Selector xpath='//*' data='<meta name="twitter:title" content="B...'>,
 <Selector xpath='//*' data='<meta name="twitter:description" cont...'>,
 <Selector xpath='//*' data='<meta name="twitter:image" content="h...'>,
 <Selector xpath='//*' data='<link rel="shortcut icon" href="../fa...'>,
 <Selector xpath='//*' data='<link rel="stylesheet" href="https://...'>,
 <Selector xpath='//*' data='<script src="https://code.jquery.com/...'>,
 <Selector xpath='//*' data='<script src="https://cdnjs.cloudflare...'>,
 <Selector xpath='//*' data='<script src="https://stackpath.bootst...'>,
 <Selector xpath='//*' data='<link href="https://fonts.googleapis....'>,
 <Selector xpath='//*' data='<link rel="stylesheet" type="text/css...'>,
 <Selector xpath='//*' data='<meta name="viewport" content="width=...'>,
 <Selector xpath='//*' data='<script>(function(w,d,s,l,i){w[l]=w[l...'>,
 <Selector xpath='//*' data='<body>\n<!-- Google Tag Manager (noscr...'>,
 <Selector xpath='//*' data='<noscript><iframe src="https://www.go...'>,
 <Selector xpath='//*' data='<iframe src="https://www.googletagman...'>,
 <Selector xpath='//*' data='<link href="../_lib/selectize/css/sel...'>,
 <Selector xpath='//*' data='<script src="https://ajax.googleapis....'>,
 <Selector xpath='//*' data='<script src="../_lib/selectize/js/sta...'>,
 <Selector xpath='//*' data='<link rel="stylesheet" href="https://...'>,
 <Selector xpath='//*' data='<div class="container-fluid main">\n  ...'>,
 <Selector xpath='//*' data='<header>\n   <div class="row row-fluid...'>,
 <Selector xpath='//*' data='<div class="row row-fluid">\n        \n...'>,
 <Selector xpath='//*' data='<div class="col-12 no-padding">\n   <n...'>,
 <Selector xpath='//*' data='<nav class="navbar navbar-dark menu">...'>,
 <Selector xpath='//*' data='<div class="pull-left hidden-xs">\n   ...'>,
 <Selector xpath='//*' data='<div class="pull-left">\n             ...'>,
 <Selector xpath='//*' data='<a href="../esports/home/"><img src="...'>,
 <Selector xpath='//*' data='<img src="../_img/gol_esports.png" cl...'>,
 <Selector xpath='//*' data='<a href="../esports/home/"><img src="...'>,
 <Selector xpath='//*' data='<img src="../_img/gol_esports_mini.pn...'>,
'...']

Upvotes: 1

Related Questions