CODEWITHSUNDEEP
shellweb-scrapingxpathxidelhacker-news-api
Mévatlavé Kraspek
Mévatlavé Kraspek

Reputation: 681

xidel: wrong order of results on hacker news

To scrape hacker news, I use:

xidel -e '//span[@class="titleline"]/a/@href|//span[@class="titleline"]' https://news.ycombinator.com/newest

But the output in not in the expected order, the URL come after the text, so it's very difficult to parse.

Does I miss something to have the good order?

I have:

There Is No Such Thing as a Microservice (youtube.com)
https://www.youtube.com/watch?v=FXCLLsCGY0s

I expect:

https://www.youtube.com/watch?v=FXCLLsCGY0s
There Is No Such Thing as a Microservice (youtube.com)

Or even better

https://www.youtube.com/watch?v=FXCLLsCGY0s There Is No Such Thing as a Microservice (youtube.com)

Upvotes: 1

Views: 121

Answers (3)

Reino
Reino

Reputation: 3443

Please see "Using / on sequences rather than on sets" on why this is happening and why you should be using the XPath 3 mapping operator ! in this case:

$ xidel -s "https://news.ycombinator.com/newest" -e '
  //span[@class="titleline"]/a ! (@href,.)
'

(also please specify input first)

For a simple string-concatenation this isn't necessary:

-e '//span[@class="titleline"]/a/join((@href,.))'
-e '//span[@class="titleline"]/a/concat(@href," ",.)'
-e '//span[@class="titleline"]/a/x"{@href} {.}"'

(Bonus) Output to JSON:

$ xidel -s "https://news.ycombinator.com/newest" -e '
  array{
    //span[@class="titleline"]/a/{
      "title":.,
      "url":@href
    }
  }
'

Upvotes: 2

Mévatlavé Kraspek
Mévatlavé Kraspek

Reputation: 681

Found a better way:

$ xidel -e '//span[@class="titleline"]/a/@href|//span[@class="titleline"]/a/text()' https://news.ycombinator.com/newest

Upvotes: 0

LMC
LMC

Reputation: 12822

Nodes are returned in document order not in XPath order so additional parsing is needed. With xmllint and awk

xmllint --html --recover --xpath '//span[@class="titleline"]/a/@href|//span[@class="titleline"]/a/text()' tmp.html 2>/dev/null|\
gawk 'BEGIN{RS="\n? href="; FS="\n"}{ print $1, $2}' | tr -d '"'

Result

https://github.com/thesephist/ink Ink: Minimal, functional programming language inspired by modern JavaScript, Go
https://controlleddigitallending.org/whitepaper/ A White Paper on Controlled Digital Lending of Library Books
item?id=35471687 Ask HN: Connect Guitar to Tesla?

Note: Xpath in the answer does not need awksince href comes before a/text() in document order as OP expected. Added for reference on how to change ouput order.

Upvotes: 1

Related Questions