CODEWITHSUNDEEP
xidel
Mr. Curious
Mr. Curious

Reputation: 927

Convert Output of Xidel Pattern Match into Json Objects

Curious whether I can use Xidel pattern from a file to convert an html list into an array of json objects.

Given this example HTML:

<div class="watch-sidebar-body">
  <ul id="watch-related" class="video-list">
    <li class="video-list-item">
      <div class="content-wrapper">
        <a href="/watch?v=XPt3uMaqG7c">
          <span class="title">
            10 Best Super Bowl 2018 Commercials
          </span>
          <span class="accessible-description">
            - Duration: 11:07.
          </span>
          <span class="stat attribution"><span class="">Dalibor Truhlar</span></span>
          <span class="stat view-count">546,346 views</span>
        </a>
      </div>
    </li>
    <li class="video-list-item">
    ...

By using this pattern from a file:

<ul id="watch-related" class="video-list">
<t:loop>
<li>
    <a>
      <span class="title">{$title}</span>
      <span class="accessible-description">{$duration := extract(., "[0-9]+:[0-9]+")}</span>
      <span>
        <span>{$author}</span>
      </span>
      <span class="view-count">{$views := extract(., "[0-9,]+")}</span>
      {url := @href}
    </a>
</li>
</t:loop>
</ul>

I would like to produce the following json:

[{ "title" : "10 Best Super Bowl 2018 Commercials"
 , "duration" : "11:07"
 , "author" : "Dalibor Truhlar"
 , "views" : "546,346"
 , "url" : "/watch?v=XPt3uMaqG7c"
 }
,{ "title" : "Funny Commercial Compilation"
 , "duration" : "9:33"
 , "author" : "Gaming Coyote"
 , "views" : "9,449,290"
 , "url" : "/watch?v=BTka0cgf99c"
 }
...

The pattern properly matches the HTML and extracts the data, but I can't get the json output illustrated above.

When I run the command

curl -s https://www.youtube.com/watch\?v\=HE9nLWFZ6ac | xidel  - --silent --extract-file=yt.xq

I'm just getting the dump of all variables on stdout:

title := 10 Best Super Bowl 2018 Commercials
duration := 11:07
author := Dalibor Truhlar
views := 548,710
url := /watch?v=XPt3uMaqG7c
title := Funny Commercial Compilation
duration := 9:33
author := Gaming Coyote
views := 9,451,516
url := /watch?v=BTka0cgf99c

But how do I go from this to an array of json objects?

--output-format=json-wrapped doesn't help, since it converts each variable into it's own array, instead of zipping them into objects.

I know it is possible to create the desired json output with an xpath expression on command line, but I'm specifically interested in learning how to get that output from a pattern stored in a file.

Upvotes: 2

Views: 610

Answers (1)

BeniBela
BeniBela

Reputation: 16917

You can create an explicit object in the pattern:

<ul id="watch-related" class="video-list">
<t:loop>
<li>
    <a>

      {$out := {}}

      <span class="title">{$out.title}</span>
      <span class="accessible-description">{$out.duration := extract(., "[0-9]+:[0-9]+")}</span>
      <span>
        <span>{$out.author}</span>
      </span>
      <span class="view-count">{$out.views := extract(., "[0-9,]+")}</span>
      {$out.url := @href}
    </a>
</li>
</t:loop>
</ul>

and call it with | xidel - --silent --dot-notation=on --extract-file=yt.xq

Upvotes: 2

Related Questions