CODEWITHSUNDEEP
perlmojolicious
Chuck H
Chuck H

Reputation: 183

Trouble pinpointing child elements while using Mojo::DOM

I'm trying to extract text from an old vBulletin forum using WWW::Mechanize and Mojo::DOM.

vBulletin doesn't use HTML and CSS for semantic markup, and I'm having trouble using Mojo::DOM->children to get at certain elements.

These vBulletin posts are structured differently depending on their content.

Single message:

<div id="postid_12345">The quick brown fox jumps over the lazy dog.<div>

Single message quoting another user:

<div id="postid_12345">
    <div>
    <table>
        <tr>
            <td>
            <div>Quote originally posted by Bob</div>
            <div>Everyone knows the sky is blue.</div>
            </td>
        </tr>
    </table>
    </div>

 I disagree with you, Bob. It's obviously green.
</div>

Single message with spoilers:

<div id="postid_12345">
    <div class="spoiler">Yoda is Luke's father!</div>
</div>

Single message quoting another user, with spoilers:

<div id="postid_12345">
    <div>
    <table>
        <tr>
            <td>
            <div>Quote originally posted by Fred</div>
            <div class="spoiler">Yoda is Luke's father!</div>
            </td>
        </tr>
    </table>
    </div>
    <div class="spoiler">No waaaaay!</div>
</div>

Assuming the above HTML and an array packed with the necessary post IDs:

for (@post_ids) {
    $mech->get($full_url_of_specific_forum_post);
    my $dom = Mojo::DOM->new($mech->content);
    my $div_id = 'postid_' . $_;

    say $dom->at($div_id)->children('div')->first;
    say $dom->at($div_id)->text;
}

Using $dom->at($div_id)->all_text gives me everything in an unbroken line, which makes it difficult to tell what's quoted and what's original in the post.

Using $dom->at($div_id)->text skips all of the child elements, so quoted text and spoilers are not picked up.

I've tried variations of $dom->at($div_id)->children('div')->first, but this gives me everything, including the HTML.

Ideally, I'd like to be able to pick up all the text in each post, with each child element on its own line, e.g.

 POSTID12345:
 + Quote originally posted by Bob
 + Everyone knows the sky is blue. 
 I disagree with you, Bob. It's obviously green.

I'm new to Mojo and rusty with Perl. I wanted to solve this on my own, but after looking over the documentation and fiddling with it for a few hours, my brain is mush and I'm at a loss. I'm just not getting how Mojo::DOM and Mojo::Collections work.

Any help will be greatly appreciated.

Upvotes: 4

Views: 901

Answers (2)

Joel Berger
Joel Berger

Reputation: 20280

Looking at the source of Mojo::DOM, basically the all_text method recursively walks the DOM and extracts all text. Use that source to write your own walking the DOM function. Its recursive function depends on returning a single string, in yours you might have it return an array with whatever context you need.

EDIT:

After some discussion on IRC, the web scraping example has been updated, it might help you guide you. http://mojolicio.us/perldoc/Mojolicious/Guides/Cookbook#Web_scraping

Upvotes: 3

creaktive
creaktive

Reputation: 5220

There is a module to flattern HTML tree, HTML::Linear. The explanation of purpose for flatterning HTML tree is a bit long and boring, so here's a picture showing the output of the xpathify tool, bound with that module:

screenshot

As you see, HTML tree nodes become single key/value list, where the key is the XPath for that node, and the value is the node's text attribute. In a few keystrokes, this is how you use HTML::Linear:

#!/usr/bin/env perl
use strict;
use utf8;
use warnings;

use Data::Printer;
use HTML::Linear;

my $hl = HTML::Linear->new;
$hl->parse_file(q(vboard.html));

for my $el ($hl->as_list) {
    my $hash = $el->as_hash;
    next unless keys %{$hash};
    p $hash;
}

Upvotes: 2

Related Questions