CODEWITHSUNDEEP
javascriptnode.jsscreen-scraping
metalaureate
metalaureate

Reputation: 7732

How to convert HTML page to plain text in node.js?

I know this has been asked before but I can't find a good answer for node.js

I need server-side to extract the plain text (no tags, script, etc.) from an HTML page that is fetched.

I know how to do it client-side with jQuery (get the .text() contents of the body tag), but do not know how to do this on the server side.

I've tried https://npmjs.org/package/html-to-text but this doesn't handle scripts.

  var { convert } = require('html-to-text');
    var request = require('request');
    request.get(url, function (error, result) {
        var text = convert(result.body, {
            wordwrap: 130
        });
    });

I've tried phantom.js but can't find a way to just get plain text.

Upvotes: 18

Views: 32056

Answers (5)

Poyoman
Poyoman

Reputation: 1966

For those searching for a regex solution, here is my one

const HTMLPartToTextPart = (HTMLPart) => (
  HTMLPart
    .replace(/\n/ig, '')
    .replace(/<style[^>]*>[\s\S]*?<\/style[^>]*>/ig, '')
    .replace(/<head[^>]*>[\s\S]*?<\/head[^>]*>/ig, '')
    .replace(/<script[^>]*>[\s\S]*?<\/script[^>]*>/ig, '')
    .replace(/<\/\s*(?:p|div)>/ig, '\n')
    .replace(/<br[^>]*\/?>/ig, '\n')
    .replace(/<[^>]*>/ig, '')
    .replace('&nbsp;', ' ')
    .replace(/[^\S\r\n][^\S\r\n]+/ig, ' ')
);

Note: If you need to handle edge cases (see comments bellow) you will prefer html parsing solutions instead of regex one.

Upvotes: 7

Brad
Brad

Reputation: 163742

As another answer suggested, use JSDOM, but you don't need jQuery. Try this:

JSDOM.fragment(sourceHtml).textContent

Upvotes: 7

Geroj
Geroj

Reputation: 31

You can use TextVersionJS (http://textversionjs.com) to generate the plain text version of an HTML string. It's pure javascript (with tons of RegExps) so you can use it in the browser and in node.js as well.

This library may work for your needs, but it's NOT the same as getting the text of an element in the browser. Its purpose is to create a text version of an HTML email. This means that things like images are included. For example, given the following HTML and code snippet:

var textVersion = require("textversionjs");
var htmlText = "<html>" +
                    "<body>" +
                        "Lorem ipsum <a href=\"http://foo.foo\">dolor</a> sic <strong>amet</strong><br />" +
                        "Lorem ipsum <img src=\"http://foo.jpg\" alt=\"foo\" /> sic <pre>amet</pre>" +
                        "<p>Lorem ipsum dolor <br /> sic amet</p>" +
                        "<script>" +
                            "alert(\"nothing\");" +
                        "</script>" +
                    "</body>" +
                "</html>";
var plainText = textVersion.htmlToPlainText(htmlText);

The variable plainText will contain this string:

Lorem ipsum [dolor] (http://foo.foo) sic amet
Lorem ipsum ![foo] (http://foo.jpg) sic amet
Lorem ipsum dolor
sic amet

Note that it does properly ignore script tags. You'll find the latest version of the source code on GitHub.

Upvotes: 3

hgoebl
hgoebl

Reputation: 12987

Use jsdom and jQuery (server-side).

With jQuery you can delete all scripts, styles, templates and the like and then you can extract the text.

Example

(This is not tested with jsdom and node, only in Chrome)

jQuery('script').remove()
jQuery('noscript').remove()
jQuery('body').text().replace(/\s{2,9999}/g, ' ')

Upvotes: 8

Grimnoff
Grimnoff

Reputation: 486

Why not just get textContent of the body tag?

var body = document.getElementsByTagName('body')[0];
var bodyText = body.textContent;

Upvotes: -7

Related Questions