CODEWITHSUNDEEP
node.jsweb-crawlerphantomjssingle-page-application
bendulum
bendulum

Reputation: 1857

PhantomJS not serving JS and CSS files correctly

I have a node server running and listening to Port 8080. There is a mod_rewrite rule active that redirects Bots to this Port.

RewriteCond %{HTTP_USER_AGENT} (googlebot|adsbot-google|bingbot|msnbot|psbot|gigabot|twitterbot|linkedinbot|yahoo-mmcrawler|pingdom\.com_bot) [NC]
RewriteRule ^ http://127.0.0.1:8080%{REQUEST_URI} [P]

The node script relies on a phantomjs script to open any URLs the Bots request and return the contents.

The code taken from here http://backbonetutorials.com/seo-for-single-page-apps/

I tested this by running phantomjs on my local machine and requesting my web page directly with the same phantomjs code. The results are the same as what "crawl as google" indicates (google webmaster tools), which is that CSS and JS files are not properly served by phantomjs.

The CSS file contains only 

<html><head></head><body></body></html>

and no actual CSS contents. The JS file has 

<html><head></head><body><pre style="word-wrap: break-word; white-space: pre-wrap;">

inserted at the very beginning before any JS code.

As a result when google tries to crawl and render the page the layout is broken, and JS errors are thrown.

Any hints? Thanks.

Upvotes: 4

Views: 699

Answers (1)

morriq
morriq

Reputation: 424

Alright. I have found solution.

You use probablly page.content for each request. You need to use it only for html files. Here is my code:

    if (url.indexOf('.html') > -1) cb(page.content);
    else cb(page.plainText);

Second solution requires changes in your htaccess.

RewriteCond %{HTTP_USER_AGENT} (googlebot|adsbot-google|bingbot|msnbot|psbot|gigabot|twitterbot|linkedinbot|yahoo-mmcrawler|pingdom\.com_bot) [NC]
RewriteCond %{REQUEST_URI} (.*).html(.*) <---- ADDED THIS ONE
RewriteRule ^ http://%{HTTP_HOST}:3004%{REQUEST_URI} [P]

Upvotes: 2

Related Questions