CODEWITHSUNDEEP
javascriptwebcross-domainweb-crawler
Deepak
Deepak

Reputation: 57

Making to make a web crawler and a link tree purely in Javascript

In one of my assignments I have to make a web crawler, purely in JavaScript. That means, the input will be an URL and output will be a tree of all the links starting from that page. I used the plugin https://github.com/padolsey/jQuery-Plugins/tree/master/cross-domain-ajax/ which uses YQL and it worked for fetching all the links on the site. (Like I did for the Google's Homepage , http://deepakpathak.in/langoor/linkfinder.html). However I am having trouble making a tree structure out of it. Is there any other simpler way to make a tree of links of a given website, and that in Javascript?

Upvotes: 0

Views: 945

Answers (1)

Spoike
Spoike

Reputation: 121772

You're not mentioning what kind of tree you want to do, should the output be a tree list component on a website of some sort or do you want to put it in a database?

However you can use plain old javascript objects and arrays to create your tree (look up any notes on the tree data structure to see how they work there are many ways to represent them). To get you started a basic tree can look like this (with a createNode function):

var createNode = function(content) {
    return {
        'nodes': [], // children are put here
        'content': content // puts the content here
    };
};

var treeroot = createNode();
    // create the root node of the tree
    // content is undefined

treeroot.nodes.push(createNode(myTreeNode)); 
    // accesses the nodes array and pushes a new node into the root node
    // content in that node will be whatever "myTreeNode" is

You do have to write traversal algorithms yourself as javascript doesn't have any functions to handle trees. Alternatively create your tree using the DOM itself (as it is a tree data structure).

Upvotes: 1

Related Questions