CODEWITHSUNDEEP
c#html-agility-pack
Yuki Kutsuya
Yuki Kutsuya

Reputation: 4088

Working with HtmlAgilityPack

I'm trying to get a link and another element from an HTML page, but I don't really know what to do. This is what I have right now:

var client = new HtmlWeb(); // Initialize HtmlAgilityPack's functions.
var url = "http://p.thedgtl.net/index.php?tag=-1&title={0}&author=&o=u&od=d&page=-1&"; // The site/page we are indexing.
var doc = client.Load(string.Format(url, textBox1.Text)); // Index the whole DB.
var nodes = doc.DocumentNode.SelectNodes("//a[@href]"); // Get every url.

string authorName = "";
string fileName = "";
string fileNameWithExt;

foreach (HtmlNode link in nodes)
{
    string completeUrl = link.Attributes["href"].Value; // The complete plugin download url.

    #region Get all jars

    if (completeUrl.Contains(".jar")) // Check if the url contains .jar
    {
        fileNameWithExt = completeUrl.Substring(completeUrl.LastIndexOf('/') + 1); // Get the filename with extension.
        fileName = fileNameWithExt.Remove(fileNameWithExt.LastIndexOf('.')); ; // Get the filename without extension.
        Console.WriteLine(fileName);
    }

    #endregion

    #region Get all Authors

    if (completeUrl.Contains("?author=")) // Check if the url contains .jar
    {
        authorName = completeUrl.Substring(completeUrl.LastIndexOf('=') + 1); // Get the filename with extension.
        Console.WriteLine(authorName);
    }

    #endregion
}

I am trying to get all the filenames and authors next to each other, but now everything is like randomly placed, why?

Can someone help me with this? Thanks!

Upvotes: 1

Views: 796

Answers (1)

Jeff Mercado
Jeff Mercado

Reputation: 134811

If you look at the HTML, it's very unfortunate it is not well-formed. There's a lot of open tags and the way HAP structures it is not like a browser, it interprets the majority of the document as deeply nested. So you can't just simply iterate through the rows of the table like you would in the browser, it gets a lot more complicated than that.

When dealing with such documents, you have to change your queries quite a bit. Rather than searching through child elements, you have to search through descendants adjusting for the change.

var title = System.Web.HttpUtility.UrlEncode(textBox1.Text);
var url = String.Format("http://p.thedgtl.net/index.php?title={0}", title);

var web = new HtmlWeb();
var doc = web.Load(url);

// select the rows in the table
var xpath = "//div[@class='content']/div[@class='pluginList']/table[2]";
var table = doc.DocumentNode.SelectSingleNode(xpath);

// unfortunately the `tr` tags are not closed so HAP interprets
// this table having a single row with multiple descendant `tr`s
var rows = table.Descendants("tr")
    .Skip(1); // skip header row

var query =
    from row in rows
    // there may be a row with an embedded ad
    where row.SelectSingleNode("td/script") == null
    // each row has 6 columns so we need to grab the next 6 descendants
    let columns = row.Descendants("td").Take(6).ToList()
    let titleText = columns[1].Elements("a").Select(a => a.InnerText).FirstOrDefault()
    let authorText = columns[2].Elements("a").Select(a => a.InnerText).FirstOrDefault()
    let downloadLink = columns[5].Elements("a").Select(a => a.GetAttributeValue("href", null)).FirstOrDefault()
    select new
    {
        Title = titleText ?? "",
        Author = authorText ?? "",
        FileName = Path.GetFileName(downloadLink ?? ""),
    };

So now you can just iterate through the query and write out what you want for each of the rows.

foreach (var item in query)
{
    Console.WriteLine("{0} ({1})", item.FileName, item.Author);
}

Upvotes: 1

Related Questions