HTML DOM not working with JavaScript page

Question

Getting HTML DOM from websites that work with client-side (JavaScript)

Answer 1

It doesn't work because

<div class="currTableContainer" id="divCurrTableContainer">

is empty. There is nothing in it. You wont find a table in it. Hence the nullpointer. The HTML you're looking for is generated by a script(ajax).

Check the loaded HTML to confirm.

Because I was bored I decided to finish your homework. You can't use HAP in this case. You have to use the ajax service they provided. Also better since the data is prone to change a lot. The code that is used on the site(ajax script) looks like this:

$("document").ready(function () {



    $.ajax({
        type: "POST",
        url: '/_layouts/15/LINKDev.CIB.CurrenciesFunds/FundsCurrencies.aspx/GetCurrencies',
        async: true,
        data: "{'lang':'" + document.getElementById("ctl00_ctl48_g_5d7fc52f_a66d_4aa2_8d6c_c01fb4b38cb2_hdnLang").value + "'}",
        contentType: "application/json; charset=utf-8",
        dataType: "json",
        success: function (msg) {
            if (msg.d != null && msg.d.length > 0) {
                var contentHTML = "<table class='currTable' cellspacing='0' rules='all' style='border-width:0px;border-collapse:collapse;'>"
                        + "<tbody><tr class='currHeaderRow'>"
                            + "<th scope='col'>" + document.getElementById("ctl00_ctl48_g_5d7fc52f_a66d_4aa2_8d6c_c01fb4b38cb2_hdnCurrency").value + "</th><th scope='col'>" + document.getElementById("ctl00_ctl48_g_5d7fc52f_a66d_4aa2_8d6c_c01fb4b38cb2_hdnBuy").value + "</th><th scope='col'>" + document.getElementById("ctl00_ctl48_g_5d7fc52f_a66d_4aa2_8d6c_c01fb4b38cb2_hdnSell").value + "</th>"
                        + "</tr>";

                for (var i = 0; i < msg.d.length; i++) {
                    if (msg.d[i].CurrencyID.length > 0) {
                        contentHTML += "<tr class='currRow'>"
                                + "<td>" + msg.d[i].CurrencyID + "</td><td>" + msg.d[i].BuyRate + "</td><td class='lastCell'>" + msg.d[i].SellRate + "</td>"
                            + "</tr>";
                    }
                }
                contentHTML += "</tbody></table>";
                $("#divCurrTableContainer").html(contentHTML);
                if ($(".bannerElements").length > 0)
                    FixCurrenciesRatesScroll();
            }
        },
        error: function (msg) {
        }
    });


});

As you can see the script uses an URL from a different part of the site to update their currency. In order to fetch it you just make http request with the following JSON {"lang":"en"}. I have translated the script into it's equivalent in C# below. The response from the http request will be JSON formatted. Which means you will have to create a class that can be used to serialize the data. I recommend you look at Newtonsoft for this since I can't do all your homework.

using System.Text;
using System.Threading.Tasks;
using System.IO;
using System.Net;

namespace test
{
    class Program
    {
    public static void Main(string[] args)
    {
        try
        {
            string webAddr = "http://www.cibeg.com/_layouts/15/LINKDev.CIB.CurrenciesFunds/FundsCurrencies.aspx/GetCurrencies";

            var httpWebRequest = (HttpWebRequest)WebRequest.Create(webAddr);
            httpWebRequest.ContentType = "application/json";
            httpWebRequest.Method = "POST";
            httpWebRequest.Expect = "application/json";

            string datastring = "{\"lang\":\"en\"}";
            ASCIIEncoding encoder = new ASCIIEncoding();
            byte[] data = encoder.GetBytes(datastring);

            httpWebRequest.ContentLength = data.Length;
            httpWebRequest.GetRequestStream().Write(data, 0, data.Length);

            var httpResponse = (HttpWebResponse)httpWebRequest.GetResponse();
            using (var streamReader = new StreamReader(httpResponse.GetResponseStream()))
            {
                var responseText = streamReader.ReadToEnd();
                Console.WriteLine(responseText);

                //Now you have your response.
                //or false depending on information in the response     
            }
        }
        catch (WebException ex)
        {
            Console.WriteLine(ex.Message);
        }
        Console.ReadKey();
    }