Extracting values and labels from a webpage table that is made up of divs using hmtlagilitypack

Question

There is a webpage in which I want to extract the labels and values that are in a table, however the table is not conveniently defined with the usual "tr" and "td" html tags, but instead it uses "div class.." I cannot seem to find code examples for this case. The following is a snippet of the HTML code that makes the table I am after.

<div ng-controller="stockDetailSummary" ng-cloak>
    <div class="stockDetail" ng-show="quoteFound">
        <div ng-show="DataLoaded">

             <h2>Stock Scorecard</h2>
                <div class="Column Left">
            <div class="row">
                <div class="label">Market Cap</div>
                <div class="value" ng-if="quoteObj.currentMarketCap">{{quoteObj.currentMarketCap}}</div>
                <div class="value" ng-if="!quoteObj.currentMarketCap">- -</div>
            </div>
            <div class="row alt">
                <div class="label">Yield</div>
                <div class="value" ng-if="quoteObj.lastDividentReported">{{quoteObj.lastDividentReported | number:2 }}%<span ng-if="quoteObj.yield">%</span></div>
                <div class="value" ng-if="!quoteObj.lastDividentReported">- -</div>
            </div>  
            <div class="row">
                <div class="label">Quarterly Dividend</div>
                <div class="value" ng-if="quoteObj.dividend">{{quoteObj.dividend | number:2 }}</div>
                <div class="value" ng-if="!quoteObj.dividend">- -</div>
            </div>
            <div class="row alt">
                <div class="label">Open</div>
                <div class="value" ng-if="quoteObj.open">{{quoteObj.open | number:2 }}</div>
                <div class="value" ng-if="!quoteObj.open">- -</div>
            </div>
            <div class="row">
                <div class="label">High</div>
                <div class="value" ng-if="quoteObj.high">{{quoteObj.high | number:2 }}</div>
                <div class="value" ng-if="!quoteObj.high">- -</div>
            </div>  
            <div class="row alt">
                <div class="label">Low</div>
                <div class="value" ng-if="quoteObj.low">{{quoteObj.low | number:2 }}</div>
                <div class="value" ng-if="!quoteObj.low">- -</div>
            </div>
            <div class="row">
                <div class="label">Previous Close</div>
                <div class="value" ng-if="quoteObj.previousClosePrice">{{quoteObj.previousClosePrice | number:2}}</div>
                <div class="value" ng-if="!quoteObj.previousClosePrice">- -</div>
            </div>
            <div class="row alt">
                <div class="label">Volume</div>
                <div class="value" ng-if="quoteObj.totalVolume">{{quoteObj.totalVolume | number}}</div>
                <div class="value" ng-if="!quoteObj.totalVolume">- -</div>
            </div>
            <div class="row">
                <div class="label">52-Week High</div>
                <div class="value" ng-if="quoteObj.yearHigh">{{quoteObj.yearHigh | number:2 }}</div>
                <div class="value" ng-if="!quoteObj.yearHigh">- -</div>
            </div>
            <div class="row alt">
                <div class="label">52-Week Low</div>
                <div class="value" ng-if="quoteObj.yearLow">{{quoteObj.yearLow | number:2 }}</div>
                <div class="value" ng-if="!quoteObj.yearLow">- -</div>
            </div>
        </div>
        <div class="Column Right">
            <div class="row">
                <div class="label">1 Year Total % Return</div>
                <div class="value" ng-if="quoteObj.oneYearReturn">{{quoteObj.oneYearReturn | number:2}}</div>
                <div class="value" ng-if="!quoteObj.oneYearReturn">- -</div>
            </div>
            <div class="row alt">
                <div class="label">Earnings Per Share</div>
                <div class="value" ng-if="quoteObj.eps">{{quoteObj.eps | number:2 }}</div>
                <div class="value" ng-if="!quoteObj.eps">- -</div>
            </div>
            <div class="row">
                <div class="label">Price/Earnings Ratio</div>
                <div class="value"ng-if="quoteObj.peRatio">{{quoteObj.peRatio | number:2 }}</div>
                <div class="value" ng-if="!quoteObj.peRatio">- -</div>
            </div>
            <div class="row alt">
                <div class="label">Price-to-Book Ratio</div>
                <div class="value" ng-if="quoteObj.priceToBookRatio">{{quoteObj.priceToBookRatio | number:2}}</div>
                <div class="value" ng-if="!quoteObj.priceToBookRatio">- -</div>
            </div>
            <div class="row">
                <div class="label">Price-to-Sales Ratio</div>
                <div class="value" ng-if="quoteObj.priceToSalesRatio">{{quoteObj.priceToSalesRatio | number:2}}</div>
                <div class="value" ng-if="!quoteObj.priceToSalesRatio">- -</div>
            </div>
            <div class="row alt">
                <div class="label">Shares Outstanding</div>
                <div class="value" ng-if="quoteObj.shareOutstanding">{{quoteObj.shareOutstanding | number:2}} M</div>
                <div class="value" ng-if="!quoteObj.shareOutstanding">- -</div>
            </div>
            <div class="row">
                <div class="label">30 Day Average Volume</div>
                <div class="value" ng-if="quoteObj.thirtyDayAvgVolume">{{quoteObj.thirtyDayAvgVolume}}</div>
                <div class="value" ng-if="!quoteObj.thirtyDayAvgVolume">- -</div>
            </div>
            <div class="row alt">
                <div class="label">Exchange</div>
                <div class="value" ng-if="quoteObj.exchange">{{quoteObj.exchange}}</div>
                <div class="value" ng-if="!quoteObj.exchange">- -</div>
            </div>
            <div class="row">
                <div class="label">Currency</div>
                <div class="value" ng-if="quoteObj.currency">{{quoteObj.currency}}</div>
                <div class="value" ng-if="!quoteObj.currency">- -</div>
            </div>
            <div class="row alt">
                <div class="label">Expected Reporting Date</div>
                <div class="value" ng-if="quoteObj.expectedReportDate">{{quoteObj.expectedReportDate | date:'MMM d, yyyy'}}</div>
                <div class="value" ng-if="!quoteObj.expectedReportDate">- -</div>
            </div>
        </div>
        <div class="clear"></div>
    </div>      
    </div>
    <div class="not-found" ng-show="!quoteFound">
        <p>There was a problem retrieving the data</p>
    </div>
</div>

You can also view it at Bloomberg Stock Quote for ENB:CT

I found the xpath for the table I am after, and tried the following code:

            HtmlAgilityPack.HtmlWeb web = new HtmlAgilityPack.HtmlWeb();
            HtmlAgilityPack.HtmlDocument jkdoc = web.Load("https://www.bnnbloomberg.ca/stock/ENB:CT");
            HtmlNode jnode = jkdoc.DocumentNode.SelectSingleNode("/html[1]/body[1]/div[3]/div[3]/section[1]/div[3]/div[2]/div[3]/div[1]/div[1]/div[1]/div[1]/div[1]/table[1]");
            string value = jnode.InnerText;
 Console.WriteLine(value);

The string shown on the screen contains the labels for the various sections, but no actual values. Instead I get something like {{revenue | replaceEmptyValue}}

Is this some sort of timing issue where I have to wait for the page to fully load and populate the values?

Answer 1

The table seems to just be a template for the Angular JavaScript framework. That means the relevant scripts needs to be executed for it to actually be properly rendered and filled with the data. HtmlAgilityPack will only parse the HTML it directly receives from the server, meaning the unrendered template markup.

This question basically comes down to the same thing, and might help you in the right direction.