CODEWITHSUNDEEP
c#httpwebrequesthttpwebresponse
Jaume
Jaume

Reputation: 3780

C# get html from url. Error (429) unknown

I am using the following code to retrieve the html code from a url. It is working fine for a url as:

"The remote server returned an error: (429) unknown.'"

What could be the error or how to get more info about the error?

private void getHtmlFromUrl() {

    string urlAddress = "https://trends.google.es/trends/explore?q=test&geo=US";

    HttpWebRequest request = (HttpWebRequest)WebRequest.Create(urlAddress);
    HttpWebResponse response = (HttpWebResponse)request.GetResponse();

    if (response.StatusCode == HttpStatusCode.OK)
    {
        Stream receiveStream = response.GetResponseStream();
        StreamReader readStream = null;

        if (String.IsNullOrWhiteSpace(response.CharacterSet))
            readStream = new StreamReader(receiveStream);
        else
            readStream = new StreamReader(receiveStream, Encoding.GetEncoding(response.CharacterSet));

        string htmlData = readStream.ReadToEnd();

        response.Close();
        readStream.Close();
    }

}

Upvotes: 0

Views: 827

Answers (2)

Moshe levy
Moshe levy

Reputation: 1

I highly recommend the GoogleTrendsApi package, which allows you to connect to this Google API, and receive data from it in a really simple way, certainly compared to the way of integrating Python into your code.

Full disclosure: I am its creator.

Upvotes: 0

Mohammed Amine Belbachir
Mohammed Amine Belbachir

Reputation: 732

It's no longer possible to get the source code of google services without an API because specifically the trends service makes many calls when you visit only trends.google.es/trends/explore?q=test hence the error 429.

Do not waste your time digging in proxies, browser emulation or bots none will work. The best way is to use Google API for c# .

Example Projects:

https://github.com/thegreymatter/GoogleTrends (Deprecated)

https://developers.google.com/api-client-library/dotnet/guide/aaa_oauth (Deprecated)

New Solution!

(We install python then convert our py script to an exe file to use from .net code. The good news is no api is needed with this method)

1- Install Python: https://www.python.org/downloads/windows/

2- Install Pytrends using:

pip install pytrends

3- Create a test.py and add support to arguments:

from pytrends.request import TrendReq
import sys

# Only need to run this once, the rest of requests will use the same session.
pytrend = TrendReq()
# Set your region
pytrend.geo = 'ES'
pytrend.build_payload(kw_list=[sys.argv[1]]) # Also supports an array of keywords e.g. kw_list=['test', 'music']
related_queries_dict = pytrend.related_queries()
print(related_queries_dict)
#Check the documentation for other available queries https://github.com/GeneralMills/pytrends#api-methods

Now if you cmd.exe (do not use PowerShell command won't work there) in the same location of test.py use the following command:

test.py test

Screenshot Results output of test.py test

4- Now let's convert test.py to an .exe (windows console file)

(a) Install Pyinstaller:

pip install pyinstaller

(b) Compile test.py to test.exe

pyinstaller test.py -F

5- Your "test.exe" is inside \dist\ directory. Let's see if it works:

test.exe test output

Yes it does. (Keep in mind that the filesize of test.exe is 80 mb since it employs many libraries.)

Now all you have left to do is Process.Start the "test.exe" filename with argument "test" to read the output from your csharp app.

You can use IronPython instead to make python commands through your csharp code

Upvotes: 1

Related Questions