CODEWITHSUNDEEP
c#xml
Luke4792
Luke4792

Reputation: 455

Can I speed up my code for downloading XML pages?

I have a program that needs to download ~200 XML pages of ~250 lines. It does this by grabbing a list of ID's from a main page, and then it then iterates through each ID, inserting it into a URL and using a StreamWriter to write the XML page to a file. The methods look like this:

    private static void XMLUpdate(string path)
    {
        try
        {
            //create client for downloading XMLs
            var client = new WebClient();

            foreach (var node in GetId())
            {
                //stores xml text into a string
                var s = client.DownloadString("https://example" + node + "/xml");

                // assign to the output
                var file = new StreamWriter(path + "\\" + node + ".xml");
                file.WriteLine(s);
                file.Close();
            }
        }
        catch (Exception ex)
        {
            MessageBox.Show(ex.Message);
        }
    }

    private static string[] GetId()
    {
        var idList = new List<string>();
        var request = WebRequest.Create("https://example/xml");

        var i = 0;

        using (var response = request.GetResponse())
        using (var xmlReader = XmlReader.Create(response.GetResponseStream()))
        {
            while (xmlReader.Read())
            {
                xmlReader.ReadToFollowing("n" + i);
                //go through each of the n nodes in the xmldocument and get the name and id
                if (xmlReader.NodeType != XmlNodeType.Element || xmlReader.Name != "n" + i) continue;
                xmlReader.ReadToFollowing("id");

                if (xmlReader.NodeType == XmlNodeType.Element && xmlReader.Name == "id")
                {
                    var id = xmlReader.ReadElementContentAsString();
                    idList.Add(id);
                }

                i += 1;
            }
        }

        var IDs = idList.ToArray();

        return IDs;
    }

At the moment the program takes a long time to download everything I need. Is there something I can do to speed up the process?

Upvotes: 0

Views: 55

Answers (1)

Patrick Hofman
Patrick Hofman

Reputation: 157098

Yes, you can run things in parallel, by using Parallel.ForEach:

Parallel.ForEach
( GetId()
, new ParallelOptions() { MaxDegreeOfParallelism = 32 } // run 32 in parallel
, node =>
  {
    //stores xml text into a string
    var client = new WebClient();
    var s = client.DownloadString("https://example" + node + "/xml");

    // assign to the output
    var file = new StreamWriter(path + "\\" + node + ".xml");
    file.WriteLine(s);
    file.Close();
  }
);

You can tweak the MaxDegreeOfParallelism as you want and your service can handle.

Upvotes: 1

Related Questions