Xml simplification/extraction of distinct values - possible LINQ

Question

Sorry for this long post....But i have a headache from this task.

I have a mile long xml document where I need to extract a list, use distinct values, and pass for transformation to web.

I have completed the task using xslt and keys, but the effort is forcing the server to its knees.

Description: hundreds of products in xml, all with a number of named and Id'ed cattegories, all categories with at least one subcategory with name and id.

The categories are unique with ID, all subcategories are unique WITHIN that category:

Simplified example form the huge file (left our tons of info irrelevant to the task):

<?xml version="1.0" encoding="utf-8"?>
<root>
<productlist>
<product id="1">
<name>Some Product</name>
<categorylist>
<category id="1">
<name>cat1</name>
<subcategories>
<subcat id="1">
<name>subcat1</name>
</subcat>
<subcat id="2">
<name>subcat1</name>
</subcat>
</subcategories>
</category>
<category id="2">
<name>cat1</name>
<subcategories>
<subcat id="1">
<name>subcat1</name>
</subcat>
</subcategories>
</category>
<category id="3">
<name>cat1</name>
<subcategories>
<subcat id="1">
<name>subcat1</name>
</subcat>
</subcategories>
</category>
</categorylist>
</product>
<product id="2">
<name>Some Product</name>
<categorylist>
<category id="1">
<name>cat1</name>
<subcategories>
<subcat id="2">
<name>subcat2</name>
</subcat>
<subcat id="4">
<name>subcat4</name>
</subcat>
</subcategories>
</category>
<category id="2">
<name>cat2</name>
<subcategories>
<subcat id="1">
<name>subcat1</name>
</subcat>
</subcategories>
</category>
<category id="3">
<name>cat3</name>
<subcategories>
<subcat id="1">
<name>subcat1</name>
</subcat>
</subcategories>
</category>
</categorylist>
</product>
</productlist>
</root>

DESIRED RESULT:

<?xml version="1.0" encoding="utf-8"?>
<root>
<maincat id="1">
<name>cat1</name>
<subcat id="1"><name>subcat1</name></subcat>
<subcat id="2"><name>subcat2</name></subcat>
<subcat id="3"><name>subcat3</name></subcat>
</maincat>
<maincat id="2">
<name>cat2</name>
<subcat id="1"><name>differentsubcat1</name></subcat>
<subcat id="2"><name>differentsubcat2</name></subcat>
<subcat id="3"><name>differentsubcat3</name></subcat>
</maincat>
<maincat id="2">
<name>cat2</name>
<subcat id="1"><name>differentsubcat1</name></subcat>
<subcat id="2"><name>differentsubcat2</name></subcat>
<subcat id="3"><name>differentsubcat3</name></subcat>
</maincat>
</root>

(original will from 2000 products produce 10 categories with from 5 to 15 subcategories)

Things tried:

1. Xslt with keys - works fine, but pooooor performance

2. Played around with linq:

          IEnumerable<XElement> mainCats =
                   from Category1 in doc.Descendants("product").Descendants("category") select Category1;
   
               var cDoc = new XDocument(new XDeclaration("1.0", "utf-8", null), new XElement("root"));
               cDoc.Root.Add(mainCats);
               cachedCategoryDoc = cDoc.ToString();
   

   Result was a "categories only" (not distinct values of categories or subcategories)

Applied the same xlst to that, and got fairly better performance..... but still far from usable...

Can i apply some sort of magic with the linq statement to have the desired output??

A truckload of good karma goes out to the ones that can point me in det right direction..

//Steen

NOTE:

• I am not stuck on using linq/XDocument if anyone has better options
• Currently on .net 3.5, can switch to 4 if needed

Answer 1

This will parse your xml into a dictionary of categories with all the distinct subcategory names. It uses XPath from this library: https://github.com/ChuckSavage/XmlLib/

XElement root = XElement.Load(file);
string[] cats = root.XGet("//category/name", string.Empty).Distinct().ToArray();
Dictionary<string, string[]> dict = new Dictionary<string, string[]>();
foreach (string cat in cats)
{
    // Get all the categories by name and their subcat names
    string[] subs = root
        .XGet("//category[name={0}]/subcategories/subcat/name", string.Empty, cat)
        .Distinct().ToArray();
    dict.Add(cat, subs);
}

Or the parsing as one statement:

Dictionary<string, string[]> dict = root
    .XGet("//category/name", string.Empty)
    .Distinct()
    .ToDictionary(cat => cat, cat => root
        .XGet("//category[name={0}]/subcategories/subcat/name", string.Empty, cat)
        .Distinct().ToArray());

I give you the task of assembling your resulting xml from the dictionary.

Answer 2

If I understood your question corectly, here's a LINQ atempt.

The query below parses your XML data and creates a custom type which represents a category and contains the subcategories of that element.

After parsing, the data is grouped by category Id to get distinct subcategories for each category.

var doc = XElement.Load("path to the file");
var results = doc.Descendants("category")
    .Select(cat => new
    {
        Id = cat.Attribute("id").Value,
        Name = cat.Descendants("name").First().Value,
        Subcategories = cat.Descendants("subcat")
            .Select(subcat => new
            {
                Id = subcat.Attribute("id").Value,
                Name = subcat.Descendants("name").First().Value
            })
     })
     .GroupBy(x=>x.Id)
     .Select(g=>new
     {
         Id = g.Key,
         Name = g.First().Name,
         Subcategories = g.SelectMany(x=>x.Subcategories).Distinct()
     });

From the results above you can create your document using the code below:

var cdoc = new XDocument(new XDeclaration("1.0", "utf-8", null), new XElement("root")); 
cdoc.Root.Add(
    results.Select(x=>
    {
        var element = new XElement("maincat", new XAttribute("id", x.Id));
        element.Add(new XElement("name", x.Name));
        element.Add(x.Subcategories.Select(c=>
        {
            var subcat = new XElement("subcat", new XAttribute("id", c.Id));
            subcat.Add(new XElement("name", c.Name));
            return subcat;
        }).ToArray());
        return element;
    }));

Answer 3

Try this i have done something for it.. attributes are missing you can add them using XElement ctor

 var doc = XDocument.Load(reader);
                    IEnumerable<XElement> mainCats =
                        doc.Descendants("product").Descendants("category").Select(r =>
                            new XElement("maincat", new XElement("name", r.Element("name").Value),
                                r.Descendants("subcat").Select(s => new XElement("subcat", new XElement("name", s.Element("name").Value)))));


                    var cDoc = new XDocument(new XDeclaration("1.0", "utf-8", null), new XElement("root"));
                    cDoc.Root.Add(mainCats);
                    var cachedCategoryDoc = cDoc.ToString();

Regards.