How do I get a Wyam pipeline of documents based of a comma-delimited meta value from a previous pipeline?

Question

I have a Wyam pipeline called "Posts" filled with documents. Some of these documents have a Tags meta value, which is a comma-delimited list of tags. For example, let's say it has three documents, with Tags meta of:

gumby,pokey
gumby,oscar
oscar,kermit

I want a new pipeline filled with one document for each unique tag found in all documents in the "Posts" pipeline. These documents should have the tag in a meta value called TagName.

So, the above values should result in a new pipeline consisting of four documents, with the TagName meta values of:

gumby
pokey
oscar
kermit

Here is my solution. This technically works, but I feel like it's inefficient, and I'm pretty sure there has to be a better way.

Documents(c => c.Documents["Posts"]
    .Select(d => d.String("Tags", string.Empty))
    .SelectMany(s => s.Split(",".ToCharArray()))
    .Select(s => s.Trim().ToLower())
    .Distinct()
    .Select(s => c.GetNewDocument(
        string.Empty,
        new List<KeyValuePair<string, object>>()
        {
            new KeyValuePair<string, object>("TagName", s)
        }
     ))
 )

So, I'm calling Documents and passing in a ContextConfig which:

1. Gets the documents from "Posts" (I have a collection of documents)
2. Selects the Tags meta value (now I have a collection of strings)
3. Splits this on the comma (a bigger collection of strings)
4. then trims and lower cases (still a collection of strings)
5. De-dupes it (a smaller collection of strings)
6. Then creates a new document for each value in the list, with am empty body and a TagName value for the string (I should end up with a collection of new documents)

Again, this works. But is there a better way?

Answer 1

That's actually not bad at all - part of the challenge here is getting the comma-separated list of tags into something that can be processed by a LINQ expression or similar. That part is probably unavoidable and accounts for 3 of the lines in your expression.

That said, Wyam does provide a little help here with the ToLookup() extension (see the bottom of this page: http://wyam.io/getting-started/concepts).

Here's how that might look (this code is from a self-contained LINQPad script and would need to be adjusted for use in a Wyam config file):

public void Main()
{   
    Engine engine = new Engine();
    engine.Pipelines.Add("Posts",
        new PostsDocuments(),
        new Meta("TagArray", (doc, ctx) => doc.String("Tags")
            .ToLowerInvariant().Split(',').Select(x => x.Trim()).ToArray())
    );
    engine.Pipelines.Add("Tags",
        new Documents(ctx => ctx.Documents["Posts"]
            .ToLookup<string>("TagArray")
            .Select(x => ctx.GetNewDocument(new MetadataItems { { "TagName", x.Key } }))),
        new Execute((doc, ctx) =>
        {
            Console.WriteLine(doc["TagName"]);
            return null;
        })
    );
    engine.Execute();
}

public class PostsDocuments : IModule
{
    public IEnumerable<IDocument> Execute(IReadOnlyList<IDocument> inputs, IExecutionContext context)
    {
        yield return context.GetNewDocument(new MetadataItems { { "Tags", "gumby,pokey" } });
        yield return context.GetNewDocument(new MetadataItems { { "Tags", "gumby,oscar" } });
        yield return context.GetNewDocument(new MetadataItems { { "Tags", "oscar,kermit" } });
    }
}

Output:

gumby
pokey
oscar
kermit

A lot of that is just housekeeping to set up the fake environment for testing. The important part that you're looking for is this:

engine.Pipelines.Add("Tags",
    new Documents(ctx => ctx.Documents["Posts"]
        .ToLookup<string>("TagArray")
        .Select(x => ctx.GetNewDocument(new MetadataItems { { "TagName", x.Key } }))),
    // ...
);

Note that we still have to do the work of getting the comma delimited tags list into an array - it's just happening earlier up in the "Posts" pipeline.