Design pattern for dynamic C# object

Question

I have a queue that processes objects in a while loop. They are added asynchronously somewhere.. like this:

myqueue.pushback(String value);

And they are processed like this:

while(true)
{
    String path =  queue.pop();
    if(process(path))
    {
        Console.WriteLine("Good!");
    }
    else
    {
        queue.pushback(path);
    }
}

Now, the thing is that I'd like to modify this to support a TTL-like (time to live) flag, so the file path would be added o more than n times.

How could I do this, while keeping the bool process(String path) function signature? I don't want to modify that.

I thought about holding a map, or a list that counts how many times the process function returned false for a path and drop the path from the list at the n-th return of false. I wonder how can this be done more dynamically, and preferably I'd like the TTL to automatically decrement itself at each new addition to the process. I hope I am not talking trash. Maybe using something like this

class JobData
{
   public string path;
   public short ttl;

   public  static implicit operator String(JobData jobData) {jobData.ttl--; return jobData.path;}
}

Answer 1

I like the idea of a JobData class, but there's already an answer demonstrating that, and the fact that you're working with file paths give you another possible advantage. Certain characters are not valid in file paths, and so you could choose one to use as a delimiter. The advantage here is that the queue type remains a string, and so you would not have to modify any of your existing asynchronous code. You can see a list of reserved path characters here:

http://en.wikipedia.org/wiki/Filename#Reserved_characters_and_words

For our purposes, I'll use the percent (%) character. Then you can modify your code as follows, and nothing else needs to change:

const int startingTTL = 100;
const string delimiter = "%";

while(true)
{
    String[] path =  queue.pop().Split(delimiter.ToCharArray());
    int ttl = path.Length > 1?--int.Parse(path[1]):startingTTL;

    if(process(path[0]))
    {
        Console.WriteLine("Good!");
    }
    else if (ttl > 0)
    {
        queue.pushback(string.Format("{0}{1}{2}", path[0], delimiter,ttl));             
    }
    else
    {
        Console.WriteLine("TTL expired for path: {0}" path[0]);
    }
}

Again, from a pure architecture standpoint, a class with two properties is a better design... but from a practical standpoint, YAGNI: this option means you can avoid going back and changing other asynchronous code that pushes into the queue. That code still only needs to know about the strings, and will work with this unmodified.

One more thing. I want to point out that this is a fairly tight loop, prone to running away with a cpu core. Additionally, if this is the .Net queue type and your tight loop gets ahead of your asynchronous produces to empty the queue, you'll throw an exception, which would break out of the while(true) block. You can solve both issues with code like this:

while(true)
{

    try 
    {
        String[] path =  queue.pop().Split(delimiter.ToCharArray());
        int ttl = path.Length > 1?--int.Parse(path[1]):startingTTL;

        if(process(path[0]))
        {
            Console.WriteLine("Good!");
        }
        else if (ttl > 0)
        {
            queue.pushback(string.Format("{0}{1}{2}", path[0], delimiter,ttl));             
        }
        else
        {
            Console.WriteLine("TTL expired for path: {0}" path[0]);
        }
    }
    catch(InvalidOperationException ex)
    {
        //Queue.Dequeue throws InvalidOperation if the queue is empty... sleep for a bit before trying again
        Thread.Sleep(100);
    }
}

Answer 2

If the constraint is that bool process(String path) cannot be touched/changed then put the functionality into myqueue. You can keep its public signatures of void pushback(string path) and string pop(), but internally you can track your TTL. You can either wrap the string paths in a JobData-like class that gets added to the internal queue, or you can have a secondary Dictionary keyed by path. Perhaps even something as simple as saving the last poped path and if the subsequent push is the same path you can assume it was a rejected/failed item. Also, in your pop method you can even discard a path that has been rejected too many time and internally fetch the next path so the calling code is blissfully unaware of the issue.

Answer 3

You could abstract/encapsulate the functionality of the "job manager". Hide the queue and implementation from the caller so you can do whatever you want without the callers caring. Something like this:

public static class JobManager
{
    private static Queue<JobData> _queue;

    static JobManager() { Task.Factory.StartNew(() => { StartProcessing(); }); }

    public static void AddJob(string value)
    {
        //TODO: validate

        _queue.Enqueue(new JobData(value));
    }

    private static StartProcessing()
    {
        while (true)
        {
            if (_queue.Count > 0)
            {
                JobData data = _queue.Dequeue();
                if (!process(data.Path))
                {
                    data.TTL--;
                    if (data.TTL > 0)
                        _queue.Enqueue(data);
                }
            }
            else
            {
                Thread.Sleep(1000);
            }
        }
    }

    private class JobData
    {
        public string Path { get; set; }
        public short TTL { get; set; }

        public JobData(string value)
        {
            this.Path = value;
            this.TTL = DEFAULT_TTL;
        }
    }

}

Then your processing loop can handle the TTL value.

Edit - Added a simple processing loop. This code isn't thread safe, but should hopefully give you an idea.