Combine LINQ queries

Question

I'm in the process of making a small query thing for a set of results of files.

public class f_results
    {
        public String name { get; set; }
        public DateTime cdate { get; set; }
        public DateTime mdate { get; set; }
        public DateTime adate { get; set; }
        public Int64 size { get; set; }
    }

I have a screen on which users can select what they want. At the moment I go through a filter system:

    foundfiles = new BindingList<f_results>(totalresults.Find(fname.Text,true));
    if (fsize.Text.Trim() != "")
    {
        try
        {
            Int64 sz = Int64.Parse(fsize.Text);
            List<f_results> y = (from p in foundfiles where p.size >= sz orderby p.size descending select p  ).ToList();
            foundfiles = new BindingList<f_results>(y);
        }
        catch
        { }
    }
    if (adate.Text.Trim() != "")
    {
        try
        {
            List<f_results> y;
            DateTime test = DateTime.Parse(adate.Text);
            if ((adateop.Text) == ">")
            {
                y = (from p in foundfiles where p.adate >= test select p).ToList();
            }
            else
                y = (from p in foundfiles where p.adate <= test select p).ToList();
            foundfiles = new BindingList<f_results>(y);
        }
        catch
        { }
    }

    if (mdate.Text.Trim() != "")
    {
        try
        {
            List<f_results> y;
            DateTime test = DateTime.Parse(mdate.Text);
            if ((mdateop.Text) == ">")
            {
                y = (from p in foundfiles where p.mdate >= test select p).ToList();
            }
            else
                y = (from p in foundfiles where p.mdate <= test select p).ToList();
            foundfiles = new BindingList<f_results>(y);
        }
        catch
        { }
    }

    if (cdate.Text.Trim() != "")
    {
        try
        {
            List<f_results> y;
            DateTime test = DateTime.Parse(cdate.Text);
            if ((cdateop.Text) == ">")
            {
                y = (from p in foundfiles where p.cdate >= test select p).ToList();
            }
            else
                y = (from p in foundfiles where p.cdate <= test select p).ToList();
            foundfiles = new BindingList<f_results>(y);
        }
        catch
        { }
    }

At the end, I have my results the way I want them, but I'm looking to process about 72 TB of file data so, there are plenty of files and plenty of directories in my list (totalresults is a type results, which contains a list of files (f_results) and directories (results).. Find then iterates down and returns a massive list of f_results which match a given regex.

Is there a way to make my LINQ query one query? Given not all options maybe used, e.g. they may just want files > x, or not used since.. or.. etc.

I did consider making flags for the test, and so on, as it's the test part that's the most important.. is that the better way, or is there better? Or does it not matter much in the wash?

Answer 1

At least I suggest to remove the .ToList() invokes everywhere. As LINQ has a deferred invocation, it will iterate once, even you have:

var foundfiles = from p in foundfiles where p.size >= sz select p ;
foundfiles = from p in foundfiles where p.mdate >= test select p

Update (in that case order by should be put after all filters)

But if you write:

var foundfiles = (from p in foundfiles where p.size >= sz orderby p.size descending select p).ToList() ;
foundfiles = (from p in foundfiles where p.mdate >= test select p).ToList();

It will iterate two times - and that might be a serious performance problem.

But I don't think the code will look much simpler if you make this code as a single query.

Also, why are you catching all exceptions? You shouldn't do that.

Answer 2

OK - I wanted to semi answer my own question..

I can combine to a single query, the following works very well.. Ideal? Possibly not!

Am going to now look at BrokenGlass' suggestion which looks nice and tidy!

   Boolean flag_size = false;
    Boolean flag_adate = false;
    Boolean flag_cdate = false;
    Boolean flag_mdate = false;
    Int64 sz=0;
    DateTime adatetest=DateTime.Now;
    DateTime cdatetest = DateTime.Now;
    DateTime mdatetest = DateTime.Now;
    String mop = mdateop.Text;
    String aop = adateop.Text;
    String cop = cdateop.Text;

    if (fsize.Text.Trim() != "")
    {
        try
        {
            sz = Int64.Parse(fsize.Text);
            flag_size = true;
        }
        catch { }
    }

    if (adate.Text.Trim() != "")
    {
        try
        {
            adatetest = DateTime.Parse(adate.Text);
            flag_adate = true;
        }
        catch
        { }
    }
    if (cdate.Text.Trim() != "")
    {
        try
        {
            cdatetest = DateTime.Parse(cdate.Text);
            flag_cdate = true;
        }
        catch
        { }
    }
    if (mdate.Text.Trim() != "")
    {
        try
        {
            mdatetest = DateTime.Parse(mdate.Text);
            flag_mdate = true;
        }
        catch
        { }
    }


    foundfiles = new BindingList<f_results>(totalresults.Find(fname.Text, true));


            List<f_results> y = (from p in foundfiles.AsParallel() 
                                 where  (!flag_size || (flag_size && p.size >= sz)) &&
                                        (!flag_mdate || (flag_mdate && mop == ">" && p.mdate >= mdatetest) || (flag_mdate && mop == "< " && p.mdate >= mdatetest)) &&
                                        (!flag_adate || (flag_adate && aop == ">" && p.adate >= adatetest) || (flag_adate && aop == "< " && p.adate >= adatetest)) &&
                                        (!flag_cdate || (flag_cdate && cop == ">" && p.cdate >= cdatetest) || (flag_cdate && cop == "< " && p.cdate >= cdatetest))
                                 orderby p.size descending 
                                 select p).ToList();

            foundfiles = new BindingList<f_results>(y);

Answer 3

You could generate your filters beforehand, then apply them all at once - you would only have to iterate your initial enumeration once, something like this (shortened):

IEnumerable<f_results> foundfiles = new List<f_results>();
var filters = new List<Func<f_results, bool>>();

if (fsize.Text.Trim() != "")
{
    long sz = long.Parse(fsize.Text);
    filters.Add(x => x.size >= sz);
}

if (adate.Text.Trim() != "")
{
    DateTime test = DateTime.Parse(adate.Text);
    filters.Add(x => x.adate >= test);
}

foreach (var filter in filters)
{
    var filterToApply = filter;
    foundfiles = foundfiles.Where(filterToApply);
}
finalResults = new BindingList<f_results>(foundfiles);

More importantly don't call ToList() until you have processed all filters, otherwise you keep iterating through your full result list over and over.