Find items with duplicate values over mutiple properties

Question

It's hard to explain my issue without giving a concrete example. There might be a similar question on here but I wasn't able to find it because I'm having trouble wording it in searchable terms.

Basically I need to find items in a list that have any duplicate values over multiple properties. In other words, any value in the original list has to be unique regardless of which property it is in.

Here is a simple example I could come up with that describes the problem really well:

There is a list of dates for holidays with an extra property for an optional replacement date (ex: for when the holiday falls in the weekend). Each date in this list has to be unique, so I'd like to find the items that contain duplicate dates.

PART1: return a list of duplicate dates
PART2: return a list of all the items with a duplicate date

I believe this is a great example because one of the properties is nullable which might make it even a little more difficult.

Model:

public class Holiday
{
    public Holiday(DateTime hDate, string descr, DateTime? rDate = null)
    {
        HolidayDate = hDate;
        Description = descr;
        ReplacementDate = rDate;
    }

    public DateTime HolidayDate { get; set; }
    public string Description { get; set; }
    public DateTime? ReplacementDate { get; set; }
}

Here is some example data to get you started (and hopefully clear up any confusion)

var list = new List<Holiday>()
{
    new Holiday(new DateTime(2016,1,1),"NEW YEAR 2016"),
    new Holiday(new DateTime(2016,3,27),"EASTER MONDAY 2016", new DateTime(2016,3,28)),
    new Holiday(new DateTime(2016,12,25),"CHRISTMAS DAY 2016", new DateTime(2016,12,26)),
    new Holiday(new DateTime(2017,1,1),"NEW YEAR 2017", new DateTime(2017,1,2)),
    new Holiday(new DateTime(2017,4,17),"EASTER MONDAY 2017"),
    new Holiday(new DateTime(2017,12,25),"CHRISTMAS DAY 2017"),
    new Holiday(new DateTime(2018,1,1),"NEW YEAR 2018"),
    new Holiday(new DateTime(2018,1,1),"DUPLICATE 1"), //example of a duplicate
    new Holiday(new DateTime(2018,1,2),"DUPLICATE 2", new DateTime(2016,1,1)), //example of a duplicate
    new Holiday(new DateTime(2018,1,3),"DUPLICATE 3", new DateTime(2017,1,2)), //example of a duplicate
    new Holiday(new DateTime(2018,1,4),"DUPLICATE 4", new DateTime(2018,1,4)), //example of a duplicate

};

var result = list; //add your code that finds the items with a duplicate value, preferably a linq query

//PART1: return list of the actual duplicate values
//duplcateDates =  "2016-01-01, 2017-01-02, 2018-01-01, 2018-01-04";

//PART2: return a list with the items that have a duplicate item
var reultString = string.Join(", ", result.Select(q => q.Description));
//resultString = "NEW YEAR 2016, DUPLICATE 2, NEW YEAR 2017, DUPLICATE 3, NEW YEAR 2018, DUPLICATE 1, DUPLICATE 4";

For those of you that are lazy and don't feel like working out this example, as long as you understood my issue and can help me by using your own similar example or point me in the right direction, I'd really appreciate it.

Any solution that doesn't involve writing a specific foreach or for-loop that checks each property individually for duplicates will be accepted as an answer.
I'd like to be able to apply the solution of this problem to different similar situations without having to write an entire block of code iterating trough the possibilities.

This is why I'd like to know if this is possible by linq queries. However if you have a generic method or extension that solves problems like this, please share!

Answer 1

Here you have a generic extension which returns a list of a custom object that holds a reference to the generic object, the value that is a duplicate, and whether it is a self duplicate or not.

Here is the code that you need to add in a new file:

public class MainObject<TRef, TProp> where TProp : struct
{
    public TRef Reference { get; set; }
    public TProp? Value { get; set; }

    public bool SelfDuplicate { get; set; }

}

public static class Extensions
{
    public static IEnumerable<MainObject<TIn, TProp>> GetDuplicates<TIn, TProp>(this IEnumerable<TIn> @this) where TProp : struct
    {
        var type = typeof(TIn);
        // get the properties of the object type that match the property type
        var props = type.GetProperties(BindingFlags.Instance | BindingFlags.Public).Where(prop => prop.PropertyType == typeof(TProp) || prop.PropertyType == typeof(TProp?)).ToList();

        // convert the input enumerable to a list so we don't iterate it multiple times
        var list = @this as IList<TIn> ?? @this.ToList();

        // create a list to hold all duplicates
        var duplicates = new List<MainObject<TIn, TProp>>();

        foreach (var item1 in list)
        {
            var isSelfDupe = item1.IsDuplicate<TIn, TProp>(props);
            if (isSelfDupe.IsDupe)
            {
                duplicates.Add(new MainObject<TIn, TProp>
                {
                    Reference = item1,
                    SelfDuplicate = true,
                    Value = (TProp?)isSelfDupe.Property1.GetValue(item1)
                });
            }

            foreach (var item2 in list)
            {
                if (ReferenceEquals(item1, item2)) continue;

                var isDuplicate = item1.IsDuplicate<TIn, TProp>(item2, props);
                if (isDuplicate.IsDupe)
                {
                    duplicates.Add(new MainObject<TIn, TProp>
                    {
                        Reference = item1,
                        SelfDuplicate = false,
                        Value = (TProp?)isDuplicate.Property1.GetValue(item1)
                    });
                }
            }
        }

        return duplicates.Distinct().ToList();
    }

    private static IsDuplicateResult<TIn> IsDuplicate<TIn, TProp>(this TIn obj1, IEnumerable<PropertyInfo> props) where TProp : struct
    {
        var propList = props as IList<PropertyInfo> ?? props.ToList();

        var valueList = propList.Select(prop => prop.GetValue(obj1)).ToList();
        foreach (var p1 in propList)
        {
            foreach (var p2 in propList)
            {
                if (ReferenceEquals(p1, p2)) continue;

                if (EqualityComparer<TProp?>.Default.Equals((TProp?)p1.GetValue(obj1), (TProp?)p2.GetValue(obj1)))
                {
                    return new IsDuplicateResult<TIn> { IsDupe = true, Reference = obj1, Property1 = p1, Property2 = p2 };
                }
            }
        }
        return new IsDuplicateResult<TIn> { IsDupe = false };
    }

    private static IsDuplicateResult<TIn> IsDuplicate<TIn, TProp>(this TIn obj1, TIn obj2, IEnumerable<PropertyInfo> props) where TProp : struct
    {
        var propList = props as IList<PropertyInfo> ?? props.ToList();

        var dict1 = propList.ToDictionary(prop => prop, prop => prop.GetValue(obj1));
        var dict2 = propList.ToDictionary(prop => prop, prop => prop.GetValue(obj2));

        foreach (var k1 in dict1.Keys)
        {
            foreach (var k2 in dict2.Keys)
            {
                if (dict1[k1] == null || dict2[k2] == null) continue;

                if (EqualityComparer<TProp?>.Default.Equals((TProp?)dict1[k1], (TProp?)dict2[k2]))
                {
                    return new IsDuplicateResult<TIn> { IsDupe = true, Reference = obj1, Property1 = k1, Property2 = k2 };
                }
            }
        }
        return new IsDuplicateResult<TIn> { IsDupe = false };
    }

    private class IsDuplicateResult<TIn>
    {
        public bool IsDupe { get; set; }

        public TIn Reference { get; set; }

        public PropertyInfo Property1 { get; set; }
        public PropertyInfo Property2 { get; set; }
    }
}

Using this is pretty simple and straightforward, all you have to do is call the extension method.

var result = holidayList.GetDuplicates<Holiday, DateTime>().ToList();
// printing the results
Console.WriteLine(string.Join(", ", result.Select(q => q.Reference.Description)));
// printing the values
var values = result.Select(r => r.Value).Distinct();
Console.WriteLine(string.Join("\r\n", values.Select(q => $"{q:dd/MM/yyyy}")));

DotNetFiddle link: https://dotnetfiddle.net/OGwpb4

DotNetFiddle link (with property annotations): https://dotnetfiddle.net/vzjS2t

You might need some more fiddling around with the functions, to see if anything goes wrong as I didn't have much time to do so but, I think this should get you into the right direction :)

Answer 2

You can use IEnumerable and IEnumerator

using System;
using System.Collections;
using System.Collections.Generic;
using System.Linq;
using System.Text;

namespace ConsoleApplication1
{

    class Program
    {
        static void Main(string[] args)
        {
            Holiday holiday = new Holiday();

            foreach (Holiday day in Holiday.holidays)
            {
                foreach (object field in  new HolidayField(day))
                {
                    int a = 1;
                }
                HolidayField hf = new HolidayField(day);
                for (HolidayField.Field field = HolidayField.Field.HOLIDAY_DATE; field < HolidayField.Field.END; field++)
                {
                    object o = hf.GetEnumerator(field);
                }
            }
            var groups = Holiday.holidays.GroupBy(x => x.HolidayDate).ToList();


        }
    }
    public class HolidayField : IEnumerator, IEnumerable
    {
        public Holiday holiday;

        public enum Field
        {
            RESET,
            HOLIDAY_DATE,
            DESCRIPTION,
            REPLACEMENT_DATE,
            END
        }
        Field _holidayProperties;

        public HolidayField(Holiday holiday)
        {
            this.holiday = holiday;

        }

        public Field GetIndex()
        {
            return _holidayProperties;
        }

        public object GetEnumerator(Field field)
        {
            return GetCurrent(holiday, field);
        }

        public HolidayField GetEnumerator()
        {
            return this;
        }

        // Implementation for the GetEnumerator method.
        IEnumerator IEnumerable.GetEnumerator()
        {
            return (IEnumerator) GetEnumerator();
        }



        public object Current
        {
            get
            {
                return GetCurrent(holiday, _holidayProperties);
            }
        }
        public object GetCurrent(Holiday holiday, Field field)
        {
            object results = null;

            switch (field)
            {
                case Field.DESCRIPTION:
                    results = (object)holiday.Description;
                    break;
                case Field.HOLIDAY_DATE:
                    results = (object)holiday.HolidayDate;
                    break;
                case Field.REPLACEMENT_DATE:
                    results = (object)holiday.ReplacementDate;
                    break;
            }

            return results;
        }
        public bool MoveNext()
        {
            _holidayProperties += 1;
            return _holidayProperties == Field.END ? false : true;
        }

        public void Reset()
        {
            _holidayProperties = Field.RESET;
        }
        public void Dispose()
        {
        }

    }
    public class Holiday
    {
        public static List<Holiday> holidays = new List<Holiday>()
        {
            new Holiday(new DateTime(2016,1,1),"NEW YEAR 2016"),
            new Holiday(new DateTime(2016,3,27),"EASTER MONDAY 2016", new DateTime(2016,3,28)),
            new Holiday(new DateTime(2016,12,25),"CHRISTMAS DAY 2016", new DateTime(2016,12,26)),
            new Holiday(new DateTime(2017,1,1),"NEW YEAR 2017", new DateTime(2017,1,2)),
            new Holiday(new DateTime(2017,4,17),"EASTER MONDAY 2017"),
            new Holiday(new DateTime(2017,12,25),"CHRISTMAS DAY 2017"),
            new Holiday(new DateTime(2018,1,1),"NEW YEAR 2018"),
            new Holiday(new DateTime(2018,1,1),"DUPLICATE 1"), //example of a duplicate
            new Holiday(new DateTime(2018,1,2),"DUPLICATE 2", new DateTime(2016,1,1)), //example of a duplicate
            new Holiday(new DateTime(2018,1,3),"DUPLICATE 3", new DateTime(2017,1,2)), //example of a duplicate
            new Holiday(new DateTime(2018,1,4),"DUPLICATE 4", new DateTime(2018,1,4)), //example of a duplicate

        };

        public DateTime HolidayDate { get; set; }
        public string Description { get; set; }
        public DateTime? ReplacementDate { get; set; }


        public Holiday() { }
        public Holiday(DateTime hDate, string descr, DateTime? rDate = null)
        {
            HolidayDate = hDate;
            Description = descr;
            ReplacementDate = rDate;
        }

    }


}

Answer 3

In case if anyone, in the future, looking for a generic solution.

Note: This solution will cast nullable properties to their non-nullable types and compare their values too.

Generic solution

For easier implementation we will define two Methods:

1. IsSelfDuplicate: Which will detect if the object has two properties with the same value.

   private bool IsSelfDuplicate(object myObj)
   {
       PropertyInfo[] props = myObj.GetType().GetProperties();
       for (int i = 0; i < props.Length; i++)
       {
           if (props[i].GetValue(myObj) == null)
               continue;
   
           Type iType = (Nullable.GetUnderlyingType(props[i].PropertyType) ?? props[i].PropertyType);
           object iValue = Convert.ChangeType(props[i].GetValue(myObj), iType);
   
           for (int j = i + 1; j < props.Length; j++)
           {
               if (props[j].GetValue(myObj) == null)
                   continue;
   
               Type jType = (Nullable.GetUnderlyingType(props[j].PropertyType) ?? props[j].PropertyType);
               object jValue = Convert.ChangeType(props[j].GetValue(myObj), jType);
   
               if (iType == jType && iValue.ToString() == jValue.ToString())
                   return true;
           }
       }
       return false;
   }
   

2. IsDuplicate: Which will detect if two objects, each has a property's value equals in the other.

   private bool IsDuplicate(object obj1, object obj2)
   {
       PropertyInfo[] props = obj1.GetType().GetProperties();
   
       for (int i = 0; i < props.Length; i++)
       {
           if (props[i].GetValue(obj1) == null)
               continue;
   
           Type iType = (Nullable.GetUnderlyingType(props[i].PropertyType) ?? props[i].PropertyType);
           object iValue = Convert.ChangeType(props[i].GetValue(obj1), iType);
   
           for (int j = 0; j < props.Length; j++)
           {
               if (props[j].GetValue(obj2) == null)
                   continue;
   
               Type jType = (Nullable.GetUnderlyingType(props[j].PropertyType) ?? props[j].PropertyType);
               object jValue = Convert.ChangeType(props[j].GetValue(obj2), jType);
   
               if (iType == jType && iValue.ToString() == jValue.ToString())
                   return true;
           }
       }
       return false;
   }
   

To use the IsSelfDuplicate Method, simply use:

List<Holiday> selfDuplicate = list.Where(l => IsSelfDuplicate(l)).ToList();

To use the IsDuplicate Method, iterate between your list of objects:

List<Holiday> duplicates = new List<Holiday>();

for (int i = 0; i < list.Count; i++)
    for (int j = i + 1; j < list.Count; j++)
    {
         if (IsDuplicate(list[i], list[j]))
         {
              duplicates.Add(list[i]);
              duplicates.Add(list[j]);
              break;
         }
    }

Finally, here is a sample list of objects, as the OP provided it:

var list = new List<Holiday>()
{
     new Holiday(new DateTime(2016,1,1),"NEW YEAR 2016"),
     new Holiday(new DateTime(2016,3,27),"EASTER MONDAY 2016", new DateTime(2016,3,28)),
     new Holiday(new DateTime(2016,12,25),"CHRISTMAS DAY 2016", new DateTime(2016,12,26)),
     new Holiday(new DateTime(2017,1,1),"NEW YEAR 2017", new DateTime(2017,1,2)),
     new Holiday(new DateTime(2017,4,17),"EASTER MONDAY 2017"),
     new Holiday(new DateTime(2017,12,25),"CHRISTMAS DAY 2017"),
     new Holiday(new DateTime(2018,1,1),"NEW YEAR 2018"),
     new Holiday(new DateTime(2018,1,1),"DUPLICATE 1"), //example of a duplicate
     new Holiday(new DateTime(2018,1,2),"DUPLICATE 2", new DateTime(2016,1,1)), //example of a duplicate
     new Holiday(new DateTime(2018,1,3),"DUPLICATE 3", new DateTime(2017,1,2)), //example of a duplicate
     new Holiday(new DateTime(2018,1,4),"DUPLICATE 4", new DateTime(2018,1,4)), //example of a duplicate
};

Now, you will have two results for self and non-self (if I can call it) duplicates. Enjoy ;)

Answer 4

Got one, too: Not sure, if its possible without collecting the data, first.

//PART0: collecting data 
var holidayDateDates = list.Select(x => x.HolidayDate).ToList();
var replacementDates = list.Select(x => x.ReplacementDate).Where(y => y != null).ToList().ConvertAll<DateTime>(x => x.Value);
holidayDateDates.AddRange(replacementDates);
//PART1: return list of the actual duplicate values
var result = holidayDateDates.GroupBy(x => x)
    .Where(g => g.Count() > 1)
    .Select(y => y.Key)
    .ToList();
var duplicateDates = string.Join(", ", result.Select(c => c.ToString("yyyy-MM-dd")));

//PART2: return a list with the items that have a duplicate item
var dummytime = new DateTime();// this will never be in the list and kills all nulls, see below
var duplicateHolidays = list.Where(x => result.Contains(x.HolidayDate) || result.Contains(x.ReplacementDate??dummytime));
var resultString = string.Join(", ", duplicateHolidays.Select(q => q.Description));

Answer 5

You can flatten collection so that each date (holiday and replacement) is represeted by seprate item, then group by date, like this:

// flatten
var result = list.SelectMany(c => new[] {
    // always include HolidayDate, it's not nullable
    new {
        Date = c.HolidayDate,
        Instance = c
    },
    // include replacement date if not null
    c.ReplacementDate != null ? new {
        Date = c.ReplacementDate.Value,
        Instance = c
    }: null
})
// filter out null items (that were created for null replacement dates)
.Where(c => c != null)
.GroupBy(c => c.Date)
.Where(c => c.Count() > 1)
.ToArray();

// keys of groups are duplicate dates
var duplcateDates = String.Join(", ", result.Select(c => c.Key.ToString("yyyy-MM-dd")));

var reultString = string.Join(", ", 
      // flatten groups extracting all items
      result.SelectMany(c => c)
      // filter out duplicates
     .Select(c => c.Instance).Distinct()
     .Select(c => c.Description));