Remove duplicates from ArrayLists

Question

I have an ArrayList of custom objects. I want to remove duplicate entries.

The objects have three fields: title, subtitle, and id. If a subtitle occurs multiple times, I only need the first item with thats subtitle (ignore the remaining object with that subtitle).

Answer 1

Update for Java8:

Using Java8 streams you can also do this pretty trivially.

ArrayList<String> deduped;
deduped = yourArrayList.stream()
             .distinct()
             .collect(Collectors.toCollection(ArrayList::new));

This also has the advantage over ArrayList → Set → ArrayList of maintaining ordering.

Answer 2

List<Item> result = new ArrayList<Item>();
Set<String> titles = new HashSet<String>();

for(Item item : originalList) {
    if(titles.add(item.getTitle()) {
        result.add(item);
    }
}

add() of the Set returns false if the element already exists.

Answer 3

Another method using Java 8 streams you can also do pretty cool:

List<Customer> CustomerLists;
List<Customer> unique = CustomerLists.stream().collect(collectingAndThen(
        toCollection(() -> new TreeSet<>(comparingLong(Customer::getId))),
        ArrayList::new));

Answer 4

List<YourObject> all = ******** // this is the object that you have already  and filled it.
List<YourObject> noRepeat= new ArrayList<YourObject>();

for (YourObject al: all) {
    boolean isPresent = false;
    // check if the current objects subtitle already exists in noRepeat
    for (YourObject nr : noRepeat) {
        if (nr.getName().equals(al.getName()) {
            isFound = true;//yes we have already
            break;
        }
    }

    if (!isPresent)
        noRepeat.add(al); // we are adding if we don't have already
}

take one new ArrayList Object of same type
one by one add all the old arraylists elements into this new arraylist object but before adding every object check in the new arraylist that if there is any object with the same subtitle.if new arraylist contains such subtitle don't add it. otherwise add it

Answer 5

In Java 8, you can also do something like this:

yourList.stream().collect(
     Collectors.toMap(
         obj -> obj.getSubtitle(),
         Function.identity(), 
         (o1,o2) -> o1))
    .values();

The trick is to collect stream to map and provide key collision resolver lambda ((o1,o2) -> o1) which always returns its first parameter. The result is a Collection, not a List but you can easily convert it to a List:

new ArrayList(resultCollection);

Answer 6

The solution depends on circumstances.

If you don't have much data then go with a Set Set<T> unique = new HashSet<>(yourList); (use LinkedHashSet if you care about the order. It creates a new collection, but usually it's not a problem.

When you want to modify existing list and don't want to/can't create a new collection, you can remove duplicates like here:

List<Integer> numbers =
    new ArrayList<>(asList(1, 1, 2, 1, 2, 3, 5));

System.out.println("Numbers: " + numbers);
ListIterator<Integer> it = numbers.listIterator();
while (it.hasNext()) {
    int i = it.nextIndex();
    Integer current = it.next();
    for (int j = 0; j < i; ++j) {
        if (current.equals(numbers.get(j))) {
            it.remove();
            break;
        }
    }
}
System.out.println("Unique: " + numbers);

It works in O(n^2), but it works. Similar implementation, but simpler, is when the list is sorted - works in O(n) time. Both implementations are explained at Farenda: remove duplicates from list - various implementations.

Answer 7

You can put the content of the ArrayList in a TreeSet using a custom Comparator which should return 0 if the two subtitles are the same. After that you can convert the Set in a List and have the List without "duplicates". Here is an example for Object, of course you should use the correct class and logic.

public void removeDuplicates(List<Object> l) {
    // ... the list is already populated
    Set<Object> s = new TreeSet<Object>(new Comparator<Object>() {

        @Override
        public int compare(Object o1, Object o2) {
            // ... compare the two object according to your requirements
            return 0;
        }
    });
            s.addAll(l);
    List<Object> res = Arrays.asList(s.toArray());
}

Answer 8

Use Collections.sort() to sort and use a simple for cycle to catch doubles, e.g.:

Collections.sort(myList);
A previous = null;
for (A elem: myList) {
    if (elem.compareTo(previous) == 0) continue;
    previous = elem;

    [... process unique element ...]
}

This presumes that you'll implement Comparable in your type A.

Answer 9

Removes any duplicates in a collection, while preserving the order if it is an ordered collection. Efficient enough for most cases.

public static <I, T extends Collection<I>> T removeDuplicates(T collection)
{
    Set<I> setItems = new LinkedHashSet<I>(collection);
    collection.clear();
    collection.addAll(setItems);

    return collection;
}

Answer 10

private static List<Integer> removeDuplicates(List<Integer> list) {
    ArrayList<Integer> uniqueList = new ArrayList<Integer>();
    for (Integer i : list) {
        if (!inArray(i, uniqueList)) {
            uniqueList.add(i);
        }
    }

    return uniqueList;
}

private static boolean inArray(Integer i, List<Integer> list) {
    for (Integer integer : list) {
        if (integer == i) {
            return true;
        }
    }

    return false;
}

Answer 11

You can use an O(n^2) solution: Use list.iterator() to iterate the list once, and on each iteration, iterate it again to check if there are duplicates. If there are - call iterator.remove(). A variation of this is to use guava's Iterables.filter(list, predicate) where your filtering logic is in the predicate.

Another way (perhaps better) would be to define the equals(..) and hashCode(..) methods to handle your custom equality logic, and then simply construct a new HashSet(list). This will clear duplicates.

Answer 12

If I understand correctly you have an ArrayList<Custom>, let's call it list. Your Custom class has a subtitle field, let's say with a getSubtitle() method that returns String. You want to keep only the first unique subtitle and remove any remaining duplicates. Here's how you can do that:

Set<String> subtitles = new HashSet<String>();
for (Iterator<Custom> it = list.iterator(); it.hasNext(); ) {
    if (!subtitles.add(it.next().getSubtitle())) {
        it.remove();
    }
}

Answer 13

List list = (...);

//list may contain duplicates.

//remove duplicates if any
Set setItems = new LinkedHashSet(list);
list.clear();
list.addAll(setItems);

You may need to override "equals()" so that 2 elements are considered equals if they have the same subtitle (or tite and subtitle maybe ?)

Answer 14

I would suggest using a Set

http://download.oracle.com/javase/6/docs/api/java/util/Set.html

Which by its nature cannot contain duplicate items. You can create a new set from your original ArrayList using

Set myset = new HashSet(myArrayList);

Alternatively, just use a Set from the start, and don't use an ArrayList as it is not performing the function that you require.