CODEWITHSUNDEEP
javalistcollectionsarraylistduplicates
user25778
user25778

Reputation: 6061

How do I remove repeated elements from ArrayList?

I have an ArrayList<String>, and I want to remove repeated strings from it. How can I do this?

Upvotes: 589

Views: 1046482

Answers (30)

akhil_mittal
akhil_mittal

Reputation: 24197

Suppose we have a list of String like:

List<String> strList = new ArrayList<>(5);
// insert up to five items to list.

Then we can remove duplicate elements in multiple ways.

Prior to Java 8

List<String> deDupStringList = new ArrayList<>(new HashSet<>(strList));

Note: If we want to maintain the insertion order then we need to use LinkedHashSet in place of HashSet

Using Guava

List<String> deDupStringList2 = Lists.newArrayList(Sets.newHashSet(strList));

Using Java 8

List<String> deDupStringList3 = strList.stream().distinct().collect(Collectors.toList());

Note: In case we want to collect the result in a specific list implementation e.g. LinkedList then we can modify the above example as:

List<String> deDupStringList3 = strList.stream().distinct()
                 .collect(Collectors.toCollection(LinkedList::new));

We can use parallelStream also in the above code but it may not always give expected performance benefits. Check this question for more.

Upvotes: 81

Kirguduck
Kirguduck

Reputation: 806

Kotlin

val list = listOf('a', 'A', 'b', 'B', 'A', 'a')
println(list.distinct()) // [a, A, b, B]
println(list.distinctBy { it.uppercaseChar() }) // [a, b]

from here kotlinlang

Upvotes: 0

Gil SH
Gil SH

Reputation: 3868

Here is a solution that works with any object:

public static <T> List<T> clearDuplicates(List<T> messages,Comparator<T> comparator) {
    List<T> results = new ArrayList<T>();
    for (T m1 : messages) {
        boolean found = false;
        for (T m2 : results) {
            if (comparator.compare(m1,m2)==0) {
                found=true;
                break;
            }
        }
        if (!found) {
            results.add(m1);
        }
    }
    return results;
}

Upvotes: 0

satish
satish

Reputation: 1651

In Java, List permits ordered access of their elements. They can have duplicates because their lookup key is the position not some hash code, every element can be modified while they remain in the list where as Set represents a collection of unique elements and while elements are in set, they must not be modified.While there is no restriction preventing you from modifying elements in a set, if an element is modified, then it could become forever lost in the set.

public static void main(String[] args) {
       List<String> l = new ArrayList<String>();
       l.add("A");
       l.add("B");
       l.add("C");
       l.add("A");
       System.out.println("Before removing duplicates: ");
       for (String s : l) {
            System.out.println(s);
       }
       Set<String> set = new HashSet<String>(l);
       List<String> newlist = new ArrayList<String>(set);
       System.out.println("after removing duplicates: ");
       for (String s : newlist) {
            System.out.println(s);
       }
  }

for reference, refer this link How to remove duplicates from ArrayList

Upvotes: -1

Ravi Vital
Ravi Vital

Reputation: 1

Would something like this work better ?

public static void removeDuplicates(ArrayList<String> list) {
    Arraylist<Object> ar     = new Arraylist<Object>();
    Arraylist<Object> tempAR = new Arraylist<Object>();
    while (list.size()>0){
        ar.add(list(0));
        list.removeall(Collections.singleton(list(0)));
    }
    list.addAll(ar);
}

That should maintain the order and also not be quadratic in run time.

Upvotes: 0

Saurabh Gaddelpalliwar
Saurabh Gaddelpalliwar

Reputation: 1645

If you are using model type List< T>/ArrayList< T> . Hope,it's help you.

Here is my code without using any other data structure like set or hashmap

for (int i = 0; i < Models.size(); i++){
for (int j = i + 1; j < Models.size(); j++) {       
 if (Models.get(i).getName().equals(Models.get(j).getName())) {    
 Models.remove(j);
   j--;
  }
 }
}

Upvotes: 6

user2868724
user2868724

Reputation: 279

this can solve the problem:

private List<SomeClass> clearListFromDuplicateFirstName(List<SomeClass> list1) {

     Map<String, SomeClass> cleanMap = new LinkedHashMap<String, SomeClass>();
     for (int i = 0; i < list1.size(); i++) {
         cleanMap.put(list1.get(i).getFirstName(), list1.get(i));
     }
     List<SomeClass> list = new ArrayList<SomeClass>(cleanMap.values());
     return list;
}

Upvotes: 27

stbn
stbn

Reputation: 425

Here's a way that doesn't affect your list ordering:

ArrayList l1 = new ArrayList();
ArrayList l2 = new ArrayList();

Iterator iterator = l1.iterator();

while (iterator.hasNext()) {
    YourClass o = (YourClass) iterator.next();
    if(!l2.contains(o)) l2.add(o);
}

l1 is the original list, and l2 is the list without repeated items (Make sure YourClass has the equals method according to what you want to stand for equality)

Upvotes: 30

linker
linker

Reputation: 891

If you want your list to automatically ignore duplicates and preserve its order, you could create a HashList(a HashMap embedded List).

public static class HashList<T> extends ArrayList<T>{
        private HashMap <T,T> hashMap;
        public HashList(){
            hashMap=new HashMap<>();
        }

        @Override
        public boolean add(T t){
            if(hashMap.get(t)==null){
                hashMap.put(t,t);
                return super.add(t);
            }else return false;
        }

        @Override
        public boolean addAll(Collection<? extends T> c){
            HashList<T> addup=(HashList<T>)c;
            for(int i=0;i<addup.size();i++){
                add(addup.get(i));
            }return true;
        }

    }

Usage Example:

HashList<String> hashlist=new HashList<>();
hashList.add("hello");
hashList.add("hello");
System.out.println(" HashList: "+hashlist);

Upvotes: 0

saif
saif

Reputation: 1214

Set<String> strSet = strList.stream().collect(Collectors.toSet());

Is the easiest way to remove your duplicates.

Upvotes: 0

jonathan-stafford
jonathan-stafford

Reputation: 11827

If you don't want duplicates in a Collection, you should consider why you're using a Collection that allows duplicates. The easiest way to remove repeated elements is to add the contents to a Set (which will not allow duplicates) and then add the Set back to the ArrayList:

Set<String> set = new HashSet<>(yourList);
yourList.clear();
yourList.addAll(set);

Of course, this destroys the ordering of the elements in the ArrayList.

Upvotes: 1111

Vinze
Vinze

Reputation: 2539

As said before, you should use a class implementing the Set interface instead of List to be sure of the unicity of elements. If you have to keep the order of elements, the SortedSet interface can then be used; the TreeSet class implements that interface.

Upvotes: 1

seekingStillness
seekingStillness

Reputation: 5113

This is the right one (if you are concerned about the overhead of HashSet.

 public static ArrayList<String> removeDuplicates (ArrayList<String> arrayList){
    if (arrayList.isEmpty()) return null;  //return what makes sense for your app
    Collections.sort(arrayList, String.CASE_INSENSITIVE_ORDER);
    //remove duplicates
    ArrayList <String> arrayList_mod = new ArrayList<>();
    arrayList_mod.add(arrayList.get(0));
    for (int i=1; i<arrayList.size(); i++){
        if (!arrayList.get(i).equals(arrayList.get(i-1))) arrayList_mod.add(arrayList.get(i));
    }
    return arrayList_mod;
}

Upvotes: 0

Sameer Shrestha
Sameer Shrestha

Reputation: 123

Time Complexity : O(n) : Without Set

private static void removeDup(ArrayList<String> listWithDuplicateElements) {
    System.out.println(" Original Duplicate List :" + listWithDuplicateElements);
    List<String> listWithoutDuplicateElements = new ArrayList<>(listWithDuplicateElements.size());

    listWithDuplicateElements.stream().forEach(str -> {
        if (listWithoutDuplicateElements.indexOf(str) == -1) {
            listWithoutDuplicateElements.add(str);
        }
    });     

    System.out.println(" Without Duplicate List :" + listWithoutDuplicateElements);
}

Upvotes: 0

HamidReza
HamidReza

Reputation: 1934

you can use nested loop in follow :

ArrayList<Class1> l1 = new ArrayList<Class1>();
ArrayList<Class1> l2 = new ArrayList<Class1>();

        Iterator iterator1 = l1.iterator();
        boolean repeated = false;

        while (iterator1.hasNext())
        {
            Class1 c1 = (Class1) iterator1.next();
            for (Class1 _c: l2) {
                if(_c.getId() == c1.getId())
                    repeated = true;
            }
            if(!repeated)
                l2.add(c1);
        }

Upvotes: 2

Gujjula Ramesh Reddy
Gujjula Ramesh Reddy

Reputation: 87

This is used for your Custom Objects list 

   public List<Contact> removeDuplicates(List<Contact> list) {
    // Set set1 = new LinkedHashSet(list);
    Set set = new TreeSet(new Comparator() {

        @Override
        public int compare(Object o1, Object o2) {
            if (((Contact) o1).getId().equalsIgnoreCase(((Contact) o2).getId()) /*&&
                    ((Contact)o1).getName().equalsIgnoreCase(((Contact)o2).getName())*/) {
                return 0;
            }
            return 1;
        }
    });
    set.addAll(list);

    final List newList = new ArrayList(set);
    return newList;
}

Upvotes: 1

Nenad Bulatović
Nenad Bulatović

Reputation: 7444

You can also do it this way, and preserve order:

// delete duplicates (if any) from 'myArrayList'
myArrayList = new ArrayList<String>(new LinkedHashSet<String>(myArrayList));

Upvotes: 30

Craig P. Motlin
Craig P. Motlin

Reputation: 26758

If you're willing to use a third-party library, you can use the method distinct() in Eclipse Collections (formerly GS Collections).

ListIterable<Integer> integers = FastList.newListWith(1, 3, 1, 2, 2, 1);
Assert.assertEquals(
    FastList.newListWith(1, 3, 2),
    integers.distinct());

The advantage of using distinct() instead of converting to a Set and then back to a List is that distinct() preserves the order of the original List, retaining the first occurrence of each element. It's implemented by using both a Set and a List.

MutableSet<T> seenSoFar = UnifiedSet.newSet();
int size = list.size();
for (int i = 0; i < size; i++)
{
    T item = list.get(i);
    if (seenSoFar.add(item))
    {
        targetCollection.add(item);
    }
}
return targetCollection;

If you cannot convert your original List into an Eclipse Collections type, you can use ListAdapter to get the same API.

MutableList<Integer> distinct = ListAdapter.adapt(integers).distinct();

Note: I am a committer for Eclipse Collections.

Upvotes: 5

Hardip
Hardip

Reputation: 360

ArrayList<String> city=new ArrayList<String>();
city.add("rajkot");
city.add("gondal");
city.add("rajkot");
city.add("gova");
city.add("baroda");
city.add("morbi");
city.add("gova");

HashSet<String> hashSet = new HashSet<String>();
hashSet.addAll(city);
city.clear();
city.addAll(hashSet);
Toast.makeText(getActivity(),"" + city.toString(),Toast.LENGTH_SHORT).show();

Upvotes: 2

Timofey Gorshkov
Timofey Gorshkov

Reputation: 5124

There is also ImmutableSet from Guava as an option (here is the documentation):

ImmutableSet.copyOf(list);

Upvotes: 21

neo7
neo7

Reputation: 674

Here is my answer without using any other data structure like set or hashmap etc.

public static <T> ArrayList<T> uniquefy(ArrayList<T> myList) {

    ArrayList <T> uniqueArrayList = new ArrayList<T>();
    for (int i = 0; i < myList.size(); i++){
        if (!uniqueArrayList.contains(myList.get(i))){
            uniqueArrayList.add(myList.get(i));
        }
    }

    return uniqueArrayList;
}

Upvotes: 0

Manash Ranjan Dakua
Manash Ranjan Dakua

Reputation: 311

public static void main(String[] args){
    ArrayList<Object> al = new ArrayList<Object>();
    al.add("abc");
    al.add('a');
    al.add('b');
    al.add('a');
    al.add("abc");
    al.add(10.3);
    al.add('c');
    al.add(10);
    al.add("abc");
    al.add(10);
    System.out.println("Before Duplicate Remove:"+al);
    for(int i=0;i<al.size();i++){
        for(int j=i+1;j<al.size();j++){
            if(al.get(i).equals(al.get(j))){
                al.remove(j);
                j--;
            }
        }
    }
    System.out.println("After Removing duplicate:"+al);
}

Upvotes: 12

abahgat
abahgat

Reputation: 13530

Although converting the ArrayList to a HashSet effectively removes duplicates, if you need to preserve insertion order, I'd rather suggest you to use this variant

// list is some List of Strings
Set<String> s = new LinkedHashSet<>(list);

Then, if you need to get back a List reference, you can use again the conversion constructor.

Upvotes: 326

infoj
infoj

Reputation: 681

Java 8 streams provide a very simple way to remove duplicate elements from a list. Using the distinct method. If we have a list of cities and we want to remove duplicates from that list it can be done in a single line - 

 List<String> cityList = new ArrayList<>();
 cityList.add("Delhi");
 cityList.add("Mumbai");
 cityList.add("Bangalore");
 cityList.add("Chennai");
 cityList.add("Kolkata");
 cityList.add("Mumbai");

 cityList = cityList.stream().distinct().collect(Collectors.toList());

How to remove duplicate elements from an arraylist

Upvotes: 30

siva
siva

Reputation: 11

        List<String> result = new ArrayList<String>();
        Set<String> set = new LinkedHashSet<String>();
        String s = "ravi is a good!boy. But ravi is very nasty fellow.";
        StringTokenizer st = new StringTokenizer(s, " ,. ,!");
        while (st.hasMoreTokens()) {
            result.add(st.nextToken());
        }
         System.out.println(result);
         set.addAll(result);
        result.clear();
        result.addAll(set);
        System.out.println(result);

output:
[ravi, is, a, good, boy, But, ravi, is, very, nasty, fellow]
[ravi, is, a, good, boy, But, very, nasty, fellow]

Upvotes: 1

sharkbait
sharkbait

Reputation: 3050

The @jonathan-stafford solution is OK. But this don't preserve the list order.

If you want preserve the list order you have to use this:

public static <T> void removeDuplicate(List <T> list) {
Set <T> set = new HashSet <T>();
List <T> newList = new ArrayList <T>();
for (Iterator <T>iter = list.iterator();    iter.hasNext(); ) {
   Object element = iter.next();
   if (set.add((T) element))
      newList.add((T) element);
   }
   list.clear();
   list.addAll(newList);
}

It's only to complete the answer. Very good!

Upvotes: 0

sambhu
sambhu

Reputation: 121

Code:

List<String> duplicatList = new ArrayList<String>();
duplicatList = Arrays.asList("AA","BB","CC","DD","DD","EE","AA","FF");
//above AA and DD are duplicate
Set<String> uniqueList = new HashSet<String>(duplicatList);
duplicatList = new ArrayList<String>(uniqueList); //let GC will doing free memory
System.out.println("Removed Duplicate : "+duplicatList);

Note: Definitely, there will be memory overhead.

Upvotes: 2

M Kaweepatt Churcharoen
M Kaweepatt Churcharoen

Reputation: 216

This three lines of code can remove the duplicated element from ArrayList or any collection.

List<Entity> entities = repository.findByUserId(userId);

Set<Entity> s = new LinkedHashSet<Entity>(entities);
entities.clear();
entities.addAll(s);

Upvotes: 3

Thananjayan N
Thananjayan N

Reputation: 77

If you want to remove duplicates from ArrayList means find the below logic,

public static Object[] removeDuplicate(Object[] inputArray)
{
    long startTime = System.nanoTime();
    int totalSize = inputArray.length;
    Object[] resultArray = new Object[totalSize];
    int newSize = 0;
    for(int i=0; i<totalSize; i++)
    {
        Object value = inputArray[i];
        if(value == null)
        {
            continue;
        }

        for(int j=i+1; j<totalSize; j++)
        {
            if(value.equals(inputArray[j]))
            {
                inputArray[j] = null;
            }
        }
        resultArray[newSize++] = value;
    }

    long endTime = System.nanoTime()-startTime;
    System.out.println("Total Time-B:"+endTime);
    return resultArray;
}

Upvotes: 0

SparkOn
SparkOn

Reputation: 8956

    ArrayList<String> list = new ArrayList<String>();
    HashSet<String> unique = new LinkedHashSet<String>();
    HashSet<String> dup = new LinkedHashSet<String>();
    boolean b = false;
    list.add("Hello");
    list.add("Hello");
    list.add("how");
    list.add("are");
    list.add("u");
    list.add("u");

    for(Iterator iterator= list.iterator();iterator.hasNext();)
    {
        String value = (String)iterator.next();
        System.out.println(value);

        if(b==unique.add(value))
            dup.add(value);
        else
            unique.add(value);


    }
    System.out.println(unique);
    System.out.println(dup);

Upvotes: 0

Related Questions