Gradle Build Failed - Duplicate files copied in APK META-INF

Question

I get this error as soon as I enter the elasticsearch dependency,

compile 'org.elasticsearch.client:transport:5.0.0'

Detailed error logcat:

Error:Execution failed for task ':app:transformResourcesWithMergeJavaResForDevDebug'.
> com.android.build.api.transform.TransformException: com.android.builder.packaging.DuplicateFileException: Duplicate files copied in APK META-INF/io.netty.versions.properties
    File1: C:\Users\Dell\.gradle\caches\modules-2\files-2.1\io.netty\netty-buffer\4.1.5.Final\b5fb6bccda4d63d4a74c9faccdf32f77ab66abc1\netty-buffer-4.1.5.Final.jar
    File2: C:\Users\Dell\.gradle\caches\modules-2\files-2.1\io.netty\netty-handler\4.1.5.Final\6262900ee9487e62560030a136160df953b1cd6b\netty-handler-4.1.5.Final.jar
    File3: C:\Users\Dell\.gradle\caches\modules-2\files-2.1\io.netty\netty-transport\4.1.5.Final\37126b370722ff9631ee13c91139aacec0a71d1d\netty-transport-4.1.5.Final.jar
    File4: C:\Users\Dell\.gradle\caches\modules-2\files-2.1\io.netty\netty-codec-http\4.1.5.Final\87bda1b9ec7e3f75ca721fc87735cbedad2aa1a\netty-codec-http-4.1.5.Final.jar
    File5: C:\Users\Dell\.gradle\caches\modules-2\files-2.1\io.netty\netty-resolver\4.1.5.Final\5f367bedcdc185a727fda3296b9a18014cdc22c4\netty-resolver-4.1.5.Final.jar
    File6: C:\Users\Dell\.gradle\caches\modules-2\files-2.1\io.netty\netty-codec\4.1.5.Final\66bbf9324fa36467d041083f89328e2a24ec4f67\netty-codec-4.1.5.Final.jar
    File7: C:\Users\Dell\.gradle\caches\modules-2\files-2.1\io.netty\netty-common\4.1.5.Final\607f8433d8782445e72abe34e43a7e57e86a5e6c\netty-common-4.1.5.Final.jar

Gradle file:

    apply plugin: 'com.android.application'
    apply plugin: 'com.neenbedankt.android-apt'

    android {
        compileSdkVersion 25
        buildToolsVersion '25.0.3'

        defaultConfig {
            applicationId "..."
            minSdkVersion 18
            targetSdkVersion 25
            versionCode 6
            versionName "20170616.06"
            multiDexEnabled true
            ndk {
                abiFilter "armeabi-v7a"
            }

        }

        buildTypes {
            release {
                minifyEnabled false
                proguardFiles getDefaultProguardFile('proguard-android.txt'), 'proguard-rules.pro'
                //    signingConfig signingConfigs.release
            }
        }
        productFlavors {
            dev {
                applicationId "..."
                minSdkVersion 21
            }
            prod {
                applicationId "..."
            }
        }
        packagingOptions {
            exclude 'META-INF/LICENSE'
            exclude 'META-INF/DEPENDENCIES'
            exclude 'META-INF/LICENSE-FIREBASE.txt'
            exclude 'META-INF/NOTICE'
            exclude 'META-INF/DEPENDENCIES.txt'
            exclude 'META-INF/LICENSE.txt'
            exclude 'META-INF/NOTICE.txt'
            exclude 'META-INF/NOTICE'
            exclude 'META-INF/LICENSE'
            exclude 'META-INF/DEPENDENCIES'
            exclude 'META-INF/notice.txt'
            exclude 'META-INF/license.txt'
            exclude 'META-INF/dependencies.txt'
            exclude 'META-INF/LGPL2.1'

        }
        lintOptions {
            abortOnError false
        }
    }
    repositories {
        jcenter()
        maven { url "https://dl.bintray.com/drummer-aidan/maven" }
        mavenCentral()
    }
    configurations {
        compile.exclude group: "org.apache.httpcomponents", module: "httpclient"
    }
    dependencies {
        compile fileTree(dir: 'libs', include: ['*.jar'])
        // https://mvnrepository.com/artifact/commons-io/commons-io
        compile group: 'commons-io', name: 'commons-io', version: '2.4'

        compile 'com.android.support:appcompat-v7:' + rootProject.ext.supportLibraryVersion
        compile 'com.android.support:cardview-v7:' + rootProject.ext.supportLibraryVersion
        compile 'com.android.support:recyclerview-v7:' + rootProject.ext.supportLibraryVersion
        compile 'com.android.support:support-v4:' + rootProject.ext.supportLibraryVersion
        compile 'com.android.support:design:' + rootProject.ext.supportLibraryVersion
        compile 'com.android.support:percent:' + rootProject.ext.supportLibraryVersion
        compile 'com.android.support:animated-vector-drawable:' + rootProject.ext.supportLibraryVersion
        compile 'com.android.support:support-annotations:' + rootProject.ext.supportLibraryVersion
        compile 'com.android.support:support-compat:' + rootProject.ext.supportLibraryVersion
        compile 'com.android.support:support-core-ui:' + rootProject.ext.supportLibraryVersion
        compile 'com.android.support:support-v13:' + rootProject.ext.supportLibraryVersion
        compile 'com.android.support:support-v4:' + rootProject.ext.supportLibraryVersion'        

        compile 'com.google.firebase:firebase-storage:' + rootProject.ext.firebaseVersion
        compile 'com.google.firebase:firebase-common:' + rootProject.ext.firebaseVersion
        compile 'com.google.firebase:firebase-auth:' + rootProject.ext.firebaseVersion
        compile 'com.google.firebase:firebase-database:' + rootProject.ext.firebaseVersion
        compile 'com.google.firebase:firebase-crash:' + rootProject.ext.firebaseVersion
        compile 'com.google.firebase:firebase-core:' + rootProject.ext.firebaseVersion
        // this line must be included to integrate with Firebase
        compile 'com.google.firebase:firebase-messaging:' + rootProject.ext.firebaseVersion
        // this line must be included to use FCM
        compile 'com.google.firebase:firebase-ads:' + rootProject.ext.firebaseVersion

        compile 'com.google.android.gms:play-services-auth:' + rootProject.ext.firebaseVersion
        compile 'com.google.android.gms:play-services-plus:' + rootProject.ext.firebaseVersion
        compile 'com.google.android.gms:play-services-location:' + rootProject.ext.firebaseVersion
        compile 'com.google.android.gms:play-services-places:' + rootProject.ext.firebaseVersion


    compile "im.ene.toro2:toro-ext-exoplayer2:${toroVersion}"
    compile group: 'com.google.code.gson', name: 'gson', version: '2.8.0'


    compile 'com.jakewharton:butterknife:' + rootProject.ext.butterknifeVersion
    apt 'com.jakewharton:butterknife-compiler:' + rootProject.ext.butterknifeVersion

    //RxAndroid
    compile 'io.reactivex:rxandroid:' + rootProject.ext.rxandroidVersion
    compile 'io.reactivex:rxjava:' + rootProject.ext.rxjavaVersion
    compile 'com.squareup.retrofit2:adapter-rxjava:' + rootProject.ext.retroRxAdapterVersion
    compile 'com.squareup.picasso:picasso:2.5.2'
    compile 'com.google.firebase:firebase-auth:10.0.1'
    compile 'com.firebaseui:firebase-ui-database:0.4.0'
    compile 'com.google.firebase:firebase-invites:10.2.1'
    compile 'com.firebase:firebase-client-android:2.5.2+'
    compile 'com.facebook.android:facebook-android-sdk:[4,5)'
    compile 'com.android.support:multidex:1.0.1'
    compile 'pub.devrel:easypermissions:0.1.5'
    compile 'com.github.bumptech.glide:glide:3.6.0'
    compile 'de.hdodenhof:circleimageview:2.0.0'
    compile 'com.fasterxml.jackson.core:jackson-databind:2.7.3'
    compile 'com.danikula:videocache:2.6.4'
    compile 'com.android.support.constraint:constraint-layout:1.0.2'
    compile 'com.dinuscxj:circleprogressbar:1.1.1'
    compile 'com.afollestad.material-dialogs:core:0.9.4.5'
    compile 'com.afollestad:easyvideoplayer:0.3.0'
    compile 'com.googlecode.mp4parser:isoparser:1.1.22'
    compile 'javax.inject:javax.inject:1'
    compile 'com.google.firebase:firebase-storage:10.0.1'
    testCompile 'junit:junit:4.12'
    compile 'com.google.api-client:google-api-client:1.21.0'
    compile 'com.google.apis:google-api-services-people:v1-rev2-1.21.0'
    compile 'org.elasticsearch.client:transport:5.0.0'

    compile project(':ffmpeg4android_lib')

}


apply plugin: 'com.google.gms.google-services'

PS: There are a lot of similar posts already on SO, but none of them helped me

Can someone help me with this? Also, please tell me a generic way to solve these issues.

Answer 1

Do not forget to put

packagingOptions {
    pickFirst  'META-INF/*'
}

to all build gradle files if You use modules.

Answer 2

It is not advisable to use ElasticSearch directly into the android application. The cons of doing so are:

1. Anyone can decompile your application and get your ElasticSearch instance REST API link and play with your indexes the way they want and even modify them or delete them. The API link has to be publicly accessible for users to access.
2. There will be dependency conflicts like this question has, between Android SDK dependencies and ElasticSearch dependencies.
3. If you want to customize the way your requests are being built, you will have to release an update for the Application.

A better way to go about this is, make your own API to handle requests for your ElasticSearch Instance. From the App, make requests on that instance, that instance in turn makes a request on your ElasticSearch instance. When someone hits your API, you can get the variables for the search and build your own query. Pros of this method are:

1. Your ElasticSearch link can be private and nobody can delete your indexes.
2. You do not need to update the code for the Apps when you want to modify the way your queries are being built.
3. You do not get dependency conflicts.

Answer 3

This github issue ticket suggests to use

 packagingOptions {
        pickFirst 'META-INF/INDEX.LIST'
        pickFirst 'META-INF/LICENSE'
        pickFirst 'META-INF/io.netty.versions.properties'
    }

Answer 4

add the code below at the android section

packagingOptions {
    pickFirst  'META-INF/*'
}