Lucene 4 on Android error

Question

I've had this error when using using Lucene 4 with Android API 19 (4.4.2).

Caused by: java.lang.IllegalArgumentException: A SPI class of type org.apache.lucene.codecs.Codec with name 'Lucene46' does not exist. You need to add the corresponding JAR file supporting this SPI to your classpath.The current classpath supports the following names: []
        at org.apache.lucene.util.NamedSPILoader.lookup(NamedSPILoader.java:109)
        at org.apache.lucene.codecs.Codec.forName(Codec.java:95)
        at org.apache.lucene.codecs.Codec.<clinit>(Codec.java:122) at org.apache.lucene.index.LiveIndexWriterConfig.<init>(LiveIndexWriterConfig.java:122) at org.apache.lucene.index.IndexWriterConfig.<init>(IndexWriterConfig.java:165)

which is a similar error as reported by https://stackoverflow.com/questions/18944634/running-lucene-4-4-on-android

I migrated to android-maven-plugin on IntelliJ because it was a suggested solution to my problem but I am still getting the issue. Here is my Maven pom.xml

<?xml version="1.0" encoding="UTF-8"?><project xmlns="http://maven.apache.org/POM/4.0.0"
     xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
     xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>

<groupId>com.johnbohne.chishiki</groupId>
<artifactId>Chishiki</artifactId>
<version>1.0-SNAPSHOT</version>

<packaging>jar</packaging>

<name>Chishiki</name>
<build>
    <sourceDirectory>src</sourceDirectory>
    <finalName>${project.artifactId}</finalName>
    <pluginManagement>
        <plugins>
            <plugin>
                <groupId>com.jayway.maven.plugins.android.generation2</groupId>
                <artifactId>android-maven-plugin</artifactId>
                <version>3.8.2</version>
                <extensions>true</extensions>
            </plugin>
        </plugins>
    </pluginManagement>
    <plugins>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId><artifactId>maven-compiler-plugin</artifactId><version>2.3.2</version>
            <configuration>
                <source>1.5</source><target>1.5</target>
            </configuration>
        </plugin>
    </plugins>
</build>

I know it's possible to use Lucene 3.0 with Android, but I'm not so sure about Lucene 4.0. Does anyone have any idea on how to get Lucene 4.0 working in Android?

Answer 1

Here is a way to run it (it's for sure not the best solution, but it works):

1. Remove org/apache/lucene/util/SPIClassIterator from the lucene-core
2. Drop a patched version into your source tree
3. Add the META-INF/services files to your assets folder
4. Merge duplicated services files by hand

I'm using the following (patched) version

package org.apache.lucene.util;

/*
 * Licensed to the Apache Software Foundation (ASF) under one or more
 * contributor license agreements.  See the NOTICE file distributed with
 * this work for additional information regarding copyright ownership.
 * The ASF licenses this file to You under the Apache License, Version 2.0
 * (the "License"); you may not use this file except in compliance with
 * the License.  You may obtain a copy of the License at
 *
 *     http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */

import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.net.URL;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collections;
import java.util.Enumeration;
import java.util.Iterator;
import java.util.Locale;
import java.util.NoSuchElementException;
import java.util.ServiceConfigurationError;

/**
 * Helper class for loading SPI classes from classpath (META-INF files).
 * This is a light impl of {@link java.util.ServiceLoader} but is guaranteed to
 * be bug-free regarding classpath order and does not instantiate or initialize
 * the classes found.
 *
 * @lucene.internal
 */
public final class SPIClassIterator<S> implements Iterator<Class<? extends S>> {
  private static final String META_INF_SERVICES = "/assets/META-INF/services/";

  private final Class<S> clazz;
  private final ClassLoader loader;
  private final Enumeration<URL> profilesEnum;
  private Iterator<String> linesIterator;

  public static <S> SPIClassIterator<S> get(Class<S> clazz) {
    return new SPIClassIterator<S>(clazz, Thread.currentThread().getContextClassLoader());
  }

  public static <S> SPIClassIterator<S> get(Class<S> clazz, ClassLoader loader) {
    return new SPIClassIterator<S>(clazz, loader);
  }

  /** Utility method to check if some class loader is a (grand-)parent of or the same as another one.
   * This means the child will be able to load all classes from the parent, too. */
  public static boolean isParentClassLoader(final ClassLoader parent, ClassLoader child) {
    while (child != null) {
      if (child == parent) {
        return true;
      }
      child = child.getParent();
    }
    return false;
  }

  private SPIClassIterator(Class<S> clazz, ClassLoader loader) {
    this.clazz = clazz;
    final String fullName = META_INF_SERVICES + clazz.getName();
    this.profilesEnum = 
      Collections.enumeration(
        Arrays.asList(new URL[]{getClass().getResource(fullName)}));
    this.loader = (loader == null) ? ClassLoader.getSystemClassLoader() : loader;
    this.linesIterator = Collections.<String>emptySet().iterator();
  }

  private boolean loadNextProfile() {
    ArrayList<String> lines = null;
    while (profilesEnum.hasMoreElements()) {
      if (lines != null) {
        lines.clear();
      } else {
        lines = new ArrayList<String>();
      }
      final URL url = profilesEnum.nextElement();
      try {
        final InputStream in = url.openStream();
        IOException priorE = null;
        try {
          final BufferedReader reader = new BufferedReader(new InputStreamReader(in, "UTF-8"));
          String line;
          while ((line = reader.readLine()) != null) {
            final int pos = line.indexOf('#');
            if (pos >= 0) {
              line = line.substring(0, pos);
            }
            line = line.trim();
            if (line.length() > 0) {
              lines.add(line);
            }
          }
        } catch (IOException ioe) {
          priorE = ioe;
        } finally {
          IOUtils.closeWhileHandlingException(priorE, in);
        }
      } catch (IOException ioe) {
        throw new ServiceConfigurationError("Error loading SPI class list from URL: " + url, ioe);
      }
      if (!lines.isEmpty()) {
        this.linesIterator = lines.iterator();
        return true;
      }
    }
    return false;
  }

  @Override
  public boolean hasNext() {
    return linesIterator.hasNext() || loadNextProfile();
  }

  @Override
  public Class<? extends S> next() {
    // hasNext() implicitely loads the next profile, so it is essential to call this here!
    if (!hasNext()) {
      throw new NoSuchElementException();
    }
    assert linesIterator.hasNext();
    final String c = linesIterator.next();
    try {
      // don't initialize the class (pass false as 2nd parameter):
      return Class.forName(c, false, loader).asSubclass(clazz);
    } catch (ClassNotFoundException cnfe) {
      throw new ServiceConfigurationError(String.format(Locale.ROOT, "A SPI class of type %s with classname %s does not exist, "+
        "please fix the file '%s%1$s' in your classpath.", clazz.getName(), c, META_INF_SERVICES));
    }
  }

  @Override
  public void remove() {
    throw new UnsupportedOperationException();
  }

}

The required changes?

1. META_INF_SERVICES points to /assets/META-INF/services/ instead of META-INF/services/
2. The constructor is different

EDIT: This is for lucene 4.7.1. I'm by now just sure that it can create an empty directory, I'm not sure if indexing works fully.

EDIT 2: I can now confirm that it is indeed working. At least manual Query creation and index creation work as expected.

EDIT 3: I have just finished my own contact merger based on this. The source code is available at github/rtreffer/ContactMerger if you are interested.

EDIT 4: I'm now able to build without a patched jar. I've simply dropped the SPI loader in the src folder, disabled library pre-dexing and let dx do it's job. It's slower but the src / classes implementation wins and I can pull all dependencies from maven central. See the build.gradle for details.