Get a list of resources from classpath directory

Question

I am looking for a way to get a list of all resource names from a given classpath directory, something like a method List<String> getResourceNames (String directoryName).

For example, given a classpath directory x/y/z containing files a.html, b.html, c.html and a subdirectory d, getResourceNames("x/y/z") should return a List<String> containing the following strings:['a.html', 'b.html', 'c.html', 'd'].

It should work both for resources in filesystem and jars.

I know that I can write a quick snippet with Files, JarFiles and URLs, but I do not want to reinvent the wheel. My question is, given existing publicly available libraries, what is the quickest way to implement getResourceNames? Spring and Apache Commons stacks are both feasible.

Answer 1

I found that a lot of solutions only work when a) executing from a jar or b) executing in a development environment, but none seem to support both scenarios. Additionally, I had a scenario where some resources were located within my project while others were located in external jars (such as dependencies).

This is this the solution I created to address these scenarios:

public class ResourceWalker {

    private Path rootPath;
    private boolean inJar;
    private Class<?> context;

    public ResourceWalker(Class<?> context) throws IOException {
        if (context.isAnonymousClass())
            throw new IllegalArgumentException("Context cannot be an anonymous class");
        if (context.isLocalClass())
            throw new IllegalArgumentException("Context cannot be a local class");
        if (context.isSynthetic())
            throw new IllegalArgumentException("Context cannot be synthetic");

        this.context = context;
        URL contextUrl = context.getResource('/' + context.getName().replace('.', '/') + ".class");

        if (this.inJar = contextUrl.getProtocol().equals("jar")) {
            URLConnection connection = contextUrl.openConnection();
            if (!(connection instanceof JarURLConnection))
                throw new RuntimeException("Cannot determine jar for context: " + contextUrl);
            this.rootPath = Paths.get(((JarURLConnection) connection).getJarFile().getName());
        } else {
            try {
                this.rootPath = Paths.get(context.getProtectionDomain().getCodeSource().getLocation().toURI());
            } catch (URISyntaxException e) {
                throw new RuntimeException(e);
            }
        }

    }

    public void walkTree(String resourcePath, FileVisitor<String> visitor) throws IOException {
        if (this.inJar) {
            try (FileSystem system = FileSystems.newFileSystem(this.rootPath, null)) {
                Path path = system.getPath(resourcePath);
                Files.walkFileTree(path, new ResourceVisitorAdapter(visitor));
            }
        } else if (resourcePath.equals("/")) {
            this.walk(resourcePath, resourcePath, visitor);
        } else {
            this.walk(resourcePath, resourcePath + "/", visitor);
        }
    }

    private FileVisitResult walk(String path, String base, FileVisitor<String> visitor) throws IOException {
        FileVisitResult result = visitor.preVisitDirectory(path, null);
        switch (result) {
            case SKIP_SUBTREE:
                return FileVisitResult.CONTINUE;
            case SKIP_SIBLINGS:
            case TERMINATE:
                return result;
            default:
                break;
        }

        try (BufferedReader reader = new BufferedReader(
                new InputStreamReader(this.context.getResourceAsStream(path)))) {
            String entry;
            while ((entry = reader.readLine()) != null) {
                entry = base + entry;

                if (this.isDirectory(entry))
                    result = this.walk(entry, entry + "/", visitor);
                else
                    result = visitor.visitFile(entry, null);

                switch (result) {
                    case SKIP_SIBLINGS:
                    case TERMINATE:
                        return result;
                    default:
                        break;
                }
            }
        }

        return visitor.postVisitDirectory(path, null);
    }

    private boolean isDirectory(String entry) {
        if (entry.charAt(0) == '/')
            entry = entry.substring(1);
        return Files.isDirectory(this.rootPath.resolve(entry));
    }

    private static class ResourceVisitorAdapter implements FileVisitor<Path> {

        private FileVisitor<String> delegate;

        private ResourceVisitorAdapter(FileVisitor<String> delegate) {
            this.delegate = delegate;
        }

        @Override
        public FileVisitResult preVisitDirectory(Path dir, BasicFileAttributes attrs) throws IOException {
            return this.delegate.preVisitDirectory(dir.toString(), attrs);
        }

        @Override
        public FileVisitResult visitFile(Path file, BasicFileAttributes attrs) throws IOException {
            return this.delegate.visitFile(file.toString(), attrs);
        }

        @Override
        public FileVisitResult visitFileFailed(Path file, IOException exc) throws IOException {
            return this.delegate.visitFileFailed(file.toString(), exc);
        }

        @Override
        public FileVisitResult postVisitDirectory(Path dir, IOException exc) throws IOException {
            return this.delegate.postVisitDirectory(dir.toString(), exc);
        }

    }

}

---

You'll notice that the constructor takes a class parameter. This is to help the scanner identify where the target resources live. For example, if I have an external jar emojis.jar which contains

1. A resource folder: /images/emojis
2. A class: com.goof.emojis.Emojis

I can scan the contents of the emojis folder by providing Emojis.class as the "context" class:

FileVisitor<String> visitor = ...;
ResourceWalker walker = new ResourceWalker(com.goof.emojis.Emojis.class);
walker.walkTree("/images/emojis", visitor);

---

I wanted to comment on this line:

URL contextUrl = context.getResource('/' + context.getName().replace('.', '/') + ".class");

You may be wondering why I am using this approach to locate the jar's disk location when I could just use ProtectionDomain#getCodeSource. I learned the hard way that classes loaded with the bootstrap class loader will always return the same CodeSource, even if they originated from different locations. My workaround above was inspired by this thread: Determine which JAR file a class is from.

Hopefully this helps.

Answer 2

Using Reflections.

This answer was valid when this questions was asked. Subsequently I don't think the Reflections project has been maintained too consistently. I am leaving the answer here, but I would personally look at some of the other answers now for example the ones using classgraph.

---

Original answer:

Using Reflections.

Get everything on the classpath:

Reflections reflections = new Reflections(null, new ResourcesScanner());
Set<String> resourceList = reflections.getResources(x -> true);

Another example - get all files with extension .csv from some.package:

Reflections reflections = new Reflections("some.package", new ResourcesScanner());
Set<String> resourceList = reflections.getResources(Pattern.compile(".*\\.csv"));

Please note: (update)

This answer was written in 2015 and probably used Reflections 0.9.10. Subsequently I've tried the later versions of the Reflections library and struggled getting this answer to work. I also struggled to get examples from their own documentation to work... Personally I'm not sure what the maintainers of the project is doing... Maybe it's an error on my part.

The Reflections project was migrated from the Google Code Archive at version 0.9.9-RC1 as is evident [here]. The project is currently being maintained on github [here].

Answer 3

import org.springframework.beans.factory.annotation.Value;
import org.springframework.core.io.Resource;
import org.springframework.stereotype.Component;

@Component
public class FilesService {
    @Value("classpath:files/*.txt")
    private Resource[] files;

    @Value("classpath:files/specificFile.txt")
    private Resource file;

    //code to work with files
}

As you see it's very simple. And it works in runtime, when your jar was run on the server.

Answer 4

For root directory:

List<String> fileNames = IOUtils.readLines(MyClass.class.getClassLoader().getResourceAsStream("."), StandardCharsets.UTF_8);

Answer 5

path-matching-resource-pattern-resolver

I extracted Spring Framework's PathMatchingResourcePatternResolver into a standalone library for those that can't or don't want to import the whole Spring Framework library.

GitHub

https://github.com/SecretX33/path-matching-resource-pattern-resolver

Add the dependency

Maven

<dependency>
    <groupId>io.github.secretx33</groupId>
    <artifactId>path-matching-resource-pattern-resolver</artifactId>
    <version>0.1</version>
</dependency>

Gradle

implementation 'io.github.secretx33:path-matching-resource-pattern-resolver:0.1'

Gradle (KTS)

implementation("io.github.secretx33:path-matching-resource-pattern-resolver:0.1")

Usage example

Find a single resource

ResourcePatternResolver resolver = new PathMatchingResourcePatternResolver();
Resource resource = resolver.getResource("classpath:com/example/config.properties");

if (resource.exists()) {
    // Process the resource
}

Find multiple resources

ResourcePatternResolver resolver = new PathMatchingResourcePatternResolver();
Resource[] resources = resolver.getResources("classpath:com/example/**/*.xml");

for (Resource resource : resources) {
    // Process the resource
}

Answer 6

The most robust mechanism for listing all resources in the classpath is currently to use this pattern with ClassGraph, because it handles the widest possible array of classpath specification mechanisms, including the new JPMS module system. (I am the author of ClassGraph.)

List<String> resourceNames;
try (ScanResult scanResult = new ClassGraph().acceptPaths("x/y/z").scan()) {
    resourceNames = scanResult.getAllResources().getNames();
}

Answer 7

Expanding on Luke Hutchinsons answer above, using his ClassGraph library, I was able to easily get a list of all files in a Resource folder with almost no effort at all.

Let's say that in your resource folder, you have a folder called MyImages. This is how easy it is to get a URL list of all the files in that folder:

import io.github.classgraph.ClassGraph;
import io.github.classgraph.ResourceList;
import io.github.classgraph.ScanResult;

public static LinkedList<URL> getURLList (String folder) {
    LinkedList<URL> urlList    = new LinkedList<>();
    ScanResult      scanResult = new ClassGraph().enableAllInfo().scan();
    ResourceList    resources  = scanResult.getAllResources();
    for (URL url : resources.getURLs()) {
        if (url.toString().contains(folder)) {
            urlList.addLast(url);
        }
    }
    return urlList;
}

Then you simply do this:

LinkedList<URL> myURLFileList = getURLList("MyImages");

The URLs can then be loaded into streams or use Apache's FileUtils to copy the files somewhere else like this:

String outPath = "/My/Output/Path";
for(URL url : myURLFileList) {
    FileUtils.copyURLToFile(url, new File(outPath, url.getFile()));
}

I think ClassGraph is a pretty slick library for making tasks like this very simple and easy to comprehend.

Answer 8

Used a combination of Rob's response.

final String resourceDir = "resourceDirectory/";
List<String> files = IOUtils.readLines(Thread.currentThread().getClass().getClassLoader().getResourceAsStream(resourceDir), Charsets.UTF_8);

for (String f : files) {
  String data = IOUtils.toString(Thread.currentThread().getClass().getClassLoader().getResourceAsStream(resourceDir + f));
  // ... process data
}

Answer 9

Custom Scanner

Implement your own scanner. For example:

(limitations of this solution are mentioned in the comments)

private List<String> getResourceFiles(String path) throws IOException {
    List<String> filenames = new ArrayList<>();

    try (
            InputStream in = getResourceAsStream(path);
            BufferedReader br = new BufferedReader(new InputStreamReader(in))) {
        String resource;

        while ((resource = br.readLine()) != null) {
            filenames.add(resource);
        }
    }

    return filenames;
}

private InputStream getResourceAsStream(String resource) {
    final InputStream in
            = getContextClassLoader().getResourceAsStream(resource);

    return in == null ? getClass().getResourceAsStream(resource) : in;
}

private ClassLoader getContextClassLoader() {
    return Thread.currentThread().getContextClassLoader();
}

Spring Framework

Use PathMatchingResourcePatternResolver from Spring Framework.

Ronmamo Reflections

The other techniques might be slow at runtime for huge CLASSPATH values. A faster solution is to use ronmamo's Reflections API, which precompiles the search at compile time.

Answer 10

My way, no Spring, used during a unit test:

URI uri = TestClass.class.getResource("/resources").toURI();
Path myPath = Paths.get(uri);
Stream<Path> walk = Files.walk(myPath, 1);
for (Iterator<Path> it = walk.iterator(); it.hasNext(); ) {
    Path filename = it.next();   
    System.out.println(filename);
}

Answer 11

Neither of answers worked for me even though I had my resources put in resources folders and followed the above answers. What did make a trick was:

@Value("file:*/**/resources/**/schema/*.json")
private Resource[] resources;

Answer 12

I think you can leverage the [Zip File System Provider][1] to achieve this. When using FileSystems.newFileSystem it looks like you can treat the objects in that ZIP as a "regular" file.

In the linked documentation above:

Specify the configuration options for the zip file system in the java.util.Map object passed to the FileSystems.newFileSystem method. See the [Zip File System Properties][2] topic for information about the provider-specific configuration properties for the zip file system.

Once you have an instance of a zip file system, you can invoke the methods of the [java.nio.file.FileSystem][3] and [java.nio.file.Path][4] classes to perform operations such as copying, moving, and renaming files, as well as modifying file attributes.

The documentation for the jdk.zipfs module in [Java 11 states][5]:

The zip file system provider treats a zip or JAR file as a file system and provides the ability to manipulate the contents of the file. The zip file system provider can be created by [FileSystems.newFileSystem][6] if installed.

Here is a contrived example I did using your example resources. Note that a .zip is a .jar, but you could adapt your code to instead use classpath resources:

Setup

cd /tmp
mkdir -p x/y/z
touch x/y/z/{a,b,c}.html
echo 'hello world' > x/y/z/d
zip -r example.zip x

Java

import java.io.IOException;
import java.net.URI;
import java.nio.file.FileSystem;
import java.nio.file.FileSystems;
import java.nio.file.Files;
import java.util.Collections;
import java.util.stream.Collectors;

public class MkobitZipRead {

  public static void main(String[] args) throws IOException {
    final URI uri = URI.create("jar:file:/tmp/example.zip");
    try (
        final FileSystem zipfs = FileSystems.newFileSystem(uri, Collections.emptyMap());
    ) {
      Files.walk(zipfs.getPath("/")).forEach(path -> System.out.println("Files in zip:" + path));
      System.out.println("-----");
      final String manifest = Files.readAllLines(
          zipfs.getPath("x", "y", "z").resolve("d")
      ).stream().collect(Collectors.joining(System.lineSeparator()));
      System.out.println(manifest);
    }
  }

}

Output

Files in zip:/
Files in zip:/x/
Files in zip:/x/y/
Files in zip:/x/y/z/
Files in zip:/x/y/z/c.html
Files in zip:/x/y/z/b.html
Files in zip:/x/y/z/a.html
Files in zip:/x/y/z/d
-----
hello world

Answer 13

This should work (if spring is not an option):

public static List<String> getFilenamesForDirnameFromCP(String directoryName) throws URISyntaxException, UnsupportedEncodingException, IOException {
    List<String> filenames = new ArrayList<>();

    URL url = Thread.currentThread().getContextClassLoader().getResource(directoryName);
    if (url != null) {
        if (url.getProtocol().equals("file")) {
            File file = Paths.get(url.toURI()).toFile();
            if (file != null) {
                File[] files = file.listFiles();
                if (files != null) {
                    for (File filename : files) {
                        filenames.add(filename.toString());
                    }
                }
            }
        } else if (url.getProtocol().equals("jar")) {
            String dirname = directoryName + "/";
            String path = url.getPath();
            String jarPath = path.substring(5, path.indexOf("!"));
            try (JarFile jar = new JarFile(URLDecoder.decode(jarPath, StandardCharsets.UTF_8.name()))) {
                Enumeration<JarEntry> entries = jar.entries();
                while (entries.hasMoreElements()) {
                    JarEntry entry = entries.nextElement();
                    String name = entry.getName();
                    if (name.startsWith(dirname) && !dirname.equals(name)) {
                        URL resource = Thread.currentThread().getContextClassLoader().getResource(name);
                        filenames.add(resource.toString());
                    }
                }
            }
        }
    }
    return filenames;
}

Answer 14

So in terms of the PathMatchingResourcePatternResolver this is what is needed in the code:

@Autowired
ResourcePatternResolver resourceResolver;

public void getResources() {
  resourceResolver.getResources("classpath:config/*.xml");
}

Answer 15

If you use apache commonsIO you can use for the filesystem (optionally with extension filter):

Collection<File> files = FileUtils.listFiles(new File("directory/"), null, false);

and for resources/classpath:

List<String> files = IOUtils.readLines(MyClass.class.getClassLoader().getResourceAsStream("directory/"), Charsets.UTF_8);

If you don't know if "directoy/" is in the filesystem or in resources you may add a

if (new File("directory/").isDirectory())

or

if (MyClass.class.getClassLoader().getResource("directory/") != null)

before the calls and use both in combination...

Answer 16

With Spring it's easy. Be it a file, or folder, or even multiple files, there are chances, you can do it via injection.

This example demonstrates the injection of multiple files located in x/y/z folder.

import org.springframework.beans.factory.annotation.Value;
import org.springframework.core.io.Resource;
import org.springframework.stereotype.Service;

@Service
public class StackoverflowService {
    @Value("classpath:x/y/z/*")
    private Resource[] resources;

    public List<String> getResourceNames() {
        return Arrays.stream(resources)
                .map(Resource::getFilename)
                .collect(Collectors.toList());
    }
}

It does work for resources in the filesystem as well as in JARs.

Answer 17

The Spring framework's PathMatchingResourcePatternResolver is really awesome for these things:

private Resource[] getXMLResources() throws IOException
{
    ClassLoader classLoader = MethodHandles.lookup().getClass().getClassLoader();
    PathMatchingResourcePatternResolver resolver = new PathMatchingResourcePatternResolver(classLoader);

    return resolver.getResources("classpath:x/y/z/*.xml");
}

Maven dependency:

<dependency>
    <groupId>org.springframework</groupId>
    <artifactId>spring-core</artifactId>
    <version>LATEST</version>
</dependency>

Answer 18

Based on @rob 's information above, I created the implementation which I am releasing to the public domain:

private static List<String> getClasspathEntriesByPath(String path) throws IOException {
    InputStream is = Main.class.getClassLoader().getResourceAsStream(path);

    StringBuilder sb = new StringBuilder();
    while (is.available()>0) {
        byte[] buffer = new byte[1024];
        sb.append(new String(buffer, Charset.defaultCharset()));
    }

    return Arrays
            .asList(sb.toString().split("\n"))          // Convert StringBuilder to individual lines
            .stream()                                   // Stream the list
            .filter(line -> line.trim().length()>0)     // Filter out empty lines
            .collect(Collectors.toList());              // Collect remaining lines into a List again
}

While I would not have expected getResourcesAsStream to work like that on a directory, it really does and it works well.

Answer 19

Here is the code
Source: forums.devx.com/showthread.php?t=153784

import java.io.File;
import java.io.IOException;
import java.util.ArrayList;
import java.util.Collection;
import java.util.Enumeration;
import java.util.regex.Pattern;
import java.util.zip.ZipEntry;
import java.util.zip.ZipException;
import java.util.zip.ZipFile;

/**
 * list resources available from the classpath @ *
 */
public class ResourceList{

    /**
     * for all elements of java.class.path get a Collection of resources Pattern
     * pattern = Pattern.compile(".*"); gets all resources
     * 
     * @param pattern
     *            the pattern to match
     * @return the resources in the order they are found
     */
    public static Collection<String> getResources(
        final Pattern pattern){
        final ArrayList<String> retval = new ArrayList<String>();
        final String classPath = System.getProperty("java.class.path", ".");
        final String[] classPathElements = classPath.split(System.getProperty("path.separator"));
        for(final String element : classPathElements){
            retval.addAll(getResources(element, pattern));
        }
        return retval;
    }

    private static Collection<String> getResources(
        final String element,
        final Pattern pattern){
        final ArrayList<String> retval = new ArrayList<String>();
        final File file = new File(element);
        if(file.isDirectory()){
            retval.addAll(getResourcesFromDirectory(file, pattern));
        } else{
            retval.addAll(getResourcesFromJarFile(file, pattern));
        }
        return retval;
    }

    private static Collection<String> getResourcesFromJarFile(
        final File file,
        final Pattern pattern){
        final ArrayList<String> retval = new ArrayList<String>();
        ZipFile zf;
        try{
            zf = new ZipFile(file);
        } catch(final ZipException e){
            throw new Error(e);
        } catch(final IOException e){
            throw new Error(e);
        }
        final Enumeration e = zf.entries();
        while(e.hasMoreElements()){
            final ZipEntry ze = (ZipEntry) e.nextElement();
            final String fileName = ze.getName();
            final boolean accept = pattern.matcher(fileName).matches();
            if(accept){
                retval.add(fileName);
            }
        }
        try{
            zf.close();
        } catch(final IOException e1){
            throw new Error(e1);
        }
        return retval;
    }

    private static Collection<String> getResourcesFromDirectory(
        final File directory,
        final Pattern pattern){
        final ArrayList<String> retval = new ArrayList<String>();
        final File[] fileList = directory.listFiles();
        for(final File file : fileList){
            if(file.isDirectory()){
                retval.addAll(getResourcesFromDirectory(file, pattern));
            } else{
                try{
                    final String fileName = file.getCanonicalPath();
                    final boolean accept = pattern.matcher(fileName).matches();
                    if(accept){
                        retval.add(fileName);
                    }
                } catch(final IOException e){
                    throw new Error(e);
                }
            }
        }
        return retval;
    }

    /**
     * list the resources that match args[0]
     * 
     * @param args
     *            args[0] is the pattern to match, or list all resources if
     *            there are no args
     */
    public static void main(final String[] args){
        Pattern pattern;
        if(args.length < 1){
            pattern = Pattern.compile(".*");
        } else{
            pattern = Pattern.compile(args[0]);
        }
        final Collection<String> list = ResourceList.getResources(pattern);
        for(final String name : list){
            System.out.println(name);
        }
    }
}  

If you are using Spring Have a look at PathMatchingResourcePatternResolver