CODEWITHSUNDEEP
spring-integration
normalindon1
normalindon1

Reputation: 55

RecursiveLeafOnlyDirectoryScanner

I'm using a RecursiveLeafOnlyDirectoryScanner as a scanner in inbound-channel-adapter to launch a spring batch job, but getting heap space issue frequently. Below is my intgeration context:

<int-file:inbound-channel-adapter id="poll_IAG_File" scanner="recursiveScanner" directory="/tmp/DIR1/" channel="files" filter="compositeFilter">
    <int:poller id="pollerFlatFile" fixed-delay="10000" />
</int-file:inbound-channel-adapter>

Heap size in IBM websphere is : 1024(min) -2048(max). Need to know appropriate configurations for inbound channnel adapter or suggestions to avoid the heap issue.

Below is the system logs of IBM websphere. 

java.lang.OutOfMemoryError: Java heap space
        at java.lang.StringCoding.trim(StringCoding.java:455)
        at java.lang.StringCoding.encode(StringCoding.java:624)
        at java.lang.String.getBytes(String.java:694)
        at java.io.UnixFileSystem.list(Native Method)
        at java.io.File.list(File.java:984)
        at java.io.File.listFiles(File.java:1062)
        at org.springframework.integration.file.RecursiveLeafOnlyDirectoryScanner.listEligibleFiles(RecursiveLeafOnlyDirectoryScanner.java:35)
        at org.springframework.integration.file.RecursiveLeafOnlyDirectoryScanner.listEligibleFiles(RecursiveLeafOnlyDirectoryScanner.java:39)
        at org.springframework.integration.file.RecursiveLeafOnlyDirectoryScanner.listEligibleFiles(RecursiveLeafOnlyDirectoryScanner.java:39)
        at org.springframework.integration.file.RecursiveLeafOnlyDirectoryScanner.listEligibleFiles(RecursiveLeafOnlyDirectoryScanner.java:39)
        at org.springframework.integration.file.DefaultDirectoryScanner.listFiles(DefaultDirectoryScanner.java:64)
        at org.springframework.integration.file.FileReadingMessageSource.scanInputDirectory(FileReadingMessageSource.java:276)
        at org.springframework.integration.file.FileReadingMessageSource.receive(FileReadingMessageSource.java:250)
        at org.springframework.integration.endpoint.SourcePollingChannelAdapter.receiveMessage(SourcePollingChannelAdapter.java:111)
        at org.springframework.integration.endpoint.AbstractPollingEndpoint.doPoll(AbstractPollingEndpoint.java:184)
        at org.springframework.integration.endpoint.AbstractPollingEndpoint.access$000(AbstractPollingEndpoint.java:51)
        at org.springframework.integration.endpoint.AbstractPollingEndpoint$1.call(AbstractPollingEndpoint.java:143)
        at org.springframework.integration.endpoint.AbstractPollingEndpoint$1.call(AbstractPollingEndpoint.java:141)
        at org.springframework.integration.endpoint.AbstractPollingEndpoint$Poller$1.run(AbstractPollingEndpoint.java:273)
        at org.springframework.integration.util.ErrorHandlingTaskExecutor$1.run(ErrorHandlingTaskExecutor.java:52)
        at org.springframework.core.task.SyncTaskExecutor.execute(SyncTaskExecutor.java:49)
        at org.springframework.integration.util.ErrorHandlingTaskExecutor.execute(ErrorHandlingTaskExecutor.java:49)
        at org.springframework.integration.endpoint.AbstractPollingEndpoint$Poller.run(AbstractPollingEndpoint.java:268)
        at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:53)
        at org.springframework.scheduling.concurrent.ReschedulingRunnable.run(ReschedulingRunnable.java:81)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:450)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:314)
        at java.util.concurrent.FutureTask.run(FutureTask.java:149)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:109)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:217)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
        at java.lang.Thread.run(Thread.java:790)

Upvotes: 0

Views: 460

Answers (1)

Artem Bilan
Artem Bilan

Reputation: 121542

/**
 * DirectoryScanner that lists all files inside a directory and subdirectories,
 * without limit. This scanner should not be used with directories that contain
 * a vast number of files or on deep trees, as all the file names will be read
 * into memory and the scanning will be done recursively.
 *
 * @deprecated in favor of {@link FileReadingMessageSource#setUseWatchService(boolean)} (when using Java 7 or later)
 */
@Deprecated
public class RecursiveLeafOnlyDirectoryScanner extends DefaultDirectoryScanner {

But I don't think that it is an issue for you because even if RecursiveLeafOnlyDirectoryScanner loads all the files to the memory that doesn't mean that File object is so big.

I'm pretty sure that the problem is somewhere downstream, where you read a content of those files and don't free memory after usage. That might be an <aggregator> with the SimpleMessageStore (by default). That might be any custom FileListFilter which consults files content and store it in its local cache. And so on.

I suggests you to run some JVM tool to analyze the memory for objects which eat all your heap.

Upvotes: 1

Related Questions