spring integration file splitter sending file markers as payload

Question

I have spring xd source module which split text file line by line.I wanted to see file name and count the number of lines in file hence I am using file splitter with filemarkers.But problem is the if i have one record in file the file count is coming as payload and 3 lines are coming (1 record +2 from file markers start and end)hence my processor which is expecting the payload as a file record is getting some filemarkers.How can i make them as headers and not appear in payload

<?xml version="1.0" encoding="UTF-8"?>
    <beans xmlns="http://www.springframework.org/schema/beans"
           xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
           xmlns:int="http://www.springframework.org/schema/integration"
           xmlns:int-aws="http://www.springframework.org/schema/integration/aws"
           xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
            http://www.springframework.org/schema/integration http://www.springframework.org/schema/integration/spring-integration.xsd
            http://www.springframework.org/schema/integration/aws http://www.springframework.org/schema/integration/aws/spring-integration-aws-1.0.xsd">

        <int:poller fixed-delay="${fixed-delay}" default="true"/>

        <bean id="credentials" class="org.springframework.integration.aws.core.BasicAWSCredentials">
            <property name="accessKey" value="${accessKey}"/>
            <property name="secretKey" value="${secretKey}"/>
        </bean>

        <bean
                class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
            <property name="location">
                <value>dms-aws-s3-nonprod.properties</value>
            </property>
        </bean>

        <bean id="clientConfiguration" class="com.amazonaws.ClientConfiguration">
            <property name="proxyHost" value="${proxyHost}"/>
            <property name="proxyPort" value="${proxyPort}"/>
            <property name="preemptiveBasicProxyAuth" value="false"/>
        </bean>
    <bean id="s3Operations" class="org.springframework.integration.aws.s3.core.CustomC1AmazonS3Operations">
        <constructor-arg index="0" ref="credentials"/>
        <constructor-arg index="1" ref="clientConfiguration"/>
        <property name="awsEndpoint" value="s3.amazonaws.com"/>
        <property name="temporaryDirectory" value="${temporaryDirectory}"/>
        <property name="awsSecurityKey"  value="${awsSecurityKey}"/>
    </bean>



    <!-- aws-endpoint="https://s3.amazonaws.com"  -->
    <int-aws:s3-inbound-channel-adapter aws-endpoint="s3.amazonaws.com"
                                        bucket="${bucket}"
                                        s3-operations="s3Operations"
                                        credentials-ref="credentials"
                                        file-name-wildcard="${file-name-wildcard}"
                                        remote-directory="${remote-directory}"
                                        channel="splitChannel"
                                        local-directory="${local-directory}"
                                        accept-sub-folders="false"
                                        delete-source-files="true"
                                        archive-bucket="${archive-bucket}"
                                        archive-directory="${archive-directory}">
    </int-aws:s3-inbound-channel-adapter>

    int-file:splitter input-channel="splitChannel" output-channel="output" markers="true"/>

    <int:channel id="output"/>

    xd-shell>stream create feedTest16 --definition "aws-s3-source |processor| log" --deploy

The FileSplitter.FileMarker END message will contain a desired lineCount.

Answer 1

It's not possible; we could eliminate the start marker but the problem is we don't know that we've reached the end of file without doing the next read (which is when the end marker is emitted if we've reached EOF).

You could add a <filter/> to skip the start marker but there's no way to identify that the last "real" message is indeed the last.

You could add a transformer to transform the END marker to, say, an empty String.

We could, I suppose, add an option to the FileSplitter to read-ahead, but it doesn't do that now.

Feel free to open an Improvement JIRA Issue.

You could also create a custom splitter.