Camel read files from FTP and store in local resources folder

Question

I'm trying to build a simple route that reads from a FTP folder, and stores it in a local resources folder. I am able to connect to the FTP endpoint, but nothing happens after that.

To start my program I use: mvn clean compile camel:run

I'm not really sure what I should do next to debug this anymore.

Terminal output:

INFO | Apache Camel 2.20.0 (CamelContext: camel-1) is starting
 INFO | JMX is enabled
 INFO | Type converters loaded (core: 192, classpath: 4)
 INFO | StreamCaching is not in use. If using streams then its recommended to enable stream caching. See more details at http://camel.apache.org/stream-caching.html
 INFO | Route: route1 started and consuming from: ftp://xx.xx.xx.xx/ftp/xx/xx?password=xxxxxx&username=xx
 INFO | Total 1 routes, of which 1 are started
 INFO | Apache Camel 2.20.0 (CamelContext: camel-1) started in 0.333 seconds

pom.xml

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>com.nettport</groupId>
    <artifactId>camel_download_file_from_ftp_and_get_name_and_content</artifactId>
    <version>1.0-SNAPSHOT</version>




    <dependencies>
        <!-- Camel -->
        <dependency>
            <groupId>org.apache.camel</groupId>
            <artifactId>camel-core</artifactId>
            <version>2.20.0</version>
        </dependency>
        <dependency>
            <groupId>org.apache.camel</groupId>
            <artifactId>spi-annotations</artifactId>
            <version>2.20.0</version>
        </dependency>
        <dependency>
            <groupId>org.apache.camel</groupId>
            <artifactId>camel-jms</artifactId>
            <version>2.20.0</version>
        </dependency>

        <!-- Spring -->
        <dependency>
            <groupId>org.apache.camel</groupId>
            <artifactId>camel-spring</artifactId>
            <version>2.20.0</version>
        </dependency>
        <!-- //Spring -->

        <!-- FTP -->
        <dependency>
            <groupId>org.apache.camel</groupId>
            <artifactId>camel-ftp</artifactId>
            <version>2.20.0</version>
        </dependency>
        <!-- //FTP -->

        <!-- Quartz -->
        <dependency>
            <groupId>org.apache.camel</groupId>
            <artifactId>camel-quartz</artifactId>
            <version>2.20.0</version>
        </dependency>
        <!-- //Quartz -->

        <!-- ActiveMq -->
        <dependency>
            <groupId>org.apache.activemq</groupId>
            <artifactId>activemq-all</artifactId>
            <version>5.15.1</version>
        </dependency>


        <!-- Logging -->
        <dependency>
            <groupId>org.slf4j</groupId>
            <artifactId>slf4j-log4j12</artifactId>
            <version>1.8.0-alpha2</version>
        </dependency>
        <!-- //Logging -->

    </dependencies>


    <build>
        <plugins>
            <!-- Allows the routes to be run via 'mvn camel:run' -->
            <plugin>
                <groupId>org.apache.camel</groupId>
                <artifactId>camel-maven-plugin</artifactId>
                <version>2.20.0</version>
            </plugin>
        </plugins>
    </build>

</project>

src/main/java/com/nettport/ReceiverRoute.java:

package com.nettport;

import org.apache.camel.builder.RouteBuilder;

public class ReceiverRoute extends RouteBuilder {

    private String receiverFtpEndpoint;

    public void configure() throws Exception {

        // lets shutdown faster in case of in-flight messages stack up
        getContext().getShutdownStrategy().setTimeout(10);

        from(receiverFtpEndpoint)
        .log("### FTP Receiver Route started and consuming ###")
        .to("file:data/work_in_progress")
        .log("Downloaded file ${file:name} complete.");
    }


    public void setReceiverFtpEndpoint(String receiverFtpEndpoint) {
        this.receiverFtpEndpoint = receiverFtpEndpoint;
    }
}

META-INF/spring/camel-contxt.xml

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xsi:schemaLocation="
       http://www.springframework.org/schema/beans
       http://www.springframework.org/schema/beans/spring-beans-3.0.xsd
       http://camel.apache.org/schema/spring
       http://camel.apache.org/schema/spring/camel-spring.xsd">


    <bean id="receiverRoute" class="com.nettport.ReceiverRoute">
        <property name="receiverFtpEndpoint" value="ftp://xx.xx.xx.xx/ftp/xx/xx?username=xx&amp;password=xx"/>
    </bean>


    <camelContext xmlns="http://camel.apache.org/schema/spring">
        <routeBuilder ref="receiverRoute"/>
    </camelContext>

</beans>

Answer 1

Ok, colleagues. In case if somebody has troubles like I have set up a simple route that copies the files form FTP to my local disk folder and it does nothing try the following:

• Make sure you DO NOT have suspicious WARN or ERROR level messages in your logs.

• Make sure you have specified the camel-ftp or camel-ftp-starter in your pom.xml. I have set up a Spring Boot application and I have the following dependency in my pom.xml

  <dependency>
      <groupId>org.apache.camel</groupId>
      <artifactId>camel-ftp-starter</artifactId>
      <version>3.0.0-M2</version>
  </dependency>
  

• Make sure you have read this documentation:

  • https://github.com/apache/camel/blob/master/components/readme.adoc
  • http://camel.apache.org/ftp2.html
• Make sure you have setup everything correctly at your FTP server side:
  • You have a correct schema
  • You have set the user and a password up
• Make sure the directory you want to DOWNLOAD the files FROM (from your FTP server) is NOT EMPTY
  • In my case (see below) the folder I use to copy the files FROM is /home/lordnighton
  • Use the FileZilla client to check the state of folder (it also helps you to copy the files to your FTP server)
• Make sure you FTP server is up and running

Run the commands one by one to spin up an FTP server as a Docker container

    $ docker pull stilliard/pure-ftpd:hardened
    $ docker run -e FTP_USER_NAME=lord -e FTP_USER_PASS=nighton -e FTP_USER_HOME=/home/lordnighton  -d --name ftpd_server -p 21:21 -p 30000-30009:30000-30009 -e "PUBLICHOST=localhost" stilliard/pure-ftpd:hardened
    $ ftp ftp://lord:nighton@localhost:21 # check the connection with user/password

• Specify the route in your class that extends RouteBuilder:

  from("ftp://lord@localhost:21/home/lordnighton?password=nighton&passiveMode=true")
      .to("file:src/main/resources/data/from-ftp");
  

• You can also switch the logging levels of camel's components to TRACE to look through more verbose logs:

  log4j.logger.org.apache.commons.net=TRACE
  log4j.logger.org.apache.camel.component.file=TRACE
  log4j.logger.org.apache.camel.component.ftp=TRACE
  

• Sometimes it is helpful to switch the passiveMode to true:

  from("ftp://lord@localhost:21/home/lordnighton?password=nighton&passiveMode=true")
  

• Sometimes it is also helpful to look into this example -- https://github.com/apache/camel/tree/master/examples/camel-example-ftp

Good luck in debugging!