How to persist nifi flowfiles by using docker-compose?

Question

right now I'm using nifi and its processors for some streaming stuff (mqtt listener, json evaluating, text replacement, write into db ...). I'm trying to persist the flowfiles and therefore I did some volume mapping (see below). But it doesn't work; after restarting the container it seems the flowfiles arent't saved ...

Could anybody give me a hint how to solve that problem?

nifi:
image: apache/nifi
restart: on-failure
ports:
  - "8000:8000"
networks:
  - traefik
environment:
  - NIFI_WEB_HTTP_PORT=8000
volumes:
  - nifi_conf:/opt/nifi/conf
  - nifi_state:/data/nifi/state
  - nifi_db:/opt/nifi/database_repository
  - nifi_flowfile:/opt/nifi/flowfile_repository
  - nifi_content:/opt/nifi/content_repository
  - nifi_provenance:/opt/nifi/provenance_repository 


volumes:
 nifi_provenance:{}
 nifi_flowfile: {}
 nifi_content: {}
 nifi_db: {}
 nifi_state: {}
 nifi_conf: {}

Thanks.

Answer 1

Well the question is old, but still seems to get some attention. So, let me add my learnings and some background info regarding the current NiFi 2.2.

Actually NiFi 2.2's current docker image keeps all data in /opt/nifi, but it's not working to reference just this folder because the Apache Nifi 2.2 Dockerfile defines 9 (!) separate folders as explicit mount points. You need to reference them explicitly to persist all that data:

services:
  nifi:
    image: apache/nifi:latest
    volumes:
      - nifi:/opt/nifi/nifi-current/logs
      - nifi:/opt/nifi/nifi-current/conf
      - nifi:/opt/nifi/nifi-current/database_repository
      - nifi:/opt/nifi/nifi-current/flowfile_repository
      - nifi:/opt/nifi/nifi-current/content_repository
      - nifi:/opt/nifi/nifi-current/provenance_repository
      - nifi:/opt/nifi/nifi-current/python_extensions
      - nifi:/opt/nifi/nifi-current/nar_extensions
      - nifi:/opt/nifi/nifi-current/state

volumes:
  nifi:

As others experienced, mounting an empty volume or binding folder over a container's folder will hide the previous files in the container and NiFi might not start up, because it's missing its initial configuration files. Luckily, docker will copy the container's data into the volume only if the volume is new (see Docker Volumes volume-nocopy). So, if you are in trouble, just remove/rename the volume and "docker compose up" again!

Answer 2

FINAL EDIT: After many tests and trials, the only way I have found to persist Nifi with docker swarm has been the following:

Step 1: create nifi_data volume

$ docker volume create nifi_data

Step 2: Start the stack with the following configuration

version: "3.7"
services:
  nifi:
    image: apache/nifi:1.9.2
    ports:
      - target: 8080
        published: 9090
        protocol: tcp
        mode: host
    environment:
      - NIFI_WEB_HTTP_HOST=0.0.0.0
      #- NIFI_HOME=/home/nifi
      #- NIFI_LOG_DIR=/home/nifi/logs

    volumes:
      - nifi_data:/home/nifi

volumes:
  nifi_data:
    external: true

Step 3: Enter the container

$ docker exec -it <container_id> bash

Step 4: Copy the current in the home

$ cd /opt/nifi/nifi-current
$ cp -r ./* /home/nifi

Step 5: Remove Deployed Stack

docker stack rm nifi

Step 6: Deploy stack with the following configuration (just removed the #)

version: "3.7"
services:
  nifi:
    image: apache/nifi:1.9.2
    ports:
      - target: 8080
        published: 9090
        protocol: tcp
        mode: host
    environment:
      - NIFI_WEB_HTTP_HOST=0.0.0.0
      - NIFI_HOME=/home/nifi
      - NIFI_LOG_DIR=/home/nifi/logs

    volumes:
      - nifi_data:/home/nifi

volumes:
  nifi_data:
    external: true

Answer 3

version: '3.7'

services:

  zookeeper:  # the configuration manager
    hostname: zookeeper
    container_name: zookeeper
    image: 'bitnami/zookeeper:latest'
    environment:
      - ALLOW_ANONYMOUS_LOGIN=yes

  nifi:
    image: apache/nifi:latest
    ports:
      - 8080 # Unsecured HTTP Web Port
    environment:
      - NIFI_WEB_HTTP_PORT=8080
      - NIFI_CLUSTER_IS_NODE=true
      - NIFI_CLUSTER_NODE_PROTOCOL_PORT=8082
      - NIFI_ZK_CONNECT_STRING=zookeeper:2181
      - NIFI_ELECTION_MAX_WAIT=1 min
    volumes:
      - ./nifi/state:/opt/nifi/nifi-current/state
      - ./nifi/db:/opt/nifi/nifi-current/database_repository
      - ./nifi/flowfile:/opt/nifi/nifi-current/flowfile_repository
      - ./nifi/content:/opt/nifi/nifi-current/content_repository
      - ./nifi/provenance:/opt/nifi/nifi-current/provenance_repository
      - ./nifi/logs:/opt/nifi/nifi-current/logs

Answer 4

One update from my side. Apache Nifi changed their directories after 1.8.0. So you should use the following:

volumes:
      - ./nifi_state:/opt/nifi/nifi-current/state
      - ./nifi_db:/opt/nifi/nifi-current/database_repository
      - ./nifi_flowfile:/opt/nifi/nifi-current/flowfile_repository
      - ./nifi_content:/opt/nifi/nifi-current/content_repository
      - ./nifi_provenance:/opt/nifi/nifi-current/provenance_repository   

Answer 5

Alternatively you could only use docker-compose stop instead of docker-compose down which will not remove your container and thus keeping the volumes mounted.

This means you do not have to do any mapping of volumes and can just use this basic docker-compose file:

version: '2'
services:
  futa-nifi-lsc:
    environment:
      - NIFI_WEB_HTTP_PORT=9000
    image: apache/nifi:1.8.0
    volumes:
      - ./jdbc_driver:/opt/jdbc_driver
      - ./checkin_files:/opt/checkin_files
      - ./truststore:/opt/truststore
    ports:
      - "9000:9000"

For more information read this article here.

Answer 6

you could map docker container folders directly to the host machine like this:

services:
  nifi:
    ...
    volumes:
      - ./conf:/opt/conf
      - ./nifi_state:/data/nifi/state
      ...

no additional volume definition required

note that under windows with virtualbox this feature works only in the current user directory.