Edge device management using NiFI and MiNiFi

Question

I am looking for device management of Edge devices from Central server remotely.

Example:- I am using number of Raspberry PI devices [say 100] to aggregate sensor data and pushing that data to the central server With MQTT service.

I am looking for some sort managing Raspberry devices remotely from central server. Like upgrade software component / Linux version on all 100 of Raspberry devices.

Can I accomplish these type of use cases by using MiNiFi in Edge device and NiFi in Central server? What is the best way of doing this?

Please share your thoughts.

Thank you.

With Best Regards, Praveen.B

Answer 1

If anybody is still struggling out there, found this article to be precisely doing what this post is asking for.

I have managed to get the Agents and C2 Server working, update the central configuration in C2 and thereby all the agents pulling latest configuration and reloading the same.

MiNiFi Agents bootstrap.conf

#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
#
#   http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

# Java command to use when running MiNiFi
java=java

# Username to use when running MiNiFi. This value will be ignored on Windows.
run.as=

# Configure where MiNiFi's lib and conf directories live
# When running as a Windows service set full paths instead of relative paths
lib.dir=./lib
conf.dir=./conf

# How long to wait after telling MiNiFi to shutdown before explicitly killing the Process
graceful.shutdown.seconds=20

# The location for the configuration file
# When running as a Windows service use the full path to the file
nifi.minifi.config=./conf/config.yml

# Notifiers to use for the associated agent, comma separated list of class names
#nifi.minifi.notifier.ingestors=org.apache.nifi.minifi.bootstrap.configuration.ingestors.FileChangeIngestor
#nifi.minifi.notifier.ingestors=org.apache.nifi.minifi.bootstrap.configuration.ingestors.RestChangeIngestor
nifi.minifi.notifier.ingestors=org.apache.nifi.minifi.bootstrap.configuration.ingestors.PullHttpChangeIngestor

# File change notifier configuration

# Path of the file to monitor for changes.  When these occur, the FileChangeNotifier, if configured, will begin the configuration reloading process
#nifi.minifi.notifier.ingestors.file.config.path=./conf/config.yml
# How frequently the file specified by 'nifi.minifi.notifier.file.config.path' should be evaluated for changes.
#nifi.minifi.notifier.ingestors.file.polling.period.seconds=5

# Rest change notifier configuration

# Port on which the Jetty server will bind to, keep commented for a random open port
#nifi.minifi.notifier.ingestors.receive.http.port=8338

#Pull HTTP change notifier configuration

# Hostname on which to pull configurations from
nifi.minifi.notifier.ingestors.pull.http.hostname=localhost
# Port on which to pull configurations from
nifi.minifi.notifier.ingestors.pull.http.port=10080
# Path to pull configurations from
nifi.minifi.notifier.ingestors.pull.http.path=/c2/config
# Query string to pull configurations with
nifi.minifi.notifier.ingestors.pull.http.query=class=raspi3
# Period on which to pull configurations from, defaults to 5 minutes if commented out
nifi.minifi.notifier.ingestors.pull.http.period.ms=10000

# Periodic Status Reporters to use for the associated agent, comma separated list of class names
nifi.minifi.status.reporter.components=org.apache.nifi.minifi.bootstrap.status.reporters.StatusLogger

# Periodic Status Logger configuration

# The FlowStatus query to submit to the MiNiFi instance
nifi.minifi.status.reporter.log.query=instance:health,bulletins
# The log level at which the status will be logged
nifi.minifi.status.reporter.log.level=INFO
# The period (in milliseconds) at which to log the status
nifi.minifi.status.reporter.log.period=15000

# Disable JSR 199 so that we can use JSP's without running a JDK
java.arg.1=-Dorg.apache.jasper.compiler.disablejsr199=true

# JVM memory settings
java.arg.2=-Xms256m
java.arg.3=-Xmx256m

# Enable Remote Debugging
#java.arg.debug=-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=8000

java.arg.4=-Djava.net.preferIPv4Stack=true

# allowRestrictedHeaders is required for Cluster/Node communications to work properly
java.arg.5=-Dsun.net.http.allowRestrictedHeaders=true
java.arg.6=-Djava.protocol.handler.pkgs=sun.net.www.protocol

# Sets the provider of SecureRandom to /dev/urandom to prevent blocking on VMs
java.arg.7=-Djava.security.egd=file:/dev/urandom


# The G1GC is still considered experimental but has proven to be very advantageous in providing great
# performance without significant "stop-the-world" delays.
#java.arg.13=-XX:+UseG1GC

#Set headless mode by default
java.arg.14=-Djava.awt.headless=true

C2 Server's minifi-c2-context.xml

<?xml version="1.0" encoding="UTF-8"?>
<!--
  ~ Licensed to the Apache Software Foundation (ASF) under one or more
  ~ contributor license agreements.  See the NOTICE file distributed with
  ~ this work for additional information regarding copyright ownership.
  ~ The ASF licenses this file to You under the Apache License, Version 2.0
  ~ (the "License"); you may not use this file except in compliance with
  ~ the License.  You may obtain a copy of the License at
  ~
  ~     http://www.apache.org/licenses/LICENSE-2.0
  ~
  ~ Unless required by applicable law or agreed to in writing, software
  ~ distributed under the License is distributed on an "AS IS" BASIS,
  ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  ~ See the License for the specific language governing permissions and
  ~ limitations under the License.
  -->
<beans default-lazy-init="true"
       xmlns="http://www.springframework.org/schema/beans"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-3.1.xsd">

    <bean id="configService" class="org.apache.nifi.minifi.c2.service.ConfigService" scope="singleton">
        <constructor-arg>
            <list>
                <bean class="org.apache.nifi.minifi.c2.provider.cache.CacheConfigurationProvider">
                    <constructor-arg>
                        <list>
                            <value>text/yml</value>
                        </list>
                    </constructor-arg>
                    <constructor-arg>
                        <bean class="org.apache.nifi.minifi.c2.cache.filesystem.FileSystemConfigurationCache">
                            <constructor-arg>
                                <value>./files</value>
                            </constructor-arg>
                            <constructor-arg>
                                <value>${class}/${class}</value>
                            </constructor-arg>
                        </bean>
                    </constructor-arg>
                </bean>
            </list>
        </constructor-arg>
        <constructor-arg>
            <bean class="org.apache.nifi.minifi.c2.security.authorization.GrantedAuthorityAuthorizer">
                <constructor-arg value="classpath:authorizations.yaml"/>
            </bean>
        </constructor-arg>
    </bean>
</beans>

I manually update the central configuration as I want to be in control of when to update it. However, you can also setup the NiFiRestConfigurationProvider by providing the NiFi url. (already present in the context file, just uncomment it)

Answer 2

Praveen - good question. This is in-line with the objectives for MiNiFi in its role of end to end flow management as part of the Apache NiFi project. We're not there yet for code distribution but some of the central command and control aspects are being worked on in the community right now. Feel free to come join the mailing list and discuss what you'd like to see and contribute as much as you are comfortable. thanks

Answer 3

The getting-started guide of MiNiFi (as of June 2017) looks like there is no extra feature for that.

You should be able to build a flow from central to edge devices, update $MINIFI_HOME/conf/config.yml and restart the MiNiFi agent. I think this was a suggestion from this video:

https://www.youtube.com/watch?v=s9eIHD96OhM

From my point of view this is not a feasible solution in a real world scenario where you need full remote deployment control over a device. Just think about mandatory security updates.

I also research for a good IoT device management solution, maybe resin.io is for you. resin.io gives you the possibility to manage devices and deploy docker images in an efficient way. The admin GUI runs currently only as a SaaS/cloud-solution.

An alternative with on premise support is mender.io.