Apache NiFi and Docker notes

Build our own Dockerfile to insert our NAR

...because we're converting away from the usual installation of Apache NiFi to a Docker image. We need to add our own, custom NAR to that standard image.

FROM    apache/nifi:latest
LABEL   maintainer="[email protected]"
USER    nifi
ADD     nar/target/custom-nar-*.nar /opt/nifi/nifi-current/lib
EXPOSE  9090
ENV     PATH=/opt/nifi/nifi-current/bin:$PATH
CMD     [ "nifi.sh", "start" ]

FROM —pulls (includes) dependencies from other Docker image configurations.
LABEL —who the maintainer/guilty party is.
USER —the system user that should be doing the work.
ADD —is used to copy files or subdirectories into a Docker image. ADD has the effect of adding a new layer (image) to the existing image.
EXPOSE —expose the listed ports for mapping to the host via the -p option when launching a container. This is more documentary than effectual, but we'll add it. It means basically that port 9090 inside our container will be the one Tomcat listens on. The way this really happens though is via the -p option on the docker run command.
ENV —update the environment in order that CMD may find the command it's going to execute.
CMD —used to run NiFi in the image, along with any arguments. Allows you to set a default command that will be executed only when the container is run. However, if the container is run with a command on the command line, this command is ignored.

Unanswered questions...

What of...

...properties inside /opt/nifi/conf/nifi.properties that must be changed. Are there any?
...security (authentication, authorization, etc.) settings normally installed by Ansible?

Look for Apache NiFi Docker images

russ@tirion ~/dev/dustom $ docker search nifi
NAME                          DESCRIPTION                                             STARS     OFFICIAL   AUTOMATED
apache/nifi              Unofficial convenience binaries and Docker i…  236              [OK]
apache/nifi-registry          Unofficial convenience binaries for Apache N…           32
xemuliam/nifi                 Docker image for Apache NiFi. Created from N…           28                   [OK]
mkobit/nifi                   Unofficial Apache NiFi (https://nifi.apache.…           22                   [OK]
hortonworks/nifi              nifi                                                    14
apache/nifi-minifi            Unofficial convenience binaries for Apache N…           9
apache/nifi-toolkit           Unofficial convenience binaries for NiFi                5
...

Here are the basic steps...

...once there is a Dockerfile to a) build, run, then lauch a shell into it.

$ docker build     -t custom-nar .
$ docker run   --name custom-nar -p 9090:9090 -d -e NIFI_WEB_HTTP_PORT='9090' custom-nar
$ docker exec     -it custom-nar /bin/bash

Build our new image

russ@tirion ~/dev/dustom $ docker build -t custom-nar
russ@tirion ~/dev/custom $ docker run --name custom-nar -p 9090:9090 -d -e NIFI_WEB_HTTP_PORT='9090' apache/nifi:latest
Sending build context to Docker daemon    270MB
Step 1/7 : FROM    apache/nifi:latest
 ---> 0070c5c5d9fd
Step 2/7 : LABEL   maintainer="[email protected]"
 ---> Using cache
 ---> f644af7dd2ff
Step 3/7 : USER    nifi
 ---> Using cache
 ---> d79990826e4b
Step 4/7 : ADD     nar/target/custom-nar-*.nar /opt/nifi/nifi-current/lib
 ---> Using cache
 ---> c3d3f067569e
Step 5/7 : EXPOSE  9090
 ---> Using cache
 ---> 4f9e90e6f96b
Step 6/7 : ENV     PATH=/opt/nifi/nifi-current/bin:$PATH
 ---> Running in 4bea9835cd5b
Removing intermediate container 4bea9835cd5b
 ---> b9ac2a7c64f9
Step 7/7 : CMD     [ "nifi.sh", "start" ]
 ---> Running in e957f4023aff
Removing intermediate container e957f4023aff
 ---> acef39061dfd
Successfully built acef39061dfd
Successfully tagged custom-nar:latest
a1f37f7505b637c2e853db9450357b87376f0bd24dfbdcce6f3165ad520de95e
russ@tirion ~/sandboxes/custom-nar $ docker ps
CONTAINER ID   IMAGE                COMMAND                 CREATED         STATUS         PORTS                                                                                NAMES
a1f37f7505b6   apache/nifi:latest   "../scripts/start.sh"   6 seconds ago   Up 6 seconds   8000/tcp, 8080/tcp, 8443/tcp, 10000/tcp, 0.0.0.0:9090->9090/tcp, :::9090->9090/tcp   custom-nar

Verifying that NiFi is running...

I open a browser and set the address line to http://localhost:9090/nifi. I go to the hamburger menu and choose About which gives me version 1.15.3. I do not have that version in my filesystem, so there can be no mistake that the container is up and running.

Looking around in the container...

russ@tirion ~/dev/custom $ docker exec -it custom-nar /bin/bash
nifi@a1f37f7505b6:/opt/nifi/nifi-current$ pwd
/opt/nifi/nifi-current
nifi@a1f37f7505b6:/opt/nifi/nifi-current$ alias ll='ls -alg'
nifi@a1f37f7505b6:/opt/nifi/nifi-current$ cd ..
nifi@a1f37f7505b6:/opt/nifi$ ll
total 28
drwxr-xr-x 1 nifi 4096 Jan 18 19:08 .
drwxr-xr-x 1 root 4096 Dec 24 04:38 ..
lrwxrwxrwx 1 nifi   22 Jan 18 19:08 nifi-1.15.3 -> /opt/nifi/nifi-current
drwxr-xr-x 1 nifi 4096 Jan 27 20:47 nifi-current
lrwxrwxrwx 1 nifi   30 Jan 18 19:06 nifi-toolkit-1.15.3 -> /opt/nifi/nifi-toolkit-current
drwxr-xr-x 6 nifi 4096 Jan 22  2020 nifi-toolkit-current
drwxr-xr-x 1 nifi 4096 Dec 24 04:38 scripts

A quick experience...

...running Apache NiFi in Docker. Here are a few commands. This only demonstrates that Apache NiFi has built a working Docker image.

$ docker run --name nifi -p 9090:9090 -d -e NIFI_WEB_HTTP_PORT='9090' apache/nifi:latest
Unable to find image 'apache/nifi:latest' locally
latest: Pulling from apache/nifi
647acf3d48c2: Pull complete
b02967ef0034: Pull complete
e1ad2231829e: Pull complete
3accde8486ae: Pull complete
39bc74563c28: Pull complete
f0455cc186e3: Pull complete
252fb07dac48: Pull complete
084571fbe2db: Pull complete
9900bd2e7ed6: Pull complete
4065d32db4c6: Pull complete
22f79ad58453: Pull complete
cd20d4557bc1: Pull complete
4f4fb700ef54: Pull complete
Digest: sha256:6089e8b2e354c7597f37d66a11d8c256a52a0cf301ac2e9884b5dce9e44d9b1b
Status: Downloaded newer image for apache/nifi:latest
e8e43f72c878c3c3d0b122690c272a8a0fd54077014aeab90c601cff3ab3e9e5

$ docker ps
CONTAINER ID   IMAGE                COMMAND                 CREATED         STATUS          PORTS                                                                                NAMES
0ecf6dbf11b9   apache/nifi:latest   "../scripts/start.sh"   5 minutes ago   Up 15 seconds   8000/tcp, 8080/tcp, 8443/tcp, 10000/tcp, 0.0.0.0:9090->9090/tcp, :::9090->9090/tcp   nifi

$ docker restart nifi
nifi

$ docker stop nifi
nifi

$ docker rm nifi
nifi

$ docker rm -f $(docker ps -a -q)
0ecf6dbf11b9

Re-rolling my own NiFi Docker container

I needed to build a custom Docker image containing Apache NiFi 1.13.2 for distribution of my product (two NARs containing custom processors). I have some ancient, custom processors in NARs that will not load beginning in NiFi 1.14.0.

Normally I would begin it so:

FROM apache/nifi:1.13.2

However, at present, I need to replace Java 1.8 in the Docker container with Java 11 because I have a new, custom processor that wants to use the new Java HTTP client which is unavailable prior to Java 11.

For this reason, I think I need to hand-craft an image identical to apache/nifi:1.13.2, but sitting atop Java 11. To figure out where to go to get the Dockerfile used to build this image, I did this:

Download Apache NiFi sources

Go to Apache nifi Downloads.
Click on release archives; choose 1.13.2.
Download nifi-1.13.2-source-release.zip.
I saved it to ~/Downloads/1.13.2, then exploded it into my filesystem:
```
~/Downloads/1.13.2 $ unzip nifi-1.13.2-source-release.zip
```

Modifying NiFi's Docker image

Using find . -name Dockerfile, I located three possible instances of Dockerfile, but chose one and descended into that subdirectory:
```
~/Downloads/1.13.2 $ cd nifi-1.13.2/nifi-docker/dockerhub
```

There, I changed Dockerfile to writeable and opened it to make changes:

FROM  openjdk:11-jre                 # from openjdk:8-jre
LABEL maintainer="Apache NiFi <[email protected]>"
LABEL site="https://nifi.apache.org"

ARG   UID=1000
ARG   GID=1000
ARG   NIFI_VERSION=1.13.2            # from 1.13.1
...

I changed the JRE to 11 and the NiFi version in Dockerfile and also in DockerImage.txt (after making this file writeable) which DockerBuild.sh consumes. I made DockerBuild.sh executable. Then, I ran it:
```
~/Downloads/1.13.2/nifi-1.13.2/nifi-docker/dockerhub $ chmod u+x DockerBuild.sh
~/Downloads/1.13.2/nifi-1.13.2/nifi-docker/dockerhub $ ./DockerBuild.sh
```

After waiting a while, all appeared to proceed as I would expect and without any errors:

~/Downloads/1.13.2/nifi-1.13.2/nifi-docker/dockerhub $ docker images
REPOSITORY    TAG        IMAGE ID       CREATED          SIZE
apache/nifi   1.13.2     8c18038f152a   10 minutes ago   2.06GB

Building the new image

Next, I'll build my own, production container image from this one because I roll two custom NARs into the container image:

russ@tirion ~/sandboxes/pipeline/docker $ docker images
REPOSITORY    TAG        IMAGE ID       CREATED          SIZE
pipeline      latest     15db5d8182a8   2 minutes ago    2.38GB
apache/nifi   1.13.2     8c18038f152a   30 minutes ago   2.06GB

...and attempt to launch it:

~/sandboxes/pipeline/docker $ docker build -t pipeline .
~/sandboxes/pipeline/docker $ docker run --publish 5001:8080 pipeline:latest
...
NiFi running with PID 34.

Java home: /usr/local/openjdk-11
NiFi home: /opt/nifi/nifi-current

Bootstrap Config File: /opt/nifi/nifi-current/conf/bootstrap.conf

2023-06-07 17:15:07,480 INFO [main] org.apache.nifi.bootstrap.Command Starting Apache NiFi...
...
2023-06-07 17:19:38,437 INFO [main] org.apache.nifi.BootstrapListener Successfully initiated communication with Bootstrap
2023-06-07 17:19:38,437 INFO [main] org.apache.nifi.NiFi Controller initialization took 8819332596 nanoseconds (8 seconds).
2023-06-07 17:19:56,690 INFO [pool-17-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Initiating checkpoint of FlowFile Repository
2023-06-07 17:19:56,691 INFO [pool-17-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Successfully checkpointed FlowFile Repository with 0 records in 0 milliseconds

Running the new image

Because the previous operation runs in the console, elsewhere (in another console)...

~/sandboxes/pipeline/docker $ docker ps -a
CONTAINER ID   IMAGE             COMMAND                  CREATED         STATUS        PORTS                                                                      NAMES
1bed096ee9a5   pipeline:latest   "../scripts/start.sh…"   2 minutes ago   Up 2 minutes  8000/tcp, 8443/tcp, 10000/tcp, 0.0.0.0:5001->8080/tcp, :::5001->8080/tcp   affectionate_bose
~/sandboxes/pipeline/docker $ docker exec -it 1bed096ee9a5 bash
nifi@1bed096ee9a5:/opt/nifi/nifi-current$ env | grep JAVA_HOME
JAVA_HOME=/usr/local/openjdk-11

This looks good because:
- I see that Java home is reported as being /usr/local/openjdk-11
- Upon launch, that's clearly what's reported in the container at the command line.
- /opt/nifi/nifi-1.13.2/extensions contains my custom NARs, which load also.

Command-line summary

$ unzip nifi-1.13.2-source-release.zip           —explode NiFi download
$ chmod u+w Dockerfile DockerImage.txt           —make files writeable
$ chmod u+x DockerBuild.sh                       —make script executable
$ gvim Dockerfile DockerImage.txt                —edit for Java 11 and NiFi version
$ ./DockerBuild.sh                               —build NiFi 1.13.2 images by hand

$ docker build -t pipeline .                     —build and tag to wrap into my container
$ docker images                                  —see images in local repository
$ docker run --publish 5001:8080 pipeline:latest —run container
$ docker ps -a                                   —see running container
$ docker exec -it container-id bash              —see inside container