A simple NiFi processor project

Russell Bateman
August 2016
last update:

Though I've been developing custom NiFi processors for the better part of a year now, I've never got around to making notes on the (rather confusing) set-up of a project in the filesystem and, especially, the pom.xml files needed to pull it off.

The trick is to get the multiple pom.xml files set up. You can't do it with one because you must built the processor into its own JAR, then roll the JAR into a NAR for deployment to NiFi.

I named my project "Trick" so there would be no mistake, confusion or ambiguity as to what is what here. The important source artifacts are:

  1. processor/src/main/.../Trick.java —code for the custom NiFi processor
  2. processor/src/main/resources/.../org.apache.nifi.processor.Processor —resource that tells NiFi where your processor is
  3. processor/src/test/.../TrickTest.java —code for testing Trick class
  4. pom.xml —the root pom.xml
  5. processor/pom.xml —builds the processor JAR
  6. nar/pom.xml —rolls the processor JAR into the NAR

Note that there is nothing sacred about the subdirectory names, nar, processor, nor the project directory name, trick.

Here's what the project looks like when finished, but before any build. (I've got some IntelliJ IDEA files in here—they don't count):

~/dev/trick $ tree
.
├── .idea
├── nar
│   ├── pom.xml
│   └── trick-nar.iml
├── pom.xml
├── processor
│   ├── pom.xml
│   ├── src
│   │   ├── main
│   │   │   ├── java
│   │   │   │   └── com
│   │   │   │       └── etretatlogiciels
│   │   │   │           └── nifi
│   │   │   │               └── processor
│   │   │   │                   └── Trick.java
│   │   │   └── resources
│   │   │       └── META_INF
│   │   │           └── services
│   │   │               └── org.apache.nifi.processor.Processor
│   │   └── test
│   │       └── java
│   │           └── com
│   │               └── etretatlogiciels
│   │                   └── nifi
│   │                       └── processor
│   │                           └── TrickTest.java
│   └── trick-processor.iml
└── trick.iml

18 directories, 9 files

Or, if clearer, this illustration (which is missing the Java test file):

The Maven build is done this way, to produce the NAR for deployment (not showing the copious Maven output, but only what the project filesystem looks like afterward). Your mileage may vary, but it should look something very much like this. The NAR file is clearly highlighted:

~/dev/trick $ mvn clean test package
.
.
.
~/dev/trick $ tree
.
├── .idea
├── nar
│   ├── pom.xml
│   ├── target
│   │   ├── classes
│   │   │   └── META-INF
│   │   │       └── nar
│   │   │           └── com.etretatlogiciels.nifi.processor
│   │   │               └── trick-nar
│   │   │                   └── nar.properties
│   │   ├── maven-archiver
│   │   │   └── pom.properties
│   │   └── trick-nar-1.0-SNAPSHOT.nar
│   └── trick-nar.iml
├── pom.xml
├── processor
│   ├── pom.xml
│   ├── src
│   │   ├── main
│   │   │   ├── java
│   │   │   │   └── com
│   │   │   │       └── etretatlogiciels
│   │   │   │           └── nifi
│   │   │   │               └── processor
│   │   │   │                   └── Trick.java
│   │   │   └── resources
│   │   │       └── META_INF
│   │   │           └── services
│   │   │               └── org.apache.nifi.processor.Processor
│   │   └── test
│   │       └── java
│   │           └── com
│   │               └── etretatlogiciels
│   │                   └── nifi
│   │                       └── processor
│   │                           └── TrickTest.java
│   ├── target
│   │   ├── classes
│   │   │   ├── com
│   │   │   │   └── etretatlogiciels
│   │   │   │       └── nifi
│   │   │   │           └── processor
│   │   │   │               └── Trick.class
│   │   │   └── META_INF
│   │   │       └── services
│   │   │           └── org.apache.nifi.processor.Processor
│   │   ├── generated-sources
│   │   │   └── annotations
│   │   ├── generated-test-sources
│   │   │   └── test-annotations
│   │   ├── maven-archiver
│   │   │   └── pom.properties
│   │   ├── maven-status
│   │   │   └── maven-compiler-plugin
│   │   │       ├── compile
│   │   │       │   └── default-compile
│   │   │       │       ├── createdFiles.lst
│   │   │       │       └── inputFiles.lst
│   │   │       └── testCompile
│   │   │           └── default-testCompile
│   │   │               ├── createdFiles.lst
│   │   │               └── inputFiles.lst
│   │   ├── surefire-reports
│   │   │   ├── com.etretatlogiciels.nifi.processor.TrickTest.txt
│   │   │   └── TEST-com.etretatlogiciels.nifi.processor.TrickTest.xml
│   │   ├── test-classes
│   │   │   └── com
│   │   │       └── etretatlogiciels
│   │   │           └── nifi
│   │   │               └── processor
│   │   │                   └── TrickTest.class
│   │   └── trick-processor-1.0-SNAPSHOT.jar
│   └── trick-processor.iml
└── trick.iml

52 directories, 23 files

Here are the source files listed earlier. This processor is pretty much nonsense and doesn't do anything, and its test doesn't test anything. However, this exposé is only to help you get a) the pom.xml files and b) the layout of the project and important files.

TrickTest.java:
package com.etretatlogiciels.nifi.processor;

import org.junit.Test;
import static org.junit.Assert.assertTrue;

import org.apache.nifi.util.MockFlowFile;
import org.apache.nifi.util.TestRunner;
import org.apache.nifi.util.TestRunners;

public class TrickTest
{
  private final static String TEST_CONTENT = "This is a test.";

  @Test
  public void testProcessor()
  {
    TestRunner runner = TestRunners.newTestRunner( Trick.class );

    runner.enqueue( TEST_CONTENT );
    runner.run( 1 );
    runner.assertQueueEmpty();

    List< MockFlowFile > results  = runner.getFlowFilesForRelationship( RELATIONSHIP );
    MockFlowFile         flowfile = results.get( 0 );
    String               content  = new String( runner.getContentAsByteArray( flowfile ) );

    assertTrue( "Changed flowfile content?", content.equals( TEST_CONTENT ) );
  }
}
Trick.java:
package com.etretatlogiciels.nifi.processor;

import java.util.ArrayList;
import java.util.Collections;
import java.util.HashSet;
import java.util.List;
import java.util.Set;

import org.apache.nifi.components.PropertyDescriptor;
import org.apache.nifi.flowfile.FlowFile;
import org.apache.nifi.annotation.documentation.CapabilityDescription;
import org.apache.nifi.annotation.documentation.Tags;
import org.apache.nifi.processor.AbstractProcessor;
import org.apache.nifi.processor.ProcessContext;
import org.apache.nifi.processor.ProcessSession;
import org.apache.nifi.processor.ProcessorInitializationContext;
import org.apache.nifi.processor.Relationship;
import org.apache.nifi.processor.exception.ProcessException;
import org.apache.nifi.processor.util.StandardValidators;

@Tags( { "trick" } )
@CapabilityDescription( "Trick processor." )
public class Trick extends AbstractProcessor
{
  public static final PropertyDescriptor TRICK_PROPERTY = new PropertyDescriptor.Builder()
            .name( "Trick Property" )
            .description( "Just a property." )
            .required( true )
            .addValidator( StandardValidators.NON_EMPTY_VALIDATOR )
            .build();

  public static final Relationship TRICK_RELATIONSHIP = new Relationship.Builder()
            .name( "Trick relationship" )
            .description( "Just a relationship." )
            .build();

  private List< PropertyDescriptor > properties;
  private Set< Relationship >        relationships;

  @Override protected void init( final ProcessorInitializationContext context )
  {
    final List< PropertyDescriptor > props = new ArrayList<>();
    props.add( TRICK_PROPERTY );
    this.properties = Collections.unmodifiableList( props );

    final Set< Relationship > relationships = new HashSet<>();
    relationships.add( TRICK_RELATIONSHIP );
    this.relationships = Collections.unmodifiableSet( relationships );
  }

  @Override
  public Set< Relationship > getRelationships()
  {
    return this.relationships;
  }

  @Override
  public final List< PropertyDescriptor > getSupportedPropertyDescriptors()
  {
    return properties;
  }

  @Override
  public void onTrigger( final ProcessContext context, final ProcessSession session )
        throws ProcessException
  {
    FlowFile flowfile = session.get();

    if( flowfile == null )
    {
      context.yield();
      return;
    }

    session.transfer( flowfile, TRICK_RELATIONSHIP );
  }
}
org.apache.nifi.processor.Processor:
com.etretatlogiciels.nifi.processor.Trick

The project's pom.xml files, for the project root, the source code subdirectory and the NAR-builder.

project root pom.xml:

Mostly, this defines the modules to build the source code and bundle the NAR. It also holds the plug-in that Maven calls to build the NAR.

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
      xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
      xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
                  http://maven.apache.org/maven-v4_0_0.xsd">
  <modelVersion>4.0.0</modelVersion>

  <!-- root pom.xml -->
  <groupId>com.etretatlogiciels.nifi.processor</groupId>
  <artifactId>trick</artifactId>
  <version>1.0-SNAPSHOT</version>
  <packaging>pom</packaging>
  <name>Etretat Logiciels NiFi Processor</name>

  <properties>
    <nifi.version>0.7.0</nifi.version>
    <slf4j.version>1.7.21</slf4j.version>
    <junit.version>4.12</junit.version>
    <maven.install.skip>true</maven.install.skip>
    <maven.deploy.skip>true</maven.deploy.skip>
  </properties>

  <modules>
    <module>processor</module>
    <module>nar</module>
  </modules>

  <build>
    <plugins>
      <plugin>
        <!-- what builds the NAR file... -->
        <groupId>org.apache.nifi</groupId>
        <artifactId>nifi-nar-maven-plugin</artifactId>
        <version>1.0.1-incubating</version>
        <extensions>true</extensions>
      </plugin>
    </plugins>
  </build>
</project>
processor subdirectory's pom.xml:

Notice that we're going to build trick-1.0.SNAPSHOT.jar. This will be supplied to the NAR file build later.

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
      xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
      xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
                  http://maven.apache.org/maven-v4_0_0.xsd">
  <modelVersion>4.0.0</modelVersion>

  <parent>
    <groupId>com.etretatlogiciels.nifi.processor</groupId>
    <artifactId>trick</artifactId>
    <version>1.0-SNAPSHOT</version>
  </parent>

  <!-- processor/pom.xml -->
  <artifactId>trick-processor</artifactId>
  <version>1.0-SNAPSHOT</version>
  <packaging>jar</packaging>

  <dependencies>
    <dependency>
      <groupId>org.apache.nifi</groupId>
      <artifactId>nifi-api</artifactId>
      <version>${nifi.version}</version>
    </dependency>
    <dependency>
      <groupId>org.apache.nifi</groupId>
      <artifactId>nifi-processor-utils</artifactId>
      <version>${nifi.version}</version>
    </dependency>
    <dependency>
      <groupId>org.apache.nifi</groupId>
      <artifactId>nifi-mock</artifactId>
      <version>${nifi.version}</version>
      <scope>test</scope>
    </dependency>
    <dependency>
      <groupId>org.slf4j</groupId>
      <artifactId>slf4j-simple</artifactId>
      <version>${slf4j.version}</version>
      <scope>test</scope>
    </dependency>
    <dependency>
      <groupId>junit</groupId>
      <artifactId>junit</artifactId>
      <version>${junit.version}</version>
      <scope>test</scope>
    </dependency>
  </dependencies>

  <build>
    <plugins>
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-compiler-plugin</artifactId>
        <version>3.5.1</version>
        <configuration>
          <source>1.8</source>
          <target>1.8</target>
        </configuration>
      </plugin>
    </plugins>
  </build>
</project>
NAR bundle subdirectory's pom.xml:

Notice that we're going to build trick-1.0.SNAPSHOT.nar.

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
      xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
      xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
                  http://maven.apache.org/maven-v4_0_0.xsd">
  <modelVersion>4.0.0</modelVersion>

  <!-- nar/pom.xml -->
  <parent>
    <groupId>com.etretatlogiciels.nifi.processor</groupId>
    <artifactId>trick</artifactId>
    <version>1.0-SNAPSHOT</version>
  </parent>

  <artifactId>trick-nar</artifactId>
  <version>1.0-SNAPSHOT</version>
  <packaging>nar</packaging>

  <dependencies>
    <dependency>
      <groupId>com.etretatlogiciels.nifi.processor</groupId>
      <artifactId>trick-processor</artifactId>
      <version>1.0-SNAPSHOT</version>
    </dependency>
  </dependencies>
</project>

One last note. When ready to use the new processor, you have only to drop the .nar file into ${NIFI_ROOT}/lib, then restart NiFi.