org.apache.nifi.processors.hadoop

Class GetHDFS

java.lang.Object

org.apache.nifi.components.AbstractConfigurableComponent

org.apache.nifi.processor.AbstractSessionFactoryProcessor

org.apache.nifi.processor.AbstractProcessor

org.apache.nifi.processors.hadoop.AbstractHadoopProcessor

org.apache.nifi.processors.hadoop.GetHDFS

All Implemented Interfaces:

ConfigurableComponent, Processor

Direct Known Subclasses:

GetHDFSSequenceFile

@TriggerWhenEmpty
 @InputRequirement(value=INPUT_FORBIDDEN)
 @Tags(value={"hadoop","HDFS","get","fetch","ingest","source","filesystem","restricted"})
 @CapabilityDescription(value="Fetch files from Hadoop Distributed File System (HDFS) into FlowFiles. This Processor will delete the file from HDFS after fetching it.")
 @WritesAttribute(attribute="filename",description="The name of the file that was read from HDFS.") @WritesAttribute(attribute="path",description="The path is set to the relative path of the file\'s directory on HDFS. For example, if the Directory property is set to /tmp, then files picked up from /tmp will have the path attribute set to \"./\". If the Recurse Subdirectories property is set to true and a file is picked up from /tmp/abc/1/2/3, then the path attribute will be set to \"abc/1/2/3\".")
 @SeeAlso(value={PutHDFS.class,ListHDFS.class})
 @Restricted(value="Provides operator the ability to retrieve and delete any file that NiFi has access to in HDFS or the local filesystem.")
public class GetHDFS
extends AbstractHadoopProcessor

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
protected static class	GetHDFS.ProcessorConfiguration Holder for a snapshot in time of some processor properties that are passed around.

Nested classes/interfaces inherited from class org.apache.nifi.processors.hadoop.AbstractHadoopProcessor

AbstractHadoopProcessor.HdfsResources, AbstractHadoopProcessor.ValidationResources

Field Summary

Fields

Modifier and Type	Field and Description
static PropertyDescriptor	BATCH_SIZE
static PropertyDescriptor	BUFFER_SIZE
static int	BUFFER_SIZE_DEFAULT
static java.lang.String	BUFFER_SIZE_KEY
static PropertyDescriptor	FILE_FILTER_REGEX
static PropertyDescriptor	FILTER_MATCH_NAME_ONLY
static PropertyDescriptor	IGNORE_DOTTED_FILES
static PropertyDescriptor	KEEP_SOURCE_FILE
static PropertyDescriptor	MAX_AGE
static int	MAX_WORKING_QUEUE_SIZE
static PropertyDescriptor	MIN_AGE
static PropertyDescriptor	POLLING_INTERVAL
protected GetHDFS.ProcessorConfiguration	processorConfig
static PropertyDescriptor	RECURSE_SUBDIRS
static Relationship	REL_SUCCESS

Fields inherited from class org.apache.nifi.processors.hadoop.AbstractHadoopProcessor

ABSOLUTE_HDFS_PATH_ATTRIBUTE, ADDITIONAL_CLASSPATH_RESOURCES, COMPRESSION_CODEC, DIRECTORY, HADOOP_CONFIGURATION_RESOURCES, KERBEROS_RELOGIN_PERIOD, kerberosProperties, properties

Constructor Summary

Constructors

Constructor and Description
GetHDFS()

Method Summary

Modifier and Type	Method and Description
protected java.util.Collection<ValidationResult>	customValidate(ValidationContext context) Allows subclasses to perform their own validation on the already set properties.
java.util.Set<Relationship>	getRelationships()
protected java.util.List<PropertyDescriptor>	getSupportedPropertyDescriptors() Allows subclasses to register which property descriptor objects are supported.
void	onScheduled(ProcessContext context)
void	onTrigger(ProcessContext context, ProcessSession session)
protected java.util.Set<org.apache.hadoop.fs.Path>	performListing(ProcessContext context) Do a listing of HDFS if the POLLING_INTERVAL has lapsed.
protected void	processBatchOfFiles(java.util.List<org.apache.hadoop.fs.Path> files, ProcessContext context, ProcessSession session)
protected java.util.Set<org.apache.hadoop.fs.Path>	selectFiles(org.apache.hadoop.fs.FileSystem hdfs, org.apache.hadoop.fs.Path dir, java.util.Set<org.apache.hadoop.fs.Path> filesVisited) Poll HDFS for files to process that match the configured file filters.

Methods inherited from class org.apache.nifi.processors.hadoop.AbstractHadoopProcessor

abstractOnScheduled, abstractOnStopped, checkHdfsUriForTimeout, getCompressionCodec, getConfiguration, getFileSystem, getFileSystem, getFileSystemAsUser, getKerberosProperties, getPathDifference, getUserGroupInformation, init, isTicketOld, preProcessConfiguration, tryKerberosRelogin

Methods inherited from class org.apache.nifi.processor.AbstractProcessor

onTrigger

Methods inherited from class org.apache.nifi.processor.AbstractSessionFactoryProcessor

getControllerServiceLookup, getIdentifier, getLogger, getNodeTypeProvider, initialize, isConfigurationRestored, isScheduled, toString, updateConfiguredRestoredTrue, updateScheduledFalse, updateScheduledTrue

Methods inherited from class org.apache.nifi.components.AbstractConfigurableComponent

equals, getPropertyDescriptor, getPropertyDescriptors, getSupportedDynamicPropertyDescriptor, hashCode, onPropertyModified, validate

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Methods inherited from interface org.apache.nifi.components.ConfigurableComponent

getPropertyDescriptor, getPropertyDescriptors, onPropertyModified, validate

Field Detail

BUFFER_SIZE_KEY

public static final java.lang.String BUFFER_SIZE_KEY

See Also:

Constant Field Values

BUFFER_SIZE_DEFAULT

public static final int BUFFER_SIZE_DEFAULT

See Also:

Constant Field Values

MAX_WORKING_QUEUE_SIZE

public static final int MAX_WORKING_QUEUE_SIZE

See Also:

Constant Field Values

REL_SUCCESS

public static final Relationship REL_SUCCESS

RECURSE_SUBDIRS

public static final PropertyDescriptor RECURSE_SUBDIRS

KEEP_SOURCE_FILE

public static final PropertyDescriptor KEEP_SOURCE_FILE

FILE_FILTER_REGEX

public static final PropertyDescriptor FILE_FILTER_REGEX

FILTER_MATCH_NAME_ONLY

public static final PropertyDescriptor FILTER_MATCH_NAME_ONLY

IGNORE_DOTTED_FILES

public static final PropertyDescriptor IGNORE_DOTTED_FILES

MIN_AGE

public static final PropertyDescriptor MIN_AGE

MAX_AGE

public static final PropertyDescriptor MAX_AGE

BATCH_SIZE

public static final PropertyDescriptor BATCH_SIZE

POLLING_INTERVAL

public static final PropertyDescriptor POLLING_INTERVAL

BUFFER_SIZE

public static final PropertyDescriptor BUFFER_SIZE

processorConfig

protected GetHDFS.ProcessorConfiguration processorConfig

Constructor Detail

GetHDFS

public GetHDFS()

Method Detail

getRelationships

public java.util.Set<Relationship> getRelationships()

Specified by:

getRelationships in interface Processor

Overrides:

getRelationships in class AbstractSessionFactoryProcessor

Returns:

Set of all relationships this processor expects to transfer a flow file to. An empty set indicates this processor does not have any destination relationships. Guaranteed non null.

getSupportedPropertyDescriptors

protected java.util.List<PropertyDescriptor> getSupportedPropertyDescriptors()

Description copied from class: AbstractConfigurableComponent

Allows subclasses to register which property descriptor objects are supported. Default return is an empty set.

Overrides:

getSupportedPropertyDescriptors in class AbstractHadoopProcessor

Returns:

PropertyDescriptor objects this processor currently supports

customValidate

protected java.util.Collection<ValidationResult> customValidate(ValidationContext context)

Description copied from class: AbstractConfigurableComponent

Allows subclasses to perform their own validation on the already set properties. Since each property is validated as it is set this allows validation of groups of properties together. Default return is an empty set. This method will be called only when it has been determined that all property values are valid according to their corresponding PropertyDescriptor's validators.

Overrides:

customValidate in class AbstractHadoopProcessor

Parameters:

context - provides a mechanism for obtaining externally managed values, such as property values and supplies convenience methods for operating on those values

Returns:

Collection of ValidationResult objects that will be added to any other validation findings - may be null

onScheduled

@OnScheduled
public void onScheduled(ProcessContext context)
                              throws java.io.IOException

Throws:

java.io.IOException

onTrigger

public void onTrigger(ProcessContext context,
                      ProcessSession session)
               throws ProcessException

Specified by:

onTrigger in class AbstractProcessor

Throws:

ProcessException

processBatchOfFiles

protected void processBatchOfFiles(java.util.List<org.apache.hadoop.fs.Path> files,
                                   ProcessContext context,
                                   ProcessSession session)

performListing

protected java.util.Set<org.apache.hadoop.fs.Path> performListing(ProcessContext context)
                                                           throws java.io.IOException

Do a listing of HDFS if the POLLING_INTERVAL has lapsed. Will return null if POLLING_INTERVAL has not lapsed. Will return an empty set if no files were found on HDFS that matched the configured filters.

Parameters:

context - context

Returns:

null if POLLING_INTERVAL has not lapsed. Will return an empty set if no files were found on HDFS that matched the configured filters

Throws:

java.io.IOException - ex

selectFiles

protected java.util.Set<org.apache.hadoop.fs.Path> selectFiles(org.apache.hadoop.fs.FileSystem hdfs,
                                                               org.apache.hadoop.fs.Path dir,
                                                               java.util.Set<org.apache.hadoop.fs.Path> filesVisited)
                                                        throws java.io.IOException

Poll HDFS for files to process that match the configured file filters.

Parameters:

hdfs - hdfs

dir - dir

filesVisited - filesVisited

Returns:

files to process

Throws:

java.io.IOException - ex