@TriggerSerially @TriggerWhenEmpty @Tags(value={"hadoop","events","inotify","notifications","filesystem"}) @WritesAttribute(attribute="mime.type",description="This is always application/json.") @WritesAttribute(attribute="hdfs.inotify.event.type",description="This will specify the specific HDFS notification event type. Currently there are six types of events (append, close, create, metadata, rename, and unlink).") @WritesAttribute(attribute="hdfs.inotify.event.path",description="The specific path that the event is tied to.") @InputRequirement(value=INPUT_FORBIDDEN) @CapabilityDescription(value="This processor polls the notification events provided by the HdfsAdmin API. Since this uses the HdfsAdmin APIs it is required to run as an HDFS super user. Currently there are six types of events (append, close, create, metadata, rename, and unlink). Please see org.apache.hadoop.hdfs.inotify.Event documentation for full explanations of each event. This processor will poll for new events based on a defined duration. For each event received a new flow file will be created with the expected attributes and the event itself serialized to JSON and written to the flow file\'s content. For example, if event.type is APPEND then the content of the flow file will contain a JSON file containing the information about the append event. If successful the flow files are sent to the \'success\' relationship. Be careful of where the generated flow files are stored. If the flow files are stored in one of processor\'s watch directories there will be a never ending flow of events. It is also important to be aware that this processor must consume all events. The filtering must happen within the processor. This is because the HDFS admin\'s event notifications API does not have filtering.") @Stateful(scopes=CLUSTER, description="The last used transaction id is stored. This is used ") @SeeAlso(value={GetHDFS.class,FetchHDFS.class,PutHDFS.class,ListHDFS.class}) public class GetHDFSEvents extends AbstractHadoopProcessor
AbstractHadoopProcessor.HdfsResources, AbstractHadoopProcessor.ValidationResourcesABSOLUTE_HDFS_PATH_ATTRIBUTE, ADDITIONAL_CLASSPATH_RESOURCES, COMPRESSION_CODEC, DIRECTORY, HADOOP_CONFIGURATION_RESOURCES, KERBEROS_RELOGIN_PERIOD, kerberosProperties, properties| Constructor and Description |
|---|
GetHDFSEvents() |
| Modifier and Type | Method and Description |
|---|---|
protected org.apache.hadoop.hdfs.client.HdfsAdmin |
getHdfsAdmin() |
java.util.Set<Relationship> |
getRelationships() |
protected java.util.List<PropertyDescriptor> |
getSupportedPropertyDescriptors()
Allows subclasses to register which property descriptor objects are
supported.
|
void |
onSchedule(ProcessContext context) |
void |
onTrigger(ProcessContext context,
ProcessSession session) |
abstractOnScheduled, abstractOnStopped, checkHdfsUriForTimeout, customValidate, getCompressionCodec, getConfiguration, getFileSystem, getFileSystem, getFileSystemAsUser, getKerberosProperties, getPathDifference, getUserGroupInformation, init, isTicketOld, preProcessConfiguration, tryKerberosReloginonTriggergetControllerServiceLookup, getIdentifier, getLogger, getNodeTypeProvider, initialize, isConfigurationRestored, isScheduled, toString, updateConfiguredRestoredTrue, updateScheduledFalse, updateScheduledTrueequals, getPropertyDescriptor, getPropertyDescriptors, getSupportedDynamicPropertyDescriptor, hashCode, onPropertyModified, validateclone, finalize, getClass, notify, notifyAll, wait, wait, waitgetPropertyDescriptor, getPropertyDescriptors, onPropertyModified, validateprotected java.util.List<PropertyDescriptor> getSupportedPropertyDescriptors()
AbstractConfigurableComponentgetSupportedPropertyDescriptors in class AbstractHadoopProcessorpublic java.util.Set<Relationship> getRelationships()
getRelationships in interface ProcessorgetRelationships in class AbstractSessionFactoryProcessor@OnScheduled public void onSchedule(ProcessContext context)
public void onTrigger(ProcessContext context, ProcessSession session) throws ProcessException
onTrigger in class AbstractProcessorProcessExceptionprotected org.apache.hadoop.hdfs.client.HdfsAdmin getHdfsAdmin()