org.apache.nifi.provenance

Class PersistentProvenanceRepository

java.lang.Object

org.apache.nifi.provenance.PersistentProvenanceRepository

All Implemented Interfaces:

ProvenanceEventRepository, ProvenanceRepository

public class PersistentProvenanceRepository
extends java.lang.Object
implements ProvenanceRepository

Field Summary

Fields

Modifier and Type	Field and Description
static java.lang.String	EVENT_CATEGORY
static java.util.regex.Pattern	INDEX_PATTERN
static java.util.regex.Pattern	LOG_FILENAME_PATTERN
static int	MAX_INDEXING_FAILURE_COUNT
static int	MAX_JOURNAL_ROLLOVER_RETRIES
static int	MAX_UNDELETED_QUERY_RESULTS
static java.util.regex.Pattern	NUMBER_PATTERN

Constructor Summary

Constructors

Constructor and Description
PersistentProvenanceRepository() default no args constructor for service loading only.
PersistentProvenanceRepository(NiFiProperties nifiProperties)
PersistentProvenanceRepository(RepositoryConfiguration configuration, int rolloverCheckMillis)

Method Summary

Modifier and Type	Method and Description
void	authorize(ProvenanceEventRecord event, NiFiUser user)
void	close() Closes the repository, freeing any resources
protected IndexingAction	createIndexingAction() This method is protected and exists for testing purposes.
protected RecordWriter[]	createWriters(RepositoryConfiguration config, long initialRecordId)
StandardProvenanceEventRecord.Builder	eventBuilder() Returns a ProvenanceEventBuilder that is capable of building ProvenanceEventRecords
java.util.List<ProvenanceEventRecord>	filterUnauthorizedEvents(java.util.List<ProvenanceEventRecord> events, NiFiUser user)
protected java.util.List<java.io.File>	filterUnavailableFiles(java.util.List<java.io.File> journalFiles)
java.util.Collection<java.nio.file.Path>	getAllLogFiles()
RepositoryConfiguration	getConfiguration()
long	getContainerCapacity(java.lang.String containerName)
java.util.Set<java.lang.String>	getContainerNames()
long	getContainerUsableSpace(java.lang.String containerName)
ProvenanceEventRecord	getEvent(long id) Retrieves the Provenance Event with the given ID.
ProvenanceEventRecord	getEvent(long id, NiFiUser user)
java.util.List<ProvenanceEventRecord>	getEvents(long firstRecordId, int maxRecords) Returns a List of all ProvenanceEventRecords in the repository starting with the given ID.
java.util.List<ProvenanceEventRecord>	getEvents(long firstRecordId, int maxRecords, NiFiUser user) Returns a List of all ProvenanceEventRecords in the repository starting with the given ID.
protected IndexManager	getIndexManager()
protected int	getJournalCount()
int	getMaxAttributeCharacters()
java.lang.Long	getMaxEventId()
ProvenanceEventRepository	getProvenanceEventRepository()
protected long	getRolloverRetryMillis()
java.util.List<SearchableField>	getSearchableAttributes()
java.util.List<SearchableField>	getSearchableFields()
long	getSize(java.util.List<java.io.File> logFiles, long timeCutoff) Returns the size, in bytes, of the Repository storage
void	initialize(EventReporter eventReporter, Authorizer authorizer, ProvenanceAuthorizableFactory resourceFactory, IdentifierLookup idLookup) Performs any initialization needed.
boolean	isAuthorized(ProvenanceEventRecord event, NiFiUser user)
boolean	isShutdownComplete()
protected java.util.Set<java.io.File>	recoverJournalFiles()
void	registerEvent(ProvenanceEventRecord event) Adds the given event to the repository and returns a new event for which the event id has been populated.
void	registerEvents(java.lang.Iterable<ProvenanceEventRecord> events) Adds the given events to the repository.
java.util.Set<ProvenanceEventRecord>	replaceUnauthorizedWithPlaceholders(java.util.Set<ProvenanceEventRecord> events, NiFiUser user)
AsyncLineageSubmission	retrieveLineageSubmission(java.lang.String lineageIdentifier, NiFiUser user)
QuerySubmission	retrieveQuerySubmission(java.lang.String queryIdentifier, NiFiUser user)
AsyncLineageSubmission	submitExpandChildren(long eventId, NiFiUser user) Submits a request to expand the children of the event with the given id.
AsyncLineageSubmission	submitExpandParents(long eventId, NiFiUser user) Submits a request to expand the parents of the event with the given id.
ComputeLineageSubmission	submitLineageComputation(long eventId, NiFiUser user) Submits a Lineage Computation to be completed and returns the AsynchronousLineageResult that indicates the status of the request and the results, if the computation is complete.
AsyncLineageSubmission	submitLineageComputation(java.lang.String flowFileUuid, NiFiUser user) Submits a Lineage Computation to be completed and returns the AsynchronousLineageResult that indicates the status of the request and the results, if the computation is complete.
QuerySubmission	submitQuery(Query query, NiFiUser user) Submits an asynchronous request to process the given query, returning an identifier that can be used to fetch the results at a later time
void	waitForRollover() Blocks the calling thread until the repository rolls over.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

EVENT_CATEGORY

public static final java.lang.String EVENT_CATEGORY

See Also:

Constant Field Values

NUMBER_PATTERN

public static final java.util.regex.Pattern NUMBER_PATTERN

INDEX_PATTERN

public static final java.util.regex.Pattern INDEX_PATTERN

LOG_FILENAME_PATTERN

public static final java.util.regex.Pattern LOG_FILENAME_PATTERN

MAX_UNDELETED_QUERY_RESULTS

public static final int MAX_UNDELETED_QUERY_RESULTS

See Also:

Constant Field Values

MAX_INDEXING_FAILURE_COUNT

public static final int MAX_INDEXING_FAILURE_COUNT

See Also:

Constant Field Values

MAX_JOURNAL_ROLLOVER_RETRIES

public static final int MAX_JOURNAL_ROLLOVER_RETRIES

See Also:

Constant Field Values

Constructor Detail

PersistentProvenanceRepository

public PersistentProvenanceRepository()

default no args constructor for service loading only.

PersistentProvenanceRepository

public PersistentProvenanceRepository(NiFiProperties nifiProperties)
                               throws java.io.IOException

Throws:

java.io.IOException

PersistentProvenanceRepository

public PersistentProvenanceRepository(RepositoryConfiguration configuration,
                                      int rolloverCheckMillis)
                               throws java.io.IOException

Throws:

java.io.IOException

Method Detail

getIndexManager

protected IndexManager getIndexManager()

initialize

public void initialize(EventReporter eventReporter,
                       Authorizer authorizer,
                       ProvenanceAuthorizableFactory resourceFactory,
                       IdentifierLookup idLookup)
                throws java.io.IOException

Description copied from interface: ProvenanceRepository

Performs any initialization needed. This should be called only by the framework.

Specified by:

initialize in interface ProvenanceRepository

Parameters:

eventReporter - to report to

authorizer - the authorizer to use for authorizing individual events

resourceFactory - the resource factory to use for generating Provenance Resource objects for authorization purposes

idLookup - a mechanism for looking up identifiers in the flow

Throws:

java.io.IOException - if unable to initialize

createWriters

protected RecordWriter[] createWriters(RepositoryConfiguration config,
                                       long initialRecordId)
                                throws java.io.IOException

Throws:

java.io.IOException

getMaxAttributeCharacters

public int getMaxAttributeCharacters()

Returns:

the maximum number of characters that any Event attribute should contain. If the event contains more characters than this, the attribute may be truncated on retrieval

eventBuilder

public StandardProvenanceEventRecord.Builder eventBuilder()

Description copied from interface: ProvenanceEventRepository

Returns a ProvenanceEventBuilder that is capable of building ProvenanceEventRecords

Specified by:

eventBuilder in interface ProvenanceEventRepository

Returns:

builder

registerEvent

public void registerEvent(ProvenanceEventRecord event)

Description copied from interface: ProvenanceEventRepository

Adds the given event to the repository and returns a new event for which the event id has been populated. Depending on the implementation, the returned event may or may not be the same event given

Specified by:

registerEvent in interface ProvenanceEventRepository

Parameters:

event - to register

registerEvents

public void registerEvents(java.lang.Iterable<ProvenanceEventRecord> events)

Description copied from interface: ProvenanceEventRepository

Adds the given events to the repository.

This interface makes no assumptions about whether or not the registration of the Collection are atomic. This detail is implementation-specific.

Specified by:

registerEvents in interface ProvenanceEventRepository

Parameters:

events - to register

isAuthorized

public boolean isAuthorized(ProvenanceEventRecord event,
                            NiFiUser user)

authorize

public void authorize(ProvenanceEventRecord event,
                      NiFiUser user)

filterUnauthorizedEvents

public java.util.List<ProvenanceEventRecord> filterUnauthorizedEvents(java.util.List<ProvenanceEventRecord> events,
                                                                      NiFiUser user)

replaceUnauthorizedWithPlaceholders

public java.util.Set<ProvenanceEventRecord> replaceUnauthorizedWithPlaceholders(java.util.Set<ProvenanceEventRecord> events,
                                                                                NiFiUser user)

getEvents

public java.util.List<ProvenanceEventRecord> getEvents(long firstRecordId,
                                                       int maxRecords)
                                                throws java.io.IOException

Description copied from interface: ProvenanceEventRepository

Returns a List of all ProvenanceEventRecords in the repository starting with the given ID. The first ID in the repository will always be 0 or higher. This method performs no authorization of the events.

Specified by:

getEvents in interface ProvenanceEventRepository

Parameters:

firstRecordId - id of the first record to retrieve

maxRecords - maximum number of records to retrieve

Returns:

records

Throws:

java.io.IOException - if error reading from repository

getEvents

public java.util.List<ProvenanceEventRecord> getEvents(long firstRecordId,
                                                       int maxRecords,
                                                       NiFiUser user)
                                                throws java.io.IOException

Description copied from interface: ProvenanceRepository

Returns a List of all ProvenanceEventRecords in the repository starting with the given ID. The first ID in the repository will always be 0 or higher. Each event that is found will be authorized against the given NiFiUser. If the user does not have authorization for the event, the event will not be returned.

Specified by:

getEvents in interface ProvenanceRepository

Parameters:

firstRecordId - id of the first record to retrieve

maxRecords - maximum number of records to retrieve

user - the NiFi user that the events should be authorized against

Returns:

records

Throws:

java.io.IOException - if error reading from repository

getConfiguration

public RepositoryConfiguration getConfiguration()

getContainerNames

public java.util.Set<java.lang.String> getContainerNames()

Specified by:

getContainerNames in interface ProvenanceRepository

Returns:

the names of all Containers that exist for this Provenance Repository

getContainerCapacity

public long getContainerCapacity(java.lang.String containerName)
                          throws java.io.IOException

Specified by:

getContainerCapacity in interface ProvenanceRepository

Parameters:

containerName - name of container to check capacity on

Returns:

the maximum number of bytes that can be stored in the storage mechanism that backs the container with the given name

Throws:

java.io.IOException - if unable to check capacity

getContainerUsableSpace

public long getContainerUsableSpace(java.lang.String containerName)
                             throws java.io.IOException

Specified by:

getContainerUsableSpace in interface ProvenanceRepository

Parameters:

containerName - to check space on

Returns:

the number of bytes available to be used used by the storage mechanism that backs the container with the given name

Throws:

java.io.IOException - if unable to check space

close

public void close()
           throws java.io.IOException

Description copied from interface: ProvenanceEventRepository

Closes the repository, freeing any resources

Specified by:

close in interface ProvenanceEventRepository

Throws:

java.io.IOException - if failure closing repository

isShutdownComplete

public boolean isShutdownComplete()

getSize

public long getSize(java.util.List<java.io.File> logFiles,
                    long timeCutoff)

Returns the size, in bytes, of the Repository storage

Parameters:

logFiles - the log files to consider

timeCutoff - if a log file's last modified date is before timeCutoff, it will be skipped

Returns:

the size of all log files given whose last mod date comes after (or equal to) timeCutoff

waitForRollover

public void waitForRollover()

Blocks the calling thread until the repository rolls over. This is intended for unit testing.

getJournalCount

protected int getJournalCount()

Returns:

the number of journal files that exist across all storage directories

getRolloverRetryMillis

protected long getRolloverRetryMillis()

recoverJournalFiles

protected java.util.Set<java.io.File> recoverJournalFiles()
                                                   throws java.io.IOException

Throws:

java.io.IOException

filterUnavailableFiles

protected java.util.List<java.io.File> filterUnavailableFiles(java.util.List<java.io.File> journalFiles)

createIndexingAction

protected IndexingAction createIndexingAction()

This method is protected and exists for testing purposes. This allows unit tests to extend this class and override the createIndexingAction so that they can mock out the Indexing Action to throw Exceptions, count events indexed, etc.

getSearchableFields

public java.util.List<SearchableField> getSearchableFields()

Specified by:

getSearchableFields in interface ProvenanceRepository

Returns:

a list of all fields that can be searched via the ProvenanceRepository.submitQuery(Query, NiFiUser) method

getSearchableAttributes

public java.util.List<SearchableField> getSearchableAttributes()

Specified by:

getSearchableAttributes in interface ProvenanceRepository

Returns:

a list of all FlowFile attributes that can be searched via the ProvenanceRepository.submitQuery(Query, NiFiUser) method

submitQuery

public QuerySubmission submitQuery(Query query,
                                   NiFiUser user)

Description copied from interface: ProvenanceRepository

Submits an asynchronous request to process the given query, returning an identifier that can be used to fetch the results at a later time

Specified by:

submitQuery in interface ProvenanceRepository

Parameters:

query - to submit

user - the NiFi User to authorize the events against

Returns:

an identifier that can be used to fetch the results at a later time

submitLineageComputation

public ComputeLineageSubmission submitLineageComputation(long eventId,
                                                         NiFiUser user)

Description copied from interface: ProvenanceRepository

Submits a Lineage Computation to be completed and returns the AsynchronousLineageResult that indicates the status of the request and the results, if the computation is complete. If the given user does not have authorization to view one of the events in the lineage, a placeholder event will be used instead that provides none of the event details except for the identifier of the component that emitted the Provenance Event. It is necessary to include this node in the lineage view so that the lineage makes sense, rather than showing disconnected graphs when the user is not authorized for all components' provenance events. This method is preferred to ProvenanceRepository.submitLineageComputation(String, NiFiUser) because it is much more efficient, but the former may be used if a particular Event ID is not available.

Specified by:

submitLineageComputation in interface ProvenanceRepository

Parameters:

eventId - the numeric ID of the event that the lineage is for

user - the NiFi User to authorize events against

Returns:

a ComputeLineageSubmission object that can be used to check if the computing is complete and if so get the results

submitLineageComputation

public AsyncLineageSubmission submitLineageComputation(java.lang.String flowFileUuid,
                                                       NiFiUser user)

Description copied from interface: ProvenanceRepository

Submits a Lineage Computation to be completed and returns the AsynchronousLineageResult that indicates the status of the request and the results, if the computation is complete. If the given user does not have authorization to view one of the events in the lineage, a placeholder event will be used instead that provides none of the event details except for the identifier of the component that emitted the Provenance Event. It is necessary to include this node in the lineage view so that the lineage makes sense, rather than showing disconnected graphs when the user is not authorized for all components' provenance events.

Specified by:

submitLineageComputation in interface ProvenanceRepository

Parameters:

flowFileUuid - the UUID of the FlowFile for which the Lineage should be calculated

user - the NiFi User to authorize events against

Returns:

a ComputeLineageSubmission object that can be used to check if the computing is complete and if so get the results

submitExpandChildren

public AsyncLineageSubmission submitExpandChildren(long eventId,
                                                   NiFiUser user)

Description copied from interface: ProvenanceRepository

Submits a request to expand the children of the event with the given id. If the given user is not authorized to access any event, a placeholder will be used instead that contains only the ID of the component that emitted the event.

Specified by:

submitExpandChildren in interface ProvenanceRepository

Parameters:

eventId - the one-up id of the Event

user - the NiFi user to authorize events against

Returns:

a submission which can be checked for status

submitExpandParents

public AsyncLineageSubmission submitExpandParents(long eventId,
                                                  NiFiUser user)

Description copied from interface: ProvenanceRepository

Submits a request to expand the parents of the event with the given id. If the given user is not authorized to access any event, a placeholder will be used instead that contains only the ID of the component that emitted the event.

Specified by:

submitExpandParents in interface ProvenanceRepository

Parameters:

eventId - the one-up id of the Event to expand

user - the NiFi user to authorize events against

Returns:

a submission which can be checked for status

retrieveLineageSubmission

public AsyncLineageSubmission retrieveLineageSubmission(java.lang.String lineageIdentifier,
                                                        NiFiUser user)

Specified by:

retrieveLineageSubmission in interface ProvenanceRepository

Parameters:

lineageIdentifier - identifier of lineage to compute

user - the user who is retrieving the lineage submission

Returns:

the ComputeLineageSubmission associated with the given identifier

retrieveQuerySubmission

public QuerySubmission retrieveQuerySubmission(java.lang.String queryIdentifier,
                                               NiFiUser user)

Specified by:

retrieveQuerySubmission in interface ProvenanceRepository

Parameters:

queryIdentifier - of the query

user - the user who is retrieving the query

Returns:

the QueryResult associated with the given identifier, if the query has finished processing. If the query has not yet finished running, returns null

getEvent

public ProvenanceEventRecord getEvent(long id)
                               throws java.io.IOException

Description copied from interface: ProvenanceEventRepository

Retrieves the Provenance Event with the given ID. The event will be returned only if the given user is authorized to access the event. Otherwise, an AccessDeniedException or ResourceNotFoundException will be thrown, as appropriate

Specified by:

getEvent in interface ProvenanceEventRepository

Parameters:

id - to lookup

Returns:

the Provenance Event Record with the given ID, if it exists, or null otherwise

Throws:

java.io.IOException - if failure while retrieving event

getEvent

public ProvenanceEventRecord getEvent(long id,
                                      NiFiUser user)
                               throws java.io.IOException

Specified by:

getEvent in interface ProvenanceRepository

Throws:

java.io.IOException

getProvenanceEventRepository

public ProvenanceEventRepository getProvenanceEventRepository()

Specified by:

getProvenanceEventRepository in interface ProvenanceRepository

Returns:

the ProvenanceEventRepository backing this ProvenanceRepository

getAllLogFiles

public java.util.Collection<java.nio.file.Path> getAllLogFiles()

getMaxEventId

public java.lang.Long getMaxEventId()

Specified by:

getMaxEventId in interface ProvenanceEventRepository

Returns:

the largest ID of any event that is queryable in the repository. If no queryable events exists, returns null