@SupportsBatching @SeeAlso(value={FetchS3Object.class,DeleteS3Object.class,ListS3.class}) @InputRequirement(value=INPUT_REQUIRED) @Tags(value={"Amazon","S3","AWS","Archive","Put"}) @CapabilityDescription(value="Puts FlowFiles to an Amazon S3 Bucket\nThe upload uses either the PutS3Object method or PutS3MultipartUpload methods. The PutS3Object method send the file in a single synchronous call, but it has a 5GB size limit. Larger files are sent using the multipart upload methods that initiate, transfer the parts, and complete an upload. This multipart process saves state after each step so that a large upload can be resumed with minimal loss if the processor or cluster is stopped and restarted.\nA multipart upload consists of three steps\n 1) initiate upload,\n 2) upload the parts, and\n 3) complete the upload.\nFor multipart uploads, the processor saves state locally tracking the upload ID and parts uploaded, which must both be provided to complete the upload.\nThe AWS libraries select an endpoint URL based on the AWS region, but this can be overridden with the \'Endpoint Override URL\' property for use with other S3-compatible endpoints.\nThe S3 API specifies that the maximum file size for a PutS3Object upload is 5GB. It also requires that parts in a multipart upload must be at least 5MB in size, except for the last part. These limits are establish the bounds for the Multipart Upload Threshold and Part Size properties.") @DynamicProperty(name="The name of a User-Defined Metadata field to add to the S3 Object", value="The value of a User-Defined Metadata field to add to the S3 Object", description="Allows user-defined metadata to be added to the S3 object as key/value pairs", supportsExpressionLanguage=true) @ReadsAttribute(attribute="filename", description="Uses the FlowFile\'s filename as the filename for the S3 object") @WritesAttribute(attribute="s3.bucket",description="The S3 bucket where the Object was put in S3") @WritesAttribute(attribute="s3.key",description="The S3 key within where the Object was put in S3") @WritesAttribute(attribute="s3.contenttype",description="The S3 content type of the S3 Object that put in S3") @WritesAttribute(attribute="s3.version",description="The version of the S3 Object that was put to S3") @WritesAttribute(attribute="s3.etag",description="The ETag of the S3 Object") @WritesAttribute(attribute="s3.uploadId",description="The uploadId used to upload the Object to S3") @WritesAttribute(attribute="s3.expiration",description="A human-readable form of the expiration date of the S3 object, if one is set") @WritesAttribute(attribute="s3.sseAlgorithm",description="The server side encryption algorithm of the object") @WritesAttribute(attribute="s3.usermetadata",description="A human-readable form of the User Metadata of the S3 object, if any was set") public class PutS3Object extends AbstractS3Processor
| Modifier and Type | Class and Description |
|---|---|
protected static class |
PutS3Object.MultipartState |
| Modifier and Type | Field and Description |
|---|---|
static PropertyDescriptor |
CONTENT_TYPE |
static PropertyDescriptor |
EXPIRATION_RULE_ID |
static long |
MAX_S3_PUTOBJECT_SIZE |
static long |
MIN_S3_PART_SIZE |
static PropertyDescriptor |
MULTIPART_PART_SIZE |
static PropertyDescriptor |
MULTIPART_S3_AGEOFF_INTERVAL |
static PropertyDescriptor |
MULTIPART_S3_MAX_AGE |
static PropertyDescriptor |
MULTIPART_THRESHOLD |
static java.lang.String |
NO_SERVER_SIDE_ENCRYPTION |
static java.lang.String |
PERSISTENCE_ROOT |
static java.util.List<PropertyDescriptor> |
properties |
static PropertyDescriptor |
SERVER_SIDE_ENCRYPTION |
static PropertyDescriptor |
STORAGE_CLASS |
BUCKET, CANNED_ACL, FULL_CONTROL_USER_LIST, KEY, OWNER, READ_ACL_LIST, READ_USER_LIST, SIGNER_OVERRIDE, WRITE_ACL_LIST, WRITE_USER_LISTAWS_CREDENTIALS_PROVIDER_SERVICEACCESS_KEY, client, CREDENTIALS_FILE, DEFAULT_PROTOCOL, DEFAULT_USER_AGENT, ENDPOINT_OVERRIDE, PROXY_HOST, PROXY_HOST_PORT, region, REGION, REL_FAILURE, REL_SUCCESS, relationships, SECRET_KEY, SSL_CONTEXT_SERVICE, TIMEOUT| Constructor and Description |
|---|
PutS3Object() |
| Modifier and Type | Method and Description |
|---|---|
protected void |
abortS3MultipartUpload(com.amazonaws.services.s3.AmazonS3Client s3,
java.lang.String bucket,
com.amazonaws.services.s3.model.MultipartUpload upload) |
protected void |
ageoffS3Uploads(ProcessContext context,
com.amazonaws.services.s3.AmazonS3Client s3,
long now) |
protected PutS3Object.MultipartState |
getLocalState(java.lang.String s3ObjectKey) |
protected PutS3Object.MultipartState |
getLocalStateIfInS3(com.amazonaws.services.s3.AmazonS3Client s3,
java.lang.String bucket,
java.lang.String s3ObjectKey) |
protected java.io.File |
getPersistenceFile() |
protected com.amazonaws.services.s3.model.MultipartUploadListing |
getS3AgeoffListAndAgeoffLocalState(ProcessContext context,
com.amazonaws.services.s3.AmazonS3Client s3,
long now) |
protected PropertyDescriptor |
getSupportedDynamicPropertyDescriptor(java.lang.String propertyDescriptorName)
Used to allow subclasses to determine what PropertyDescriptor if any to
use when a property is requested for which a descriptor is not already
registered.
|
protected java.util.List<PropertyDescriptor> |
getSupportedPropertyDescriptors()
Allows subclasses to register which property descriptor objects are
supported.
|
protected boolean |
localUploadExistsInS3(com.amazonaws.services.s3.AmazonS3Client s3,
java.lang.String bucket,
PutS3Object.MultipartState localState) |
void |
onTrigger(ProcessContext context,
ProcessSession session) |
protected void |
persistLocalState(java.lang.String s3ObjectKey,
PutS3Object.MultipartState currState) |
protected void |
removeLocalState(java.lang.String s3ObjectKey) |
createACL, createCannedACL, createClient, createClient, createGrantee, createGrantees, getUrlForObjectgetCredentialsProvider, onScheduled, onScheduledUsingControllerService, onShutDowncreateConfiguration, customValidate, getClient, getCredentials, getRegion, getRelationships, initializeRegionAndEndpoint, onShutdownonTriggergetControllerServiceLookup, getIdentifier, getLogger, getNodeTypeProvider, init, initialize, isConfigurationRestored, isScheduled, toString, updateConfiguredRestoredTrue, updateScheduledFalse, updateScheduledTrueequals, getPropertyDescriptor, getPropertyDescriptors, hashCode, onPropertyModified, validateclone, finalize, getClass, notify, notifyAll, wait, wait, waitgetPropertyDescriptor, getPropertyDescriptors, onPropertyModified, validatepublic static final long MIN_S3_PART_SIZE
public static final long MAX_S3_PUTOBJECT_SIZE
public static final java.lang.String PERSISTENCE_ROOT
public static final java.lang.String NO_SERVER_SIDE_ENCRYPTION
public static final PropertyDescriptor EXPIRATION_RULE_ID
public static final PropertyDescriptor CONTENT_TYPE
public static final PropertyDescriptor STORAGE_CLASS
public static final PropertyDescriptor MULTIPART_THRESHOLD
public static final PropertyDescriptor MULTIPART_PART_SIZE
public static final PropertyDescriptor MULTIPART_S3_AGEOFF_INTERVAL
public static final PropertyDescriptor MULTIPART_S3_MAX_AGE
public static final PropertyDescriptor SERVER_SIDE_ENCRYPTION
public static final java.util.List<PropertyDescriptor> properties
protected java.util.List<PropertyDescriptor> getSupportedPropertyDescriptors()
AbstractConfigurableComponentgetSupportedPropertyDescriptors in class AbstractConfigurableComponentprotected PropertyDescriptor getSupportedDynamicPropertyDescriptor(java.lang.String propertyDescriptorName)
AbstractConfigurableComponentUsed to allow subclasses to determine what PropertyDescriptor if any to use when a property is requested for which a descriptor is not already registered. By default this method simply returns a null descriptor. By overriding this method processor implementations can support dynamic properties since this allows them to register properties on demand. It is acceptable for a dynamically generated property to indicate it is required so long as it is understood it is only required once set. Dynamic properties by definition cannot be required until used.
This method should be side effect free in the subclasses in terms of how often it is called for a given property name because there is guarantees how often it will be called for a given property name.
Default is null.
getSupportedDynamicPropertyDescriptor in class AbstractConfigurableComponentpropertyDescriptorName - used to lookup if any property descriptors exist for that nameprotected java.io.File getPersistenceFile()
protected boolean localUploadExistsInS3(com.amazonaws.services.s3.AmazonS3Client s3,
java.lang.String bucket,
PutS3Object.MultipartState localState)
protected PutS3Object.MultipartState getLocalStateIfInS3(com.amazonaws.services.s3.AmazonS3Client s3, java.lang.String bucket, java.lang.String s3ObjectKey) throws java.io.IOException
java.io.IOExceptionprotected PutS3Object.MultipartState getLocalState(java.lang.String s3ObjectKey) throws java.io.IOException
java.io.IOExceptionprotected void persistLocalState(java.lang.String s3ObjectKey,
PutS3Object.MultipartState currState)
throws java.io.IOException
java.io.IOExceptionprotected void removeLocalState(java.lang.String s3ObjectKey)
throws java.io.IOException
java.io.IOExceptionpublic void onTrigger(ProcessContext context, ProcessSession session)
onTrigger in class AbstractProcessorprotected void ageoffS3Uploads(ProcessContext context, com.amazonaws.services.s3.AmazonS3Client s3, long now)
protected com.amazonaws.services.s3.model.MultipartUploadListing getS3AgeoffListAndAgeoffLocalState(ProcessContext context, com.amazonaws.services.s3.AmazonS3Client s3, long now)
protected void abortS3MultipartUpload(com.amazonaws.services.s3.AmazonS3Client s3,
java.lang.String bucket,
com.amazonaws.services.s3.model.MultipartUpload upload)