Package org.cdlib.mrt.s3v2.aws

Class AWSS3V2Cloud

java.lang.Object

org.cdlib.mrt.s3.service.CloudStoreAbs

org.cdlib.mrt.s3v2.aws.AWSS3V2Cloud

All Implemented Interfaces:

CloudStoreInf

public class AWSS3V2Cloud
extends CloudStoreAbs
implements CloudStoreInf

Specific SDSC Storage Cloud handling

Author:

dloy

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static enum	AWSS3V2Cloud.S3Type

Nested classes/interfaces inherited from interface org.cdlib.mrt.s3.service.CloudStoreInf

CloudStoreInf.CloudAPI

Field Summary

Fields

Modifier and Type	Field	Description
protected static final String	MESSAGE
protected static final String	NAME

Fields inherited from class org.cdlib.mrt.s3.service.CloudStoreAbs

cloudProp, endpointHostname, logger

Constructor Summary

Constructors

Modifier	Constructor	Description
protected	AWSS3V2Cloud(V2Client v2Client, org.cdlib.mrt.utility.LoggerInf log)

Method Summary

Modifier and Type	Method	Description
protected static void	addProp(Properties prop, String key, String value)
boolean	awsConvertStorageClass(String bucket, String key, software.amazon.awssdk.services.s3.model.StorageClass targetStorageClass, CloudResponse response)	Convert from one AWS storage class to another
CloudResponse	awsDelete(CloudResponse response)
void	awsGet(String container, String key, File outFile, CloudResponse response)
InputStream	awsGetInputStream(String container, String key, CloudResponse response)
void	awsHandleException(CloudResponse response, Exception exception)
void	awsListAfter(String bucketName, String startAfter, int maxKeys, CloudResponse response)
protected void	awsListPrefix(String bucket, String listPrefix, int maxEntries, CloudResponse response)	List entries
void	awsRestore(String container, String key, File outFile, CloudResponse response)
CloudResponse	convertStorageClass(String bucket, String key, software.amazon.awssdk.services.s3.model.StorageClass targetStorageClass)
boolean	convertStorageClass(String bucket, String key, software.amazon.awssdk.services.s3.model.StorageClass targetStorageClass, CloudResponse response)	Convert from one AWS storage class to another
CloudResponse	deleteManifest(String bucketName, org.cdlib.mrt.core.Identifier objectID)	Delete cloud manifest
CloudResponse	deleteObject(String bucket, String key)	Delete object
CloudResponse	deleteObject(String bucketName, org.cdlib.mrt.core.Identifier objectID, int versionID, String fileID)	Delete cloud object
static AWSS3V2Cloud	getAWS(org.cdlib.mrt.utility.LoggerInf log)
CloudResponse	getCloudResponse(String bucketName, String key)
org.cdlib.mrt.utility.LoggerInf	getLogger()
InputStream	getManifest(String bucket, org.cdlib.mrt.core.Identifier objectID)	Retrieve cloud manifest
InputStream	getManifest(String bucket, org.cdlib.mrt.core.Identifier objectID, CloudResponse response)	Retrieve cloud manifest
static AWSS3V2Cloud	getMinio(String accessKey, String secretKey, String endpoint, org.cdlib.mrt.utility.LoggerInf log)
void	getObject(String container, String key, File outFile, CloudResponse response)	Retrieve content into a file
InputStream	getObject(String bucketName, String key, CloudResponse response)	Retrieve cloud object
InputStream	getObject(String name, org.cdlib.mrt.core.Identifier objectID, int versionID, String fileID, CloudResponse response)	Retrieve cloud Object
CloudResponse	getObjectList(String bucketName)	Return info on bucket/container
CloudResponse	getObjectList(String bucketName, String key)	Return content that matches for the length of the key
CloudResponse	getObjectList(String bucketName, String key, int limit)	Return a set number of entry metadata based on start key
CloudResponse	getObjectList(String bucketName, org.cdlib.mrt.core.Identifier objectID, Integer versionID, String fileID)
CloudResponse	getObjectListAfter(String bucketName, String afterKey, int limit)	Return list of S3 keys after a start key
Properties	getObjectMeta(String bucketName, String key)	Retrieve metadata for file
InputStream	getObjectStreaming(String bucketName, String key, CloudResponse response)	Retrieve cloud object in streaming mode
CloudResponse	getPreSigned(long expirationMinutes, String bucketName, String key, String contentType, String contentDisp)	Returns an S3 presigned URL
InputStream	getRangeStream(String bucketName, String key, long start, long stop, CloudResponse response)	Return part of a cloud object as stream
Properties	getRetryObjectMeta(String bucketName, String key, int retryCnt)
AWSS3V2Cloud.S3Type	getS3Type()
static AWSS3V2Cloud	getSDSC(String accessKey, String secretKey, String endpoint, org.cdlib.mrt.utility.LoggerInf log)
StateHandler.RetState	getState(String bucketName)	Return state of cloud store manager
software.amazon.awssdk.services.s3.model.StorageClass	getStorageClass()
software.amazon.awssdk.services.s3.model.StorageClass	getStorageClass(String bucketName, String key)
CloudStoreInf.CloudAPI	getType()	Return the API type
static AWSS3V2Cloud	getWasabi(String accessKey, String secretKey, String endpoint, org.cdlib.mrt.utility.LoggerInf log)
Boolean	isAlive(String bucketName)	Determine if host:port is available
boolean	isAlphaNumericKey()	Because of earlier SDSC bug only alpha-numerics could be used in a key.
CloudResponse	putManifest(String bucketName, org.cdlib.mrt.core.Identifier objectID, File inputFile)	Upload cloud manifest
CloudResponse	putObject(String bucket, String key, File inputFile)	Upload cloud object
CloudResponse	putObject(String bucketName, String key, File inputFile, Properties fileMeta)
CloudResponse	putObject(String bucketName, org.cdlib.mrt.core.Identifier objectID, int versionID, String fileID, File inputFile)	Upload cloud object
CloudResponse	putObject(String bucketName, org.cdlib.mrt.core.Identifier objectID, int versionID, String fileID, File inputFile, Properties fileMeta)
CloudResponse	putObject(CloudResponse response, File inputFile)
void	restoreObject(String container, String key, File outFile, CloudResponse response)	Restore nearline content
void	retryAWSConvertStorageClass(String bucket, String key, software.amazon.awssdk.services.s3.model.StorageClass targetStorageClass)
void	setS3Type(AWSS3V2Cloud.S3Type s3Type)
void	setStorageClass(String storageClassS)
void	setStorageClass(software.amazon.awssdk.services.s3.model.StorageClass storageClass)

Methods inherited from class org.cdlib.mrt.s3.service.CloudStoreAbs

dumpException, getCloudProp, handleException, isAliveTest, isValidFile, validateDigest, validateMd5

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.cdlib.mrt.s3.service.CloudStoreInf

getCloudProp, validateDigest, validateMd5

Field Details

NAME

protected static final String NAME

See Also:

• Constant Field Values

MESSAGE

protected static final String MESSAGE

See Also:

• Constant Field Values

Constructor Details

AWSS3V2Cloud

protected AWSS3V2Cloud(V2Client v2Client,
 org.cdlib.mrt.utility.LoggerInf log)
                throws org.cdlib.mrt.utility.TException

Throws:

org.cdlib.mrt.utility.TException

Method Details

getAWS

public static AWSS3V2Cloud getAWS(org.cdlib.mrt.utility.LoggerInf log)
                           throws org.cdlib.mrt.utility.TException

Throws:

org.cdlib.mrt.utility.TException

getMinio

public static AWSS3V2Cloud getMinio(String accessKey,
 String secretKey,
 String endpoint,
 org.cdlib.mrt.utility.LoggerInf log)
                             throws org.cdlib.mrt.utility.TException

Throws:

org.cdlib.mrt.utility.TException

getSDSC

public static AWSS3V2Cloud getSDSC(String accessKey,
 String secretKey,
 String endpoint,
 org.cdlib.mrt.utility.LoggerInf log)
                            throws org.cdlib.mrt.utility.TException

Throws:

org.cdlib.mrt.utility.TException

getWasabi

public static AWSS3V2Cloud getWasabi(String accessKey,
 String secretKey,
 String endpoint,
 org.cdlib.mrt.utility.LoggerInf log)
                              throws org.cdlib.mrt.utility.TException

Throws:

org.cdlib.mrt.utility.TException

putObject

public CloudResponse putObject(CloudResponse response,
 File inputFile)
                        throws org.cdlib.mrt.utility.TException

Throws:

org.cdlib.mrt.utility.TException

putObject

public CloudResponse putObject(String bucket,
 String key,
 File inputFile)
                        throws org.cdlib.mrt.utility.TException

Description copied from interface: CloudStoreInf

Upload cloud object

Specified by:

putObject in interface CloudStoreInf

Parameters:

bucket - s3 bucket - rackspace container

key - entry key to be added

inputFile - file to be uploaded

Returns:

information about upload

Throws:

org.cdlib.mrt.utility.TException

putObject

public CloudResponse putObject(String bucketName,
 org.cdlib.mrt.core.Identifier objectID,
 int versionID,
 String fileID,
 File inputFile)
                        throws org.cdlib.mrt.utility.TException

Description copied from interface: CloudStoreInf

Upload cloud object

Specified by:

putObject in interface CloudStoreInf

Parameters:

bucketName - s3 bucket - rackspace container

objectID - Object identifier

versionID - Version identifier

fileID - File/component identifier that may contain a directory element

inputFile - file to be uploaded

Returns:

information about upload

Throws:

org.cdlib.mrt.utility.TException

putObject

public CloudResponse putObject(String bucketName,
 String key,
 File inputFile,
 Properties fileMeta)
                        throws org.cdlib.mrt.utility.TException

Throws:

org.cdlib.mrt.utility.TException

putObject

public CloudResponse putObject(String bucketName,
 org.cdlib.mrt.core.Identifier objectID,
 int versionID,
 String fileID,
 File inputFile,
 Properties fileMeta)
                        throws org.cdlib.mrt.utility.TException

Throws:

org.cdlib.mrt.utility.TException

putManifest

public CloudResponse putManifest(String bucketName,
 org.cdlib.mrt.core.Identifier objectID,
 File inputFile)
                          throws org.cdlib.mrt.utility.TException

Description copied from interface: CloudStoreInf

Upload cloud manifest

Specified by:

putManifest in interface CloudStoreInf

Parameters:

bucketName - s3 bucket - rackspace container

objectID - Object identifier

inputFile - manifest file to be uploaded

Returns:

information about upload

Throws:

org.cdlib.mrt.utility.TException

awsDelete

public CloudResponse awsDelete(CloudResponse response)
                        throws org.cdlib.mrt.utility.TException

Throws:

org.cdlib.mrt.utility.TException

deleteObject

public CloudResponse deleteObject(String bucket,
 String key)
                           throws org.cdlib.mrt.utility.TException

Description copied from interface: CloudStoreInf

Delete object

Specified by:

deleteObject in interface CloudStoreInf

Parameters:

bucket - s3 bucket - rackspace container

key - cloud object key

Returns:

Throws:

org.cdlib.mrt.utility.TException

deleteObject

public CloudResponse deleteObject(String bucketName,
 org.cdlib.mrt.core.Identifier objectID,
 int versionID,
 String fileID)
                           throws org.cdlib.mrt.utility.TException

Description copied from interface: CloudStoreInf

Delete cloud object

Specified by:

deleteObject in interface CloudStoreInf

Parameters:

bucketName - s3 bucket - rackspace container

objectID - Object identifier

versionID - Version identifier

fileID - File/component identifier that may contain a directory element

Returns:

Throws:

org.cdlib.mrt.utility.TException

deleteManifest

public CloudResponse deleteManifest(String bucketName,
 org.cdlib.mrt.core.Identifier objectID)
                             throws org.cdlib.mrt.utility.TException

Description copied from interface: CloudStoreInf

Delete cloud manifest

Specified by:

deleteManifest in interface CloudStoreInf

Parameters:

bucketName - s3 bucket - rackspace container

objectID - Object identifier

Returns:

Throws:

org.cdlib.mrt.utility.TException

awsGet

public void awsGet(String container,
 String key,
 File outFile,
 CloudResponse response)
            throws org.cdlib.mrt.utility.TException

Throws:

org.cdlib.mrt.utility.TException

awsGetInputStream

public InputStream awsGetInputStream(String container,
 String key,
 CloudResponse response)
                              throws org.cdlib.mrt.utility.TException

Throws:

org.cdlib.mrt.utility.TException

awsRestore

public void awsRestore(String container,
 String key,
 File outFile,
 CloudResponse response)
                throws org.cdlib.mrt.utility.TException

Throws:

org.cdlib.mrt.utility.TException

convertStorageClass

public boolean convertStorageClass(String bucket,
 String key,
 software.amazon.awssdk.services.s3.model.StorageClass targetStorageClass,
 CloudResponse response)
                            throws org.cdlib.mrt.utility.TException

Convert from one AWS storage class to another

Parameters:

bucket - AWS bucket

key - AWS key

targetStorageClass - resulting storage class for conversion

response - Broad response information

Returns:

true=complete

Throws:

org.cdlib.mrt.utility.TException

awsConvertStorageClass

public boolean awsConvertStorageClass(String bucket,
 String key,
 software.amazon.awssdk.services.s3.model.StorageClass targetStorageClass,
 CloudResponse response)
                               throws org.cdlib.mrt.utility.TException

Convert from one AWS storage class to another

Parameters:

bucket - AWS bucket

key - AWS key

targetStorageClass - resulting storage class for conversion

response - Broad response information

Returns:

true=complete

Throws:

org.cdlib.mrt.utility.TException

retryAWSConvertStorageClass

public void retryAWSConvertStorageClass(String bucket,
 String key,
 software.amazon.awssdk.services.s3.model.StorageClass targetStorageClass)
                                 throws Exception

Throws:

Exception

convertStorageClass

public CloudResponse convertStorageClass(String bucket,
 String key,
 software.amazon.awssdk.services.s3.model.StorageClass targetStorageClass)
                                  throws org.cdlib.mrt.utility.TException

Throws:

org.cdlib.mrt.utility.TException

restoreObject

public void restoreObject(String container,
 String key,
 File outFile,
 CloudResponse response)
                   throws org.cdlib.mrt.utility.TException

Description copied from interface: CloudStoreInf

Restore nearline content

Specified by:

restoreObject in interface CloudStoreInf

Parameters:

key - cloud key

outFile - target file of response

response -

Throws:

org.cdlib.mrt.utility.TException

getObject

public InputStream getObject(String name,
 org.cdlib.mrt.core.Identifier objectID,
 int versionID,
 String fileID,
 CloudResponse response)
                      throws org.cdlib.mrt.utility.TException

Description copied from interface: CloudStoreInf

Retrieve cloud Object

Specified by:

getObject in interface CloudStoreInf

Parameters:

name - s3 bucket - rackspace container

objectID - Object identifier

versionID - Version identifier

fileID - File/component identifier that may contain a directory element

response - input stream of retrieved object

Returns:

Throws:

org.cdlib.mrt.utility.TException

getObject

public InputStream getObject(String bucketName,
 String key,
 CloudResponse response)
                      throws org.cdlib.mrt.utility.TException

Description copied from interface: CloudStoreInf

Retrieve cloud object

Specified by:

getObject in interface CloudStoreInf

Specified by:

getObject in class CloudStoreAbs

Parameters:

bucketName - s3 bucket - rackspace container

key - cloud object key for object to be returned

response -

Returns:

input stream of retrieved object

Throws:

org.cdlib.mrt.utility.TException

getObjectStreaming

public InputStream getObjectStreaming(String bucketName,
 String key,
 CloudResponse response)
                               throws org.cdlib.mrt.utility.TException

Description copied from interface: CloudStoreInf

Retrieve cloud object in streaming mode

Specified by:

getObjectStreaming in interface CloudStoreInf

Parameters:

bucketName - s3 bucket - rackspace container

key - cloud object key for object to be returned

response -

Returns:

input stream of retrieved object

Throws:

org.cdlib.mrt.utility.TException

getObject

public void getObject(String container,
 String key,
 File outFile,
 CloudResponse response)
               throws org.cdlib.mrt.utility.TException

Description copied from interface: CloudStoreInf

Retrieve content into a file

Specified by:

getObject in interface CloudStoreInf

Parameters:

container - s3 bucket - rackspace container

key - cloud object key for object to be returned

outFile - file to receive content

response -

Throws:

org.cdlib.mrt.utility.TException

getObjectMeta

public Properties getObjectMeta(String bucketName,
 String key)

Description copied from interface: CloudStoreInf

Retrieve metadata for file

Specified by:

getObjectMeta in interface CloudStoreInf

Parameters:

bucketName - s3 bucket - rackspace container

key - cloud object key for object to be returned

Returns:

exists: named properties; does not exist: null

getRetryObjectMeta

public Properties getRetryObjectMeta(String bucketName,
 String key,
 int retryCnt)

getStorageClass

public software.amazon.awssdk.services.s3.model.StorageClass getStorageClass(String bucketName,
 String key)
                                                                      throws org.cdlib.mrt.utility.TException

Throws:

org.cdlib.mrt.utility.TException

getCloudResponse

public CloudResponse getCloudResponse(String bucketName,
 String key)
                               throws org.cdlib.mrt.utility.TException

Throws:

org.cdlib.mrt.utility.TException

addProp

protected static void addProp(Properties prop,
 String key,
 String value)

getManifest

public InputStream getManifest(String bucket,
 org.cdlib.mrt.core.Identifier objectID)
                        throws org.cdlib.mrt.utility.TException

Description copied from interface: CloudStoreInf

Retrieve cloud manifest

Specified by:

getManifest in interface CloudStoreInf

Parameters:

bucket - s3 bucket - rackspace container

objectID - Object identifier

Returns:

input stream of retrieved manifest

Throws:

org.cdlib.mrt.utility.TException

getManifest

public InputStream getManifest(String bucket,
 org.cdlib.mrt.core.Identifier objectID,
 CloudResponse response)
                        throws org.cdlib.mrt.utility.TException

Description copied from interface: CloudStoreInf

Retrieve cloud manifest

Specified by:

getManifest in interface CloudStoreInf

Parameters:

bucket - s3 bucket - rackspace container

objectID - Object identifier

response -

Returns:

input stream of retrieved object

Throws:

org.cdlib.mrt.utility.TException

awsListPrefix

protected void awsListPrefix(String bucket,
 String listPrefix,
 int maxEntries,
 CloudResponse response)
                      throws org.cdlib.mrt.utility.TException

List entries

Parameters:

bucket - - non AWS bucket name

listPrefix - what to truncate on as if aws prefix not there

maxEntries - maximum entries to return

response - filled CloudResponse

Throws:

org.cdlib.mrt.utility.TException

awsListAfter

public void awsListAfter(String bucketName,
 String startAfter,
 int maxKeys,
 CloudResponse response)
                  throws org.cdlib.mrt.utility.TException

Throws:

org.cdlib.mrt.utility.TException

getObjectList

public CloudResponse getObjectList(String bucketName,
 org.cdlib.mrt.core.Identifier objectID,
 Integer versionID,
 String fileID)
                            throws org.cdlib.mrt.utility.TException

Throws:

org.cdlib.mrt.utility.TException

getObjectList

public CloudResponse getObjectList(String bucketName,
 String key)
                            throws org.cdlib.mrt.utility.TException

Description copied from interface: CloudStoreInf

Return content that matches for the length of the key

Specified by:

getObjectList in interface CloudStoreInf

Specified by:

getObjectList in class CloudStoreAbs

Parameters:

bucketName - s3 bucket - rackspace container

key - prefix of content to be returned

Returns:

CloudResponse with list of matching entries

Throws:

org.cdlib.mrt.utility.TException

getObjectList

public CloudResponse getObjectList(String bucketName,
 String key,
 int limit)
                            throws org.cdlib.mrt.utility.TException

Description copied from interface: CloudStoreInf

Return a set number of entry metadata based on start key

Specified by:

getObjectList in interface CloudStoreInf

Parameters:

bucketName - s3 bucket - rackspace container

key - content key of the first output

limit - number of returned entries

Returns:

CloudResponse with list of sequential entries

Throws:

org.cdlib.mrt.utility.TException

getObjectListAfter

public CloudResponse getObjectListAfter(String bucketName,
 String afterKey,
 int limit)
                                 throws org.cdlib.mrt.utility.TException

Description copied from interface: CloudStoreInf

Return list of S3 keys after a start key

Specified by:

getObjectListAfter in interface CloudStoreInf

Parameters:

bucketName - s3 bucket

afterKey - start return list after this value

limit - number of keys to return

Returns:

cloud response with keys

Throws:

org.cdlib.mrt.utility.TException

getObjectList

public CloudResponse getObjectList(String bucketName)
                            throws org.cdlib.mrt.utility.TException

Description copied from interface: CloudStoreInf

Return info on bucket/container

Specified by:

getObjectList in interface CloudStoreInf

Parameters:

bucketName - s3 bucket - rackspace container

Returns:

Throws:

org.cdlib.mrt.utility.TException

getState

public StateHandler.RetState getState(String bucketName)
                               throws org.cdlib.mrt.utility.TException

Description copied from interface: CloudStoreInf

Return state of cloud store manager

Specified by:

getState in interface CloudStoreInf

Parameters:

bucketName - bucket container

Returns:

Throws:

org.cdlib.mrt.utility.TException

getStorageClass

public software.amazon.awssdk.services.s3.model.StorageClass getStorageClass()

setStorageClass

public void setStorageClass(software.amazon.awssdk.services.s3.model.StorageClass storageClass)

setStorageClass

public void setStorageClass(String storageClassS)

isAlphaNumericKey

public boolean isAlphaNumericKey()

Description copied from interface: CloudStoreInf

Because of earlier SDSC bug only alpha-numerics could be used in a key. true=alpha-numeric key, false=ASCII (current default)

Specified by:

isAlphaNumericKey in interface CloudStoreInf

Returns:

awsHandleException

public void awsHandleException(CloudResponse response,
 Exception exception)
                        throws org.cdlib.mrt.utility.TException

Throws:

org.cdlib.mrt.utility.TException

isAlive

public Boolean isAlive(String bucketName)

Description copied from interface: CloudStoreInf

Determine if host:port is available

Specified by:

isAlive in interface CloudStoreInf

Returns:

true=alive; false=not alive; null=test not performed

getPreSigned

public CloudResponse getPreSigned(long expirationMinutes,
 String bucketName,
 String key,
 String contentType,
 String contentDisp)
                           throws org.cdlib.mrt.utility.TException

Description copied from interface: CloudStoreInf

Returns an S3 presigned URL

Specified by:

getPreSigned in interface CloudStoreInf

Parameters:

expirationMinutes - minutes for the signed URL to work

bucketName - cloud container

key - cloud key

contentType - optional ContentType for this file

contentDisp - optional ContentDisposition for this file

Returns:

CloudResponse where response.getReturnURL returns the presigned URL

Throws:

org.cdlib.mrt.utility.TException

getRangeStream

public InputStream getRangeStream(String bucketName,
 String key,
 long start,
 long stop,
 CloudResponse response)
                           throws org.cdlib.mrt.utility.TException

Description copied from interface: CloudStoreInf

Return part of a cloud object as stream

Specified by:

getRangeStream in interface CloudStoreInf

Parameters:

bucketName - cloud container

key - cloud key

start - byte location to start from zero

stop - byte location to stop from zero

response - Res[pmse tp reqiest

Returns:

input stream for segment

Throws:

org.cdlib.mrt.utility.TException

getType

public CloudStoreInf.CloudAPI getType()

Description copied from interface: CloudStoreInf

Return the API type

Specified by:

getType in interface CloudStoreInf

Returns:

AWS-S3, SDSC-Swift, Cloudhost, Pairtree

getLogger

public org.cdlib.mrt.utility.LoggerInf getLogger()

getS3Type

public AWSS3V2Cloud.S3Type getS3Type()

setS3Type

public void setS3Type(AWSS3V2Cloud.S3Type s3Type)