Class AWSS3Cloud

All Implemented Interfaces:
CloudStoreInf

public class AWSS3Cloud extends CloudStoreAbs implements CloudStoreInf
Specific SDSC Storage Cloud handling
Author:
dloy
  • Field Details

  • Constructor Details

    • AWSS3Cloud

      protected AWSS3Cloud(com.amazonaws.services.s3.AmazonS3Client s3Client, String endPoint, org.cdlib.mrt.utility.LoggerInf logger) throws org.cdlib.mrt.utility.TException
      Throws:
      org.cdlib.mrt.utility.TException
    • AWSS3Cloud

      protected AWSS3Cloud(com.amazonaws.regions.Regions region, org.cdlib.mrt.utility.LoggerInf logger) throws org.cdlib.mrt.utility.TException
      Throws:
      org.cdlib.mrt.utility.TException
    • AWSS3Cloud

      protected AWSS3Cloud(String endPoint, org.cdlib.mrt.utility.LoggerInf logger) throws org.cdlib.mrt.utility.TException
      Throws:
      org.cdlib.mrt.utility.TException
  • Method Details

    • getAWSS3Region

      public static AWSS3Cloud getAWSS3Region(String storageClass, String regionS, org.cdlib.mrt.utility.LoggerInf logger) throws org.cdlib.mrt.utility.TException
      Throws:
      org.cdlib.mrt.utility.TException
    • getAWSS3

      public static AWSS3Cloud getAWSS3(String storageClass, org.cdlib.mrt.utility.LoggerInf logger) throws org.cdlib.mrt.utility.TException
      Throws:
      org.cdlib.mrt.utility.TException
    • getMinio

      public static AWSS3Cloud getMinio(String accessKey, String secretKey, String endPoint, org.cdlib.mrt.utility.LoggerInf logger) throws org.cdlib.mrt.utility.TException
      Throws:
      org.cdlib.mrt.utility.TException
    • getWasabi

      public static AWSS3Cloud getWasabi(String accessKey, String secretKey, String endPoint, String regionName, org.cdlib.mrt.utility.LoggerInf logger) throws org.cdlib.mrt.utility.TException
      Throws:
      org.cdlib.mrt.utility.TException
    • getAWSS3

      public static AWSS3Cloud getAWSS3(org.cdlib.mrt.utility.LoggerInf logger) throws org.cdlib.mrt.utility.TException
      Throws:
      org.cdlib.mrt.utility.TException
    • getDefault

      public static AWSS3Cloud getDefault(String endPoint, org.cdlib.mrt.utility.LoggerInf logger) throws org.cdlib.mrt.utility.TException
      Throws:
      org.cdlib.mrt.utility.TException
    • getTransferManager

      public com.amazonaws.services.s3.transfer.TransferManager getTransferManager()
    • amazonS3Client

      public static com.amazonaws.services.s3.AmazonS3Client amazonS3Client(String accessKey, String secretKey, String endPoint, com.amazonaws.regions.Regions regions)
    • amazonS3ClientDefault

      public static com.amazonaws.services.s3.AmazonS3Client amazonS3ClientDefault(com.amazonaws.regions.Regions region)
    • amazonS3ClientDefault

      public static com.amazonaws.services.s3.AmazonS3Client amazonS3ClientDefault(String endPoint)
    • getClientConfiguration

      protected static com.amazonaws.ClientConfiguration getClientConfiguration()
    • getClientConfigurationDefault

      protected static com.amazonaws.ClientConfiguration getClientConfigurationDefault()
    • putObject

      public CloudResponse putObject(CloudResponse response, File inputFile) throws org.cdlib.mrt.utility.TException
      Throws:
      org.cdlib.mrt.utility.TException
    • putObject

      public CloudResponse putObject(String bucket, String key, File inputFile) throws org.cdlib.mrt.utility.TException
      Description copied from interface: CloudStoreInf
      Upload cloud object
      Specified by:
      putObject in interface CloudStoreInf
      Parameters:
      bucket - s3 bucket - rackspace container
      key - entry key to be added
      inputFile - file to be uploaded
      Returns:
      information about upload
      Throws:
      org.cdlib.mrt.utility.TException
    • putObject

      public CloudResponse putObject(String bucketName, org.cdlib.mrt.core.Identifier objectID, int versionID, String fileID, File inputFile) throws org.cdlib.mrt.utility.TException
      Description copied from interface: CloudStoreInf
      Upload cloud object
      Specified by:
      putObject in interface CloudStoreInf
      Parameters:
      bucketName - s3 bucket - rackspace container
      objectID - Object identifier
      versionID - Version identifier
      fileID - File/component identifier that may contain a directory element
      inputFile - file to be uploaded
      Returns:
      information about upload
      Throws:
      org.cdlib.mrt.utility.TException
    • putObject

      public CloudResponse putObject(String bucketName, String key, File inputFile, Properties fileMeta) throws org.cdlib.mrt.utility.TException
      Throws:
      org.cdlib.mrt.utility.TException
    • putObject

      public CloudResponse putObject(String bucketName, org.cdlib.mrt.core.Identifier objectID, int versionID, String fileID, File inputFile, Properties fileMeta) throws org.cdlib.mrt.utility.TException
      Throws:
      org.cdlib.mrt.utility.TException
    • putManifest

      public CloudResponse putManifest(String bucketName, org.cdlib.mrt.core.Identifier objectID, File inputFile) throws org.cdlib.mrt.utility.TException
      Description copied from interface: CloudStoreInf
      Upload cloud manifest
      Specified by:
      putManifest in interface CloudStoreInf
      Parameters:
      bucketName - s3 bucket - rackspace container
      objectID - Object identifier
      inputFile - manifest file to be uploaded
      Returns:
      information about upload
      Throws:
      org.cdlib.mrt.utility.TException
    • awsDelete

      public CloudResponse awsDelete(CloudResponse response) throws org.cdlib.mrt.utility.TException
      Throws:
      org.cdlib.mrt.utility.TException
    • deleteObject

      public CloudResponse deleteObject(String bucket, String key) throws org.cdlib.mrt.utility.TException
      Description copied from interface: CloudStoreInf
      Delete object
      Specified by:
      deleteObject in interface CloudStoreInf
      Parameters:
      bucket - s3 bucket - rackspace container
      key - cloud object key
      Returns:
      Throws:
      org.cdlib.mrt.utility.TException
    • deleteObject

      public CloudResponse deleteObject(String bucketName, org.cdlib.mrt.core.Identifier objectID, int versionID, String fileID) throws org.cdlib.mrt.utility.TException
      Description copied from interface: CloudStoreInf
      Delete cloud object
      Specified by:
      deleteObject in interface CloudStoreInf
      Parameters:
      bucketName - s3 bucket - rackspace container
      objectID - Object identifier
      versionID - Version identifier
      fileID - File/component identifier that may contain a directory element
      Returns:
      Throws:
      org.cdlib.mrt.utility.TException
    • deleteManifest

      public CloudResponse deleteManifest(String bucketName, org.cdlib.mrt.core.Identifier objectID) throws org.cdlib.mrt.utility.TException
      Description copied from interface: CloudStoreInf
      Delete cloud manifest
      Specified by:
      deleteManifest in interface CloudStoreInf
      Parameters:
      bucketName - s3 bucket - rackspace container
      objectID - Object identifier
      Returns:
      Throws:
      org.cdlib.mrt.utility.TException
    • setResponse

      public void setResponse(com.amazonaws.services.s3.model.ObjectMetadata metadata, CloudResponse response) throws org.cdlib.mrt.utility.TException
      Throws:
      org.cdlib.mrt.utility.TException
    • awsGet

      public void awsGet(String container, String key, File outFile, CloudResponse response) throws org.cdlib.mrt.utility.TException
      Throws:
      org.cdlib.mrt.utility.TException
    • awsRestore

      public void awsRestore(String container, String key, File outFile, CloudResponse response) throws org.cdlib.mrt.utility.TException
      Throws:
      org.cdlib.mrt.utility.TException
    • awsRestore

      public boolean awsRestore(String container, String key, CloudResponse response) throws org.cdlib.mrt.utility.TException
      Throws:
      org.cdlib.mrt.utility.TException
    • awsConvertStorageClass

      public boolean awsConvertStorageClass(String bucket, String key, com.amazonaws.services.s3.model.StorageClass targetStorageClass, CloudResponse response) throws org.cdlib.mrt.utility.TException
      Convert from one AWS storage class to another
      Parameters:
      bucket - AWS bucket
      key - AWS key
      targetStorageClass - resulting storage class for conversion
      response - Broad response information
      Returns:
      true=complete
      Throws:
      org.cdlib.mrt.utility.TException
    • retryAWSConvertStorageClass

      public void retryAWSConvertStorageClass(String bucket, String key, com.amazonaws.services.s3.model.StorageClass targetStorageClass) throws Exception
      Throws:
      Exception
    • convertStorageClass

      public CloudResponse convertStorageClass(String bucket, String key, com.amazonaws.services.s3.model.StorageClass targetStorageClass) throws org.cdlib.mrt.utility.TException
      Throws:
      org.cdlib.mrt.utility.TException
    • restoreObject

      public void restoreObject(String container, String key, File outFile, CloudResponse response) throws org.cdlib.mrt.utility.TException
      Description copied from interface: CloudStoreInf
      Restore nearline content
      Specified by:
      restoreObject in interface CloudStoreInf
      key - cloud key
      outFile - target file of response
      Throws:
      org.cdlib.mrt.utility.TException
    • getObject

      public InputStream getObject(String name, org.cdlib.mrt.core.Identifier objectID, int versionID, String fileID, CloudResponse response) throws org.cdlib.mrt.utility.TException
      Description copied from interface: CloudStoreInf
      Retrieve cloud Object
      Specified by:
      getObject in interface CloudStoreInf
      Parameters:
      name - s3 bucket - rackspace container
      objectID - Object identifier
      versionID - Version identifier
      fileID - File/component identifier that may contain a directory element
      response - input stream of retrieved object
      Returns:
      Throws:
      org.cdlib.mrt.utility.TException
    • getObject

      public InputStream getObject(String bucketName, String key, CloudResponse response) throws org.cdlib.mrt.utility.TException
      Description copied from interface: CloudStoreInf
      Retrieve cloud object
      Specified by:
      getObject in interface CloudStoreInf
      Specified by:
      getObject in class CloudStoreAbs
      Parameters:
      bucketName - s3 bucket - rackspace container
      key - cloud object key for object to be returned
      Returns:
      input stream of retrieved object
      Throws:
      org.cdlib.mrt.utility.TException
    • getObjectStreaming

      public InputStream getObjectStreaming(String bucketName, String key, CloudResponse response) throws org.cdlib.mrt.utility.TException
      Description copied from interface: CloudStoreInf
      Retrieve cloud object in streaming mode
      Specified by:
      getObjectStreaming in interface CloudStoreInf
      Parameters:
      bucketName - s3 bucket - rackspace container
      key - cloud object key for object to be returned
      Returns:
      input stream of retrieved object
      Throws:
      org.cdlib.mrt.utility.TException
    • getObject

      public void getObject(String container, String key, File outFile, CloudResponse response) throws org.cdlib.mrt.utility.TException
      Description copied from interface: CloudStoreInf
      Retrieve content into a file
      Specified by:
      getObject in interface CloudStoreInf
      Parameters:
      container - s3 bucket - rackspace container
      key - cloud object key for object to be returned
      outFile - file to receive content
      Throws:
      org.cdlib.mrt.utility.TException
    • getObjectMeta

      public Properties getObjectMeta(String bucketName, String key) throws org.cdlib.mrt.utility.TException
      Description copied from interface: CloudStoreInf
      Retrieve metadata for file
      Specified by:
      getObjectMeta in interface CloudStoreInf
      Parameters:
      bucketName - s3 bucket - rackspace container
      key - cloud object key for object to be returned
      Returns:
      exists: named properties; does not exist: null
      Throws:
      org.cdlib.mrt.utility.TException
    • getRetryObjectMeta

      public com.amazonaws.services.s3.model.ObjectMetadata getRetryObjectMeta(String bucketName, String key, int retryCnt) throws Exception
      Throws:
      Exception
    • getMaxErrRetry

      public Integer getMaxErrRetry()
    • dumpObjectMetadata

      public void dumpObjectMetadata(com.amazonaws.services.s3.model.ObjectMetadata objectMetadata) throws Exception
      Throws:
      Exception
    • dumpMeta

      public Properties dumpMeta(String bucketName, String key) throws org.cdlib.mrt.utility.TException
      Throws:
      org.cdlib.mrt.utility.TException
    • getStorageClass

      public com.amazonaws.services.s3.model.StorageClass getStorageClass(String bucketName, String key) throws org.cdlib.mrt.utility.TException
      Throws:
      org.cdlib.mrt.utility.TException
    • getCloudResponse

      public CloudResponse getCloudResponse(String bucketName, String key) throws org.cdlib.mrt.utility.TException
      Throws:
      org.cdlib.mrt.utility.TException
    • addProp

      protected static void addProp(Properties prop, String key, String value)
    • getManifest

      public InputStream getManifest(String bucket, org.cdlib.mrt.core.Identifier objectID) throws org.cdlib.mrt.utility.TException
      Description copied from interface: CloudStoreInf
      Retrieve cloud manifest
      Specified by:
      getManifest in interface CloudStoreInf
      Parameters:
      bucket - s3 bucket - rackspace container
      objectID - Object identifier
      Returns:
      input stream of retrieved manifest
      Throws:
      org.cdlib.mrt.utility.TException
    • getManifest

      public InputStream getManifest(String bucket, org.cdlib.mrt.core.Identifier objectID, CloudResponse response) throws org.cdlib.mrt.utility.TException
      Description copied from interface: CloudStoreInf
      Retrieve cloud manifest
      Specified by:
      getManifest in interface CloudStoreInf
      Parameters:
      bucket - s3 bucket - rackspace container
      objectID - Object identifier
      Returns:
      input stream of retrieved object
      Throws:
      org.cdlib.mrt.utility.TException
    • awsListPrefix

      protected void awsListPrefix(String bucket, String listPrefix, int maxEntries, CloudResponse response) throws org.cdlib.mrt.utility.TException
      List entries
      Parameters:
      bucket - - non AWS bucket name
      listPrefix - what to truncate on as if aws prefix not there
      maxEntries - maximum entries to return
      response - filled CloudResponse
      Throws:
      org.cdlib.mrt.utility.TException
    • awsListAfter

      public void awsListAfter(String bucketName, String startAfter, int maxKeys, CloudResponse response) throws org.cdlib.mrt.utility.TException
      Throws:
      org.cdlib.mrt.utility.TException
    • getObjectList

      public CloudResponse getObjectList(String bucketName, org.cdlib.mrt.core.Identifier objectID, Integer versionID, String fileID) throws org.cdlib.mrt.utility.TException
      Throws:
      org.cdlib.mrt.utility.TException
    • getObjectList

      public CloudResponse getObjectList(String bucketName, String key) throws org.cdlib.mrt.utility.TException
      Description copied from interface: CloudStoreInf
      Return content that matches for the length of the key
      Specified by:
      getObjectList in interface CloudStoreInf
      Specified by:
      getObjectList in class CloudStoreAbs
      Parameters:
      bucketName - s3 bucket - rackspace container
      key - prefix of content to be returned
      Returns:
      CloudResponse with list of matching entries
      Throws:
      org.cdlib.mrt.utility.TException
    • getObjectList

      public CloudResponse getObjectList(String bucketName, String key, int limit) throws org.cdlib.mrt.utility.TException
      Description copied from interface: CloudStoreInf
      Return a set number of entry metadata based on start key
      Specified by:
      getObjectList in interface CloudStoreInf
      Parameters:
      bucketName - s3 bucket - rackspace container
      key - content key of the first output
      limit - number of returned entries
      Returns:
      CloudResponse with list of sequential entries
      Throws:
      org.cdlib.mrt.utility.TException
    • getObjectListAfter

      public CloudResponse getObjectListAfter(String bucketName, String afterKey, int limit) throws org.cdlib.mrt.utility.TException
      Description copied from interface: CloudStoreInf
      Return list of S3 keys after a start key
      Specified by:
      getObjectListAfter in interface CloudStoreInf
      Parameters:
      bucketName - s3 bucket
      afterKey - start return list after this value
      limit - number of keys to return
      Returns:
      cloud response with keys
      Throws:
      org.cdlib.mrt.utility.TException
    • getObjectList

      public CloudResponse getObjectList(String bucketName) throws org.cdlib.mrt.utility.TException
      Description copied from interface: CloudStoreInf
      Return info on bucket/container
      Specified by:
      getObjectList in interface CloudStoreInf
      Parameters:
      bucketName - s3 bucket - rackspace container
      Returns:
      Throws:
      org.cdlib.mrt.utility.TException
    • getState

      public StateHandler.RetState getState(String bucketName) throws org.cdlib.mrt.utility.TException
      Description copied from interface: CloudStoreInf
      Return state of cloud store manager
      Specified by:
      getState in interface CloudStoreInf
      Parameters:
      bucketName - bucket container
      Returns:
      Throws:
      org.cdlib.mrt.utility.TException
    • getStorageClass

      public com.amazonaws.services.s3.model.StorageClass getStorageClass()
    • setStorageClass

      public void setStorageClass(com.amazonaws.services.s3.model.StorageClass storageClass)
    • setStorageClass

      public void setStorageClass(String storageClassS)
    • isAlphaNumericKey

      public boolean isAlphaNumericKey()
      Description copied from interface: CloudStoreInf
      Because of earlier SDSC bug only alpha-numerics could be used in a key. true=alpha-numeric key, false=ASCII (current default)
      Specified by:
      isAlphaNumericKey in interface CloudStoreInf
      Returns:
    • setSummary

      public org.cdlib.mrt.cloud.CloudList.CloudEntry setSummary(com.amazonaws.services.s3.model.S3ObjectSummary summary)
    • awsHandleException

      public void awsHandleException(CloudResponse response, Exception exception) throws org.cdlib.mrt.utility.TException
      Throws:
      org.cdlib.mrt.utility.TException
    • isAlive

      public Boolean isAlive(String bucketName)
      Description copied from interface: CloudStoreInf
      Determine if host:port is available
      Specified by:
      isAlive in interface CloudStoreInf
      Returns:
      true=alive; false=not alive; null=test not performed
    • displayResourceUrl

      public void displayResourceUrl(String bucketName, String key) throws org.cdlib.mrt.utility.TException
      Throws:
      org.cdlib.mrt.utility.TException
    • getPreSigned

      public CloudResponse getPreSigned(long expirationMinutes, String bucketName, String key, String contentType, String contentDisp) throws org.cdlib.mrt.utility.TException
      Description copied from interface: CloudStoreInf
      Returns an S3 presigned URL
      Specified by:
      getPreSigned in interface CloudStoreInf
      Parameters:
      expirationMinutes - minutes for the signed URL to work
      bucketName - cloud container
      key - cloud key
      contentType - optional ContentType for this file
      contentDisp - optional ContentDisposition for this file
      Returns:
      CloudResponse where response.getReturnURL returns the presigned URL
      Throws:
      org.cdlib.mrt.utility.TException
    • getRangeStream

      public InputStream getRangeStream(String bucketName, String key, long start, long stop, CloudResponse response) throws org.cdlib.mrt.utility.TException
      Description copied from interface: CloudStoreInf
      Return part of a cloud object as stream
      Specified by:
      getRangeStream in interface CloudStoreInf
      Parameters:
      bucketName - cloud container
      key - cloud key
      start - byte location to start from zero
      stop - byte location to stop from zero
      response - Res[pmse tp reqiest
      Returns:
      input stream for segment
      Throws:
      org.cdlib.mrt.utility.TException
    • getType

      public CloudStoreInf.CloudAPI getType()
      Description copied from interface: CloudStoreInf
      Return the API type
      Specified by:
      getType in interface CloudStoreInf
      Returns:
      AWS-S3, SDSC-Swift, Cloudhost, Pairtree
    • getS3Client

      public com.amazonaws.services.s3.AmazonS3Client getS3Client()
    • getLogger

      public org.cdlib.mrt.utility.LoggerInf getLogger()
    • setS3Client

      public void setS3Client(com.amazonaws.services.s3.AmazonS3Client s3Client)
    • showTransferProgress

      public void showTransferProgress(com.amazonaws.services.s3.transfer.Transfer xfer, String key, long inSleepSec, long fileLen)
    • getS3Type

      public AWSS3Cloud.S3Type getS3Type()
    • setS3Type

      public void setS3Type(AWSS3Cloud.S3Type s3Type)