Package org.cdlib.mrt.s3.service

Class NodeIO

java.lang.Object
org.cdlib.mrt.s3.service.NodeIO
public class NodeIO extends Object
Author:
DLoy 
The NodeIO class handles a paramertized definition for all cloud IO members.
A NodeIO name is used to specifically identify which node list to use.

Each node. line in the list defines a node and cloud definitions for that node

This process is now used on all backend Merritt services.


Example set:

./nodes-prod.properties
node.1=9001|sdsc|distrib.prod.9001.__
node.2=9103|sdsc|prod-9103
node.3=8001|ucla-prod
node.4=5001|aws-std|uc3-s3mrt5001-prd
node.5=6001|aws-near|uc3-s3mrt6001-prd
node.6=7001|aws-std|uc3-s3mrt1001-prd
node.7=7021|ch-riverside-prd|8100

 ch-riverside-prd
serviceType=cloudhost
base=https://uc3-mrtdat1-dev.cdlib.org:30443/cloudhost

./sdsc.properties
serviceType=swift
access_key=our_service:our_group
secret_key=pwd
host=cloud.sdsc.edu

./ucla-prod
serviceType=pairtree
base=/mrt-ucla-prod/dpr2/repository/node8001/store/fileCloud

./aws-std
serviceType=aws
accessMode=on-line
storageClass=Standard

./aws-near
serviceType=aws
accessMode=near-line



Process module:
Repository: https://github.com/CDLUC3/mrt-cloud
Project: s3-src
Class: org.cdlib.mrt.s3.service.NodeIO


SERVICE TYPES:

Swift
node:
node.=||

Properties:
serviceType=swift
access_key=invalid input: '<'Organization key:group key>
secret_key=
host=

Example
node.2=9103|sdsc|prod-9103
...

./sdsc.properties
serviceType=swift
access_key=our_service:our_group
secret_key=pwd
host=cloud.sdsc.edu

-------------------------------------------------------------------
Pairtree
node:
node.=|

Properties:
serviceType=pairtree
base=

Example:
node.2=8100|csh-pair
...

./csh-pair.properties
serviceType=pairtree
base=./fileCloud

-------------------------------------------------------------------
AWS
node:
node.=||


Properties:
serviceType=aws
accessMode=
  "on-line" = directly accessible content
  "near-line" = Glacier
storageClass= optional AWS storage class
  "Standard" 
  "ReducedRedundancy"

Example:
node.4=5001|aws-std|uc3-s3mrt5001-prd
...

./aws-std.properties
serviceType=aws
accessMode=on-line
storageClass=Standard

-------------------------------------------------------------------
Cloudhost 
node:
node.=||

Properties:
serviceType=cloudhost
base=

Example:
node.1=7301|ch-unm-test|8101
...

./ch-unm-test.properties
serviceType=cloudhost
base=http://oneshare-test.unm.edu:8080/cloudhost



Example set:

./nodes-prod.properties
node.1=9001|sdsc|distrib.prod.9001.__
node.2=9103|sdsc|prod-9103
node.3=8001|ucla-prod
node.4=5001|aws-std|uc3-s3mrt5001-prd
node.5=6001|aws-near|uc3-s3mrt6001-prd
node.6=7001|aws-std|uc3-s3mrt1001-prd
node.7=7021|ch-riverside-prd|8100

 ch-riverside-prd
serviceType=cloudhost
base=https://uc3-mrtdat1-dev.cdlib.org:30443/cloudhost

./sdsc.properties
serviceType=swift
access_key=our_service:our_group
secret_key=pwd
host=cloud.sdsc.edu

./ucla-prod
serviceType=pairtree
base=/mrt-ucla-prod/dpr2/repository/node8001/store/fileCloud

./aws-std
serviceType=aws
accessMode=on-line
storageClass=Standard

./aws-near
serviceType=aws
accessMode=near-line

 
 Example loading:
 
            nodeIO = NodeIO.getNodeIO(nodeName, logger);
            NodeIO.AccessNode accessNode = nodeIO.getAccessNode(nodeNumber);
            if (accessNode == null) {
               throw new TException.REQUESTED_ITEM_NOT_FOUND("test fails:"
                  + " - nodeName:" + nodeName
                  + " - nodeNumber:" + nodeNumber
               );
            }
            CloudStoreInf service = accessNode.service;
            String container = accessNode.container;
            StateHandler stateHandler = StateHandler.getStateHandler(service, container, KEY, logger);

  • Field Details

  • Constructor Details

    • NodeIO

      public NodeIO(Integer awsVersion, List<DefNode> defNodes, org.cdlib.mrt.utility.LoggerInf logger) throws org.cdlib.mrt.utility.TException
      DefNode extract of multiple nodes from list
      Parameters:
      logger -
      ssmBase - SSM path
      nodeNumber - list of nodeNumbers to resolve
      Throws:
      org.cdlib.mrt.utility.TException

    • NodeIO

      public NodeIO(String nodeName, org.cdlib.mrt.utility.LoggerInf logger) throws org.cdlib.mrt.utility.TException
      Throws:
      org.cdlib.mrt.utility.TException

    • NodeIO

      public NodeIO(String nodeName, long node, org.cdlib.mrt.utility.LoggerInf logger) throws org.cdlib.mrt.utility.TException
      Throws:
      org.cdlib.mrt.utility.TException

    • NodeIO

      public NodeIO(Properties cloudProp, org.cdlib.mrt.utility.LoggerInf logger) throws org.cdlib.mrt.utility.TException
      Throws:
      org.cdlib.mrt.utility.TException

  • Method Details

    • main

      public static void main(String[] args) throws Exception
      Throws:
      Exception

    • main_jar

      public static void main_jar(String[] args) throws Exception
      Throws:
      Exception

    • main_yaml

      public static void main_yaml(String[] args) throws Exception
      Throws:
      Exception

    • main_yaml1

      public static void main_yaml1(String[] args) throws Exception
      Throws:
      Exception

    • main_yaml2

      public static void main_yaml2(String[] args) throws Exception
      Throws:
      Exception

    • main_jarnew

      public static void main_jarnew(String[] args) throws Exception
      Throws:
      Exception

    • main_jarv1

      public static void main_jarv1(String[] args) throws Exception
      Throws:
      Exception

    • main_jarv2

      public static void main_jarv2(String[] args) throws Exception
      Throws:
      Exception

    • main_default

      public static void main_default(String[] args) throws Exception
      Throws:
      Exception

    • main_backup

      public static void main_backup(String[] args) throws Exception
      Throws:
      Exception

    • testmain

      public static void testmain(String[] args) throws Exception
      Throws:
      Exception

    • getNodeIOAccess

      public static NodeIO.AccessNode getNodeIOAccess(String nodeName, long nodeNumber, org.cdlib.mrt.utility.LoggerInf logger) throws org.cdlib.mrt.utility.TException
      Throws:
      org.cdlib.mrt.utility.TException

    • getNodeIOConfig

      public static NodeIO getNodeIOConfig(String config, org.cdlib.mrt.utility.LoggerInf logger) throws org.cdlib.mrt.utility.TException
      Throws:
      org.cdlib.mrt.utility.TException

    • configCall

      public static void configCall()

    • getCloudNode

      public static NodeIO.AccessNode getCloudNode(String nodeName, long node, org.cdlib.mrt.utility.LoggerInf logger) throws org.cdlib.mrt.utility.TException
      Throws:
      org.cdlib.mrt.utility.TException

    • getNodeIO

      public static NodeIO getNodeIO(String nodeName, org.cdlib.mrt.utility.LoggerInf logger) throws org.cdlib.mrt.utility.TException
      Throws:
      org.cdlib.mrt.utility.TException

    • getNode

      public NodeIO.AccessNode getNode(long node) throws org.cdlib.mrt.utility.TException
      Throws:
      org.cdlib.mrt.utility.TException

    • getNode

      protected long getNode(String line) throws org.cdlib.mrt.utility.TException
      Throws:
      org.cdlib.mrt.utility.TException

    • addMapEntry

      protected void addMapEntry(String line) throws org.cdlib.mrt.utility.TException
      Throws:
      org.cdlib.mrt.utility.TException

    • addMapEntry

      protected NodeIO.AccessNode addMapEntry(Long nodeNumber, String container, String nodeDescription, Properties cloudProp) throws org.cdlib.mrt.utility.TException
      Throws:
      org.cdlib.mrt.utility.TException

    • getAccessNodexxx

      public static NodeIO.AccessNode getAccessNodexxx(Long nodeNumber, String container, String nodeDescription, Properties cloudProp, org.cdlib.mrt.utility.LoggerInf logger) throws org.cdlib.mrt.utility.TException
      Throws:
      org.cdlib.mrt.utility.TException

    • getAccessNode

      public static NodeIO.AccessNode getAccessNode(String awsVersionS, Long nodeNumber, String container, String nodeDescription, Properties cloudProp, org.cdlib.mrt.utility.LoggerInf logger) throws org.cdlib.mrt.utility.TException
      Throws:
      org.cdlib.mrt.utility.TException

    • getAccessNode

      public static NodeIO.AccessNode getAccessNode(Integer awsVersion, Long nodeNumber, String container, String nodeDescription, Properties cloudProp, org.cdlib.mrt.utility.LoggerInf logger) throws org.cdlib.mrt.utility.TException
      Throws:
      org.cdlib.mrt.utility.TException

    • getAccessNode

      public NodeIO.AccessNode getAccessNode(long nodeNumber)

    • getAccessKey

      public NodeIO.AccessKey getAccessKey(String storageURLS) throws org.cdlib.mrt.utility.TException
      Throws:
      org.cdlib.mrt.utility.TException

    • getFile

      public void getFile(String storageURLS, File outfile) throws org.cdlib.mrt.utility.TException
      Throws:
      org.cdlib.mrt.utility.TException

    • getFileException

      protected void getFileException(CloudStoreInf service, String container, String key, File outfile) throws org.cdlib.mrt.utility.TException
      Throws:
      org.cdlib.mrt.utility.TException

    • getInputStream

      public InputStream getInputStream(String storageURLS) throws org.cdlib.mrt.utility.TException
      Throws:
      org.cdlib.mrt.utility.TException

    • getAwsVersion

      public Integer getAwsVersion()

    • setAwsVersion

      public void setAwsVersion(Integer awsVersion)

    • getAwsVersion

      public static Integer getAwsVersion(String awsVersionS)

    • getCollection

      public Collection<NodeIO.AccessNode> getCollection()

    • getMap

      public HashMap<Long,NodeIO.AccessNode> getMap()

    • printNodes

      public void printNodes(String header)

    • printNodes

      public void printNodes(String header, long nodeNumber)

    • setNodeName

      public void setNodeName(String nodeName)

    • getNodeName

      public String getNodeName()

    • getAccessNodesMap

      public HashMap<Long,NodeIO.AccessNode> getAccessNodesMap()

    • getAccessNodesList

      public ArrayList<NodeIO.AccessNode> getAccessNodesList()