Package org.cdlib.mrt.s3.tools

Class ExtractUnique

java.lang.Object

org.cdlib.mrt.s3.tools.ExtractUnique

public class ExtractUnique
extends Object

This routine generates a list of manifest URLS that can be used as a feed to inv zookeeper loader

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static class	ExtractUnique.ContainerInfo

Field Summary

Fields

Modifier and Type	Field	Description
protected BufferedReader	br
protected CloudStoreInf	cloud
protected String	container
protected File	containerDir
protected File	containerList
protected ArrayList<ExtractUnique.ContainerInfo>	containers
protected static final boolean	DEBUG
protected long	fileCnt
protected org.cdlib.mrt.utility.LoggerInf	logger
protected long	maxout
protected static final String	MESSAGE
protected static final String	NAME
protected static final String	NL
protected long	numContainers
protected String	startKey
static final int	UNIQUE_SIZE

Constructor Summary

Constructors

Constructor	Description
ExtractUnique(CloudStoreInf cloud, File containerDir, Long maxout, org.cdlib.mrt.utility.LoggerInf logger)	Properties Constructor IdList

Method Summary

Modifier and Type	Method	Description
String	dump(String header)
static void	main(String[] args)
void	process()
void	processContainer(String container)
void	processDistributedContainer(String distribContainer)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

NAME

protected static final String NAME

See Also:

• Constant Field Values

MESSAGE

protected static final String MESSAGE

See Also:

• Constant Field Values

DEBUG

protected static final boolean DEBUG

See Also:

• Constant Field Values

UNIQUE_SIZE

public static final int UNIQUE_SIZE

See Also:

• Constant Field Values

NL

protected static final String NL

cloud

protected CloudStoreInf cloud

container

protected String container

startKey

protected String startKey

containerDir

protected File containerDir

br

protected BufferedReader br

containerList

protected File containerList

numContainers

protected long numContainers

fileCnt

protected long fileCnt

logger

protected org.cdlib.mrt.utility.LoggerInf logger

containers

protected ArrayList<ExtractUnique.ContainerInfo> containers

maxout

protected long maxout

Constructor Details

ExtractUnique

public ExtractUnique(CloudStoreInf cloud,
 File containerDir,
 Long maxout,
 org.cdlib.mrt.utility.LoggerInf logger)
              throws org.cdlib.mrt.utility.TException

Properties Constructor IdList

Parameters:

s3ServiceProp - S3 service properties

runProp - runtime properties

Throws:

org.cdlib.mrt.utility.TException

Method Details

main

public static void main(String[] args)
                 throws IOException

Throws:

IOException

process

public void process()
             throws org.cdlib.mrt.utility.TException

Throws:

org.cdlib.mrt.utility.TException

processContainer

public void processContainer(String container)
                      throws org.cdlib.mrt.utility.TException

Throws:

org.cdlib.mrt.utility.TException

processDistributedContainer

public void processDistributedContainer(String distribContainer)
                                 throws org.cdlib.mrt.utility.TException

Throws:

org.cdlib.mrt.utility.TException

dump

public String dump(String header)