public class GlobHfs extends MultiSourceTap<Hfs,org.apache.hadoop.mapred.JobConf,org.apache.hadoop.mapred.RecordReader>
MultiSourceTap
that accepts Hadoop style 'file globing' expressions so
multiple files that match the given pattern may be used as the input sources for a given Flow
.
See FileSystem.globStatus(org.apache.hadoop.fs.Path)
for details on the globing syntax. But in short
it is similar to standard regular expressions except alternation is done via {foo,bar} instead of (foo|bar).
Note that a Flow
sourcing from GlobHfs is not currently compatible with the Cascade
scheduler. GlobHfs expects the files and paths to exist so the wildcards can be resolved into concrete values so
that the scheduler can order the Flows properly.
Note that globing can match files or directories. It may consume less resources to match directories and let
Hadoop include all sub-files immediately contained in the directory instead of enumerating every individual file.
Ending the glob path with a /
should match only directories.Hfs
,
MultiSourceTap
,
FileSystem
,
Serialized Formtaps
Constructor and Description |
---|
GlobHfs(Scheme<org.apache.hadoop.mapred.JobConf,org.apache.hadoop.mapred.RecordReader,?,?,?> scheme,
java.lang.String pathPattern)
Constructor GlobHfs creates a new GlobHfs instance.
|
GlobHfs(Scheme<org.apache.hadoop.mapred.JobConf,org.apache.hadoop.mapred.RecordReader,?,?,?> scheme,
java.lang.String pathPattern,
org.apache.hadoop.fs.PathFilter pathFilter)
Constructor GlobHfs creates a new GlobHfs instance.
|
Modifier and Type | Method and Description |
---|---|
boolean |
equals(java.lang.Object object) |
java.lang.String |
getIdentifier()
Method getIdentifier returns a String representing the resource this Tap instance represents.
|
protected Hfs[] |
getTaps()
Method getTaps returns the taps of this MultiTap object.
|
int |
hashCode() |
void |
sourceConfInit(FlowProcess<org.apache.hadoop.mapred.JobConf> process,
org.apache.hadoop.mapred.JobConf conf)
Method sourceConfInit initializes this instance as a source.
|
java.lang.String |
toString() |
getChildTaps, getModifiedTime, getNumChildTaps, getScheme, isReplace, openForRead, resourceExists
commitResource, createResource, deleteResource, getSinkFields, isSink, openForWrite, rollbackResource, sinkConfInit
createResource, deleteResource, flowConfInit, getConfigDef, getFullIdentifier, getFullIdentifier, getModifiedTime, getSinkMode, getSourceFields, getStepConfigDef, getTrace, hasConfigDef, hasStepConfigDef, id, isEquivalentTo, isKeep, isSource, isTemporary, isUpdate, openForRead, openForWrite, outgoingScopeFor, presentSinkFields, presentSourceFields, resolveIncomingOperationArgumentFields, resolveIncomingOperationPassThroughFields, resourceExists, retrieveSinkFields, retrieveSourceFields, setScheme, taps
@ConstructorProperties(value={"scheme","pathPattern"}) public GlobHfs(Scheme<org.apache.hadoop.mapred.JobConf,org.apache.hadoop.mapred.RecordReader,?,?,?> scheme, java.lang.String pathPattern)
scheme
- of type SchemepathPattern
- of type String@ConstructorProperties(value={"scheme","pathPattern","pathFilter"}) public GlobHfs(Scheme<org.apache.hadoop.mapred.JobConf,org.apache.hadoop.mapred.RecordReader,?,?,?> scheme, java.lang.String pathPattern, org.apache.hadoop.fs.PathFilter pathFilter)
scheme
- of type SchemepathPattern
- of type StringpathFilter
- of type PathFilterpublic java.lang.String getIdentifier()
Tap
getIdentifier
in class MultiSourceTap<Hfs,org.apache.hadoop.mapred.JobConf,org.apache.hadoop.mapred.RecordReader>
protected Hfs[] getTaps()
MultiSourceTap
getTaps
in class MultiSourceTap<Hfs,org.apache.hadoop.mapred.JobConf,org.apache.hadoop.mapred.RecordReader>
public void sourceConfInit(FlowProcess<org.apache.hadoop.mapred.JobConf> process, org.apache.hadoop.mapred.JobConf conf)
Tap
Flow
instance or if it participates in multiple times in a given Flow or across different Flows in
a Cascade
.
In the context of a Flow, it will be called after
FlowListener.onStarting(cascading.flow.Flow)
Note that no resources or services should be modified by this method.sourceConfInit
in class MultiSourceTap<Hfs,org.apache.hadoop.mapred.JobConf,org.apache.hadoop.mapred.RecordReader>
process
- of type FlowProcessconf
- of type Configpublic boolean equals(java.lang.Object object)
equals
in class MultiSourceTap<Hfs,org.apache.hadoop.mapred.JobConf,org.apache.hadoop.mapred.RecordReader>
public int hashCode()
hashCode
in class MultiSourceTap<Hfs,org.apache.hadoop.mapred.JobConf,org.apache.hadoop.mapred.RecordReader>
public java.lang.String toString()
toString
in class MultiSourceTap<Hfs,org.apache.hadoop.mapred.JobConf,org.apache.hadoop.mapred.RecordReader>