public class HadoopPlanner extends FlowPlanner<HadoopFlow,org.apache.hadoop.mapred.JobConf>
copyJobConf(java.util.Map, org.apache.hadoop.mapred.JobConf)
on a map properties object before constructing a new HadoopFlowConnector
.
A better practice would be to set Hadoop properties directly on the map properties object handed to the FlowConnector.
All values in the map will be passed to a new default JobConf instance to be used as defaults for all resulting
Flow instances.
For example, properties.set("mapred.child.java.opts","-Xmx512m");
would convince Hadoop
to spawn all child jvms with a heap of 512MB.assertionLevel, checkpointRootPath, debugLevel, properties
Constructor and Description |
---|
HadoopPlanner() |
Modifier and Type | Method and Description |
---|---|
HadoopFlow |
buildFlow(FlowDef flowDef)
Method buildFlow renders the actual Flow instance.
|
static void |
copyJobConf(java.util.Map<java.lang.Object,java.lang.Object> properties,
org.apache.hadoop.mapred.JobConf jobConf)
Method copyJobConf adds the given JobConf values to the given properties object.
|
static void |
copyProperties(org.apache.hadoop.mapred.JobConf jobConf,
java.util.Map<java.lang.Object,java.lang.Object> properties)
Method copyProperties adds the given Map values to the given JobConf object.
|
protected HadoopFlow |
createFlow(FlowDef flowDef) |
static org.apache.hadoop.mapred.JobConf |
createJobConf(java.util.Map<java.lang.Object,java.lang.Object> properties)
Method createJobConf returns a new JobConf instance using the values in the given properties argument.
|
static boolean |
getCollapseAdjacentTaps(java.util.Map<java.lang.Object,java.lang.Object> properties) |
org.apache.hadoop.mapred.JobConf |
getConfig() |
static boolean |
getNormalizeHeterogeneousSources(java.util.Map<java.lang.Object,java.lang.Object> properties)
Deprecated.
|
PlatformInfo |
getPlatformInfo() |
void |
initialize(FlowConnector flowConnector,
java.util.Map<java.lang.Object,java.lang.Object> properties) |
protected Tap |
makeTempTap(java.lang.String prefix,
java.lang.String name) |
static void |
setCollapseAdjacentTaps(java.util.Map<java.lang.Object,java.lang.Object> properties,
boolean collapseAdjacent)
Method setCollapseAdjacentTaps enables/disables an optimization that will identify if a sink tap and an intermediate tap
are equivalent field wise, and discard the intermediate tap for the sink tap to minimize the number of MR jobs.
|
static void |
setNormalizeHeterogeneousSources(java.util.Map<java.lang.Object,java.lang.Object> properties,
boolean doNormalize)
Deprecated.
|
createElementGraph, failOnGroupEverySplit, failOnLoneGroupAssertion, failOnMissingGroup, failOnMisusedBuffer, getProperties, handleExceptionDuringPlanning, handleJobPartitioning, handleJoins, handleNonSafeOperations, insertTempTapAfter, makeTempTap, resolveAssemblyPlanners, resolveTails, verifyAllTaps, verifyAssembly, verifyCheckpoints, verifyPipeAssemblyEndPoints, verifySourceNotSinks, verifyTaps, verifyTraps
public static void copyJobConf(java.util.Map<java.lang.Object,java.lang.Object> properties, org.apache.hadoop.mapred.JobConf jobConf)
properties
- of type MapjobConf
- of type JobConfpublic static org.apache.hadoop.mapred.JobConf createJobConf(java.util.Map<java.lang.Object,java.lang.Object> properties)
properties
- of type Mappublic static void copyProperties(org.apache.hadoop.mapred.JobConf jobConf, java.util.Map<java.lang.Object,java.lang.Object> properties)
jobConf
- of type JobConfproperties
- of type Map@Deprecated public static void setNormalizeHeterogeneousSources(java.util.Map<java.lang.Object,java.lang.Object> properties, boolean doNormalize)
properties
- of type MapdoNormalize
- of type boolean@Deprecated public static boolean getNormalizeHeterogeneousSources(java.util.Map<java.lang.Object,java.lang.Object> properties)
properties
- of type Mappublic static void setCollapseAdjacentTaps(java.util.Map<java.lang.Object,java.lang.Object> properties, boolean collapseAdjacent)
properties
- collapseAdjacent
- public static boolean getCollapseAdjacentTaps(java.util.Map<java.lang.Object,java.lang.Object> properties)
public org.apache.hadoop.mapred.JobConf getConfig()
getConfig
in class FlowPlanner<HadoopFlow,org.apache.hadoop.mapred.JobConf>
public PlatformInfo getPlatformInfo()
getPlatformInfo
in class FlowPlanner<HadoopFlow,org.apache.hadoop.mapred.JobConf>
public void initialize(FlowConnector flowConnector, java.util.Map<java.lang.Object,java.lang.Object> properties)
initialize
in class FlowPlanner<HadoopFlow,org.apache.hadoop.mapred.JobConf>
protected HadoopFlow createFlow(FlowDef flowDef)
createFlow
in class FlowPlanner<HadoopFlow,org.apache.hadoop.mapred.JobConf>
public HadoopFlow buildFlow(FlowDef flowDef)
FlowPlanner
buildFlow
in class FlowPlanner<HadoopFlow,org.apache.hadoop.mapred.JobConf>
protected Tap makeTempTap(java.lang.String prefix, java.lang.String name)
makeTempTap
in class FlowPlanner<HadoopFlow,org.apache.hadoop.mapred.JobConf>