geotrellis.spark.io.accumulo

BatchAccumuloInputFormat

class BatchAccumuloInputFormat extends InputFormatBase[Key, Value] with LazyLogging

This input format will use Accumulo [TabletLocator] to create InputSplits for each tablet that contains records from specified ranges. This is unlike AccumuloInputFormat which creates a single split per Range. The MultiRangeInputSplits are intended to be read using a BatchScanner. This drastically reduces the number of splits and consequnetly spark tasks that are produced by this InputFormat. Locality is preserved because tablets may only be hosted by a single tablet server at a given time.

Because RecordReader uses BatchScanner a number of modes are not supported: Offline, Isolated and Local Iterators. These modes are backed by specalized scanners that only support scanning through a single range.

We borrow some Accumulo machinery to set and read configurations so classOf AccumuloInputFormat should be used for modifying Configuration, as if AccumuloInputFormat will be used.

This class uses the internal Accumulo API and will likely not work across versions.

WARNING: The locality of the splits rely on reverse resolution of tserver IPs matching those of spark workers.

Linear Supertypes
LazyLogging, Logging, InputFormatBase[Key, Value], InputFormat[Key, Value], AnyRef, Any
Ordering
  1. Alphabetic
  2. By inheritance
Inherited
  1. BatchAccumuloInputFormat
  2. LazyLogging
  3. Logging
  4. InputFormatBase
  5. InputFormat
  6. AnyRef
  7. Any
  1. Hide All
  2. Show all
Learn more about member selection
Visibility
  1. Public
  2. All

Instance Constructors

  1. new BatchAccumuloInputFormat()

Value Members

  1. final def !=(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  2. final def !=(arg0: Any): Boolean

    Definition Classes
    Any
  3. final def ##(): Int

    Definition Classes
    AnyRef → Any
  4. final def ==(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  5. final def ==(arg0: Any): Boolean

    Definition Classes
    Any
  6. final def asInstanceOf[T0]: T0

    Definition Classes
    Any
  7. def clone(): AnyRef

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  8. def createRecordReader(inputSplit: InputSplit, taskAttemptContext: TaskAttemptContext): RecordReader[Key, Value]

    Definition Classes
    BatchAccumuloInputFormat → InputFormat
  9. final def eq(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  10. def equals(arg0: Any): Boolean

    Definition Classes
    AnyRef → Any
  11. def finalize(): Unit

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  12. final def getClass(): Class[_]

    Definition Classes
    AnyRef → Any
  13. def getSplits(context: JobContext): List[InputSplit]

    Definition Classes
    BatchAccumuloInputFormat → InputFormatBase → InputFormat
  14. def hashCode(): Int

    Definition Classes
    AnyRef → Any
  15. final def isInstanceOf[T0]: Boolean

    Definition Classes
    Any
  16. lazy val logger: Logger

    Attributes
    protected
    Definition Classes
    LazyLogging → Logging
  17. final def ne(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  18. final def notify(): Unit

    Definition Classes
    AnyRef
  19. final def notifyAll(): Unit

    Definition Classes
    AnyRef
  20. final def synchronized[T0](arg0: ⇒ T0): T0

    Definition Classes
    AnyRef
  21. def toString(): String

    Definition Classes
    AnyRef → Any
  22. final def wait(): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  23. final def wait(arg0: Long, arg1: Int): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  24. final def wait(arg0: Long): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

Inherited from LazyLogging

Inherited from Logging

Inherited from InputFormatBase[Key, Value]

Inherited from InputFormat[Key, Value]

Inherited from AnyRef

Inherited from Any

Ungrouped