pyspark.Accumulator

class pyspark.Accumulator(aid, value, accum_param)[source]

A shared variable that can be accumulated, i.e., has a commutative and associative “add” operation. Worker tasks on a Spark cluster can add values to an Accumulator with the += operator, but only the driver program is allowed to access its value, using value. Updates from the workers get propagated automatically to the driver program.

While SparkContext supports accumulators for primitive data types like int and float, users can also define accumulators for custom types by providing a custom AccumulatorParam object. Refer to the doctest of this module for an example.

__init__(aid, value, accum_param)[source]

Create a new Accumulator with a given initial value and AccumulatorParam object

Methods

Attributes