pyspark.streaming.StreamingContext.queueStream

StreamingContext.queueStream(rdds, oneAtATime=True, default=None)[source]

Create an input stream from a queue of RDDs or list. In each batch, it will process either one or all of the RDDs returned by the queue.

Note

Changes to the queue after the stream is created will not be recognized.

Parameters
  • rdds – Queue of RDDs

  • oneAtATime – pick one rdd each time or pick all of them once.

  • default – The default rdd if no more in rdds