pyspark.sql.DataFrame.sampleBy¶
-
DataFrame.
sampleBy
(col, fractions, seed=None)[source]¶ Returns a stratified sample without replacement based on the fraction given on each stratum.
- Parameters
col – column that defines strata
fractions – sampling fraction for each stratum. If a stratum is not specified, we treat its fraction as zero.
seed – random seed
- Returns
a new
DataFrame
that represents the stratified sample
>>> from pyspark.sql.functions import col >>> dataset = sqlContext.range(0, 100).select((col("id") % 3).alias("key")) >>> sampled = dataset.sampleBy("key", fractions={0: 0.1, 1: 0.2}, seed=0) >>> sampled.groupBy("key").count().orderBy("key").show() +---+-----+ |key|count| +---+-----+ | 0| 3| | 1| 6| +---+-----+ >>> dataset.sampleBy(col("key"), fractions={2: 1.0}, seed=0).count() 33
Changed in version 3.0: Added sampling by a column of
Column
New in version 1.5.