Stevven
Reputation: 31
How to oversample a dataframe in Pyspark?
How to oversample a dataframe in pyspark?
df.sample(fractions, seed)
Which only sample a fraction of the df, it can't oversample.
Upvotes: 3
Views: 4559
Answers (1)
Tshilidzi Mudau
Reputation: 7899
You could over-sample by making use of the sample method as follows:
df.sample(withReplacement=True, total_percent_of_upsample, seed)
sample(withReplacement, fraction, seed=None)
The True indicates that you want to sample with replacement.
Upvotes: 1
Related Questions
- Oversampling or SMOTE in Pyspark
- sampling with weight using pyspark
- How to do stratified sampling on two columns in PySpark Dataframe?
- How to sample() exact number of rows, not fraction, of PySpark DataFrame
- How to change the size and distribution of a PySpark Dataframe according to the values of its rows & columns?
- Pyspark: Take n samples with balanced classes
- pyspark: resampling pyspark dataframe on date and time
- Oversampling with Cross Validation in PySpark Pipeline
- What is the best way to oversample a dataframe preserving its statistical properties in Python 3?
- How to undersampling the majority class using pyspark