Sampling in pandas

Question

If I want to randomly sample a pandas dataframe I can use pandas.DataFrame.sample.

Suppose I randomly sample 80% of the rows. How do I automatically get the other 20% of the rows that were not picked?

Answer 1

>>> import pandas as pd, numpy as np
>>> df = pd.DataFrame({'a': [1,2,3,4,5,6,7,8,9,10], 'b': [11,12,13,14,15,16,17,18,19,20]})
>>> df
    a   b
0   1  11
1   2  12
2   3  13
3   4  14
4   5  15
5   6  16
6   7  17
7   8  18
8   9  19
9  10  20

# randomly sample 5 rows
>>> sample = df.sample(5)
>>> sample
   a   b
7  8  18
2  3  13
4  5  15
0  1  11
3  4  14

# list comprehension to get indices not in sample's indices
>>> idxs_not_in_sample = [idx for idx in df.index if idx not in sample.index]
>>> idxs_not_in_sample
[1, 5, 6, 8, 9]

# locate the rows at the indices in the original dataframe that aren't in the sample
>>> not_sample = df.loc[idxs_not_in_sample]
>>> not_sample
    a   b
1   2  12
5   6  16
6   7  17
8   9  19
9  10  20

Answer 2

As Lagerbaer explains, one can add a column with a unique index to the dataframe, or randomly shuffle the entire dataframe. For the latter,

df.reindex(np.random.permutation(df.index))

works. (np means numpy)