CODEWITHSUNDEEP
pythonpandasfloating-pointprecision
Linus Nox
Linus Nox

Reputation: 77

Rolling Std returns 0 for pandas DataFrame after multiplying with 1e-1

Does anyone know why rolling std returns 0 after it reaches a factor of 1e-8 for a float64 pandas data frame column?

Here is a code example:

import pandas as pd
import numpy as np

N = 250
pd.set_option('display.precision', 200)
df = pd.DataFrame()

df["C"] = [np.random.normal() for i in range(10000)]

print("N(0, 1)")
print(df["C"].mean())
print(df["C"].std())

# 1e0
df["std"] = df["C"].rolling(N).std()
print("Std is not 0.")
print(df["std"])

# 1e-6
df["C"] = df["C"] * 1e-6
df["std"] = df["C"].rolling(N).std()
print("Std is still not 0.")
print(df["std"])

# 1e-7
df["C"] = df["C"] * 1e-1
df["std"] = df["C"].rolling(N).std()
print("Std is sometimes (?) 0 now.")
print(df["std"])

# 1e-8
df["C"] = df["C"] * 1e-1
df["std"] = df["C"].rolling(N).std()
print("Std is always 0 now.")
print(df["std"])

1e-8 should not be the problem, as we are working with float64 here and std without rolling can handle even smaller numbers. The same issue occurs when multiplying with 1e-8 directly.

Upvotes: 0

Views: 209

Answers (1)

Nk03
Nk03

Reputation: 14949

There's one alternative via np.std:

import numpy as np
df = df["C"].rolling(N).apply(np.std)

Upvotes: 1

Related Questions