StackOverflow Questions for Tag: precision

james_womack

Reputation: 10316

Precise Financial Calculation in JavaScript. What Are the Gotchas?

javascriptdecimalprecisionfinance

Score: 169

Answers: 10

alphasierra

Reputation: 149

Replicating C Fixed-Point Math in Python

pythoncprecisionnumerical-methodsfixed-point

Score: 1

Answers: 2

WeiCODER

Reputation: 29

Different Fused Rotary Positional Embedding(RoPE) results on CPU and GPU by using float16 (half-precision float type)

c++pytorchcudaprecisionhalf-precision-float

Score: 1

Answers: 0

Chaimae Lsi

Reputation: 1

Numerical precision due to Numpy 's use of float64, what to do?

pythonnumpyprecisionnumeric

Score: -2

Answers: 0

Olof R

Reputation: 113

polynomial evaluation producing non-differentiable behavior in Python

pythonmathprecisionnumericpolynomials

Score: 0

Answers: 1

Ashish

Reputation: 8529

strange output in comparison of float with float literal

c++cfloating-pointdouble-precision

Score: 41

Answers: 9

Sibi

Reputation: 48766

Haskell's double behaviour

haskelldoubleprecision

Score: 5

Answers: 1

MothNik

Reputation: 111

How to compute sign(a² - b * c) * sqrt(abs(a² - b * c)) accurately with floating point arithmetics?

floating-pointprecisionnumericfloating-accuracy

Score: 4

Answers: 1

song xs

Reputation: 81

Is floating point math determinstic for all intel/amd cpus?

x86floating-pointx86-64precisionsse

Score: 1

Answers: 2

Hans Brende

Reputation: 8637

If xy ≠ 2ⁿ, does it follow that x y = ((x * y) / y) * y under IEEE 754 semantics?

floating-pointprecisionieee-754

Score: 7

Answers: 1

SRobertJames

Reputation: 9263

With IEEE 754 floating point numbers, is underflow still better than overflow?

floating-pointprecisionnumericnumerical-methodsunderflow

Score: 1

Answers: 3

khteh

Reputation: 3976

c++ long double (128-bit) precision

x86-64precisionc++23long-double128-bit

Score: 1

Answers: 1

Esteban Küber

Reputation: 36852

What class to use for money representation?

pythoncurrencyrounding-errorarbitrary-precisionprecision

Score: 52

Answers: 6

Peter Fick

Reputation: 11

String formatting removes floating point errors?

c#formattingprecision

Score: 1

Answers: 1

Ishisht

Reputation: 95

efficient cumulative log_sum_exp

rprecision

Score: 5

Answers: 2

Bek

Reputation: 33

Why 5.0 / 2 returns 2.5000000000000000 (scale of 16) instead of 2.5 (scale of 1)?

sqlpostgresqlprecisionnumericarbitrary-precision

Score: 0

Answers: 2

user714171

Reputation: 435

float is overflowing in openGL despite being under 32 bits

c++openglfloating-pointglslprecision

Score: 2

Answers: 1

formateu

Reputation: 167

Intel x86_64 assembly compare signed double precision floats

assemblyx86-64intelprecisionsse

Score: 3

Answers: 3

L. Perasolo

Reputation: 23

How many digits can I rely on?

floating-pointprecisionnumerical-computing

Score: 1

Answers: 2

Dmitri Nesteruk

Reputation: 23798

How to choose epsilon value for floating point?

mathfloating-pointprecisionepsilon

Score: 9

Answers: 3

PreviousPage 1Next

StackOverflow Questions for Tag: precision

Precise Financial Calculation in JavaScript. What Are the Gotchas?

Replicating C Fixed-Point Math in Python

Different Fused Rotary Positional Embedding(RoPE) results on CPU and GPU by using float16 (half-precision float type)

Numerical precision due to Numpy &#39;s use of float64, what to do?

polynomial evaluation producing non-differentiable behavior in Python

strange output in comparison of float with float literal

Haskell&#39;s double behaviour

How to compute sign(a&#178; - b * c) * sqrt(abs(a&#178; - b * c)) accurately with floating point arithmetics?

Is floating point math determinstic for all intel/amd cpus?

If x*y ≠ 2ⁿ, does it follow that x * y = ((x * y) / y) * y under IEEE 754 semantics?

With IEEE 754 floating point numbers, is underflow still better than overflow?

c++ long double (128-bit) precision

What class to use for money representation?

String formatting removes floating point errors?

efficient cumulative log_sum_exp

Why 5.0 / 2 returns 2.5000000000000000 (scale of 16) instead of 2.5 (scale of 1)?

float is overflowing in openGL despite being under 32 bits

Intel x86_64 assembly compare signed double precision floats

How many digits can I rely on?

How to choose epsilon value for floating point?

Numerical precision due to Numpy 's use of float64, what to do?

Haskell's double behaviour

How to compute sign(a² - b * c) * sqrt(abs(a² - b * c)) accurately with floating point arithmetics?

If xy ≠ 2ⁿ, does it follow that x y = ((x * y) / y) * y under IEEE 754 semantics?