CODEWITHSUNDEEP
pythoncnumerical-methods
cmmnn
cmmnn

Reputation: 325

Space efficient file type to store double precision floats

I am currently running Simulations written in C later analyzing the results using Python scripts. ATM the C Programm is writing the results (lots of double values) in a text file which is slowly but surely eating a lot of disc space.

Is there a file format which is more space efficient to store lots of numeric values? At best but not necessarily it should fulfill the following requirements

  1. Values can be appended continuously such that not all values have to be in memory at once.

  2. The file is more or less easily readable using Python.

I feel like this should be a really common question, but looking for an answer I only found descriptions of various data types within C.

Upvotes: 2

Views: 246

Answers (1)

VladP
VladP

Reputation: 537

Binary file, but please, be careful with the format of data that you are saving. If possible, reduce the width of each variable that you are using. For example, do you need to save decimal or float, or you can have just 16 or 32 bit integer? Further, yes, you may apply some of the compression scheme to compress the data before saving, and decompress it after reading, but that requires much more work, and it is probably an overkill for what you are doing.

Upvotes: 2

Related Questions