CODEWITHSUNDEEP
pythonfileooppickle
Adam
Adam

Reputation: 668

Store pkl / binary in MetaData

I am writing a function that is supposed to store a text representation of a custom class object, cl I have some code that writes to a file and takes the necessary information out of cl.

Now I need to go backwards, read the file and return a new instance of cl. The problem is, the file doesn't keep all of the important parts of cl because for the purpose of this text document parts of it are unnecessary. A .jpg file allows you to store meta data like shutter speed and location. I would like to store the parts of cl that are not supposed to be in the text portion in the meta data of a .txt or .csv file. Is there a way to explicitly write something to the metadata of a text file in Python? Additionally, would it be possible to write the byte-code .pkl representation of the entire object in the metadata?

Upvotes: 0

Views: 439

Answers (1)

Sam Hartman
Sam Hartman

Reputation: 6499

Text files don't have meta data in the same way that a jpg file does. A jpeg file is specifically designed to have ways of including meta data as extra structured information in the image. Text files aren't: every character in the text file is generally displayed to the user. Similarly, every thing in a CSV file is part of one cell in the table represented by the file.

That said, there are some things similar to text file metadata that have existed or exist over the years that might give you some ideas. I don't think any of these is ideal, but I'll give some examples to give you an idea how complex the area of meta data is and what people have done in similar situations.

  • Some filesystems have meta data associated with each file that can be extended. As an example, NTFS has streams; HFS and HFSplus have resource forks or other attributes; Linux has extended attributes on most of its filesystems. You could potentially store your pickle information in those filesystem metadata. There are disadvantages. Some filesystems don't have this meta data. Some tools for copying and manipulating files will not recognize (or intentionally strip) meta data.

  • You could have a .txt file and a .pcl file, where the .txt file contains your text representation and the .pkl file contained the other information.

  • Back in the day, some DOS programs would stop reading a text file at a DOS EOF (decimal character 26). I don't think anything behaves like that, but it's an example that there are file formats that allowed you to end the file and then still have extra data that programs could use.

  • With a format like HTML or an actual spreadsheet instead of CSV, there are ways you could include things in meta data easily.

Upvotes: 1

Related Questions