CODEWITHSUNDEEP
linuxgoiolow-level-io
Adrian Challinor
Adrian Challinor

Reputation: 9

golang write struct as raw data

I am working on a new type of database, using GO. One of the things I would like to do is have a distributed disk so that I can distribute queries over multiple machines (think Pi type architectures). This means building my own structures on raw disk.

My challenge is that I can't find a GO package that will let me write N bytes from a pointer to a structure. All the IO packages limit the access to []byte slices.

That's nice for protection, but if I have to buffer everything through a byte array via some form of encoding it will slow down the access to a specific object.

Anyone got any idea on how to do raw IO? Or am I going to have to handle GOBs as my unit of IO and suffer the penalty for encoding/decoding?

Upvotes: 0

Views: 3110

Answers (3)

Amnon
Amnon

Reputation: 334

...Or am I going to have to handle GOBs as my unit of IO and suffer the penalty for encoding/decoding?

Just use GOBs. Premature optimization is the root of all evil.

Upvotes: 0

Marc
Marc

Reputation: 21095

Big warning first: don't do it: it is neither safe nor portable

For a given struct, you can reflect over it to figure out the in-memory size of the actual struct, then unsafely cast it to a []byte using unsafe.

eg: (*[in-mem size]byte)(unsafe.Pointer(&mystruct))

This will give you something C-ish with absolutely no safety guarantees or portability.

I'll quote the Go spec:

A package using unsafe must be vetted manually for type safety and may not be portable.

You can find a lot more details in this Go and Memory layout post, including all the steps you need to unsafely treat structs as just bytes.

Overall, it's fascinating to examine how Go functions on a low level, but this is absolutely the wrong thing to do in your case. Any real data infrastructure will need storage logic way more complicated than just dumping in-memory structs to disk anyway.

Upvotes: 3

chowey
chowey

Reputation: 9836

In general, you cannot do raw IO of a Go struct (i.e. memdump). This is because many things in Go contain pointers, and the actual data is not contiguous in memory.

For example, a struct like this:

type Person struct {
    Name string
}

contains a string, which in turn contains a pointer to the bytes of the string. A raw memdump would only dump the pointer.

The solution is serialization. This is never free, although some implementations do a pretty good job.

The closest to what you are describing is something like go-memdump, but I wouldn't recommend it for production.

Otherwise, I recommend looking at a performant serialization technique. (Go's gob encoding is not the best.)

Upvotes: 2

Related Questions