CODEWITHSUNDEEP
arraysgommap
jochen
jochen

Reputation: 3940

Go: how to convert unsafe.Pointer into pointer to array with unknown length?

I am trying to write a Go program which uses mmap to map a very large file containing float32 values into memory. Here is my attempt (inspired by a previous answer, error handling omitted for brevity):

package main

import (
    "fmt"
    "os"
    "syscall"
    "unsafe"
)

func main() {
    fileName := "test.dat"
    info, _ := os.Stat(fileName)
    fileSize := info.Size()
    n := int(fileSize / 4)

    mapFile, _ := os.Open(fileName)
    defer mapFile.Close()
    mmap, _ := syscall.Mmap(int(mapFile.Fd()), 0, int(fileSize),
        syscall.PROT_READ, syscall.MAP_SHARED)
    defer syscall.Munmap(mmap)
    mapArray := (*[n]float32)(unsafe.Pointer(&mmap[0]))

    for i := 0; i < n; i++ {
        fmt.Println(mapArray[i])
    }
}

This fails with the following error message:

./main.go:21: non-constant array bound n

Since n is determined by the length of the file (not known at compile time), I cannot replace n with a constant value in the cast. How do I convert mmap into an array (or slice) of float32 values?

Upvotes: 3

Views: 1422

Answers (2)

YenForYang
YenForYang

Reputation: 3304

Unfortunately you can't get a pointer to an array in your case. This is because n is not a constant value (i.e. it's determined at runtime with fileSize/4). (Note If fileSize were constant, you could get an array.)

There are safe and unsafe alternatives though.

The safe, or some might call the "right" way -- this requires a copy, but you have control over the endianness. Here's an example:

import (
    "encoding/binary"
    "bytes"
    "unsafe" // optional
)

const SIZE_FLOAT32 = unsafe.Sizeof(float32(0)) // or 4

bufRdr := bytes.NewReader(mmap)
mapSlice := make([]float32, len(mmap)/SIZE_FLOAT32) // = fileSize/4
err := binary.Read(bufRdr, binary.LittleEndian, mapSlice) // could pass &mapSlice instead of mapSlice: same result.
// mapSlice now can be used like the mapArray you wanted.

There are a couple ways to do this unsafely, but with Go 1.17 it's pretty simple.

mapSlice := unsafe.Slice((*float32)(unsafe.Pointer(&mmap[0])), len(mmap)/SIZE_FLOAT32)

You could also use reflect.SliceHeader. There are lots of nuances to be careful of here to prevent garbage collector issues:

var mapSlice []float32 // mapSlice := []float32{} also works (important thing is that len and cap are 0)

// newSh and oldSh are here for readability (i.e. inlining these variables is ok, but makes things less readable IMO)
newSh := (*reflect.SliceHeader)(unsafe.Pointer(&mapSlice))
oldSh := (*reflect.SliceHeader)(unsafe.Pointer(&mmap))

// Note: order of assigning Data, Cap, Len is important (due to GC)
newSh.Data = oldSh.Data
newSh.Cap = oldSh.Cap/SIZE_FLOAT32
newSh.Len = oldSh.Len/SIZE_FLOAT32

runtime.KeepAlive(mmap) // ensure `mmap` is not freed up until this point.

The final unsafe way I can think of is given in @JimB's answer -- cast an mmap's Data to an unsafe.Pointer, then cast that to an arbitrarily large pointer to array, and then finally slice that array specifying to desired size and capacity.

Upvotes: 2

Mr_Pink
Mr_Pink

Reputation: 109443

You first convert to an array of a type with a static length that can fit your data, then slice that array to the correct length and capacity.

mapSlice := (*[1 << 30]float32)(unsafe.Pointer(&mmap[0]))[:n:n]

Upvotes: 3

Related Questions