CODEWITHSUNDEEP
gomemorystructmemory-leaksslice
ptimson
ptimson

Reputation: 5803

Memory usage of [][]string vs struct containing []string

When appending to a [][]string profiling shows the app uses around 145MiB of memory.

defer profile.Start(profile.MemProfile).Stop()

f, _ := os.Open("test.csv") // 100 MiB File
r := csv.NewReader(f)

var records [][]string
for {
    values, err := r.Read()
    if err == io.EOF {
        break
    }
    records = append(records, values)
}

When storing the slice in a struct and appending that the app uses around 260MiB of memory.

defer profile.Start(profile.MemProfile).Stop()

type record struct {
    values []string
}

f, _ := os.Open("test.csv") // 100 MiB File
r := csv.NewReader(f)

var records []record
for {
    values, err := r.Read()
    if err == io.EOF {
        break
    }
    r := record{values: values}
    records = append(records, r)
}

It feels like it's using double the memory in the second example. Can someone explain why the second example uses more memory?

Upvotes: 13

Views: 1075

Answers (1)

rk1234 afbce
rk1234 afbce

Reputation: 43

For those of you who are using between go 1.12 and go 1.15, debug.FreeOSMemory() would not have returned the free memory back to OS, so htop/top will show wrong number or if you rely on RSS to monitor your app, it would be wrong.

This is due to the fact that runtime in Golang (1.12 - 1.15) uses MADV_FREE rather than MADV_DONTNEED.<= Go (1.11) and Go (1.16 - which was released couple of days back) use MADV_DONTNEED.

Go 1.16 reverted back to MADV_DONTNEED.

Please find the changelog image and change log URL here.

Please upgrade to get predicatable analytics on memory usage. If you want to use GoLang(1.12-1.15) and still want to use MADV_DONTNEED, Kindly run your binaries using GODEBUG=madvdontneed=1 ./main. enter image description here

Upvotes: 1

Related Questions