Size control on logging an unknown length of parameters

Question

The Problem:

Right now, I'm logging my SQL query and the args that related to that query, but what will happen if my args weight a lot? say 100MB?

The Solution:

I want to iterate over the args and once they exceeded the 0.5MB I want to take the args up till this point and only log them (of course I'll use the entire args set in the actual SQL query).

Where am stuck:

1. I find it hard to find the size on the disk of an interface{}.
2. How can I print it? (there is a nicer way to do it than %v?)

The concern is mainly focused on the first section, how can I find the size, I need to know the type, if its an array, stack, heap, etc..

If code helps, here is my code structure (everything sits in dal pkg in util file):

package dal

import (
    "fmt"
)

const limitedLogArgsSizeB = 100000 // ~ 0.1MB

func parsedArgs(args ...interface{}) string {
    currentSize := 0
    var res string

    for i := 0; i < len(args); i++ {
        currentEleSize := getSizeOfElement(args[i])

        if !(currentSize+currentEleSize =< limitedLogArgsSizeB) {
            break
        }

        currentSize += currentEleSize

        res = fmt.Sprintf("%s, %v", res, args[i])
    }

    return "[" + res + "]"
}

func getSizeOfElement(interface{}) (sizeInBytes int) {

}

So as you can see I expect to get back from parsedArgs() a string that looks like:

"[4378233, 33, true]"

for completeness, the query that goes with it:

INSERT INTO Person (id,age,is_healthy) VALUES ($0,$1,$2)

so to demonstrate the point of all of this:

lets say the first two args are equal exactly to the threshold of the size limit that I want to log, I will only get back from the parsedArgs() the first two args as a string like this:

"[4378233, 33]"

I can provide further details upon request, Thanks :)

Answer 1

Getting the memory size of arbitrary values (arbitrary data structures) is not impossible but "hard" in Go. For details, see How to get memory size of variable in Go?

The easiest solution could be to produce the data to be logged in memory, and you can simply truncate it before logging (e.g. if it's a string or a byte slice, simply slice it). This is however not the gentlest solution (slower and requires more memory).

Instead I would achieve what you want differently. I would try to assemble the data to be logged, but I would use a special io.Writer as the target (which may be targeted at your disk or at an in-memory buffer) which keeps track of the bytes written to it, and once a limit is reached, it could discard further data (or report an error, whatever suits you).

You can see a counting io.Writer implementation here: Size in bits of object encoded to JSON?

type CounterWr struct {
    io.Writer
    Count int
}

func (cw *CounterWr) Write(p []byte) (n int, err error) {
    n, err = cw.Writer.Write(p)
    cw.Count += n
    return
}

We can easily change it to become a functional limited-writer:

type LimitWriter struct {
    io.Writer
    Remaining int
}

func (lw *LimitWriter) Write(p []byte) (n int, err error) {
    if lw.Remaining == 0 {
        return 0, io.EOF
    }

    if lw.Remaining < len(p) {
        p = p[:lw.Remaining]
    }
    n, err = lw.Writer.Write(p)
    lw.Remaining -= n
    return
}

And you can use the fmt.FprintXXX() functions to write into a value of this LimitWriter.

An example writing to an in-memory buffer:

buf := &bytes.Buffer{}
lw := &LimitWriter{
    Writer:     buf,
    Remaining:  20,
}

args := []interface{}{1, 2, "Looooooooooooong"}

fmt.Fprint(lw, args)
fmt.Printf("%d %q", buf.Len(), buf)

This will output (try it on the Go Playground):

20 "[1 2 Looooooooooooon"

As you can see, our LimitWriter only allowed to write 20 bytes (LimitWriter.Remaining), and the rest were discarded.

Note that in this example I assembled the data in an in-memory buffer, but in your logging system you can write directly to your logging stream, just wrap it in LimitWriter (so you can completely omit the in-memory buffer).

Optimization tip: if you have the arguments as a slice, you may optimize the truncated rendering by using a loop, and stop printing arguments once the limit is reached.

An example doing this:

buf := &bytes.Buffer{}
lw := &LimitWriter{
    Writer:    buf,
    Remaining: 20,
}

args := []interface{}{1, 2, "Loooooooooooooooong", 3, 4, 5}

io.WriteString(lw, "[")
for i, v := range args {
    if _, err := fmt.Fprint(lw, v, " "); err != nil {
        fmt.Printf("Breaking at argument %d, err: %v\n", i, err)
        break
    }
}
io.WriteString(lw, "]")

fmt.Printf("%d %q", buf.Len(), buf)

Output (try it on the Go Playground):

Breaking at argument 3, err: EOF
20 "[1 2 Loooooooooooooo"

The good thing about this is that once we reach the limit, we don't have to produce the string representation of the remaining arguments that would be discarded anyway, saving some CPU (and memory) resources.