CODEWITHSUNDEEP
gomemory-managementmemory-leaksbotsslice
Liam Murnux
Liam Murnux

Reputation: 51

Understanding Golang memory management with large slice of strings

I am working on a chat bot for the site Twitch.tv that is written in Go.

One of the features of the bot is a points system that rewards users for watching a particular stream. This data is stored in a SQLite3 database.

To get the viewers, the bot makes an API call to twitch and gathers all of the current viewers of a stream. These viewers are then put in a slice of strings.

Total viewers can range anywhere from a couple to 20,000 or more.

What the bot does

Code

type Viewers struct {
    Chatters struct {
        CurrentModerators []string `json:"moderators"`
        CurrentViewers    []string `json:"viewers"`
    } `json:"chatters"`
}    

func RunPoints(timer time.Duration, modifier int, conn net.Conn, channel string) {
    database := InitializeDB() // Loads database through SQLite3 driver
    var Points int
    var allUsers []string
    for range time.NewTicker(timer * time.Second).C {
        currentUsers := GetViewers(conn, channel)
        tx, err := database.Begin()
        if err != nil {
            fmt.Println("Error starting points transaction: ", err)
        }

        allUsers = append(allUsers, currentUsers.Chatters.CurrentViewers...)
        allUsers = append(allUsers, currentUsers.Chatters.CurrentModerators...)

        for _, v := range allUsers {
            userCheck := UserInDB(database, v)
            if userCheck == false {
                statement, _ := tx.Prepare("INSERT INTO points (Username, Points) VALUES (?, ?)")
                statement.Exec(v, 1)
            } else {

                err = tx.QueryRow("Select Points FROM points WHERE Username = ?", v).Scan(&Points)
                if err != nil {

                } else {
                    Points = Points + modifier
                    statement, _ := tx.Prepare("UPDATE points SET Points = ? WHERE username = ?")
                    statement.Exec(Points, v)
                }
            }
        }

        tx.Commit()
        allUsers = allUsers[:0]
        currentUsers = Viewers{} // Clear Viewer object struct

    }

Expected Behavior

When pulling thousands of viewers, naturally, I expect the system resources to get pretty high. This can turn the bot using 3.0 MB of RAM up to 20 MB+. Thousands of elements takes up a lot of space, of course!

However, something else happens.

Actual Behavior

Each time the API is called, the RAM increases as expected. But because I clear the slice, I expect it to fall back down to its 'normal' 3.0 MB of usage.

However, the amount of RAM usage increases per API call, and doesn't go back down even if the total number of viewers of a stream creases.

Thus, given a few hours, the bot will easily consume 100 + MB of ram which doesn't seem right to me.

What am I missing here? I'm fairly new to programming and CS in general, so perhaps I am trying to fix something that isn't a problem. But this almost sounds like a memory leak to me.

I have tried forcing garbage collection and freeing the memory through Golang's run time library, but this does not fix it.

Upvotes: 3

Views: 4948

Answers (2)

Adrian
Adrian

Reputation: 46433

When you reslice the slice:

allUsers = allUsers[:0]

All the elements are still in the backing array and cannot be collected. The memory is still allocated, which will save some time in the next run (it doesn't have to resize the array so much, saving slow allocations), which is the point of reslicing it to zero length instead of just dumping it.

If you want the memory released to the GC, you'd need to just dump it altogether and create a new slice every time. This would be slower, but use less memory between runs. However, that doesn't necessarily mean you'll see less memory used by the process. The GC collects unused heap objects, then may eventually free that memory to the OS, which may eventually reclaim it, if other processes are applying memory pressure.

Upvotes: 2

Verran
Verran

Reputation: 4072

To understand what's happening here, you need to understand the internals of a slice and what's happening with it. You should probably start with https://blog.golang.org/go-slices-usage-and-internals

To give a brief answer: A slice gives a view into a portion of an underlying array, and when you are attempting to truncate your slice, all you're doing is reducing the view you have over the array, but the underlying array remains unaffected and still takes up just as much memory. In fact, by continuing to use the same array, you're never going to decrease the amount of memory you're using.

I'd encourage you to read up on how this works, but as an example for why no actual memory would be freed up, take a look at the output from this simple program that demos how changes to a slice will not truncate the memory allocated under the hood: https://play.golang.org/p/PLEZba8uD-L

Upvotes: 4

Related Questions