Appending one element to nil slice increases capacity by two

Question

I have a nil slice:

var s1 []int // len(s1) == 0, cap(s1) == 0

Which I append one element to:

s2 := append(s1, 1) // len(s2) == 1, cap(s2) == 2

Why is appending one element to a nil slice increases the capacity by 2?

Printing the slices using fmt.Printf shows the following:

[] // s1
[1] // s2

I am also confused about why re-slicing s2[0:2] shows a zero which was not in the original slice nor appended to it:

[1,0] // s2[0:2]

Answer 1

There are many great answers here, but I want to give you the short version: In this particular case, Go knows better than the programmer.

Ignoring the specification and paying attention to the reasoning, there are critical fine-tuning decisions that must be made at the level of the implementation and it's better to allocate more than you need and waste extra space than to need to allocate a new memory block and copy all of your data. Especially since

• The memory allocator is unlikely to deal with sizes small than some boundary, like 32 or 64 bytes -- so it should use the whole block.
• Memory is cheap these days
• If you are dealing with large numbers of these slices to where the memory wasted is an issue, then you need to fine-tune your code instead.

And just for fun

Here's what a slice looks like inside: https://go.dev/play/p/upvwbXTPwqw

type GoSlice struct {
    Ptr uintptr
    Len int
    Cap int
}

You can see that the array is stored separately from the slice. That way when you exceed the capacity, it just needs to allocate a new array, copy your data and change the pointer.

Answer 2

The capacity growth is not under user control:

append(s S, x ...T) S  // T is the element type of S 

If the capacity of s is not large enough to fit the additional values, append allocates a new, sufficiently large underlying array that fits both the existing slice elements and the additional values. Otherwise, append re-uses the underlying array.

ref: https://golang.org/ref/spec#Appending_and_copying_slices
and see: https://golang.org/doc/effective_go.html#append

it is not increased by 2 (it is performance optimized):
test sample code with initial capacity 5 bytes then it is 16 instead of 10 (see commented output):

package main

import "fmt"

func main() {
    s := []byte{1, 2, 3, 4, 5}
    fmt.Println(cap(s)) // 5
    s = append(s, s...)
    fmt.Println(cap(s)) // 16
}

test sample code (with commented output):

package main

import (
    "fmt"
)

func main() {
    s := []int{0}
    fmt.Println(cap(s)) // 1
    s = append(s, s...)
    fmt.Println(cap(s)) // 2
}

test sample code (with commented output):

package main

import (
    "fmt"
)

func main() {
    s := []int{}
    fmt.Println(cap(s)) // 0
    s = append(s, 1)
    fmt.Println(cap(s)) // 1
}

test sample code with nil slice (with commented output):

package main

import (
    "fmt"
)

func main() {
    var s []int
    fmt.Println(cap(s)) // 0
    s = append(s, 1)
    fmt.Println(cap(s)) // 1
}

your sample code (with commented output):

package main

import "fmt"

func main() {
    var s1 []int
    s2 := append(s1, 1)
    fmt.Println(cap(s1)) // 0
    fmt.Println(cap(s2)) // 1
}

test sample code with 5 ints (with commented output):

package main

import "fmt"

func main() {
    s := []int{1, 2, 3, 4, 5}
    fmt.Println(cap(s)) // 5
    s = append(s, s...)
    fmt.Println(cap(s)) // 10
}

you can not access uninitialized indexes of slice like s2[1]:
panic: runtime error: slice bounds out of range:
test sample code (with commented output):

package main

import "fmt"

func main() {
    var s1 []int
    s2 := append(s1, 1)

    fmt.Println(cap(s1)) // 0
    fmt.Println(cap(s2)) // 1
    fmt.Println(s1)      // []
    fmt.Println(s2)      // [1]

    //fmt.Println(s2[0:2]) //panic: runtime error: slice bounds out of range
    //fmt.Println(s2[1])   //panic: runtime error: slice bounds out of range

}

Bounds Checking Elimination (or BCE) is a general term for removing redundant bound checking. Normally a go program will panic when a slice or a string is accessed outside of its bounds. There are two types of bound checking: for indexing (a[i]) and for slicing (a[i:j]). The go compiler inserts these bounds checks at every access, but in most cases they are not needed and are redundant based on the context.

Bound checking is important because it provides a defense against buffer overflow attacks and catches a common programming error early. BCE is important because: it speeds up the code, makes the binary smaller. If binaries are slowed down by bound checks then developers will have an incentive to disable bound checking (using -gcflags=-B).

ref

Answer 3

There's a bit of confusion here about capacity and length, I think. What you are looking at when you print the slice and see zero or one elements in the slice is its length, that is, the number of values the slice actually contains. The capacity of the underlying array is generally hidden from you unless you look it up with the cap() builtin.

Under the hood, slices are actually fixed length arrays. When you run out of space in the slice, Go has to make it bigger by creating a new (longer) array, and copying all the values over from the old one. If you're adding lots of values to a slice, it would be very slow to allocate memory for the new value (and copy all the old ones over) every time, so sometimes Go assumes that you're going to append more elements and goes ahead and allocates more memory than it needs so that you don't have to copy things as often. This extra memory can be used the next time you call append, and the number of values which can be stored in the slice before it has to be expanded is called its capacity. In other words, the capacity of the slice is the length of the slices backing array, and the length of a slice is independent from the capacity.

When you append a single value to your slice Go sees that it has to allocate space for this value, so it allocates twice the amount of space that it actually needs, increasing the length by 1 and the capacity by 2.

The slice you mention is because slicing acts on the underlying array: Go lets you slice beyond the length of a slice, you just can't go beyond its capacity (the length of the underlying array). For example, lets try a few things on a simple nil slice:

var s []int
fmt.Println(len(s), cap(s)) // Prints `0 0` because this is a nil slice
s = append(s, 1)
fmt.Println(len(s), cap(s)) // Prints `1 2`
fmt.Println(s[0:2])         // Prints `[1 0]` (only the first value is part of the slice, the second position in the underlying array is a zero value that is waiting to be used when the slices length grows)
fmt.Println(s[0:3])         // panic: slice bounds out of range (because we've exceeded the slices capacity)

Answer 4

Slice zero value

Zero value for slices is nil. But it doesn't mean you can't do anything with it. In Go types can actually be used when the value is nil. For example, struct methods can be called even if the pointer receiver is nil. Here's an example:

package main

import "fmt"

type foo struct {
}

func (f *foo) bar() {
    fmt.Println(1)
}

func main() {
    var f *foo
    fmt.Println(f)
    f.bar()
}

playground

The same applies to slices. len, cap, append all work even if you pass a nil slice. In case of append it basically creates a new slice for you that points to an array holding the value.

Slice capacity growth

When you add an element and you need to allocate more space for it, you don't allocate space for just one element. That's very ineficient. Instead you allocate more than actually needed.

Exactly how much more is allocated depends on the implementation and is not defined in the language spec. Usually capacity is doubled but in case of Go, at least as of v1.5, it rounded up to allocated memory block. You can find link to the source code here.

Slicing past length

Slicing past length is actually supported. You can slice beyond the length of a slice but you can't slice beyond the capacity:

Earlier we sliced s to a length shorter than its capacity. We can grow s to its capacity by slicing it again:

A slice cannot be grown beyond its capacity. Attempting to do so will cause a runtime panic, just as when indexing outside the bounds of a slice or array. Similarly, slices cannot be re-sliced below zero to access earlier elements in the array.

https://blog.golang.org/go-slices-usage-and-internals

In your case underlying array has capacity of 2. You only appended one element so the other one is equal to it's zero value. When you reslice past length Go can recognize that slice already has the needed capacity. So it returns a new slice that points to the same array but with length value set to 2. Here's an example of how it works:

package main

import "fmt"

func main() {
    var s []int
    s = append(s, 1, 2)
    fmt.Println(s, cap(s))

    s = s[:1]
    fmt.Println(s, cap(s))

    s = s[:2]
    fmt.Println(s, cap(s))
}

playground

It will print

[1 2] 2
[1] 2
[1 2] 2

You can see that even though I resliced to smaller length, second element is still preserved.

Answer 5

Go is free to provide you more capacity than you request. This improves performance by reducing the number of allocations (and possibly copying) that are required. Capacity is just the amount of space reserved before another allocation would be required.

If you append 5 elements to this slice, at least in my experiments, the capacity is 8. This shouldn't be surprising, but also shouldn't be relied on. On different platforms, or different versions of the compiler, the actual result may be different, as long as the capacity is "sufficiently large" (equal to or greater than length).

The upper index bound of a slice is defined as its capacity:

For arrays or strings, the indices are in range if 0 <= low <= high <= len(a), otherwise they are out of range. For slices, the upper index bound is the slice capacity cap(a) rather than the length. A constant index must be non-negative and representable by a value of type int; for arrays or constant strings, constant indices must also be in range. If both indices are constant, they must satisfy low <= high. If the indices are out of range at run time, a run-time panic occurs.

This is why reading past the length is not causing a panic. Even so, you shouldn't think of those zeros as part of the slice. They are indexable by the slice, but fmt.Printf(s2) would correctly not show them because they are not part of the slice. Don't subscript this way.

In general, you want to be looking at length, not capacity. Capacity mostly is readable to assist in performance optimization.