CODEWITHSUNDEEP
goslicedivision
MarcoW
MarcoW

Reputation: 535

'Slice bounds out of range'-error when splitting a slice

I want to run a bunch of goroutines to insert an insanely large amount of .csv-files into a postgres database. I read the .csv-files to an object and then try to divide the object in 10 parts. This works perfectly when the amount of files is exactly dividable by 10, but when it's not I get a panic: runtime error: slice bounds out of range.

This would be sensible if i + len(fis)/gophers returned a float, but it does not. When I use the debgging-mode of Goland-IDE, both i as well as the result of i + len(fis)/gophers are full integers.

Any clue clue as to why this error occurs? I'm mostly interested in the theory behind why this happens, but also curious if someone can come up with a solution other than using the %-operator.

Thanks in advance!

Edit: I was under the impression that the error returned on the first iteratio (at which point, according to the debugger both i and i + len(fis)/gophers are well within the bounds of len(fis). The amount of files in the directory is 2568. It turns out the error was on the last iterattion. Thanks Berry for pointing this out.

//Read files from download-folder
f, _ := os.Open(dlFolder)
fis, _ := f.Readdir(-1)
f.Close()

var wg sync.WaitGroup

gophers := 10

//Split slice of files into 10 chunks and run a goroutine for each of them
for i := 0; i < len(fis) ; i = i + len(fis)/gophers{

    wg.Add(1)

    xFiles := fis[i:i + len(fis)/gophers]
    go csvTSql(xFiles)

}

wg.Wait()

Upvotes: 2

Views: 3201

Answers (1)

sberry
sberry

Reputation: 132138

Check again. i + len(fis)/gophers will not be within bound if len(fis) is not evenly divisible by gophers. It will exceed the bounds on the last iteration.

There are a number of ways to fix this like checking if i + len(fis)/gophers > len(fis) - 1 and resetting the value if it is. You can also just use a math.Min call.

chunk := int(math.Min(float64(i+len(fis)/gophers), float64(len(fis))))
xFiles := fis[i:chunk]

Upvotes: 3

Related Questions