CODEWITHSUNDEEP
goiobuffer
Nicky Feller
Nicky Feller

Reputation: 3899

Golang - How to overcome Scan() buffer limit from bufio?

I am using the bufio scanner's .Scan() method to read text file lines. However when I reach a certain size in the file lines, the scanner no longer allows me to read, it just returns an empty line.

How should I configure the buffer to take in larger amounts of data?

bigfile.txt is just a file with many integers in a line separated by a space. For example, 40000 integers in one line. (note that it works for file lines of 10000 integers and less, but not 40000 for example)

234 544 765 45 34 67 67 87 98 43 [... n = 40000 ]

func main() {
    readInputFile("bigfile.txt")
}

func readInputFile(name string) {
    inFile, _ := os.Open(name)
    defer inFile.Close()

    scanner := bufio.NewScanner(inFile)

    for scanner.Scan() {
        line := scanner.Text()
        fmt.Printf(line)
    }
}

Upvotes: 16

Views: 14117

Answers (2)

coder
coder

Reputation: 31

A code to use dynamic buffer size. Get the file size and use it in the scanner

info, infoErr := file.Stat()
var maxSize int
scanner := bufio.NewScanner(file)
maxSize = int(info.Size())
buffer := make([]byte, 0, maxSize)
scanner.Buffer(buffer, maxSize)

For JSON add a few more bytes maxSize = maxSize + 1 or maxSize = maxSize + 10

Upvotes: 3

putu
putu

Reputation: 6444

According to documentation and corresponding source code, by default Scanner uses internal buffer in which the capacity is 64K. In your case, the internal buffer is not sufficient for storing 40000 integers. Set the buffer to be used by scanner before calling Scan, i.e. 

scanner := bufio.NewScanner(inFile)

//adjust the capacity to your need (max characters in line)
const maxCapacity = 512*1024  
buf := make([]byte, maxCapacity)
scanner.Buffer(buf, maxCapacity)

Upvotes: 32

Related Questions