CODEWITHSUNDEEP
go
Marcin Pietraszek
Marcin Pietraszek

Reputation: 3214

Error during go build/run execution

I've created a simple go script: https://gist.github.com/kbl/86ed3b2112eb80522949f0ce574a04e3

It's fetching some xml from the internet and then starts X goroutines. The X depends on file content. In my case it was 1700 goroutines.

My first execution finished with:

$ go run mathandel1.go 
2018/01/27 14:19:37 Get https://www.boardgamegeek.com/xmlapi/boardgame/162152?pricehistory=1&stats=1: dial tcp 72.233.16.130:443: socket: too many open files
2018/01/27 14:19:37 Get https://www.boardgamegeek.com/xmlapi/boardgame/148517?pricehistory=1&stats=1: dial tcp 72.233.16.130:443: socket: too many open files
exit status 1

I've tried to increase ulimit to 2048.

Now I'm getting different error, script is the same thou:

$ go build mathandel1.go 
# command-line-arguments
/usr/local/go/pkg/tool/linux_amd64/link: flushing $WORK/command-line-arguments/_obj/exe/a.out: write $WORK/command-line-arguments/_obj/exe/a.out: file too large

What is causing that error? How can I fix that?

Upvotes: 0

Views: 573

Answers (1)

Marc
Marc

Reputation: 21035

You ran ulimit 2048 which changed the maximum file size.

From man bash(1), ulimit section:

If no option is given, then -f is assumed.

This means that you now set the maximum file size to 2048 bytes, that's probably not enough for.... anything.

I'm guessing you meant to change the limit for number of open file descriptors. For this, you want to run:

ulimit -n 2048

As for the original error (before changing the maximum file size), you're launching 1700 goroutines, each performing a http get. Each creates a connection, using a tcp socket. These are covered by the open file descriptor limit.

Instead, you should be limiting the number of concurrent downloads. This can be done with a simple worker pool pattern.

Upvotes: 1

Related Questions