CODEWITHSUNDEEP
go
Camilo Crespo
Camilo Crespo

Reputation: 615

Use goroutines in combination with buffered reading to optimize reading a large file

Given a requirement where a large csv file (about 300 bytes long lines ending in /n) needs to be processed in a typical ETL: Extract, Transform, Load fashion (each line read, split and composition of a JSON inserted in a DB). Would it be beneficial to spawn one (or more) goroutines that worked together processing the file? What would need to be done to create a bufio.Scanner that started reading from a random position of the file?

Upvotes: 1

Views: 302

Answers (1)

Roy Lee
Roy Lee

Reputation: 10842

Would it be beneficial to spawn one (or more goroutines)?

Yes, absolutely. In general, you could have 3 concurrent goroutines on each E, T, L, and have them coordinated via channels.

For more insights, check out this awesome talk from Rob Pike himself:

Concurrency is not Parallelism: https://goo.gl/cp8xgF Talk Slides http://talks.golang.org/2012/waza.slide#1

Upvotes: 3

Related Questions