CODEWITHSUNDEEP
julia
RM-
RM-

Reputation: 1008

Most efficient way of writing data to a csv file with Julia

I want to write large amounts of data in Julia language. The data is generated and then stored in lists. A pseudocode is:

f = open("test.csv", "w")

for i = 1:3
    position = [j for j = i:(i + 10) ]
    string_position = string(position)
    n = length(string_position)
    write(f, string_position[2:(n - 1)]*"\n")
end

close(f)

However it seems inefficient to get the length of the strings in each iteration and then remove the first and last element of the string.

Is there a faster way?

Upvotes: 10

Views: 3094

Answers (1)

Jeff Bezanson
Jeff Bezanson

Reputation: 3207

One simple optimization is to use

write(f, string_position[2:(n - 1)], "\n")

instead of *. This writes the two objects in succession, instead of first concatenating them and then writing the result.

It might also be faster to use a SubString, which references part of another string in place without copying.

In general, it is also likely to be faster to avoid creating intermediate strings. Instead of first making a string and then writing it, write the individual items. For example

for item in position
    print(f, item, ",")
end
print(f, "\n")

I should add that there is a writecsv function in the standard library that does this for you.

Upvotes: 13

Related Questions