CODEWITHSUNDEEP
httpjulia
Alec
Alec

Reputation: 4482

Pass result of Julia `download` to memory instead of file?

With Julia 1.6's download function, the typical behavior is to output to a file. How can I save the result directly to something in memory?

E.g. I'd like something like:

result = download(url)
contains(result,"hello")

Upvotes: 4

Views: 512

Answers (3)

Andrej Oskin
Andrej Oskin

Reputation: 2342

You can also use UrlDownload.jl library. It uses HTTP.jl instead of curl, so it always keep the result in memory, and you can process it on the fly.

julia> using UrlDownload

julia> url = "https://raw.githubusercontent.com/Arkoniak/UrlDownload.jl/master/data/ext.csv"
"https://raw.githubusercontent.com/Arkoniak/UrlDownload.jl/master/data/ext.csv"

julia> urldownload(url, parser = x -> String(x))
"x,y\n1,2\n3,4\n"

Upvotes: 0

Alec
Alec

Reputation: 4482

As suggested by the help text for download, use the Downloads library; download can take an IOBuffer. Example:

result = String(take!(Downloads.download(url,IOBuffer())))

Upvotes: 2

Bill
Bill

Reputation: 6086

Julia uses the curl library, or something similar, for the download function, and that library writes to a file by default, or to stdout, not to a C or Julia string. Consider that many downloads may be large, perhaps larger than system RAM, to see why.

You could easily extend Julia to download, create a string for the download, and remove the temp file:

import Base.download

function Base.download(url::AbstractString, String)
     tmpfile = download(url)
     str = read(tmpfile, String)
     rm(tmpfile)
     return str
 end

Watch out for big files though :)

Upvotes: 0

Related Questions