CODEWITHSUNDEEP
julia
ChrlTsr
ChrlTsr

Reputation: 149

limit size JLD2 Julia

Is there a size limit to load data using JLD2 in Julia?

The same code:

using JLD2
using FileIO
exp=load("myfile.jld2");

seems to work fine until a certain size of data. For some experiments I ran, the file is 5GB and I obtain the following :

Error encountered while loading "/pathtofile/myfile.jld2". Fatal error:

ERROR: EOFError: read end of file

Stacktrace:

[1] handle_error(::EOFError, ::File{DataFormat{:JLD2}}) at /path/.julia/packages/FileIO/Y0Lre/src/error_handling.jl:80

[2] handle_exceptions(::Array{Any,1}, ::String) at /path/.julia/packages/FileIO/Y0Lre/src/error_handling.jl:75

[3] #load#27(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::Function, ::File{DataFormat{:JLD2}}) at /path/.julia/packages/FileIO/Y0Lre/src/loadsave.jl:193

[4] load(::File{DataFormat{:JLD2}}) at /path/.julia/packages/FileIO/Y0Lre/src/loadsave.jl:172

[5] #load#13(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::Function, ::String) at /path/.julia/packages/FileIO/Y0Lre/src/loadsave.jl:118

[6] load(::String) at /path/.julia/packages/FileIO/Y0Lre/src/loadsave.jl:118

[7] top-level scope at none:0

I tried using jldopen which I have no experience in using at all but I also get similar error messages.

What could I do ?

And, btw, what is the difference between load and jldopen("my file.jld2","r")?

I am using Julia Version 1.0.2 on Mac OSHighSierra 10.13.6.

Upvotes: 0

Views: 645

Answers (1)

carstenbauer
carstenbauer

Reputation: 10127

I regularily read >5GB files without problems. So I assume that, for some reason, your file wasn't written correctly. This is also indicated by the error message.

To answer your other question, the differene between load("myfile.jld2") and jldopen("myfile.jld2", "r") is that the former loads all the contents of the file into a dictionary whereas the latter opens the file and returns a JLDFile object which you can use to get fine grained access to your data file. For example it allows you to write multiple datafields to the same file (Note that multiple save commands overwrite the same file).

Demonstration:

julia> using FileIO, JLD2

julia> jldopen("myfile.jld2", "w") do f
           write(f, "x", rand(10))
           write(f, "y", "test")
       end

julia> load("myfile.jld2")
Dict{String,Any} with 2 entries:
  "x" => [0.918336, 0.608631, 0.757459, 0.935133, 0.548579, 0.909, 0.913573, 0.0278975, 0.…

  "y" => "test"

julia> load("myfile.jld2", "y")
"test"

julia> f = jldopen("myfile.jld2", "r")
JLDFile C:\Users\carsten\myfile.jld2 (read-only)
 ├─� x
 └─� y

julia> typeof(f)
JLD2.JLDFile{JLD2.MmapIO}

julia> f["x"]
10-element Array{Float64,1}:
 0.9183355611466055
 0.6086314948624771
 0.757458522055442
 0.9351333595616453
 0.5485794420648191
 0.9089998574850378
 0.9135729509843764
 0.027897482037234633
 0.5827560900500541
 0.9831034815173016

julia> close(f)

Upvotes: 0

Related Questions