CODEWITHSUNDEEP
sizelimitavro
Kevin Si
Kevin Si

Reputation: 108

What is Avro data size limit?

I couldn't find data size limit information on Avro spec.

What is the max size limit for each primitive & complex field? What is the max size limit per data block or object?

Some of of the field size limitations may depend of the language we use, for example, Java has size limit on String object.

Upvotes: 1

Views: 4411

Answers (1)

OneCricketeer
OneCricketeer

Reputation: 191884

In the Avro spec, arrays, strings, maps in general all are limited on a "variable length" "zig zag" coded long.

And a varint has no set size, as far as I know as the byte reader keeps reading the high-order bit

A variable-length format for positive integers is defined where the high-order bit of each byte indicates whether more bytes remain to be read.

While I don't know the max limit on that, even if we just base the logic purely on Java longs, then 

Long.MAX_VALUE =  9223372036854775807

and that in bytes is over 9.22 thousand Petabytes (or Exabytes), so I think you should be fine.

On the Java side of things, though, (and most other languages that have an integer-sized string-type) Strings have a much smaller size limit

How many characters can a Java String have?

Upvotes: 2

Related Questions