CODEWITHSUNDEEP
apache-arrow
Litchy
Litchy

Reputation: 673

Arrow Java ListVector writeBatch and read get empty list

I have a ListVector and value [[1,2,3,4,5]] in VectorSchemaRoot and I could see its value in IDEA.

I use following code to write the VectorSchemaRoot variable and get the byte array

    val out = new ByteArrayOutputStream()
    val writer = new ArrowStreamWriter(vectorSchemaRoot, null, out)
    writer.start()
    writer.writeBatch()
    writer.end()
    out.close()
    val byteArr = out.toByteArray

And read back

    val allocator = new RootAllocator(Int.MaxValue)
    val reader = new ArrowStreamReader(new ByteArrayInputStream(byteArr), allocator)
    while (reader.loadNextBatch()) {
      val schemaRoot = reader.getVectorSchemaRoot
      schemaRoot
    }

The schema is correct, but the list is empty []

However, I use other types of values, like char, bit, the result read from the byteArr is correct(non-empty).

How to fix the ListVector empty issue?

Upvotes: 1

Views: 381

Answers (1)

Litchy
Litchy

Reputation: 673

Finally I used just basic classes.

The StructVector, ListVector are complex classes, and according to my test, they do not bring speed or memory benefit over just using basic classes. And the documents for complex classes are very few.

Thus basic classes is recommended. And just use List of Fields to make the schema of them, could also get the structured vector.

Upvotes: 1

Related Questions