CODEWITHSUNDEEP
pythonpyarrowdremio
vikashsingh9
vikashsingh9

Reputation: 41

Arrrow Flight to send RecordBatch Stream Bytpe Array as batch over python Rest Api Response

I have a result set as a stream of Arrow record batches,i have used reader.read_chunk() to get the batchs ,i have pushed the batches to batch array and convert it to bytearray as response .for that here is the code

def getBatchStreambytes(_):
   reader = client.do_get(flight_info.endpoints[0].ticket, options)
            print('[INFO] Reading query results from Dremio Server ')
            batches = [] 
            while True:
                     try:
                            batch, metadata = reader.read_chunk()
                            print(batch.num_rows)
                            batches.append(batch)

                except Exception as exception:
                        break

        data = pa.Table.from_batches(batches)
        sink = pa.BufferOutputStream()
        writer = pa.RecordBatchStreamWriter(sink, data.schema)
        writer.write_table(data)
        writer.close()
        #print(reader.read_pandas())

        return  sink.getvalue().to_pybytes()

as API response taking more time for each batches ,how its possible to send batch as response so that api will respond with iterated batch ,how i can send chunks of batch. Here us screenshots of List of Batches i recived from flight server enter image description here

Upvotes: 0

Views: 547

Answers (0)

Related Questions