CODEWITHSUNDEEP
pythontimestampavropython-datetimetimestamp-with-timezone
Arman Malkhasyan
Arman Malkhasyan

Reputation: 11

Timestamps of type long and logical type timestamp-micros in Python

I have the following Avro schema:

schema = {
    'name': 'avro.example.Image',
    'type': 'record',
    'fields': [
        {'name': 'image_id', 'type': 'string'},
        {'name': 'image_byte', 'type': 'bytes'},
        {'name': 'update_time', "type": [ "null", {
            "type": "long",
            "logicalType": "timestamp-micros"
        }]
        },
    ]
}

and the following python script which inserts data with the schema and generates file in Avro format:

schema_parsed = make_avsc_object(schema)

with open('images.avro', 'wb') as f:
    writer = DataFileWriter(f, DatumWriter(), schema_parsed)
    writer.append({'image_id': 'image-1', 'image_byte': a, 'update_time': datetime.now()})
    writer.append({'image_id': 'image-2', 'image_byte': a, 'update_time': datetime.now()})
    writer.close()

But it returns an error:

AvroTypeException: The datum "2022-10-02 22:38:00.605558" provided for "update_time" is not an example of the schema [
  "null",
  {
    "type": "long",
    "logicalType": "timestamp-micros"
  }
]

How to generate correct timestamp of type long and logical type timestamp-micros in Python ?

Upvotes: 0

Views: 1107

Answers (1)

fxo
fxo

Reputation: 11

You need to provide a time zone to make datatime aware if it, so instead of datetime.now() you could use datetime.now(tz=timezone.utc)

Upvotes: 1

Related Questions