CODEWITHSUNDEEP
pythoncronsynchronousinfluxdb-2data-loss
lambruscoAcido
lambruscoAcido

Reputation: 578

If multiple Python processes write data into InfluxDB, only parts are written

I have multiple Python processes, running on a server (RPI5) by cron, which read data from web API's, and write it then into a common InfluxDB database - on the same bucket.

However, some of the data is lost. The code to write into Influx is:

influxdb_client = InfluxDBClient(url=url, token=token, org=org) 
...

def f(df): 
  write_api = influxdb_client.write_api()
  ...
  record = [] 
  for i in range(df.shape[0]):
    point = Point(measurement).tag("location", ...).time(...)  
    for col in list(df.columns): 
      value = df.loc[i, col]  
      point = point.field(col, value) 
      record += [point]   
  write_api.write(bucket=bucket, org=org, record=record) 

...
## Let df be a data.frame with 20-500 rows, and 10-20 columns.   
f(df)

What could be the reason of this issue? Some problem with asynchronous/synchronous?

Thx

Upvotes: 2

Views: 123

Answers (1)

Suyash
Suyash

Reputation: 335

Your reasoning seems correct, modify the write_api() to use the synchronous mode and also add error handling logic to the code and try again.

Upvotes: 1

Related Questions