CODEWITHSUNDEEP
python-3.xgoogle-cloud-rungoogle-cloud-logging
Expert wanna be
Expert wanna be

Reputation: 10624

GCP Cloud run (with python) logging to cloud logging

I'm trying to use cloud run for receiving millions of log messages through HTTPS (streaming) and sending them to cloud logging.

But I found there is some loss of data, the number of messages in cloud logging is less than cloud run receiving.

This is the sample code that I tried,

# unzip data
data = gzip.decompress(request.data)

# split by lines
logs = data.decode('UTF-8').split('\n')

# output the logs
log_cnt = 0
for log in logs:
    try:
        # output to jsonPayload
        print(json.dumps(json.loads(log_str))
        log_cnt += 1
    except Exception as e:
        logging.error(F"messsage: {str(e)}")

and If I compare the log_cnt and number of logs in cloud logging, the log_cnt is more. So some print is not finishing the delivering message.

I tried using logging API instead of print(), but the number of logs is too many for sending using logging API (12,000 calls limit for a minute), so it causes the latency very bad, and could not handle requests stably.

I dought the moving number of instances might cause it, so I test when the active instance is not changed, but still, 3-5% of messages are missing.

Is there something I can do for sending all of the messages to cloud logging without any loss?

(updated)

the line of data looks like this, (around 1kb)

{"key1": "ABCDEFGHIJKLMN","key2": "ABCDEFGHIJKLMN","key3": "ABCDEFGHIJKLMN","key4": "ABCDEFGHIJKLMN","key5": "ABCDEFGHIJKLMN","key6": "ABCDEFGHIJKLMN","key7": "ABCDEFGHIJKLMN","key8": "ABCDEFGHIJKLMN","key9": "ABCDEFGHIJKLMN","key10": "ABCDEFGHIJKLMN","key11": "ABCDEFGHIJKLMN","key12": "ABCDEFGHIJKLMN","key13": "ABCDEFGHIJKLMN","key14": "ABCDEFGHIJKLMN","key15": "ABCDEFGHIJKLMN","key16": "ABCDEFGHIJKLMN","key17": "ABCDEFGHIJKLMN","key18": "ABCDEFGHIJKLMN","key19": "ABCDEFGHIJKLMN","key20": "ABCDEFGHIJKLMN","key21": "ABCDEFGHIJKLMN","key22": "ABCDEFGHIJKLMN","key23": "ABCDEFGHIJKLMN","key24": "ABCDEFGHIJKLMN","key26": "ABCDEFGHIJKLMN","key27": "ABCDEFGHIJKLMN","key28": "ABCDEFGHIJKLMN","key29": "ABCDEFGHIJKLMN","key30": "ABCDEFGHIJKLMN","key31": "ABCDEFGHIJKLMN","key32": "ABCDEFGHIJKLMN","key33": "ABCDEFGHIJKLMN","key34": "ABCDEFGHIJKLMN","key35": "ABCDEFGHIJKLMN"}

Upvotes: 0

Views: 974

Answers (1)

guillaume blaquiere
guillaume blaquiere

Reputation: 75705

I can recommend you to use the entries.write API that allow you to write a bulk of entries in the same time. Have a look on my test in the API explorer

enter image description here

It could solve the JSON format and the multiple write in the same time. Have a try with this API and let me know if it's better for you. If not, I will remove this answer!

Upvotes: 2

Related Questions