Scaling up the number of events sent with Measurement Protocol to GA4

Question

I'm using the Measurement Protocol to send custom events to Google Analytics 4 (GA4). As far as I saw...

• Measurement Protocol offers to send multiple events per user in a single REST request.
• But we cannot send multiple events for different users in a single REST request.

In my current use case I have millions of users. Therefor I would need to execute millions of REST requests. I'm currently trying to scale up the number of events, that I can send per day.

So I'm sending those requests in a loop. After a few thousands of iterations, the Google server complains about sending to many requests:

requests.exceptions.ConnectionError: HTTPSConnectionPool(host='www.google-analytics.com', port=443): Max retries exceeded with url: [...]"

So I implemented a backoff-retry strategy... everytime I get this exception, my application waits a few seconds and sends a further request. My code looks like this (in Python):

import requests
from requests.adapters import HTTPAdapter
from requests.packages.urllib3.util.retry import Retry

url = 'https://www.google-analytics.com/mp/collect/mp/collect?api_secret=YYYY&measurement_id=G-XXXX'

for request_data in all_requests:
    # Get payload for REST API request
    payload = create_payload(request_data)

    # Define retry-backoff strategy
    n_retries = 10
    backoff_factor = 10
    status_codes = [443]

    # Execute request using retry-backoff strategy
    session = requests.Session()
    retry = Retry(connect = n_retries, backoff_factor = backoff_factor, status_forcelist = status_codes)
    adapter = HTTPAdapter(max_retries = retry)
    session.mount('http://', adapter)
    session.mount('https://', adapter)
    response = session.post(f'{url}',
                            data=payload,
                            headers={'content-type': 'application/json'},
                            timeout=10)
    if response.status_code != requests.codes.no_content:
        raise SystemExit(f'Failed to send event with status code '
                        f'({response.status_code}) and parameters: {payload}')

This works fine... but I can only send a limited number of events per day... I tried optimizing the parameters for n_retries and backoff_factor... Per day (24h) I could send something like 500 thousand events... But in my use case I need to send millions of events.

What can I do to scale it up... so that I'm able to send much more events per day? Do you have any ideas?

---

Measurement Protocol Docs https://developers.google.com/analytics/devguides/collection/protocol/ga4