Crontab scheduled Selenium script occasionally fails to start Chromedriver

Question

I'm using crontab to schedule a Selenium script with Chromedriver to run automatically. My script generates several processes, each running its own operation and independently opening its own instance of Chromedriver. However, I've noticed that there's a certain probability (almost 50%) that Chromedriver will fail to start (all processes succeed or all fail), which never occurred when I manually launch the script. What could be the cause of this issue?

Here's the code structure for reference.

#main.py

import multiprocessing as mp

if __name__ == '__main__':
    pool = mp.Pool()
    result = pool.starmap_async(book_job, parameter_list)
    pool.close()
    pool.join()

#book.py
import func_timeout # type: ignore
from func_timeout import func_set_timeout

def book_job():
    #......
    try:
        book_result = book()
        return message + book_result
    except Exception:
        traceback.print_exc()
        return message + 'book failed'
    #......

def book():
    #......
    try:
        driver = wait_book_page()
    except func_timeout.exceptions.FunctionTimedOut:
        logging.error("wait book page timeout")
        driver.quit()
        return "Time out to wait book page enabled"
    #......
    
 
@func_set_timeout(120)
def wait_book_page():

    options = webdriver.ChromeOptions()

    options.add_argument('--headless')
    options.add_argument('--disable-gpu')
    options.add_argument('--no-sandbox')

    try:
        webdriver_service = Service('/usr/bin/chromedriver')
        driver = webdriver.Chrome(service=webdriver_service,options=options)
        logging.info('chromedriver started')
    except Exception:
        traceback.print_exc()
    #......

Error message:

Traceback (most recent call last):
File "/home/***/book.py", line 93, in wait_book_page
driver = webdriver.Chrome(service=webdriver_service,options=options)
File "/home/***/miniconda3/lib/python3.9/site-packages/selenium/webdriver/chrome/webdriver.py", line 81, in **__init__
super().__init__(
File "/home/***/miniconda3/lib/python3.9/site-packages/selenium/webdriver/chromium/webdriver.py", line 103, in __init__
self.service.start()
File "/home/***/miniconda3/lib/python3.9/site-packages/selenium/webdriver/common/service.py", line 106, in start
self.assert_process_still_running()
File "/home/***/miniconda3/lib/python3.9/site-packages/selenium/webdriver/common/service.py", line 119, in assert_process_still_running
raise WebDriverException(f"Service {self.path} unexpectedly exited. Status code was: {return_code}")
selenium.common.exceptions.WebDriverException: Message: Service /usr/bin/chromedriver unexpectedly exited. Status code was: 1

Traceback (most recent call last):
File "/home/***/main.py", line 145, in book_job
book_result = book(session, venue_list, time, space_num)
File "/home/***/book.py", line 25, in book
driver = wait_book_page(session, base_url+venue_list\[0\])
File "/home/***/miniconda3/lib/python3.9/site-packages/func_timeout/dafunc.py", line 185, in \<lambda\>
return wraps(func)(lambda args, kwargs : func_timeout(defaultTimeout, func, args=args, kwargs=kwargs))
File "/home/***/miniconda3/lib/python3.9/site-packages/func_timeout/dafunc.py", line 108, in func_timeout
raise_exception(exception)
File "/home/***/miniconda3/lib/python3.9/site-packages/func_timeout/py3_raise.py", line 7, in raise_exception
raise exception\[0\] from None
File "/home/***/book.py", line 103, in wait_book_page
raise Exception('chromedriver started failed')

My crontab file:

59 8 * * * username DISPLAY=:0 /home/***/miniconda3/bin/python /home/***/main.py >> /home/***/out.log 2>&1

Environment:

• ubuntu22.04.1 LTS
• Linux 5.15.0-52-generic
• ChromeDriver 111.0.5563.146 (2b5ec366bb5b68dacca9719d5d83cb7e61fb6431-refs/branch-- heads/5563@{#1238})
• conda 23.1.0
• python 3.9.12

I reinstalled chromium and chromedriver, and tried redeploying the environment on another machine, but the same error occurs. When I executed it manually, I used the same account with the same command as in cron.

I'm new to Python and Selenium, maybe there's some stupid mistakes. Any insights or suggestions are appreciated.