CODEWITHSUNDEEP
pythonamazon-qldb
rustruss
rustruss

Reputation: 1

QLDB Python driver seems very slow

I am using the Python AWS QLDB Driver to make queries against a QLDB ledger. I've noticed that the total time for results to return is about double what the internal qldb timing statistics says it should be. Just wondering what additional overhead there is or what can be done about this.

Driver setup:

    qldb_driver = QldbDriver(
        ledger_name, retry_config=retry_config, region_name=region_name)

    def get_all_test_statement(table_name:str):
        return f"SELECT * FROM {table_name}"

Executor function:

def test_get_all(transaction_executor, statement)
    docs = []
    processing_time_milliseconds = 0

    start = datetime.now(timezone.utc)
    cursor = transaction_executor.execute_statement(statement)


    for doc in cursor:
        docs.append(doc)
        timing_information = cursor.get_timing_information()
        processing_time_milliseconds += timing_information.get('ProcessingTimeMilliseconds')


    end = datetime.now(timezone.utc)

    print("qldb timing", processing_time_milliseconds)
    print("actual timing", (end - start).total_seconds())

    return docs

Running it:

    s = get_all_test_statement("test_table")
    
    docs = qldb_driver.execute_lambda(
        lambda executor: test_get_all(executor, s)
    )

Result:

qldb timing 3.42126
actual timing 7.341538

Upvotes: 0

Views: 170

Answers (1)

Ang
Ang

Reputation: 76

the timing information doesn't include time spent on network calls and other overhead before QLDB frontend server starts the timing on server side. It's only calculating the server time of a single statement that is being executed and commands like StartSession, StartTransaction, CommitTransaction are not included through get_timing_information.

I profiled functions in python driver and the following are some numbers observed. It's obvious that the overhead comes from when driver sends out the command and receives response.

_send_command 86.24600000000001
_update_hash 0.36000000000000004
_session._execute_statement 86.479
qldb timing 11

Upvotes: 0

Related Questions