cassandra INSERT fails after a lot inserts with : "Operation timed out"

Question

I use the cassandra-c++-driver to write 100000 rows in a 100-columns table like this:

#include <cstdlib>
#include <stdio.h>
#include <cassandra.h>
#include <string>
#include <iostream>
#include <random>
#include <chrono>
#include <unistd.h>
#include <thread>
CassFuture *connect_future = NULL;
CassCluster *cluster = NULL;
CassSession *session = NULL;
std::random_device rd;
std::mt19937_64 gen(rd());
std::uniform_int_distribution<unsigned long long> dis;


int COLUMNS_COUNT = 100;
using namespace std;

void insertQ() {

    auto t1 = std::chrono::high_resolution_clock::now();


    for (int row = 0; row < 10000; ++row) {
        string columns;
        for (int i = 0; i < COLUMNS_COUNT; ++i) {
            columns += "name" + to_string(i) + " , ";
        }

        string result = "INSERT INTO mykeyspace.users2 (user_id,";
        result += columns;
        result += "lname) VALUES (";


        string values = to_string(dis(gen) % 50000000) + ",";

        for (int i = 0; i < COLUMNS_COUNT; ++i) {
            values += "'name" + to_string(dis(gen)) + "' , ";
        }

        values += " 'lname" + to_string(dis(gen) % 20) + "'";
        result += values;
        result += ");";


        CassStatement *statement = cass_statement_new(result.c_str(), 0);

        CassFuture *result_future = cass_session_execute(session, statement);
        cass_future_wait(result_future);

        if (cass_future_error_code(result_future) == CASS_OK) {
//            cout << "insert ok" << endl;
        }
        else {
            const char *message;
            size_t message_length;
            cass_future_error_message(result_future, &message, &message_length);
            fprintf(stderr, "Unable to run query: '%.*s'\n", (int) message_length,
                    message);

            cerr << "index : " << row << endl;
        }

        cass_statement_free(statement);
        cass_future_free(result_future);

        if (row % 1000 == 0)
        {
//            usleep(1000000);
//            std::this_thread::sleep_for(std::chrono::seconds(1));
//            cass_se
        }

    }
    auto t2 = std::chrono::high_resolution_clock::now();

    auto duration = std::chrono::duration_cast<std::chrono::milliseconds>(t2 - t1);

    cout << "duration: " << duration.count() << endl;


}


int main() {

    /* Setup and connect to cluster */
    connect_future = NULL;
    cluster = cass_cluster_new();
    session = cass_session_new();

    /* Add contact points */
//    cass_cluster_set_contact_points(cluster, "127.0.0.1,127.0.0.2,127.0.0.3");
    cass_cluster_set_contact_points(cluster, "127.0.0.1");

    /* Provide the cluster object as configuration to connect the session */
    connect_future = cass_session_connect(session, cluster);

    if (cass_future_error_code(connect_future) == CASS_OK) {
        CassFuture *close_future = NULL;

        insertQ();

        /* Close the session */
        close_future = cass_session_close(session);
        cass_future_wait(close_future);
        cass_future_free(close_future);
    } else {
        /* Handle error */
        const char *message;
        size_t message_length;
        cass_future_error_message(connect_future, &message, &message_length);
        fprintf(stderr, "Unable to connect: '%.*s'\n", (int) message_length,
                message);
    }

    cass_future_free(connect_future);
    cass_cluster_free(cluster);
    cass_session_free(session);

    return 0;
}

its works and writes about 90000 rows and then falls with this error:
index : 91627 Unable to run query: 'Operation timed out - received only 0 responses.' ..

and continues, I can execute 'SELECT' queries but after this 'INSERT's fails. unitl I restart the cassandra servcice.

Whats the problem?
My system: Ubuntu 14.04 x64, 8 gig ram, cassandra 2.1.4 (from cassandra debian repositories with default configrations)

thanks.

Answer 1

This error is coming back from Cassandra. It indicates that less than the amount of replicas required responded to your read/write request within the period of time configured in cassandra. Since you are not specifying a consistency level, all that is required is that one node responds and it isn't within the write timeout. The most relevant configurations to look at in cassandra.yaml are:

write_request_timeout_in_ms (default 2000ms)
read_request_timeout_in_ms (default: 5000ms)
range_request_timeout_in_ms (default: 10000ms)

Since you are doing inserts, write_request_timeout_in_ms is probably the most relevant configuration.

What's likely happening is that you are overwhelming your cassandra cluster. Have you looked a cpu utilization/disk io/memory utilization on the server while running your test?

The interesting thing is that your code only ever does 1 INSERT at a time, is this correct? I would expect that this should be fine, but maybe what is happening is that this is putting intense pressure on your memory heap in cassandra and it can't flush data fast enough, so it becomes backed up while writing to disk. You should take a look at your cassandra system.log (usually in /var/log/cassandra if on linux) and see if there are any suspicious messages about long garbage collections (look for GCInspector) or memtable pressure.