CODEWITHSUNDEEP
databasecassandranosqldatastax-enterprise
Anubhuti Pandey
Anubhuti Pandey

Reputation: 1

cassandra read query taking time in different GCP[ INDIA and GCP US ]

We are migrating to GCP INDIA region from GCP US region. We have created 3 node cluster in INDIA and there is 3 node cluster in GCP US. While transferring the some read queries to GCP INDIA, we are facing high read response latency (30 Milliseconds) of the same query which is running in GCP US under 15 milliseconds.

GCP US (API) --> read from --> GCP Cassandra (US ) --> takes 10 -15 milliseconds.

But same flow for GCP INDIA region,

GCP INDIA (API ) ---> read from GCP Cassandra (INDIA )-->takes 30 milliseconds.

While checking the read queries tracing plan with local_quorum;

---------------------------------------------------------------------------------------------------+----------------------------+
                                                                                Execute CQL3 query | 2021-07-13 11:39:26.806000 |
to_serv,eto_glusr_pref_loc FROM glusr_usr WHERE glusr_usr_id =12530; [Native-Transport-Requests-3] | 2021-07-13 11:39:26.807000 |
                                                 Preparing statement [Native-Transport-Requests-3] | 2021-07-13 11:39:26.808000 |
                                   reading digest from /XX.XX.XX.XX [Native-Transport-Requests-3] | 2021-07-13 11:39:26.809000 | 
                                                                      Row cache miss [ReadStage-2] | 2021-07-13 11:39:26.809000 |
                                                                   Row cache miss [ReadStage-2] | 2021-07-13 11:39:26.811000 | XX
                             Partition index with 0 entries found for sstable 139241 [ReadStage-2] | 2021-07-13 11:39:26.811000 |
                                                                      Row cache miss [ReadStage-2] | 2021-07-13 11:39:26.811000 |
                                       Executing single-partition query on glusr_usr [ReadStage-2] | 2021-07-13 11:39:26.811000 |
                                       Executing single-partition query on glusr_usr [ReadStage-2] | 2021-07-13 11:39:26.811000 |
                                                        Acquiring sstable references [ReadStage-2] | 2021-07-13 11:39:26.811000 |
                                                        Acquiring sstable references [ReadStage-2] | 2021-07-13 11:39:26.811000 |
          Skipped 0/13 non-slice-intersecting sstables, included 0 due to tombstones [ReadStage-2] | 2021-07-13 11:39:26.811000 |
          Skipped 0/13 non-slice-intersecting sstables, included 0 due to tombstones [ReadStage-2] | 2021-07-13 11:39:26.811000 |
                             Partition index with 0 entries found for sstable 138332 [ReadStage-2] | 2021-07-13 11:39:26.811000 |
                             Partition index with 0 entries found for sstable 138620 [ReadStage-2] | 2021-07-13 11:39:26.813000 |
                             Partition index with 0 entries found for sstable 138331 [ReadStage-2] | 2021-07-13 11:39:26.815000 |
                             Partition index with 0 entries found for sstable 139240 [ReadStage-2] | 2021-07-13 11:39:26.815000 |
                             Partition index with 0 entries found for sstable 138783 [ReadStage-2] | 2021-07-13 11:39:26.817000 |
                             Partition index with 0 entries found for sstable 139719 [ReadStage-2] | 2021-07-13 11:39:26.817000 |
                             Partition index with 0 entries found for sstable 138621 [ReadStage-2] | 2021-07-13 11:39:26.817000 |
                             Partition index with 0 entries found for sstable 139855 [ReadStage-2] | 2021-07-13 11:39:26.819000 |
                             Partition index with 0 entries found for sstable 138950 [ReadStage-2] | 2021-07-13 11:39:26.820000 |
                             Partition index with 0 entries found for sstable 139096 [ReadStage-2] | 2021-07-13 11:39:26.820000 |
                                         Bloom filter allows skipping sstable 139878 [ReadStage-2] | 2021-07-13 11:39:26.821000 |
                                         Bloom filter allows skipping sstable 140774 [ReadStage-2] | 2021-07-13 11:39:26.821000 |
                                         Bloom filter allows skipping sstable 139883 [ReadStage-2] | 2021-07-13 11:39:26.821000 |
                                         Bloom filter allows skipping sstable 140789 [ReadStage-2] | 2021-07-13 11:39:26.821000 |
                                         Bloom filter allows skipping sstable 139888 [ReadStage-2] | 2021-07-13 11:39:26.821000 |
                                         Bloom filter allows skipping sstable 140794 [ReadStage-2] | 2021-07-13 11:39:26.821000 |
                                         Bloom filter allows skipping sstable 139893 [ReadStage-2] | 2021-07-13 11:39:26.821000 |
                             Partition index with 0 entries found for sstable 140831 [ReadStage-2] | 2021-07-13 11:39:26.821000 |
                                         Bloom filter allows skipping sstable 141000 [ReadStage-2] | 2021-07-13 11:39:26.821000 |
                                         Bloom filter allows skipping sstable 141037 [ReadStage-2] | 2021-07-13 11:39:26.821000 |
                                                                      Caching 1 rows [ReadStage-2] | 2021-07-13 11:39:26.821001 |
                             Partition index with 0 entries found for sstable 139987 [ReadStage-2] | 2021-07-13 11:39:26.822000 |
                                           Merged data from memtables and 5 sstables [ReadStage-2] | 2021-07-13 11:39:26.822000 |
                                         Bloom filter allows skipping sstable 140052 [ReadStage-2] | 2021-07-13 11:39:26.822000 |
                                         Bloom filter allows skipping sstable 140097 [ReadStage-2] | 2021-07-13 11:39:26.822000 |
                                         Bloom filter allows skipping sstable 140134 [ReadStage-2] | 2021-07-13 11:39:26.822001 |
                                         Bloom filter allows skipping sstable 140135 [ReadStage-2] | 2021-07-13 11:39:26.822001 |
                                                                      Caching 1 rows [ReadStage-2] | 2021-07-13 11:39:26.822001 |
                                           Merged data from memtables and 5 sstables [ReadStage-2] | 2021-07-13 11:39:26.823000 |
                                              Read 1 live rows and 0 tombstone cells [ReadStage-2] | 2021-07-13 11:39:26.823000 |
                             Partition index with 0 entries found for sstable 140137 [ReadStage-2] | 2021-07-13 11:39:26.823000 |
                                              Read 1 live rows and 0 tombstone cells [ReadStage-2] | 2021-07-13 11:39:26.823000 |
                                                 Enqueuing response to /XX.XX.XX.XX [ReadStage-2] | 2021-07-13 11:39:26.823000 | 
 Sending REQUEST_RESPONSE message to /XX.XX.XX.XX [MessagingService-Outgoing-/XX.XX.XX.XX-Large] | 2021-07-13 11:39:26.824000 | X
    REQUEST_RESPONSE message received from /XX.XX.XX.XX [MessagingService-Incoming-/XX.XX.XX.XX] | 2021-07-13 11:39:26.825000 | X
                                         Bloom filter allows skipping sstable 140140 [ReadStage-2] | 2021-07-13 11:39:26.825000 |
                                   Processing response from /XX.XX.XX.XX [RequestResponseStage-2] | 2021-07-13 11:39:26.825000 |

My nodetool info

Key Cache              : entries 52837, size 5.21 MiB, capacity 500 MiB, 14109 hits, 49400 requests, 0.286 recent hit rate, 14400 save period in seconds
Row Cache              : entries 232, size 20.91 MiB, capacity 4.09 GiB, 78 hits, 540 requests, 0.144 recent hit rate, 0 save period in seconds

So my question is how to reduce this 30 milliseconds latency and why it is happening .

Please let me know, thank you.

Upvotes: 0

Views: 78

Answers (1)

Alex Ott
Alex Ott

Reputation: 87329

From first look I would say that you have too many SSTables, you're spending almost 12ms looking for data in them. I would suggest to check how many SSTables read in both DCs by using nodetool tablehistograms.

But according to trace, it took only 19ms to return answer. If you see 30ms response time, then you may need to capture execution metric from the driver side, and make sure that you have fast link from driver to Cassandra nodes.

Upvotes: 2

Related Questions