Reputation: 23
When I stress test on my Scylla inside a docker I can't get more than 2000 successful write per second and write requests more than that will lead to time out even though I have dedicated a lot of resources but Scylla doesn't use much of it.
the time out error:
Operation timed out for keyspace1.messagedb - received only 0 responses from 1 CL=LOCAL_SERIAL.
here is part of my docker compose:
some-scylla:
image: scylladb/scylla:latest
container_name: some-scylla
restart: always
ports:
- "9333:10000"
command: [
"--smp", "16",
"--memory", "10G",
"--experimental", "0",
"--io-setup", "0",
"--disable-version-check"
]
volumes:
- scylla3:/var/lib/scylla
- ./scylla.d/memory.conf:/etc/scylla.d/memory.conf
- ./scylla.d/io.conf:/etc/scylla.d/io.conf
- ./scylla.yaml:/etc/scylla/scylla.yaml
- ./scylla.d/io_properties.yaml:/etc/scylla.d/io_properties.yaml
- ./scylla-server:/etc/default/scylla-server
here is my memory.conf:
MEM_CONF="--memory=10G"
here is my cpuset.conf:
CPUSET="--smp 16"
when I use the same client to send request to cassandra I get 40k success write per second.
I already turned off the developer mode and configured the io.conf and io_properties.yaml using its built-in scylla_io_setup and fed it to scylla and tried tuning it in scylla.yaml
here is my scylla.yaml:
num_tokens: 1024
commitlog_sync: periodic
commitlog_sync_period_in_ms: 200
commitlog_segment_size_in_mb: 1024
schema_commitlog_segment_size_in_mb: 4096
seed_provider:
- class_name: org.apache.cassandra.locator.SimpleSeedProvider
parameters:
- seeds: "127.0.0.1"
listen_address: localhost
native_transport_port: 9042
native_shard_aware_transport_port: 19042
read_request_timeout_in_ms: 2000
write_request_timeout_in_ms: 2000
cas_contention_timeout_in_ms: 1000
endpoint_snitch: SimpleSnitch
rpc_address: localhost
rpc_port: 9160
api_port: 10000
api_address: 127.0.0.1
batch_size_warn_threshold_in_kb: 1024
batch_size_fail_threshold_in_kb: 8192
partitioner: org.apache.cassandra.dht.Murmur3Partitioner
commitlog_total_space_in_mb: -1
developer_mode: false
murmur3_partitioner_ignore_msb_bits: 12
force_schema_commit_log: true
task_ttl_in_seconds: 10
consistent_cluster_management: true
strict_is_not_null_in_views: true
api_ui_dir: /opt/scylladb/swagger-ui/dist/
api_doc_dir: /opt/scylladb/api/api-doc/
this is my client configure which is using gocqlx:
cluster := gocql.NewCluster(hosts...)
cluster.Keyspace = Keyspace
cluster.Consistency = gocql.One
cluster.ProtoVersion = 4
cluster.Port = port
cluster.RetryPolicy = &gocql.SimpleRetryPolicy{NumRetries: 1}
cluster.PoolConfig.HostSelectionPolicy = gocql.HostPoolHostPolicy(
hostpool.NewEpsilonGreedy(nil, 0, &hostpool.LinearEpsilonValueCalculator{}),
)
cluster.NumConns = 64
cluster.MaxRoutingKeyInfo = 1000
consider gocql send 2 request per connection which I can't change
and I only have one host
Upvotes: 0
Views: 177