CODEWITHSUNDEEP
pythonredis
tormich
tormich

Reputation: 43

redis - Timeout reading from socket every 20 minutes

I have a server with redis, redis-sentinel and python-tornado app. Every ~20 minutes or so connection to the redis can't be established for 1-3 seconds due to "Timeout reading from socket".

What can it be?

Redis info:

#Server
redis_version:2.8.6
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:7648891aee0aa9dc
redis_mode:standalone
os:Linux 2.6.32-431.5.1.el6.x86_64 x86_64
arch_bits:64
multiplexing_api:epoll
gcc_version:4.4.7
process_id:25110
run_id:6121f01610ff853446692fa7984b8f011a53361f
tcp_port:6379
uptime_in_seconds:9438
uptime_in_days:0
hz:10
lru_clock:620564
config_file:/etc/redis/6379.conf

# Clients
connected_clients:67
client_longest_output_list:0
client_biggest_input_buf:0
blocked_clients:0

# Memory
used_memory:1788024368
used_memory_human:1.67G
used_memory_rss:1834860544
used_memory_peak:1788713352
used_memory_peak_human:1.67G
used_memory_lua:34816
mem_fragmentation_ratio:1.03
mem_allocator:jemalloc-3.2.0

# Persistence
loading:0
rdb_changes_since_last_save:743
rdb_bgsave_in_progress:0
rdb_last_save_time:1411297206
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:28
rdb_current_bgsave_time_sec:-1
aof_enabled:1
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_last_write_status:ok
aof_current_size:669871843
aof_base_size:668274530
aof_pending_rewrite:0
aof_buffer_length:0
aof_rewrite_buffer_length:0
aof_pending_bio_fsync:0
aof_delayed_fsync:11

# Stats
total_connections_received:2969
total_commands_processed:202332538
instantaneous_ops_per_sec:30807
rejected_connections:0
sync_full:0
sync_partial_ok:0
sync_partial_err:0
expired_keys:287
evicted_keys:0
keyspace_hits:108252
keyspace_misses:536
pubsub_channels:1
pubsub_patterns:0
latest_fork_usec:4498

# Replication
role:master
connected_slaves:0
master_repl_offset:0
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0

# CPU
used_cpu_sys:3555.96
used_cpu_user:2583.51
used_cpu_sys_children:35.61
used_cpu_user_children:346.40

# Keyspace
db0:keys=781283,expires=23,avg_ttl=261371

Upvotes: 1

Views: 5476

Answers (1)

tormich
tormich

Reputation: 43

Turns out,it was hdd problem. the thing is - we took Azure's "A" series vm instance (cheapest series). And every time Redis was trying to save data to rdb, IO jumped to 100% blocking everyone till Redis is done.

For now our solution is to stop master-redis from saving to rdb (only aof). And use slave for persistent saving.

Upvotes: 2

Related Questions