CODEWITHSUNDEEP
cassandrabroken-pipe
Black Jack
Black Jack

Reputation: 11

Broken pipe when bootstrapping or repair node

I got 2 node "A" and "B" in two datacenter,

A in DC1, B in DC2.

Replication factor for DC1 = 3, for DC2 = 1.

I try join new node "C" to the DC1 but i got this error message on node "A":

ERROR [STREAM-OUT-/X.X.XX.XXX] 2015-12-31 02:39:05,252 StreamSession.java:524 - [Stream #a74ecdc0-af67-11e5-8429-a94b3a1b0e16] Streaming error occurred
org.apache.cassandra.io.FSReadError: java.io.IOException: Broken pipe
        at org.apache.cassandra.io.util.ChannelProxy.transferTo(ChannelProxy.java:144) ~[apache-cassandra-2.2.4.jar:2.2.4]
        at org.apache.cassandra.streaming.compress.CompressedStreamWriter$1.apply(CompressedStreamWriter.java:79) ~[apache-cassandra-2.2.4.jar:2.2.4]
        at org.apache.cassandra.streaming.compress.CompressedStreamWriter$1.apply(CompressedStreamWriter.java:76) ~[apache-cassandra-2.2.4.jar:2.2.4]
        at org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.applyToChannel(BufferedDataOutputStreamPlus.java:297) ~[apache-cassandra-2.2.4.jar:2.2.4]
        at org.apache.cassandra.streaming.compress.CompressedStreamWriter.write(CompressedStreamWriter.java:75) ~[apache-cassandra-2.2.4.jar:2.2.4]
        at org.apache.cassandra.streaming.messages.OutgoingFileMessage.serialize(OutgoingFileMessage.java:90) ~[apache-cassandra-2.2.4.jar:2.2.4]
        at org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:48) ~[apache-cassandra-2.2.4.jar:2.2.4]
        at org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:40) ~[apache-cassandra-2.2.4.jar:2.2.4]
        at org.apache.cassandra.streaming.messages.StreamMessage.serialize(StreamMessage.java:47) ~[apache-cassandra-2.2.4.jar:2.2.4]
        at org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.sendMessage(ConnectionHandler.java:363) ~[apache-cassandra-2.2.4.jar:2.2.4]
        at org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.run(ConnectionHandler.java:335) ~[apache-cassandra-2.2.4.jar:2.2.4]
        at java.lang.Thread.run(Thread.java:745) [na:1.8.0_65]
Caused by: java.io.IOException: Broken pipe
        at sun.nio.ch.FileChannelImpl.transferTo0(Native Method) ~[na:1.8.0_65]
        at sun.nio.ch.FileChannelImpl.transferToDirectlyInternal(FileChannelImpl.java:427) ~[na:1.8.0_65]
        at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:492) ~[na:1.8.0_65]
        at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:607) ~[na:1.8.0_65]
        at org.apache.cassandra.io.util.ChannelProxy.transferTo(ChannelProxy.java:140) ~[apache-cassandra-2.2.4.jar:2.2.4]
        ... 11 common frames omitted

I try to run nodetool repair on node A, but i get:

Repair session xxx for range xx xx failed with error Stream failed 

[2015-12-31 14:36:12,722] Repair session 75e93060-af4c-11e5-8429-a94b3a1b0e16 for range (-4171974372789419026,-4027636102743839484] failed with error Stream failed (progress: 9%)
[2015-12-31 16:48:26,913] Repair session 75f76130-af4c-11e5-8429-a94b3a1b0e16 for range (-1477833652511894033,-1464467586067376161] finished (progress: 12%)
[2015-12-31 17:19:40,353] Repair session 75f82483-af4c-11e5-8429-a94b3a1b0e16 for range (4136345906437366778,4261334469649078197] failed with error Stream failed (progress: 12%)

and in log the same Broken pipe error occur

I found some information that this can be a network problem. I check the connection between nodes with mtr, but is this enough?

Is there any other possibility to retry the repair just on the failed range?

Upvotes: 0

Views: 1075

Answers (1)

Greg
Greg

Reputation: 121

There seems to be a bug in CompressedStreamReader when it encounter sections larger than 2.14GB (because using an int instead of a long to calculate section length...)

Please see this bug https://issues.apache.org/jira/browse/CASSANDRA-10961, a patched JAR is attached and will be included in Cassandra 2.2.5

Upvotes: 2

Related Questions