CODEWITHSUNDEEP
sslgithub-webhook
paulrw
paulrw

Reputation: 41

Github webhook over https: "Service timeout"

This question is about making a github webhook work over https. It's also a question from an inexperienced person as to best practices for troubleshooting.

I have a github webhook for my staging site that points to https://staging.domain.com/git_webhook

It functions perfectly if I point it to http and not https. But with https, github responds: We couldn’t deliver this payload: Service Timeout. This happens even when SSL verification is disabled for webhook.

Using Postman and curl, the webhook functions perfectly over https.

What I've tried

  1. Check if firewall is responsible.

The server is Ubuntu 18.04 with apparmor, ufw, and fail2ban. In ufw, https is open for all. I've disabled each service and restarted apache, with no success. I haven't found any rules that single out github ip's. But I'm inexperienced, and may not understand how to look deep enough for them if they exist. I figured that simply briefly disabling the services would be enough to test if they were involved.

  1. Use tshark to examine incoming HTTPS POST request from github

tshark showed that after "Server Hello, Certificate, Server Key Exchange, Server Hello Done", there is no handshake. There are 4 or 5 [PSH,ACK] retransmissions from my server to github, with no response from github, then github closes the connection.

  1. Is this a problem with my SSL Cert?

My main domain.com has a Comodo SSL Cert that does not apply to any subdomain. my staging.domain.com has a valid Let's Encrypt SSL Cert.

When I run openssl s_client -showcerts -servername staging.domain.com -connect staging.domain.com:443 </dev/null I get the Let's Encrypt certificate that belongs to the staging.domain.com

But when I run openssl s_client -showcerts -connect staging.domain.com:443 </dev/null I get the Comodo certificate that belongs to the main domain.com

Perhaps github's webhook service can't process SNI? (Both the main and subdomain Apache virtualhosts include <ServerName ... >)

I then disabled the Comodo SSL Certificates, and expanded the Let's Encrypt certificate to include both domain.com and staging.domain.com, and restarted Apache. Still "Service Timeout" from github.

Does github not like Let's Encrypt? I ordered a free 30-day certificate from Comodo and applied it to staging.domain.com. No luck.

When I run staging.domain.com through https://www.ssllabs.com/ssltest, it shows the Let's Encrypt certificate, but interestingly, below that, it shows a "Certificate #2", which is the Comodo certificate for domain.com. And since the domain names don't match, that certificate is flagged with lots of red warnings. Is this abnormal behavior, and should I discover a way for the domain.com certificate to be undetectable when visiting (or analyzing) staging.domain.com?

At this point, I'm all out of ideas. I'd appreciate any and all guidance. This is also my first SO question, and I'm willing to take pointers on my question etiquette as well.

Edit 1: When activating the webhook push event, here is the output of sudo tshark -d tcp.port==443,ssl -f "net 140.82.112.0/20 or net 185.199.108.0/22 or net 192.30.252.0/22" (those are github's webhook IP's):

1 0.000000000 140.82.115.240 → <IP OF MY SERVER> TCP 74 64733 → 443 [SYN] Seq=0 Win=26880 Len=0 MSS=8960 SACK_PERM=1 TSval=2233744096 TSecr=0 WS=1024
2 0.000066037 <IP OF MY SERVER> → 140.82.115.240 TCP 74 443 → 64733 [SYN, ACK] Seq=0 Ack=1 Win=61936 Len=0 MSS=8860 SACK_PERM=1 TSval=2212655787 TSecr=2233744096 WS=128
3 0.000879665 140.82.115.240 → <IP OF MY SERVER> TCP 66 64733 → 443 [ACK] Seq=1 Ack=1 Win=27648 Len=0 TSval=2233744097 TSecr=2212655787
4 0.012202817 140.82.115.240 → <IP OF MY SERVER> TLSv1 313 Client Hello
5 0.012281121 <IP OF MY SERVER> → 140.82.115.240 TCP 66 443 → 64733 [ACK] Seq=1 Ack=248 Win=61696 Len=0 TSval=2212655799 TSecr=2233744109
6 0.013146175 <IP OF MY SERVER> → 140.82.115.240 TLSv1.2 2799 Server Hello, Certificate, Server Hello Done
7 0.231698984 <IP OF MY SERVER> → 140.82.115.240 TCP 2799 [TCP Retransmission] 443 → 64733 [PSH, ACK] Seq=1 Ack=248 Win=61696 Len=2733 TSval=2212656019 TSecr=2233744109
8 0.451700300 <IP OF MY SERVER> → 140.82.115.240 TCP 2799 [TCP Retransmission] 443 → 64733 [PSH, ACK] Seq=1 Ack=248 Win=61696 Len=2733 TSval=2212656239 TSecr=2233744109
9 0.895731268 <IP OF MY SERVER> → 140.82.115.240 TCP 2799 [TCP Retransmission] 443 → 64733 [PSH, ACK] Seq=1 Ack=248 Win=61696 Len=2733 TSval=2212656683 TSecr=2233744109
10 1.791706743 <IP OF MY SERVER> → 140.82.115.240 TCP 2799 [TCP Retransmission] 443 → 64733 [PSH, ACK] Seq=1 Ack=248 Win=61696 Len=2733 TSval=2212657579 TSecr=2233744109
11 3.551693664 <IP OF MY SERVER> → 140.82.115.240 TCP 2799 [TCP Retransmission] 443 → 64733 [PSH, ACK] Seq=1 Ack=248 Win=61696 Len=2733 TSval=2212659339 TSecr=2233744109
12 4.930201185 140.82.115.240 → <IP OF MY SERVER> TCP 66 64733 → 443 [FIN, ACK] Seq=248 Ack=1 Win=27648 Len=0 TSval=2233749027 TSecr=2212655799
13 4.930468118 <IP OF MY SERVER> → 140.82.115.240 TCP 66 443 → 64733 [FIN, ACK] Seq=2734 Ack=249 Win=61696 Len=0 TSval=2212660718 TSecr=2233749027
14 4.931240019 140.82.115.240 → <IP OF MY SERVER> TCP 54 64733 → 443 [RST] Seq=249 Win=0 Len=0

And when I look at the decrypted output from tshark, frame 6 seems to be the initial transmission, from my server to github, of SSL certificate info. Then frames 7 through 11 are 5 more retransmissions of the same info, all with no reply. After that, github just initiates closing the connection, with no errors.

Edit 2: I've since tried spinning up a test server with default Apache and self-signed SSL certificate. Same problem. I've also tried testing the webhook on my public facing site that has a fully functional, signed and paid-for Comodo certificate. Same problem.

Upvotes: 1

Views: 1204

Answers (1)

paulrw
paulrw

Reputation: 41

Github support took several weeks to get back to me, but eventually they were able to identify the problem:

"the MTU on your server is set very high and you're not accepting ICMP defragmentation responses (MSS=8860)."

Typing ip addr showed mtu setting for the network interface (in this case ens3) was 8900. This was the out-of-the-box setup of the Ubuntu 20.04 image I deployed.

Following instructions I found on the internet, I used ping -s XXXX -c1 distrowatch.com and found that any number for XXXX higher than 1452 would result in 100% packet loss.

1452 + 28 bytes for the header means that a 1480 mtu setting should work.

Knowing from the previous ip addr command that the network interface is ens3, I entered sudo ip link set dev ens3 mtu 1480 and attempted to redeliver the github webhook. Voilá, it worked.

On my system, the network config is not at /etc/network/interfaces but at /etc/netplan/50-cloud-init.yaml. So to make the mtu change permanent, I made a backup of that file, then edited its mtu down from 8900 to 1480. Finally, github webhook works!

Upvotes: 2

Related Questions