Does webhook delivery properly deal with multiple hosts behind a single DNS name?


#1

If I put my webhook receiver behind a CNAME that points to a couple of hosts, will the webhook delivery code understand that, and pick one at random?

I guess my question really is: if I want to improve the reliability on my side, if i want to improve the chances of receiving a webhook even if one of the servers is down or unreachable, what do I do?

What does Github recommend?


#2

@st3fan DNS is not my strong suit, but I’m going to try to help figure this one out.

Do you mean having a load balancer? Because my understanding is that if you have a load balancer, then it’s the load balancer doing the choosing. This would be invisible from the point of view of the webhook.


#3

I think the question is: what happens if DNS round-robin is used? Will HookShot re-check the DNS every time it needs to deliver a webhook?


#4

DNS round-robin would be handled at the nameserver level, I believe, not within our webhooks infrastructure.

As long as the name servers are handling the round-robin correctly and DNS caching isn’t causing all the traffic to go to one IP address, then I can only imagine that the webhooks would end up where you expect them to.

I will ask internally if we have any recommended best practices to improve reliability on the receiving end.


#5

I chatted with a colleague about best practices and the TL;DR is that we don’t have any specific recommendations. One engineer I know who uses GitHub webhooks quite heavily for his project has set things up so that all deliveries get captured in a queue, so that if he fails to process a delivery he can troubleshoot and redeliver more easily than manually navigating to a settings page on GitHub. Other folks use load balancers.

I’ve confirmed that our webhooks infrastructure does not pay attention to anything at the DNS layer—it fires off the payload to the configured URL, and that is all. If the delivery fails, then you would need to manually redeliver the payload by navigating to your hook’s setting and pressing the Redeliver button.

Troubleshooting these things can be a bit finicky, so if you are seeing problems with your deliveries then you can always contact support@github.com. Our support engineers can check the logs on our end and otherwise help figure out what is going on, whether it is on your end or ours.


#6

Thank you this is good info.