This post has been de-listed
It is no longer included in search results and normal feeds (front page, hot posts, subreddit posts, etc). It remains visible only via the author's post history.
First of, we really apologize for the issue that everybody experienced this evening, and the longer than normal time it took us to identify the root cause of the problem.
At around 8PM Geneva time, we started receiving alerts in our monitoring of servers going down. Within minutes, all servers in the entire network were reported as down. At this point, our infrastructure team immediately began looking into the problem.
We quickly realised that the VPN servers themselves were OK, and that the root cause was that our authentication servers were unable to communicate with our API.
The problem was very puzzling to our network team because for redundancy, we maintain several authentication servers in physically separate locations with different networks, and unfortunately, the problem impacted all of our authentication servers in all datacenters, which had never occurred before.
This led us to suspect there was a problem in Proton's core network (where the Proton API is located). However, coming from other networks that are not used by ProtonVPN, we had no problems talking to Proton's core network. Therefore, the issue was something particular that was specifically impacting ProtonVPN.
At this point, we began to work backwards. First, we checked the Proton API and network to make sure no security policies had been triggered that would somehow block ProtonVPN networks. After confirming that the block was not happening within our core network, we escalated the issue upstream, where we eventually found the source of the problem.
As some of you may know, the Proton network happens to be one of the most heavily attacked networks in our infrastructure, subject to frequent high volume DDoS attacks. Because of this, it is heavily protected not just by our team, but also with the help of some companies that we partner with who are specialised in mitigating DDoS attacks.
It turns out that due to the natural growth of ProtonVPN's userbase, the number of requests from the ProtonVPN authentication servers hitting our core network, has begun to look remarkably similar to a constant DDoS attack. Because of this, ProtonVPN auth servers triggered the DDoS mitigation service that is protecting our core network from attack.
As a result, network policy rules that were designed to protect our core infrastructure instead worked to aggressively block ProtonVPN servers from accessing our core network.
Once the issue was identified, we immediately worked with the upstream providers to adjust the protection policy which stopped the blockage against ProtonVPN traffic.
Unfortunately, identifying the issue, communicating with the upstream networking teams, and applying the new protection rules across the entire network took some time, and we were not able to restore service until 10PM.
Needless to say, we have now made adjustments which will prevent this from ever happening again. We apologize again for the outage and to everybody who was inconvenienced by it. If there are any questions, please let us know below.
Subreddit
Post Details
- Posted
- 7 years ago
- Reddit URL
- View post on reddit.com
- External URL
- reddit.com/r/ProtonVPN/c...