April 29th Service Outage

Dear New York-From one New Yorker to another, we would like to apologize for the outage our users experienced on April 29th. We understand how painful a disruption of your commute can be. As most of you know, PATH train service was suspended on all three lines during rush hour on April 29th. PATH users had to find an alternative way home and overwhelmed the New York Waterway infrastructure. Waterway dispatched additional vessels but the situation was compounded by their communication network going down, which took out Waterway’s ticket vending machines as well. Unfortunately our mobile ticketing app servers couldn’t handle the rapid surge, hitting levels 800% above our peak capacity. Simply put, we could not add additional resources fast enough. As one server came online, others went down.Our system is engineered for scalability and has performed at a high level in similar situations after Hurricane Sandy. The surge in traffic encountered that night was unprecedented and an experience that will only improve our processes going forward. Our emergency response plan is evolving from these lessons learned, it will be more comprehensive and improve the performance of our scaling methods. Not only have we added capacity but we have redesigned components of the backend to be more efficient and to scale faster and at lower thresholds.We are also hard at work getting 2.0 out the door, which will dramatically reduce the load on our servers by adding many offline features. We want to thank everyone for the rapid adoption of mobile ticketing in New York and we can’t wait to show you the next version.Thank You,Micah BergdaleCEO Bytemark