We updated to the new Rails version following the warnings about the vulnerabilities that had been patched in the new Rails version. Fearing the hackers we performed the updates as well. One can never be too safe.
The result: A new security issue!
One of our private beta testers on the Browserbite Recorder cross browser testing service messaged me privately a day later saying that the search feature is giving results “that should not be there”. I managed to reproduce the issue within seconds. Indeed – just performing an empty search in the requests showed all the requests from all the users. Luckily the breach did not reveal anything that was not public yet.
Quickly switching to damage control we first identified that this particular feature was touched around 4 months ago. So what had changed? Many have heard of the recent issues regarding GitHub’s infamous “Today’s e-mail incident“. We fell in the same hole but in a different manner. You know the answer to what had changed: Rails. To be honest, one should regression test the application every time there’s an environment update. Nostra culpa!
We had two options: surrender to the hackers and downgrade Rails or produce a patch. We scanned the rest of the application for potential security vulnerabilities related to the issue and it looked like there was none.
Our swift-fingered developer produced a quick patch and deployed it. Our users are safe. The incident had only one user seeing a short list of web addresses that are public anyway – so luckily no big harm done. There was no option to see the actual captures of the cross browser testing results.
Now moving on to the next question:
How can we avoid happening it in the future?
It looks like we’re stuck between two bad options.
- Roll with the updates and risk security risks.
- Don’t roll with updates and risk security risks.
We have chosen the first path since we better rely on the regression sets of the Rails developers. Everyone makes mistakes and I believe they’ve learned from it. But from now on we will do some extra regression checks after we update the environments as well. Better safe than sorry.Posted by