In the spirit of ongoing delivery of information to RevolutionEHR customers, this update is intended to provide deeper insight into the causes of, and our responses to service outages with RevolutionEHR.
Recently, our 99.9% uptime number has been met with some degree of skepticism. Since RevolutionEHR is a software service that is expected to be available 100% of every day, and that has been servicing customers since 2007, this percentage is accurate. However, even fifteen minutes of downtime during a patient day can feel like a significant part of the work day/week.
While 100% uptime is our goal, service outages are inevitable. When a temporary service outage does occur, we are committed to rapid response to restore service availability. We have captured a summary of recent service outages to provide insight into the wide variety of traceable causes for such events with RevolutionEHR.
We have also captured commonly asked questions from this week’s experience as well as general questions, and developed a Q&A section that we hope will provide answers and insight into our ever evolving efforts to prevent service outages and interruptions from ever occurring.
The Last Three Months
Over the past 90 days, RevolutionEHR engineers have been called into rapid response mode to resolve 3 very different root causes of temporary service outages:
- August 8: Data Center Network Mapping Error
- October 1: Data Center Hardware Component Failure
- October 29: RevolutionEHR Software Release
This level of frequency of outages with RevolutionEHR is highly uncommon. For instance, prior to August 2013, we experienced two very temporary service outages since January 2012. Two of our recent outages were a result of hardware or network issues, and one was related to an update release. The details of these 3 recent outages follow:
August 8: Data Center Network Mapping Error
Cause: LUN (LUN = logical unit number) remapped causing RevolutionEHR database to temporarily go offline
Fix: Data Center engineers fixed mapping error
Duration: 45 minutes
October 1: Data Center Hardware Component Failure
Cause: SCSI Card
Fix: Data Center engineers replaced SCSI card
Duration: 2.5 hours
October 29: RevolutionEHR Software Release
Cause: Audit log causing server input/output overload
Fix: RevolutionEHR engineers reverted back to release 5.4.7
Duration: 2.5 hours
Q&A from This Week’s Experience
Q: Why does it seem like there are always problems after a release?
A: As software releases have gone, there have been instances of releases that have required day-after patches but all of them have gone smoothly without interruption of service. The release-related issues of October 29 have resulted in a deepened lack of confidence that the system will operate normally on the day after a release.
Q: Didn’t you test this better before you released it?
A: Without doubt, we had an insufficiency in the quality assurance protocols used prior to the release of v6.0.0. The release was tested but ample consideration was not given to predict the data load increase that happened when the audit log enhancement started saving 10000% more data than it had in v5.4.7.
Q: Can you please make releases happen on the weekend?
A: We have taken this common request to heart. We realize that you are commonly in business during weekdays and any possible outages due to a release issue are most impactful then. RevolutionEHR is in use 7 days a week for nearly 18 hours per day as we cover from Maine to Hawaii. Serious consideration will be taken to moving releases to later in the week, such as Thursdays, to minimize any unforeseen release related impact on practices.
Q: Can’t you simply buy the necessary technology to make these experiences stop?
A: RevolutionEHR’s information technology plans have been in constant evolution since the product launch in 2007. The plan to expand the technology underneath RevolutionEHR has resulted in more computer hardware being utilized than ever before. The October 29 event did not come from weak or insufficient technology. When the October 1 episode occurred, we acknowledged that forthcoming technology expansion to provide more redundant computer technology in the background would have likely avoided any customer realization that a small hardware switch had failed. Ultimately, it is important to explain that there are no expenses spared in the investment in a strong technology infrastructure, and like Facebook and Amazon, even the best plans do not always avoid customer access outages.
General Q&A
Q: Can INSIGHT be automated to do system updates since I see issues in my practice before it’s updated?
A: INSIGHT is run on a completely different infrastructure on purpose so it allows for communication if/when a system problem arises. While automating messages to INSIGHT might be possible, it’s not simple. Our Customer Support team is often able to judge incoming issues via increases in Facebook community posts before we even have phone calls begin or the our monitoring has reached any warning thresholds.
Q: Why does my experience in these circumstances look like “I’m in, then I’m out, then I’m in, then out…”?
A: The application servers (there are many) that display RevolutionEHR are set up to allow one to fail and the user to get to another one, with at most a refresh of the browser. If more than one refresh leads to HTTP errors or login issues, then it is likely that viewing any data will be impossible.
Summary
Our highest priority is service availability. While 100% uptime is our goal, we reluctantly acknowledge that service outages are inevitable. We do everything in our power to limit service outages and interruptions, but when they do happen, we are committed to an all hands on deck rapid response to restore service availability to all. Our business is dependent on you being able to do your business, and it should be very clear that the necessary time and attention is being spent on preventive service and responsive actions to deliver the best possible customer experience.
As always, if you have questions or concerns, feel free to direct them to me personally at sjens@revolutionehr.com. Thank you, with all sincerity!
Scott Jens, OD, FAAO
Chief Executive Officer
RevolutionEHR
