On behalf of the RevolutionEHR executive team and our entire company, I wanted to take this opportunity to update you on this morning’s outage. The experiences and your feedback that you sent were taken in and digested by all of us, and I want to apologize to each of you for its impact on your business.
This morning at approximately 8:15am CST, we began to experience performance issues with the servers that deliver the RevolutionEHR system to your computer, causing it to be unavailable to most users. An intense, full-team assessment was initiated immediately, consistent with our policy to address all situations that could seriously impact you. We determined that a change made as part of the 6.0.0 release did not respond properly to high usage of the application and caused this system instability.
After several attempts to isolate and fix this problem completely, we decided to remove the 6.0.0 release and reverted back to release 5.4.7, which is the version that all users were on up until last night. By 10:45am CST, version 5.4.7 was back in operation and RevolutionEHR was again available and operating normally. This instance only affected the ability of the servers to display RevolutionEHR to the users and did not have any impact on the data stored within the system.
As always, we received passionate input from our customers with appropriately critiquing commentary. Please know our co-founding CTO, Jim Schneider, and I take every action we perform with great seriousness. We have appreciated our long-standing experience of satisfaction within our customer base, and have spent the time and money that we believed would give you a seamless, positive experience with RevolutionEHR.
So, what happened this morning to create such headaches and dissatisfaction? I take personal responsibility for a team that I directed to enhance RevolutionEHR for EHR certification, as we severely underestimated the amount of data that would be stored to support the significant increase in the amount of audit logging required for certification. Specifically, in the past we stored small bits of audit logging data – for instance, for all of yesterday, the total audit log data stored was 54MB. Today, with the increased audit logging, from 8 am – 10 am CST we had stored 5GB — an increase of nearly 10000%. The excess logging took down the data input/output system that was trying to write that much data to our database and to our replication/reporting servers.
We tested this system prior to release and did not have the load changes properly anticipated. This is something that we need to address as it is at the core of the many concerns we have heard about improving testing. In an update that we will send later, we will address this along with the many other comments that have arisen including preferred dates for releases.
We know this morning’s outage was very painful for you and it was very humbling for us. Please know that we consider any unscheduled disruption in the availability of RevolutionEHR as unacceptable. It’s not OK on any level and the availability of RevolutionEHR to all users is our highest priority.
In the coming days, we will be sharing with you more information on the specifics of this morning’s event as well as others in the recent past so we can help everyone understand that they are not all alike, and to explain the numerous efforts and resources that we are exhausting to retain your satisfaction.
Best personal regards.
Scott Jens, OD, FAAO
Chief Executive Officer
RevolutionEHR
