The dreaded Blue Screen and its aftershocks are well known by all of us. As one Alberta dental group (The “Dental Provider”) learned too late, there are other critical malfunctions that you and your systems administrator need to be aware of – in this case, an irretrievable disappearance of the patient database. On the positive side, this can be avoided. Ultimately, you are accountable for taking specific actions as owner or manager to maintain a secure database. Above all, fulfilling the ultimate privacy and goodwill of your patients and compliance with your regulator’s strict requirements.
Complacency
This case study examines an actual and recent patient database and digital imaging loss by a Dental Provider (functions and details altered to protect identity). This non-hospital surgical facility, which had a high daily volume of patients, hired its own IT company (SA) and mistakenly assumed they were familiar with and trained in backup protocols that would have prevented severe data corruption. This Dental Provider relied on their dental software provider for backups, which also failed them. When both the main system and the cloud failed, the SA had no solutions. The result was the complete loss of several months’ worth of data, costing approximately $2 million. The Dental Provider was forced to instigate workplace changes, turn away patients, and re-strategize. Benson Hunt (a specialty cyber security firm in Alberta) were consulted, and they were successful in recovering some of the lost data. They also provisioned ITIL protocols to prevent future crisis.
How to avoid this kind of crisis
Who is accountable for screening and hiring your IT to ensure appropriate skillsets and business risk management knowledge? What are the essential certifications? Who, if anyone, monitors backups and notifications? In addition to your IT, should you be hacked or suffer a database integrity issue? Who is ultimately accountable? In the case of The Dental Provider, their SA lacked the certifications and expertise necessary to install a failsafe framework.
Here are some critical actions and policies that you can implement to avoid costly crises:
1. Ensure that your systems administrator is ITIL (Information Technology Infrastructure Library) certified, preferably Version 2, 3, or 4.
a) ITIL is an international technical resource library created in the 1980s by the British Government in 1989. ITIL offers practices and protocols for systems administration to create and enable a business continuity plan. And in case of a catastrophic failure, a continuity plan must be in place. This involves hardware, software, and business processes along with implementation of a future 3-to-5-year technology plan. Unfortunately, there are few IT firms in Canada adopting these protocols. The versions ITILv2, v3, and v4 reflect improvements of the protocols over the years.
b) While there are other frameworks in existence, ITIL is favoured by Benson Hunt for the business continuity. We employ ITIL architects, and all systems are designed by ITIL analysts. Example industries include public healthcare, law firms, manufacturing and oil and gas.
2. 3/2/1 backup system: 3 copies of data (the original and 2 copies), 2 different media types (i.e., 2 storage mediums), and 1 copy stored off-site. In an ideal system, backups should be performed at least nightly. Additionally, notifications should be sent to your systems administrator, as well as to the owner and office manager. While regular notifications may be annoying to some, having multiple pairs of eyes on the process is good insurance. Dental practices are notorious for not testing backups or sending notifications. Benson Hunt’s practice is to customize notifications so that assigned management can validate backups and retrieve data independently of the systems administrator.
a) Network Attached Storage (NAS). A backup unit that cannot be tampered with on site or remotely. It is not a hard disk (HD) because a HD does not offer security features available to NAS. NAS mitigates cybersecurity risks due to multifactor authentication and data encryption.
Key features of NAS:
- Storage
- Security
- Reliability
- RAID
b) Big picture. Many SAs will repair one issue that has arisen only to leave a giant hole elsewhere because they haven’t thought through consequences of what the ITIL framework demands. It is simply lack of awareness.
3. Backups should be tested quarterly with daily notifications. In the case of the Dental Provider, this procedure was not in place. In addition, your SA should, in a controlled environment, only have the ability to deliberately corrupt the server completely and then restore it in a reasonable and timely manner.
Backups are your last line of defense, and while they may not always prevent hacks and crashes, they are a RECOVERY TOOL, after the fact. You are in constant danger of hacking by Botnets as well as covert attacks. For example, you may be sent an extraordinarily excessive number of packets that your system can’t handle, resulting in no internet, no routing, no access to data.
You and your team may be the target of social engineering, like hacking, which is rapidly becoming more commonplace. Social engineering is a technique to take advantage of human error.
There are no official statistics available for dental practices who have lost their data, for obvious reasons. The SA may disappear and/or the dental providers don’t reveal the mishap, in which case rectification nor restoration of damages is possible.
4. Granular backups. As a dental provider, you are a highly trained and skilled dentist, and your patients are of the utmost priority – including their privacy and medical information. Obviously, time is of the essence, but there are over 100 software programs designed for dental offices using SQL as the “language” for their database, and choosing the best is challenging. Hundreds of dental programs also use a variant of SQL database language for writing their program. These include AkituOne, Cleardent, Dentrix, OpenDental, Tracker, Ortho2Edge and Curve, as some examples. However most fail to incorporate granular backups wherein each patient is recognized as a granular datapoint. If such a singular datapoint fails, it can easily be restored. Conversely, without granular/singular datapoints, a dental provider would be unable to locate the specific failure. In short, choose a software that follows the protocols of ITIL, including the granular and the 3/2/1 backup system. Alternatively, your computers may well deny access to patient information/attendance, critical business information, lost accounting records, X-rays, medical data, and you will be unable to launch any software. YouTube will be your sole consolation prize.
5. Beware of dental support companies who do not employ these critical protocols. Some sell inadequate hardware not specified for unique business needs. They sell “servers” to clinics, which are standard computers that do not follow the ITIL framework. Service is minimal and when it fails – and it will – your SA will look like a hero. But only for a while.
6. Insurance coverage. Contact your insurance company to ascertain whether you have data loss or similar insurance coverage whether directly as part of your policy or indirectly as part of a package offered by your association. An added benefit to this is that many insurers will issue data loss insurance, but they will require that the dental provider adhere to certain IT operational conditions such as MFA and Encryption.
Benson Hunt maintains contact with specialty insurance companies. However, it is important to realize that losses of this nature will never be fully insurable or recoverable and the dental provider will lose much more in terms of its customers’ goodwill. So, it is best to initiate the changes that we suggest in the first place.
7. Re-examine your operations annually to keep up to date; check for loopholes and personally monitor your SA’s currency.
Note that we have not used the word “hacker” or “ransomware” in this article. Our investigations have shown that owners, SA, and stakeholders should always be vigilant about negligence, incompetence, and the consequential internally caused losses. Ransomware hackers are constantly assessing high income targets remotely for the same vulnerabilities; if you have them, they will find them.
Positive outcomes
There was a happy ending for this case study: a year after it adopted Benson Hunt’s protocols, a software vendor accidentally, or intentionally, sent a bug update to the Dental Provider, crashing the system. The Dental Provider, which employs 80 users, was ready. Within an hour, we were able to restore the server because their backup was resilient, robust, and regularly tested. Be warned: some practices have been led to believe that by merely copying the external hard drive plugged into a server, they are running properly. In reality, nothing may be working as intended.
Your SA is there to advise, provide, and work collaboratively with you. Safeguarding your patients’ information is your responsibility. So, in the end, the buck stops here – with you.
Disclaimer: Details in this case study, including the name of the company and specific technical issues, have been altered to protect client confidentiality. Any resemblance to real entities is purely coincidental.
Please note: The authors of the article are currently employed by Benson Hunt.
About the Authors
Tony Phung is a seasoned entrepreneur in technology.
Jason Tu holds ITIL certifications and over 10 years of scalable technology experiences. Together, they design custom solutions tailored for unique business needs while ensuring the designs follow Microsoft best practices and ITIL standards.