HOW MUCH DATA LOSS IS ACCEPTABLE?

Randy Hartwig & Josh Cantrell

This article is directed towards the professional IT staff in your gaming organization, so if that’s not you, the acronyms may overwhelm some, but we will try to clarify most as we move along.

We all know that you have a great IT team – top-notch Engineers and Management that are involved in all critical processes. Recently, however, you bought a brand new, very expensive, state-of-the-art Casino Management System (CMS). It came with all of the bells and whistles: Bladed Servers connected to a new Storage Array, which is also connected to that new DR-Site your CIO built last year. Performance should be an afterthought and the entire system, when deployed, is entirely automated – right?

In this series of articles, we will try to cover recovering your System from a major event.

In today’s Gaming environment, even the smallest casino with little or no IT Staff will have a complex mixture of technologies with hardware and software from numerous vendors. Your CMS might be from one vendor or you may have a complex blending of systems that were cobbled together from numerous vendors and may include Audit, Accounting, Player Tracking, Slot Accounting, Table Games, Email, BI, Marketing, Back Office, Video Surveillance and so-on.

Let us ask a couple questions: Is your Systems vendor responsible for backing-up the System they sold you? Do you have multiple Systems vendors? Are all Systems on the same Enterprise Backup solution? Will your Systems vendor (or vendors) be responsible for restoring your system in the event of a disaster? Probably not.

The real answer is that you probably have a mixture of Hardware and Software from numerous vendors and no single vendor will take responsibility for the other vendors. But, even if all your System hardware was purchased and supported by one manufacturer and all your Systems software is from one Systems vendor, you still have the issue of ensuring your data can be recovered in a timely and efficient manner. And not just recovered, but also recovered in such a way that all your Systems software and data are aligned in such a way that the data in one product can correlate to data in the other systems or products. Add on top of this the need to be concerned about the long-term archival of data across systems to satisfy Accounting, Audit, Regulatory and perhaps the IRS. The process of maintaining these systems, or moving your older data to new technologies, will be expensive and can be time consuming so you may end up with numerous Backup/Restore technologies, only complicating this issue.

This article isn’t really about backups, but returning your system to a working state. Getting your hardware and software restored is one thing, but getting your data back to a point where it is transactionally consistent and all data is aligned is another.

So, how do we get all this data to correlate? First, we have to understand our business requirements and terms like RTO and RPO.

RTO, or Recovery Time Objective, is the target time you set for the recovery of your IT and a business activity after a disaster has struck.

• E.g.: Your system went down and you have 4 hours to get it back, restored, and working.

RPO, or Recovery Point Objective, is focused on your data and your company’s level of acceptance towards that data loss. Your RPO is determined by looking at how long between data backups and the amount of data that could potentially be lost before the business is impacted.

• E.g.: Your system went down and you were able to restore your system in your 4-hour RTO, but because you only had backups from the night before, you could only restore to 10 p.m. last night. That being said, you lost 12 hours of data. Is 12 hours of data loss acceptable?

Ok, now that we know what RTO and RPO are, what exactly can we do with this information?

Well, it’s most likely that your Systems vendor will not be there with you at 2 a.m. when the System goes down, but you can probably get them on the phone. You restore your system, but the floor IT is not coming up correctly. You call your Systems vendor again and they inform you that your data is out of sync. But you just restored your system, so how could your data be out of sync?

It really becomes obvious when you understand that back-ups happen in a staggered manner; your CMS database at 4 a.m., your Player Tracking database at 5 a.m., your Point of Sale database at 5:30 and so on. Some are full backups and some are full and transactional, but that transactional backup occurs only every 4 hours.

As you can see from the times above, not all systems or databases are usually backed up at exactly the same time. It’s just not practical. They are usually staggered as this would probably require more performance than your backup system or storage array could handle. Even if it’s possible to back-up everything at the same time, you still have transactions coming in while you are backing up, so getting all systems restored to the same point in time or, at least, very close, is a big task.

In our next article, we will talk about the different types of backups you can and should perform, how often to back-up the various data file types, and ways we can try to ensure our data is in sync once the restore process begins in order to mitigate as much data loss as possible and keep those Auditors, Accountants and your Management happy.

Randy Hartwig has over 30 years of experience in Electronics & Information Technologies with the last 20+ years in the related Gaming Industry IT Technologies field. He is currently VP of Technology and founding partner at Systems Performance Partners and can be reached at http://systemsperformancepartners.net.

Josh Cantrell is Founder of and Senior Slot Systems Consultant for Sierra Gaming Consultants (www.sierragamingconsultants.com). He specializes in slot operations, slot accounting / analysis, and slot system support. He has over 25 years of experience working in casino markets around the world including those in Canada, Mexico, Macau, South Africa, and throughout South America.