Balance Scale

Back in late 2011, I wrote about the use of low-cost NAS devices as mission critical file shares. That blog post generated quite a bit of discussion with our Pro-Partner technicians around the country. While most agreed with the premise of the article, several others asked a related question:

If my client is running low on server disk storage, can I just purchase an external USB disk and attach it directly to the server?

While this seems like a reasonable solution, and it may work for some applications, its probably not the way you want to go with a mission critical server. While external storage (and NAS) devices are good for data backup and archival, they may not be great for a busy server environment.

All disks are not created equal.

A small  (2.5 inch) SATA (Serial ATA) mini-disk which gets power from an external USB port, is typically “spinning” at 5,400 RPM (rotations per minute). These disks can handle a workload of approximately 65 IOPS (input/output operations per second).

Go with a larger (3.5 inch) SATA drive with an external power supply, and you’ll get 7,200 RPM which equates to about 75 IOPS – a 15% improvement.

While these may sound like reasonable numbers, keep in mind that many business-grade servers have “beefier” throughput rates that rely on SAS (Serial Attached SCSI) disks. These disks are typically spinning at 10,000 or 15,000 RPM and have individual IOPS of approximately 140 and 175 respectively.

In other words, an external disk connected to a USB port could have a throughput rate which is less than 50% of the original internal disks. (Ouch!)

If you run low on server disk storage, and cleanup is not possible, your best bet is to add additional internal disk (if possible) or to migrate your existing data to new internal disks of a larger capacity – with the possible aid of a high-capacity NAS device !

While a faster internal disk may cost a little more than an external SATA drive – users will likely feel the difference if you go “cheap”.

At first glance, it would seem that most online backup services are pretty-much the same. They provide software you load on your computer to identify files for backup. They establish an Internet connection to a cloud-based storage server. Finally, they transmit an encrypted version of your data to off site storage for safe-keeping.

So, all things seemingly equal, it’s not surprising that some businesses go “price shopping” for service. But can (or should) the vendor who has the cheapest price, or the largest storage package, be the one to trust with your most valuable asset…your critical business data?

Perhaps not.

Generally speaking, there are two classes of online backup offerings available in the marketplace: self-managed and provider-managed – and they are significantly different.

Self-managed offerings tend to be very inexpensive and are directly marketed to home users or owners of very small businesses. They are simplistic in nature, easy to install and require a very low level of user sophistication. You’ve probably seen them advertised on television.

When operating properly, a self-managed online backup seems to do a reasonable job of backing up simple data files such as word documents, music downloads and digital pictures.

Home users don’t typically own Windows servers, don’t care about backing up SQL databases or Exchange message stores, don’t use complex financial software and don’t have regulatory concerns. For these folks a very simplistic backup service, at a very low price, seems reasonable. It’s certainly better than no backups!

A self-managed online backup service is often referred to as an “unmanaged” service. It’s unfortunate when a user of an unmanaged backup service naively assume that a charge on their credit card equates to a properly functioning backup. (Yikes!)  Little do they realize that if a data disaster occurs, there’s a very real possibility that “supposedly” protected data could actually be lost.

Any business that uses an unmanaged service risks permanent loss of critical customer or financial information that could devastate their organization. (Ouch!) Businesses should strongly consider opting for the safety-net offered by a provider- managed online backup service.

Managed online backup services are marketed to small-to-medium businesses that have critical data – but may not have the time or in-house expertise to implement and manage a business-class backup service.

A “S.M.A.R.T.” managed backup service helps ensure critical data is safe by augmenting fully-featured software with these five essential service components delivered by specially trained technicians:

Setup Service – Proper setup of the initial backup is critical. Not only must software be installed, but critical data including: files, folders, email stores, databases, vertical application data and more must be identified. Then backup jobs can then be defined, schedules set, and appropriate data retention policies established. No two computers are identical so it’s important that the service provider and customer work closely together during the setup phase.

Note: Rather than taking days or weeks to perform an initial data upload, managed service providers often choose to use portable media to physically transport encrypted data to off site storage servers. This is referred to as a “seed” backup.

Monitoring Service – Managed service providers understand that there is no such thing as “set it and forget it” in the computer world. Servers, PCs and laptops are in a constant state of change. What works today, may fail tomorrow. Log files must be regularly examined for errors, storage usage must be tracked and PC health problems (such as low disk space) must be reported.

Alerting Service – Proactive notification of problems is the cornerstone of any managed backup service. When backups run with errors, run “empty” or do not run at all, the managed service provider must alert the user, and in many cases, directly assist in issue resolution.

Recovery Service – The full benefit of a managed online backup service is often not realized until disaster strikes. It’s at this time that users are most vulnerable and often unable to think clearly. A managed service provider will calmly walk the user through the recovery process and actively assist in data restoration.

Testing Service – Periodic “end-to-end” testing of online backups is one of the best ways to assure you are prepared for a disaster. A managed service provider can help perform a “fire drill” restore of critical data files. This can help establish a realistic Recovery Time Objective (RTO) and Recovery Point Objective (RPO) for the organization. Known recovery procedures and time lines generally lead to a smoother and more predictable restoration service.

In summary, the combination of fully-featured backup software plus the value-add of a S.M.A.R.T. managed backup service provider helps ensure that important data is backed up properly – and can be restored when the business needs it!

These days, a top-notch managed online backup service with a healthy allocation of online storage can be purchased for about $1-$2 per day.

Isn’t your business data worth the cost of a good cup of coffee?

 Canaries were once regularly used in coal mines as an early warning system. Toxic gases such as carbon monoxide and carbon dioxide in the mine would kill the bird before it affected the miners.

Because canaries tend to sing much of the time, they would stop singing prior to succumbing to the gas – therefore alerting miners to the danger. Hence, the phrase “canary in a coal mine” is frequently used to refer to a person or thing which serves as an early warning of a coming crisis.[1]

Much like a chatty yellow bird, a business-class online backup service is actually a canary for your computer – protecting your valuable information from a variety of threats.

For example, at the completion of each backup job, Dr.Backup sends you an email message. The message provides you with the overall status of the backup such as successful, completed with exceptions, incomplete or empty.

If the subject line in your email message says “Successful” that generally means offsite backup of your data completed without a problem. An email message with a subject line containing the word “Exception” may be an early warning sign of problems.

Abnormal conditions detected are usually reported in a log file attached to the email. If you open this attachment, you may find indications of low disk space, inaccessible drives, incomplete backups, files left open, corrupt data, non-standard filenames, low system resources, incorrect file permissions and more.

Contact your local IT consultant or Dr.Backup support if you require assistance interpreting the information contained in your detailed log file.

If your computer is turned on and you don’t get a daily email message, then something else may be seriously wrong.

In other words, when the Dr.Backup canary isn’t singing…danger might be in the air !

[1] Wikipedia – http://en.wikipedia.org/wiki/Domestic_Canary

From time to time I get asked if it’s a good idea to use a low-cost NAS device (Network Attached Storage) as primary disk storage. In most cases, I say NO. Let me explain.

In an office environment, PC users frequently need to setup public folders so that information can be easily shared between workgroup members. One common approach is to install a dedicated Microsoft Windows server.

However, even an entry level Windows server will cost over $1,000 to purchase and probably an equal amount (or more) to configure. Ongoing management costs could also be substantial — making the Windows server a “budget-buster” for some small workgroups.

With a dedicated Windows server ruled out, one member of the team usually steps up and suggests the group consider an inexpensive NAS device. The logic is that by connecting a NAS device to the local network, end users can create a “mapping” from their local PC to a shared storage area on the NAS.

NAS devices can be extremely easy to deploy, are often managed via a web browser, and offer lots of hard disk storage for very little money. So, on the surface, a low-cost NAS device with a large storage capacity seems to be an ideal workgroup solution.

But, before you go out and buy a low-cost NAS device, let’s take a look at some of the negatives of this approach:

1. It’s “low-cost” for a reason – Low-cost NAS devices are often made with the absolute lowest quality parts. Customer support is frequently non-existent (swap out service only). Judging from my personal experience, many of these boxes fail in less than a year – and some within days of being deployed.

2. Performance can be slow – Random access performance of many NAS devices could be just 10% of what you would see compared to your own computer’s internal disk. Yes, NAS will attach to your 100 Mbps LAN, but the transfer rate is likely to be much slower.

3. Disk failure means data loss – If the disk inside of a NAS device fails, your shared workgroup folders are gone. Are you willing to put your team’s most important information on the lowest cost — and least reliable — form of storage?

4. Incompatible with live databases – Microsoft SQL Server databases are commonly used by Windows application software. You can’t use your NAS device (mapped drive) to host an SQL database. Might your applications require SQL? (now or in the near future.)

5. File create/modified times out-of-sync – The internal clock used in many low-cost NAS devices often fails to keep accurate time. This means that file timestamps may not be correct. On client systems, I often see file creation dates of January 1, 1990 12:00 AM – indicating a malfunctioning internal clock on the NAS. Incorrect file timestamps could cause various file synchronization tools to badly fail…sometimes resulting in unexpected data loss.

6. Lack of incremental backup support – Many NAS devices do not support the Windows standard “archive flag” feature. The archive flag is set on a file when content changes occur and let’s a smart backup program know if an individual file backup is needed. Without archive flag support, time consuming full backups of the entire device are often required. Any file which is left open (in use) on the NAS device cannot be backed up at all (until it is closed.)

7. Can easily be stolen – Many NAS devices are light-weight and small enough to be slipped into a coat pocket. It’s just too easy for these devices to walk out the door…along with your valuable data.

8. Maximum file size of 4,000MB (4GB) – Some low-cost NAS devices support individual files upto 4GB in size. If a large file copy to your NAS device fails, you should immediately return the device to the store where you bought it. Many multimedia files, backup archives, email archives, etc. routinely exceed 4GB in size. This restriction is just unacceptable in the 21st century!

So given all these negatives, what I would suggest?

If you have less than 10 devices in your workgroup, you might choose to nominate a lightly-used workgroup PC as your shared storage device. It’s highly unlikely that the person using this PC will even realize its dual purpose.

First, put a UPS battery on the device ($50) so that it stays permanently powered. Next, allocate some internal disk storage to be used as your shared workgroup folders. Finally, setup file security to permit shared access.

Some call this solution a “poor man’s server” but remember, before Microsoft introduced their server operating system, peer-to-peer workgroup solutions were common. One added benefit of this approach is that a group printer can also be shared — providing a place for wireless devices (like iPads) to print.

If you’ve already purchased that low-cost NAS device, why not just use it for local backup…since you can never have too much backup!

When you read the specifications on most modern disk drives, you’ll likely come across a statistic called MTBF. MTBF is the “Mean Time Between Failure” or the average time before the device is likely to fail.

Disk drives usually carry an MTBF of over 1 million hours. Given that there are 8,760 hours per year, I suppose using simple math, you could say that an ‘average’ drive should last 114 years (1,000,000/8,760).

Well, it’s pretty obvious from these numbers that nobody actually “tested” a drive to see if it would fail in 114 years…since disk drives haven’t even been around that long!

Rather, MTBF must be a theoretical calculation that an engineer “reasoned” based on some assumptions about the underlying component performance inside a drive.

The excerpts below from one recent study confirmed what we all know.

Customers replace disk drives at rates far higher than those suggested by the estimated mean time between failure (MTBF) supplied by drive vendors, according to a study of about 100,000 drives conducted by Carnegie Mellon University.

The Carnegie Mellon study examined large production systems, including high-performance computing sites and Internet services sites running SCSI, FC and SATA drives. The data sheets for those drives listed MTBF between 1 million to 1.5 million hours, which the study said should mean annual failure rates “of at most 0.88%.” However, the study showed typical annual replacement rates of between 2% and 4%, “and up to 13% observed on some systems.”

Ashish Nadkarni, a principal consultant at GlassHouse Technologies, a storage services provider in Framingham, Mass., said he isn’t surprised by the comparatively high replacement rates because of the difference between the “clean room” environment in which vendors test and the heat, dust, noise or vibrations in an actual data center.

The studies won’t change how Tom Dugan, director of technical services at Recovery Networks, a Philadelphia-based business continuity services provider, protects his data. “If they told me it was 100,000 hours, I’d still protect it the same way. If they told me if was 5 million hours I’d still protect it the same way. I have to assume every drive could fail.”

Bottom Line: Don’t put too much trust in MTBF numbers. A 4% failure rate means that 1 in 25 hard drives will malfunction in the coming year. The physical drive will likely cost less than $200 to replace. However, the information on that drive could be critical to your business and may be irreplaceable.

Backup your critical business data every day, automatically, with Dr.Backup.

—-
Excerpts from: Robert L. Scheier, Computerworld – http://www.pcworld.com/article/129558-2/study_hard_drive_failure_rates_much_higher_than_makers_estimate.html

Each month, Dr.Backup clients perform over 70,000 backup jobs resulting in millions of files being transferred onto our storage servers. Overall, the process runs very smoothly.

However, not every scheduled backup progresses to a successful completion. Sometimes jobs don’t start. Other times, a file or folder will “error out” and fail to transfer.

I estimate that each night, about 2% of all backups incur some sort of issue. Sometimes the problem is transient and clears the next day…but many times, it requires technical support intervention.

This past week, some of the problems we encountered included: a PC that was rebooted during middle of a backup, a disk drive that was completely full, a disk drive that started to malfunction, a new firewall configuration that blocked access to the Internet, file permissions on a server that were incorrectly set, Windows open file software was not functioning, and a user unintentionally backing up a 60 GB virtual machine image over a (slow) DSL connection.

We found one company that purchased a new server and completely forget that the backup software needs to be moved from the old box to the new!

Automated monitoring and support tools help Dr.Backup technicians identify the small percentage of backups that are troubled. This allows us to focus our attention on fixing (vs. finding) troubled backups.

Once we identify a problem, we notify the client technical support contact and actively work with this individual – or their IT consultant – to help ensure the next day’s backup runs clean.

Each client situation is different and must be looked at individually. This is sometimes a time-consuming and labor intensive process (which is completely ignored by most basic “econo” backup services.)

By monitoring backups – and proactively taking steps to fix problems  – we know that we are greatly enhancing our clients’ ability to recover from a data “mishap”.

All of us here at Dr.Backup understand that you depend on us to be there when all else fails. Proactive monitoring and repair of online backups is just one way we’ve got your BACK !

Needless to say, Dr.Backup‘s back office computers perform nightly online backups of our business-critical data files. However, in addition to the off-site backups, we also depend on a local backup to periodically snap an image of our critical systems.

One of our back office computers runs Microsoft Windows Server 2008. It’s a fairly new server and hasn’t really given us much trouble. Recently, things got weird so I figured it was time for a reboot.

After issuing the restart command, the console turned blue and displayed the “shutting down” message. I watched the swirling icon on the monitor for about 30 minutes, and finally figured it was time to move on — so I pushed the button and cycled the power.

Soon after, beeps and whirls could be heard. The system restarted and things seemed back to normal. But were they?

The next day, fearing that a forced reboot and failure to restart could be a harbinger of things to come, I wanted to make double sure that my local Windows disk backup was still functioning correctly.

It took hours to build our system and I certainly didn’t want to start from scratch if ‘Murphy’ was gong to rear his ugly head! After scrounging around, I found the standard windows backup status display. Below are screen shots of what I saw.

The most recent local backup did fail. Had the reboot issue not prompted me to check the backup status log, who knows when/if I ever would have noticed that there was a problem. A real emergency would have likely resulted in a sleepless night of server rebuilding!!!

As it turns out, my backup disk media was an external USB drive. When I was forced to do the power cycle, I cycled the server box – but NOT the USB drive. When the server rebooted, it did not recognize/mount the USB drive. This caused the local backups to fail.

Windows did log an innocuous “backup failed” message, which was tucked away in the voluminous event log…never to be seen by human eyes.

In this case, the fix was easy. I simply unplugged and replugged the USB drive into the computer. It was immediately recognized and available for the next backup – which was successful. (Of course the problem could have also been a failed external drive requiring hardware replacement — but not this time.)

This event got me to thinking about how many of us depend exclusively on unmanaged local backups to protect critical business data. I wonder what percentage of the local backups are “broke” right now? …and how many businesses are completely vulnerable to extreme data loss and don’t even know it.

A business-class online backup service like Dr.Backup provides a valuable safety-net for critical data files. Had Dr.Backup‘s own back office computer crashed, it would have been a nuisance to rebuilt from scratch – but at least we would have been able to download our critical data from the Internet and continue to serve our clients.

For most businesses, off-site backup should be thought of as a cost of doing business…and not just additional overhead they can do without! A meaningful cost justification can be found here.

These days, more and more businesses run mission-critical applications which depend on complex database technology. Some examples include point-of-sale processing, patient electronic medical records (EMR), inventory control systems and financial accounting packages.

If any of these applications fail, things get nasty quickly! To prevent permanent data lose and to minimize downtime, it is very important to take regular backups of all mission critical databases.

Unlike basic word processing documents or spreadsheets, backing up applications that use embedded databases is tricky. The care you take in preparing and testing a database backup may determine how easily you bring the critical application back online.

When performing off-site/online backups of a database application, there are four approaches to consider.

Option 1 – Manufacturer’s instructions. Upon request, software vendors will usually provide you with written instructions on how backup their application database(s). In some cases, the application itself is able to schedule and perform a backup to local disk which can then be moved off-site. When available, following the software vendor’s instructions is usually a  safe way to proceed.

Option 2 – SQL backup. More and more applications now store customer data in Microsoft’s SQL (Express) database server software. This database software is installed along side the client’s vertical application. Advanced online backup programs (like Dr.Backup) contain special “agent” software which can work directly with Microsoft SQL to perform a live backup without actually closing down the database application.

Option 3 – Closed file backup.  When you completely shut down an application, sometimes the underlying database will close. This permits the use of standard backup techniques to capture the files containing critical information. A shutdown/restart database procedure can often be scripted permitting online backups to execute nightly without intervention.

Option 4 - Open file backup. In some cases, it’s just not possible to easily close a mission critical application database. Rather, the underlying files must be “snapshot” archived while the application is live. Special “open file agent” software may be required to safely backup these continuously open data files. Databases which remain consistently open, are very tricky to properly backup. Often, they require a FULL backup of select folders – a feature available only in business-class online backup programs like Dr.Backup.

Selecting the proper approach – and performing a practice restore to a test machine – greatly increases your chances of quickly restoring a database application to production after a data disaster.

Each day, 100′s of new viruses are “born” on the Internet. In most cases, these nasty pieces of malware are trapped by anti-virus software before they can infect your computer. But, sometimes viruses wind up on your disk – where they are seen as ordinary data files and are candidates for backup.

If your online backup program attempts to open an infected file, access to this file can be restricted and your backup program may be forcibly terminated or simply “suspended” by the anti-virus (or malware) application. This results in “stalled” or incomplete backups.

Sometimes, suspended backups on unattended PCs can go undetected for days — or even weeks. This leaves you unprotected against a data disaster…on a computer infected by one or more viruses!

Ideally, like Dr.Backup, your online backup service will notify you each time you complete a backup AND notify you if you do NOT backup for any extended amount of time. In this way, you know that your automated backups will be there for you in an emergency.

NextUp: Firewalls can block online backups

I was recently asked what I thought was a fairly simple question: What file do I select to backup my Quickbooks company data?

Being the smart person I am…I immediately fired back a short email saying “just backup the .QBW file”. Seemed like a simple enough answer.

But then, I got to thinking that maybe it wasn’t quite such a ‘simpleton’ question. Below is a snapshot of what I got when doing a file search for “QBW” on Dr.Backup’s own back-office server.
Turns out there were multiple QBW files. One needs to be quite careful to select the proper file to backup!

The file that I *must* backup is called DrBackup.QBW. It’s the so-called “company” file and contains the critical financial information needed by the Quickbooks program. In most cases, this file will be the largest Quickbooks data file. Automatically backing up this file with Dr.Backup is highly recommended.

The DrBackup.QBW.TLG file is referred to as the database transaction log. This log is a record of all Quickbooks transactions that have occurred since the last Quickbooks full internal (local) backup. It can grow larger in size than the company file itself – sometimes to 1000′s of megabytes of storage. Making an online backup of this file is not generally required. Please make sure you don’t confuse this log file with the actual company file – you can’t recover a working Quickbooks configuration from only a transaction log.

The DrBackup.QBW.ND file shown is actually data used by the Quickbooks Database Manager software that tracks which users are currently logged into Quickbooks and making changes, i.e., multi-user mode users. Quickbooks restricts the number of simultaneous users who can access the company file based on license counts. This file can be re-created by Quickbooks on the fly – no backup is needed.

So now you have the “real” short answer on what Quickbooks file to backup.  But, there’s always more to the story.

Do you know what a Quickbooks “.QBB” file is? What about a “.LGB” file? Is it necessary to backup these files if they are on your system?

Click the comments link below and let me know what you think.