Located in the heart of hurricane territory, the Louisiana Organ Procurement Agency (LOPA) has little tolerance for downtime. That's why the organization whose business is dealing with life and death runs Neverfail for Microsoft Exchange, a high availability and disaster recovery product for e-mail.
One of 50 federally designated organ procurement organizations in the country, LOPA, founded in 1988, is responsible for tissue and organ donation in Louisiana. With about 80 employees in three offices, the organization believed its disaster recovery plan was a good one until 2005, when it was hit by back-to-back hurricanes Katrina and Rita.
"We quickly discovered that we had to revisit our disaster recovery plan," said Max Prather, director of information at LOPA in Metairie, Louisiana.
Literally a lifeline for LOPA, Exchange is used to communicate and coordinate with its regional offices and networked hospitals and is considered to be one of the organization's most mission critical applications. "We had to make sure we had a real-time copy of our data," said Prather.
Today, LOPA replicates its Exchange data to a virtual server programmed for automatic failover in an emergency.
Hurricane Katrina, one of the deadliest and most costly hurricanes in the country's history, was a real wake-up call for those without good business continuity and disaster recovery practices. LOPA was one of those that learned many lessons about recovery.
At the time, the organ procurement organization used Secure Online Backup from Data Protection Services (DPS) in Hammond, La., to back up its data. The service allows users to decide what data to back up, the frequency and timing of backups, and how long the backup is retained in DPS' vaults. Data is replicated and transmitted over an Internet connection.
According to Prather, LOPA had used the service to recover individual and blocks of files prior to Hurricane Katrina and was satisfied with the outcome. Katrina, however, was another story. LOPA's IT equipment, located on the ninth floor, was mostly spared during the storm. "But we couldn't access it. There was no power, and fuel to the generator had run out," said Prather. Not only was the elevator down, there also wasn't any air conditioning and the building was surrounded by flood waters.
While DPS could theoretically send the stored LOPA files over the Internet, it was unreasonable to try to send 100 GB of data. Instead, DPS loaded the data on a storage server and shipped it to LOPA. The organization recovered at its Lafayette location, about 130 miles away.
However, it took about 10 days to recover the organization's equipment in Metairie, set it up on a network in Lafayette and get dependable lines for communications. About 30 employees from the Metairie facility joined up with 15 employees in the Lafayette office. Work resumed using laptops and cell phones.
In another few weeks, even before LOPA had time to fully restore operations in Lafayette, Hurricane Rita came through, leaving the organization without a site to recover to. "Not only didn't we have a site to restore data, there weren't any communications and our e-mail server had no redundancy," said Prather. At the time, LOPA's Shreveport location housed one local domain controller but no facilities to maintain a network.
Despite the fact that LOPA was without e-mail for several weeks, the organization was back at work on an organ recovery case the next day using alternate means of communications. But LOPA faced difficult times nonetheless.
"From the time a donor is reported, recovery must take place within 22 hours on average," said Prather. Communications with the state's hospitals was difficult at best. "It was difficult getting information about organs that were recovered and transplanted," he said. Additionally, the organization had difficulty with billing, accounts payable and receivable, as well as payroll.
A New Plan
Once LOPA was able to return to Metairie after the hurricanes, Prather knew it was time to look for a real-time replication solution for Microsoft Exchange. "With fewer than 100 people, we knew we didn't want to deal with a cluster," he said.
LOPA identified two solutions: CA's XOsoft and Neverfail from the Neverfail Group. While Prather didn't bring either solution in-house for testing, each vendor gave an in-depth product demonstration hosted on their respective servers. "We did get to work with each system to see how it functioned," said Prather.
Ultimately, LOPA selected Neverfail for Exchange. According to Prather, while Neverfail had a more intricate setup and wasn't as user-friendly as XOsoft, the failover in the Neverfail product was seamless. "For the users, they never have to know that anything has changed using Neverfail. There's no downtime, nor do users have to close Outlook," he said. This transparency is important to an organization that tries to be invisible to the end user, especially during a stressful time such as a disaster. The XOsoft solution, on the other hand, required user intervention. The Neverfail purchase also included a dedicated technician for any questions or configuration help.
Neverfail for Exhange, according to the vendor, proactively monitors the entire Exchange mail server environment, including server hardware, the network infrastructure, operating systems and the Microsoft Exchange application. If a problem occurs, Neverfail takes a variety of preemptive and corrective actions. If a full system failover is required, a switchover is automatically made to a secondary Exchange server.
LOPA's backup machine is a virtual server for Exchange at its Shreveport office, which has since been configured as a cold site for disaster recovery. Shreveport is a five-hour drive from Metairie and more prone to winter ice storms than hurricanes, according to Prather.
Purchased in 2006, LOPA had the occasion to test the Neverfail product when a construction crew working on its new office building cut its T1 line. "I initiated a failover from the airport, where I was at the time," said Prather. The incident happened at night, and when employees arrived in the office in the morning no one knew the failover had taken place.
At that time, Prather didn't have the automatic failover set up, but it has been configured since then. The organization hasn't had to test the system since.
Microsoft Exchange is currently the only application LOPA replicates in real-time. Round-the-clock back-up for other applications using DPS currently serves LOPA well. However, Neverfail can be used to replicate data and other services in real-time too.
"As we continue to invest in virtual server technology, real-time replication will become a bigger part of our disaster recovery/business continuity plans," said Prather.