I was in Des Moines, Iowa, when I got an email from Brian McDonald, the IT Security Specialist, that said there was a fire emergency meeting at the La Farge Community Center. I started making calls and I wasn’t getting any answers. Nobody was picking up their phone. I finally got hold of Unified Communications Solution Architect, Brett Janzen. He said, “Get the marshmallows. We’re gonna make s’mores.”
I was like, “It’s that bad, huh?” and he goes, “Yeah.”
When did your part in helping come in?
Even after getting back to La Farge, there wasn’t much I could do in my area of expertise until Network Solutions Architect, Josh Krzych, shut down the servers. Josh ran out of the building and did his Irish jig and goes, “You guys have the go-ahead!” Then I worked with the DBAs (Database Administrators) to bring up all of our systems in the correct order. We have almost 300 systems.
How did you do that exactly?
We gathered at a picnic table outside the building with our laptops. Janelle Fellegy and other DBAs came up with a plan of which servers needed to go up and in what order. She would call out the name of a server – SQLERP, for example – and I would say, “Okay, turning it on!”, and then Dan Shaha would run a backup of it and then Nick Doskocil would verify that there was no lost data. It was an assembly line of verifying and maintaining and making sure that we didn’t lose anything.
How long did that take?
From the minute we got the go-ahead from Josh at 6 p.m. Wednesday, it only took us an hour and half to get the 276 systems up and running. We were very well-prepared and efficient. It was really incredible.
Having everything up meant we could concentrate on smaller things. How are we going to get phones up? How do we keep orders moving and make sure that people can work? The main goal at all times was to keep the business running.
I heard IT staff were working in tents in the parking lot.
The tents were a neat experience because if felt a bit like a war zone. We weren’t in the battle, but we were like the generals in the tents guiding all the resources.
We were in that tent right outside the HQ so that people could find us. HQ was actually still smoldering all that time. By Thursday morning it was cold and pouring rain. The tent roof kept overloading with water because of the rain, and we had to keep pushing the water off so it wouldn’t collapse .
In hindsight, was there anything about IT that you would set up differently?
I think the fire showed us that we were set up well. We fought hard to get funds to build the new data center the way we knew it should be, and it really paid off. All of our upgrades and our equipment survived the fire because of it. Four months ago, all of our data systems were in the old data center. That, of course, was destroyed in the fire. If the fire had happened four months ago, all of us in IT and Information Resources (IR) would still be working madly to get our systems running effectively.
If Chief Financial Officer Mike Bedessem promised to grant you one thing to make disaster recovery better, what would you ask him for?
Off-site data center. All of our systems would be stored in a location where we don’t have workers and preferably in a different region. It doesn’t necessarily protect it against fires, but it gives us another option of protection against disasters. Cashton and La Farge are so close together. If a tornado comes, or a meteor, or an alien invasion, we still have all of our equipment within 30 miles of each other.
Have any benefits come out of the fire?
I think so. All employees now have the same new technology, like Lync and Windows 7, to boost our communication abilities. Everybody had to jump off the deep end. I think people realized, “Hey, this new technology isn’t as different or difficult as we thought it would be.” Legacy hardware and legacy software are two things that can really cripple a company. The fire forced us to leap forward.
Were you surprised at the damage that the fire caused? Was the aftermath what you would have expected from such a disaster?
When I heard there was a fire, I expected the aftermath to be a lot worse than it was. All I could think of was that it would get to the new data center and we would be stuck. We were very fortunate that that did not happen. There are a lot of variables involved with the survival ability of the new data center. It’s hard to make an in-house data center fireproof because it is surrounded by an office building.
Were there any funny moments?
So the first day it happened I was trying to support Josh as much as possible. I kept asking, “What can I get you, man? What do you need me to do?” Finally he handed me some DMGs (Dialogic Media Gateway) and goes, “Take these to Cashton, and before you do that can you get me some dry socks?”
His socks were soaked from working in the old data center that was flooded from the sprinkler system. So I brought him some of mine from home. Come to think of it, he still has my socks.
Does IT have a crisis mode?
Definitely. Basically, a big part of IT’s day to day job is to keep our tools sharp. It is all about being ready to use your tools when you need them. We did that.
Our whole group really came together. We were efficient and ready for anything that was thrown at us. We know our stuff and it really made things easy for us once the crisis hit.
What saved us from ending up in a “worst-case scenario”?
The whole building didn’t burn. So the firefighters really saved us, in that they were able to get the fire put out.
When was your, “Okay, we are officially out of crisis mode” moment?
Once all the servers were up, I felt like we could all just take a deep breath. After that, it was a matter of getting people back to work. That process began the next day.