As relayed to Sarah Holm 

Officer’s Log: 5142013Niner    Brett

As smoke rolls into the sky, my mind races with ideas to combat the damage. It first jumps to the physical layer – “Where are things located? Wait! The old server room is where we are terminating all PSTN (Public Switch Telephone Network) connections! There is going to be water damage to the systems from the sprinklers.”

Hour 1: The fire department starts rolling out the big guns. I realize this is going to be bigger than I had first thought. I turn to our Microsoft Lync Administrator and ask, “Samantha, the extra DMG (Dialogic Media Gateway) is located in the old server room which means we have lost connectivity. What can we do to re-establish it?”

Hour 2: We shut down the servers from what is left of our wireless network. My little Dell is holding up great. While services shut down, I design a backup plan to restore our limited calling capability.

I bounce ideas off of Samantha. “Remember those 3 Test PSTN connections in Cashton? I think we can use them!” We sketch a quick design and factor in the possibility of no network access to the La Farge data center.

Good news arrives. “Brett, there is the SBA (Survivable Branch Appliance) and 2 PSTN connection cards in the Mitel!” This disaster is starting to look up.

Brett, Samantha, and others hard at work

Brett, Samantha, and others hard at work

Hour 5:  The team has made its home in the Eagle’s View conference room in Cashton. Samantha is etching our deployment plan on the big whiteboard. We hit the ground running. Samantha calls up our partners and consultants to see if we can get their help on this task.

The Unified Communication Systems are resilient by design. An IR manager asks about email. “Email lives for failover” I reply. Our Microsoft Exchange servers automatically failed over to our Cashton site. This is made possible by Microsoft Exchange Server 2010 with a feature called DAGs (Database Available Group). All the mailbox data was replicated in our Cashton system, which allowed for our Mailbox Server to take over the work load from La Farge HQ.

I look over some errors in the event logs and notice that the exchange server is trying to talk to the domain controllers that are off-line. That needs to be fixed.

I immediately go into System Administrating mode.  With a power-shell command, we are allowed to set a primary domain controller to the Cashton site. I change some DNS (Domain Name System) to a new server.  Now CROPPies can access their email. I stand back to see if I missed anything. I hear Samantha say, “If the primary witness server is down, who is going to control the cluster service?” She is right. We need to move the witness roll over to the Hub/CAS server in Cashton. She quickly makes the change via the shell, within two hours Microsoft Exchange was back. 100% healthy and accepting mail.

Hour 7: Samantha is on an emergency call with our vendor, Marco. We are getting our Test PSTNs set up to work with the old Mitel unit. If we can get the PSNTs to work on the Mitel, we can use that as a temporary gateway for PSTNs. We now have to call a Level 3 to bring the PSTNs to an active state. It’s midnight. Nothing we can really do till we can get Level 3 on the line.

Hour 14: Level 3 and Five Star have started a conference phone call. We discuss routing and failover from our 2 PSTN in La Farge that are lost. There’s a spark of hope as we hear there are recovery efforts to get the DMGs from the old server room. IT Technical Services Manager, Curt Parr and Network Administrator, Nick Korte, send over the units. We hope to clean them and get them running. We attack them with hair dryers and rubbing alcohol. No luck. 3 hours of moisture has done its work.

Samantha has called vendor after vendor to get us a new DMG. We have some coming by overnight mail. We need them to get the phones back to 100% capability.

20130517_0104

Hour 32: Our mighty new DMG arrives. Samantha loads the configuration we saved the week before. With a simple test, we are able to verify that the DMG is ready. Samantha loads the DMG in the car and drives the speed limit to Vernon Tel in Westby to install it. Within 30 minutes it is up and running and calls are routed to Microsoft Lync users. Users can now call in and out of CROPP! We celebrate with cigars. (Kidding)

Over the next two days, our new challenge is moving over 500 CROPPies to Microsoft Lync Enterprise Voice. But that is a story for another time.

Over-n-Out

Brett A Janzen

Brett Janzen