Posts Tagged ‘crisis management’

Q&A with @SunGardAS User Group Forum Keynote Speaker, Michael Leiter


Michael LeiterMichael Leiter
serves as an expert on counterterrorism, cybersecurity, and national security for NBC News and worked as the director of the National Counterterrorism Center (NCTC) from 2008 to July 2011.

On October 15, Mr. Leiter will deliver the keynote address at the annual SunGard Availability Services Business Continuity International User Group Forum at the Chicago Marriott Downtown from Oct. 14 – 16. In his address, “Leading in a Crisis: Before, During, and After,” Mr. Leiter will share lessons on instilling leadership while managing a crisis and describe his experiences with helping manage scenarios that impacted the nation’s security.

The SunGard International User Group Forum is a symposium that offers peer-to-peer sessions on business continuity (BC), real-life case studies of disaster events and success stories of business continuity plans resolving operational disruptions. Attendees will also learn about implementing BC software in an organization and view the next generation of BC Software enhancements that will shape the future of business continuity management. Follow the conversation on Twitter at #SunGardUGF

In advance of the User Group Forum, SunGard asked Mr. Leiter for his opinions on leadership, crisis management, overlooked factors, and how enterprises can learn from national security threats.

What is the most important factor in leading an organization through a crisis?

First and foremost, it’s the idea that planning is not just for a predicted future. Planning is critical for responding in a time of crisis. It allows you to understand your organization, its surroundings, and what you are faced with. And when an unpredictable or predictable event occurs that throws a wrench in works, it is that planning which allows you to respond in a crisis and change the organization’s priorities because you understand it so well.

When is leadership most important when a crisis or business disruption arises?

I’m a very strong believer in leadership from the very top at all times, but especially before and during a crisis. It affects every part of the organization. Part of the responsibility of the highest level of leadership is to create champions in every part of the organization for your business continuity and crisis plan.

Your professional background includes roles in the highest levels of government, including at the U.S. National Counterterrorism Center and the Office of the Director of National Intelligence.
Even at highly organized organizations with respected leaders, do you find that crisis planning still has a role?

When you have a crisis, the best laid plans go out window, except for those pieces that help you understand how your organization can shift and change to respond to new situations. Also, in my experience, in terms of planning during a crisis, it’s critical for a leader to understand all components of an organization and what its capabilities are. Because unfortunately, no matter the organization, many people in the organization may very well lose their cool. The more you have thought about what the organization can do and cannot do, the better position you will be in to react to that time of crisis, to adjust to changed circumstances, and then reshape the organization beyond the period of crisis to be more effective when you have new requirements upon you.

How do you apply lessons you learned from managing major national security threats as director of the U.S. National Counterterrorism Center to business continuity planning for enterprises?

Let’s take the raid on Osama bin Laden [in May 2011] as an example. This was an undertaking that required an enormous amount of planning in the run up to the raid. At various intelligence agencies, people had been working on this mission for over a decade. They had been planning and thinking and identifying every possible eventuality. In this particular case, we knew when the crisis might arise, which was when the operation would be enacted. So in last two weeks before it became active, the plans were shown to an entirely new group of people who had not been involved at all. Everything was presented to them, and we said, “Come up with all the eventualities you can and tell us all things we might be getting wrong.”

Another example is the attacks in Mumbai, India [in 2008]. In evaluating security threats, it was typical at the time to always talk to local authorities like the police. In the Mumbai attacks, it turned out the attackers used fire as a weapon while inside a large building. This was an eventuality no one had thought about.

I think any organization can learn from these examples. It’s great to have intelligent, well-informed people involved in business continuity planning, but also it’s also critical before the finalizing that plan to step back. You want to give that same information and the scenarios to a group of people outside the organization who understand the problem, but who aren’t emotionally involved to the outcomes or the plan. The goal is to try to come up with alternatives to find where the planning may have gone off the mark and to identify the problems.

What are your recommendations for initial steps for building a BC/DR plan for an organization of any size?

You have to start small. You can’t plan for all eventualities and you shouldn’t start with everything falling apart. Start with a smaller crisis, such as what happens if you lose your company email. That can be a crisis, for sure, but it’s much different that losing all your electronic storage. In national security planning, we don’t start with a nuclear attack on Washington. We ask what would happen if there were a suicide bomber in Washington, D.C. and how we would react and handle that. It’s much better to start from a smaller crisis and build out.

As a former national security leader, you had stakeholders across a wide range, such as the White House, agencies like the CIA and FBI, Congressional leaders, and, more broadly, the American public. What advice can you share with organizations about communicating effectively during a crisis to all its important stakeholders?

One thing that immediately comes to mind is that it’s very easy to assume that you understand what a customer wants when a crisis hits. In the case of my own crisis planning at the National Counterterrorism Center, I tried to understand what the President, the White House and members of Congress wanted for information. But I found it’s much better to sit down and ask them, “How do you want this information? When do you want it? What information do you want first?  Who else do you think should be informed about this?” It may be difficult for some businesses to plan this way, but I think it’s important to engage customers and explain that during all these preparations to become well positioned for eventualities, you want to understand their requirements and what information they will want to know.

And internal communications is just as important. In my experience, the vast majority of people find this kind of strategic planning to be an annoyance. When you have a crisis plan developed only by the crisis planning team, it’s helpful but not nearly as useful as one developed by a broader cross section of users.

It’s incredibly important to engage stakeholders inside the organization and, sometimes, leaders have to do that with internal stakeholders by twisting their arm a little. You want to do that in a way that reduces the workload on them, but so that they understand it’s important and you need them. In the end, you will have a much better return on investment. If you leave any part of the organization out, it’s almost guaranteed that’s the part of organization that will open the plan for the first time at the moment of crisis.

During the crisis, it’s obviously about communication, communication, communication. If you can’t effectively communicate messages to employees and leaders across the organization, you could soon be faced with a workforce that thinks there is no plan. Your business crisis will quickly become an existential crisis.

Lessons From Hurricane Crisis Management

Bob DiLossi is the Director of the SunGard Crisis Management Center. A long-time business continuity practitioner, Bob provides some commentary in this post concerning lessons gleaned from crisis management in the midst of severe weather events, such as hurricanes.

1. Does the Crisis Management Team do anything different once a hurricane has been named and a projected path is announced by the National Weather Service?
It’s important to recognize that we monitor all weather events, not just hurricanes in-season. What makes weather events unique is that sometimes, you have a warning period that allows for review of plans and preparation. Right now we are tracking a tropical storm over the Cayman Islands which may strike Texas or Louisiana, or may move in another direction. Our process is to consider potential storm direction, and contact customers who may be potentially affected. We begin by reviewing the human factors and anything that would affect the safety of employees and our ability to contact them during a crisis. Second, we discuss potential business impacts. We then put them on alert, not waiting for them to act. That act of placing them on alert often becomes an alarm for them to make sure they are taking the necessary precautions themselves. The SunGard portal then gives them visibility into our plans and status as the storm track develops.

2. What advice would you offer to SunGard customers as a storm approaches their location?
Of course, safety comes first. Immediately behind that is communications. We use our own NōtiFind product to manage calls and response tracking, as do many of our customers. Regional events such as storms can quickly become complicated from a communications perspective, both with the numbers of people to reach, and the failures in communications channels that a storm can cause. NōtiFind, integrated with LDRPS is how we manage the complexity. I also recommend that a customer never rely on just one means of communications. Land lines at home and work, cell phones, pagers and increasingly social media all serve to provide multiple channels to keep communications open.

3. Should customers do anything proactive with support vendors, such as maintenance vendors? What about with their trading partners?
Support vendors can be critical to both ongoing operations, and if needed during a recovery operation. My first critical suggestion involves fuel providers. You need a contract in place before an event, or else you become just another name on a list in the middle of the crisis. Second, review both your backup schedule and off-site transit of backups; depending on the anticipated timing of a storm, make sure that backup tapes do not sit on-site longer than necessary, and that they are stored hopefully outside the threat zone. Third, review any employee travel agreements. If you need to quickly send staff to a recovery facility, you may need to be sure that everything is in place to make that as easy as possible on your employees, such as planning the potential for emergency petty cash.  In regard to 3rd party partners and vendors, make sure you involve them in tests and validation exercises, especially during “off hours”; responses at 10 AM on a Tuesday may be very different than 3 AM on a Sunday.

4. Is there anything different in the SunGard response to a hurricane when compared to, say, a fire or power outage?
The biggest difference is that with storms, you may get some warning.  You might also get some warming with wild fires, such as we experienced this past year in the west. Most other events have no warning, and you are in a reactive mode.  So, use the idea of “hurricane season” to do a periodic review of your plan, resources and capabilities. Hopefully you are not involved in a hurricane, but you will be better prepared for other unexpected events.

A Fresh Perspective on Resilience Exercises

Reading a Harvard Business Review Blog this week triggered this thought on resilience: when conducting any validation exercise, it is important to invite “outsiders” to participate.

John Baldini, writing for HBR, noted that management coaching involves having an outsider suggest ways to improve your perspective on reality and decision making, with the suggestion to invite others into routine meetings from outside the normal attendee list. It adds energy and creates some fresh dialog.  Baldini writes: “A new perspective can allow a leader to make certain that what she sees is reality, not her perception of reality.” That statement applies equally well to resilience programs, too.

During more than twenty years in the continuity business, there are two observations that remain true even though the industry has shifted from “event-driven” to “resilience” planning. The first is that if you test the same components each time you validate your continuity plan, you really are not testing anything challenging. Ask yourself if your business changed during this same period, and the answer will always be “yes.”

The second observation is that while we generally “know” what our peers do during normal times, we are likely mistaken about who is responsible for what in the midst of a crisis. Mistakes here lead to decisions we will likely regret once the crisis is over.

Supporting a number of company-wide simulations over the past few years has proven this to be the case in virtually every type of organization, large and small, governmental and private sector. We make some basic – and often reasonable – assumptions about who makes decisions during a disaster, but it is critical that these assumptions be tested. Don’t assume Department X takes care of task 123; ask them, because they may be assuming that you are responsible for that task.

Better still, schedule an annual validation exercise that involves those outsiders. It has the dual value of increasing organizational training, while energizing the validation process. Assumptions during any crisis management activity often lead to lost time or mistaken actions.

Lessons Learned … Again

With continuing concern surrounding the damaged nuclear plants, the global community continues to watch the turmoil unfolding in Japan. In the twenty days since the Sendai earthquake and the resulting tsunami brought unimagined devastation to the Japanese nation, we are seeing just how small planet earth really is.

Global Dependencies are Felt Locally

Moving beyond the destructive impact on whole communities and the human toll too quickly seems to trivialize the impact, but at the same time, it is important that organizations on a global level recognize our interdependence. These dependencies can be seen clearly in the examination of global supply chains. Companies such as Boeing, Sony, Caterpillar and John Deere have been referenced in the news as enterprises that are feeling the supply chain impact, or anticipating parts shortages within a very short time frame. General Motors has announced production impacts from Louisiana to Spain to Germany related to dwindling supplies of Japanese components.

Forrester Research mentioned yesterday that business continuity is “… back on the agenda …” for business executives. Today the Wall Street Journal reported that the disaster plan from Tokyo Electric Power was inadequate, especially for the combined impact or earthquake and tsunami.

Earlier this week, in a conversation with Gartner Research about testing recovery plans, the point was raised that more than just worst case scenarios, planning for the combination of events raises maturity to a best practice level.

While the Japanese continue their struggle to recover on a massive scale, much of the world has begun to consider “lessons learned.” We did this following the attacks of 9-11-01, following Hurricane Katrina, and similar action is demanded to review plans as to whether the assumptions made are grounded in the new reality unfolding in the news and within the lives of the Japanese people. Business processes and interdependency have become more reliant on automation, built around more complex trading partner and business models, and subject to more rapid impacts for disruptions due to “just-in-time” processes and inventory levels.

Lesson #1: Acknowledge Increased Risk Levels

My point today is simple: resilience and risk managers in organizations of every size must acknowledge the increased risk, and adjust plans accordingly. The lessons gained from examining events in Japan should stir internal reviews by every organization with trading partners concerning risks, logistics, capitalization, insurance and diversification.
For most of us, it is difficult to fully comprehend the impact on the ground in Japan. But all businesses need to examine how complex supply relationships – from raw materials to manufacturing capacity to transportation and selling channels – would be impacted from disruptive events that threaten such relationships. The imperative becomes determining appropriate mitigating actions and procedures in light of what we see in new light following the natural disasters in Japan and other global regions.

Fitness Training and Resilience

The more physically fit we are, the more resilient our muscles and bodies are to stress and strain. The same can be said for organizational resilience programs. They may need a “trainer” to help us get them in shape, but even without that expert resource, they certainly need regular exercises.

The risks companies face today are varied, and much like exercising different muscle groups, they call for different activities to examine and strengthen against these threats. In 2010, natural disasters had an estimated $109 billion impact, more than triple the previous year; that number doubles when you add the costs of man-made disasters, such as the Gulf Oil Spill, and we quickly see the cost justification for planning for worst case scenarios.

What Shape Is Your Resilience Program In Today?

Consider: data breaches become a violation of expectations of privacy by your employees or customers. When information is exposed to the outside world that should not have been revealed, both a technical and a communications response is needed; both factor into the estimated cost, which reached $214 per breached record in 2010 according to the Ponemon Institute. The same could be said for protected health information that needs to be kept confidential, and accessed only by authorized personnel. In a conversation this week with the president of a local hospital chain, she mentioned that they have dismissed employees over HIPAA rules violations. We operate in a world where transparency is demanded (SOX), and prohibited (HIPAA). Remaining resilient in the face of such risks calls for balance between privacy and authorized access in our highly connected world.

On another level, consider the recent WikiLeaks episodes. The public disclosure of confidential information gave a new meaning to transparency, and a caution to information security managers. I’ll not debate the layers of questions that these actions triggered concerning the breach of confidence, under the claim that the public had a right to know; what is clear to me is that all organizations, both public and private, need to make certain their information security programs are up to today’s challenges and threats.

Relevant in this blog space is the impact on organizations and their resiliency, and how best to mitigate such impacts. The cyber activity following the release of confidential information led to DOS cyberattacks and the outages for major credit card networks, which had a subsequent disruptive impact on numerous businesses and their e-commerce. This risk is real, and calls for every organization to review the effectiveness of their information security programs in dealing with such incidents. GLB and HIPAA regulations call for the periodic assessment of electronic security against anticipated risks or hazards. Given the demonstrated impact to systems these past few months, this is now a risk that must be anticipated (GLB: 16 CFR 314; HIPAA: 45 CFR 160-164).

Different Risks, Different Training

Resilience and crisis management each depend on responses to risks, both actual and anticipated. Beyond the technical programs for information security and the capability to recover your operations at an alternate facility, resilience and crisis management call for effective emergency communications programs, something frequently overlooked. If your plans don’t include guidance on who should speak in the face of a disaster, what they will say and how you will preview any statements before release to the public, then it is time to update your plans. Consider drafting sample statements for the anticipated risks; the internal review of these sample statements not only better prepare your spokespeople, but also help uncover additional elements of your plan that may need to be updated.

Continued Monitoring and Exercising

Ongoing monitoring of risks and mitigation programs is important – and required by regulations. As any fitness trainer will advise you, you need to keep at your exercise program, or you will quickly fall behind.

A Case for Cybersecurity First Responders

For years in the continuity business, I’ve spoken to audiences about the fact that you can’t outsource your first response. This is true at a community level, and it is true for businesses. When a fire strikes, the red trucks and water don’t come from Washington; they come from your local volunteer or paid fire fighters. The same is true for corporate disasters; outsourced partners and support vendors will be involved, but the first response will likely come from within your organization.

Step One: Prepare in Advance

Preparing first response tactics are best done well in advance of any actual disaster. You need health and safety plans in place before any fire, flood or earthquake impacts your business. For some events, like a hurricane, you may have a few hours’ notice before the storm actually hits, but in this stage you are likely already activating your response plan – not creating it as the storm winds gather.

Cybersecurity is no different from a planning perspective, but is often overlooked. Recently, major credit card companies Visa and MasterCard experienced disruptions in service due to cyber attacks, as reported in the Huffington Post and most major news outlets. A key lesson to be learned here is that when we hear names like Visa and MasterCard, we know they have sophisticated security systems and teams of people handling their security threats 24 x 7. And even with those protective measures, they still were compromised.

Some in our industry use the phrase “… it’s not ‘if’ but ‘when’ you experience a disaster …” to justify support for their programs, or to spur management support for improving the maturity of existing programs. Cyber threats are a certainty – their probability is 100%. It has been estimated by security software vendor Symantec that more than 90% of all Internet traffic can be classified as spam or malware.

Mitigate Impact With Effective Crisis Communications

That means that cyber threats need to be classified as high frequency events with potentially high impact. Effective cybersecurity programs call for defense in layers, monitoring potential attacks, and need to include not only a technical team of first responders, but an effective management response and crisis communications. E-Commerce depends on trusted pathways that reflect security that consumers are comfortable will protect their financial information. A recent SunGard paper reported that 69% of the financial impact from security failures was due to lost revenue and customer turnover. Just as in the often used example of the Johnson and Johnson response to the Tylenol disaster some 20 years ago, effective first responders – which include crisis communications – are essential for companies to maintain the trust of their customers in any e-commerce systems and applications.

Resilience is Power …

… and water, and HVAC, and all related infrastructure components. Resilience is dependent on all these infrastructure components, along with network communications.

This became obvious this weekend with two distinct events:
- A friend shared a photo of a car that went through the side of a building while parking; in doing so, they broke water and sewage lines which prevented the building from remaining open for business occupancy for several days.
- An underground explosion and fire in Philadelphia early Monday morning – just a few blocks from my office – caused local businesses to deal with power outages and street closures when they arrived back after the weekend.

These two incidents are a reminder that every business is dependent on utilities for power and water and on telecommunications carriers for their connections to the outside world. Not long ago, I directed a continuity drill for a brokerage company, where we simulated an underground fire. Within ten days of that simulation, underground utility fires or explosions occurred in both Philly and New York.

Scenario Planning

Continuity planning is never really finished. It is a cyclical process, including establishing policies for your organization, assessing capabilities to meet those policies, training staff and validating capabilities, and then maintaining that readiness. In parallel to maintaining the readiness and capability is the ongoing question of whether your organizational needs are changing.

Regular validation is recommended by most standards and may be mandatory under many regulations. This includes updating the scenarios which you follow when conducting any validation exercise.

Disaster Statistics Point to Risks

So what are the top causes of disasters? Of 2,367 disasters supported by SunGard over the years, 1,181 (49%) were caused by hardware failures (570), weather (349) or power failures (282). Fourth on the list is terrorism at 7.4%.

With these threats in mind, it becomes easier to ensure that any scenario planning you consider to maintain and validate your plan includes those elements that have been consistently a threat to continuity, in addition to any industry-specific threats you need to anticipate.

The SunGard statistics for 2009 (last full-year) show that the industries with the greatest number of declaration events are led by financial organizations, followed by manufacturing, government, technology, services, health care and insurance. When you consider the business drivers of regulations and supply chain dependency, these industry segments demonstrate the greatest maturity in their continuity planning programs.

When developing your business continuity program, be sure to consider a broad array of possible threats. The events that will actually occur will differ, but the guidance of your plan will still inform your decision process as you recover production capability.

________________________________________________________________

Please join me at the Continuity Insights webinar discussing alternate site selection on November 23, 2010.

Q&A with: Bob DiLossi – Director of Crisis Management

Bob DiLossi is the Director of the SunGard Availability Crisis Management Center, having managed this area for the past seven years. In that time, Bob has been directly involved in hundreds of disaster exercises and actual declarations. As we recognize September as National Preparedness Month, pass this ninth anniversary of the September 11th tragedy, and anticipating the DRJ Fall World Conference (September 19-22, 2010), I had the chance to speak with Bob and get his perspective on crisis management today.

Q: Can you tell me what has changed in recent years from what you are seeing with customers?

A: Customers today practice additional scenarios, and to a greater depth of detail, than they have in the past. These scenarios reflect more of the everyday events, which lead to more realistic and more robust validation of their continuity programs. I see significantly more blending of the data center recovery process with the business processes, as evidenced by the increased number of mock disasters we have seen in the past few months, tying customers’ internal table top exercises with the SunGard Crisis Management Center.

Q: Bob, you’ve participated in literally hundreds of disasters and exercises; what do people forget most often that would help them become more effective and successful?

A: In the past, I would say that they had neglected the people aspect, the detailed processes that surround the end-user recovery. That trend, fortunately, has changed of late, perhaps driven by a greater awareness of the staff impact that has been seen in the news following events like Hurricane Katrina. The biggest challenge now is change management. There continues to be a disconnect when an organization deploys new technology. Too often, we see technology changes that support daily production workloads not reflected in recovery plans.

Q: Some customers are more effective than others in their test success; what sets them apart?

A: Probably the single most important factor is how thoroughly they exercise their plans. We advise new customers to follow a “crawl-walk-run” approach to improving their plan, but some never progress past the “walking” – testing individual components but not all their applications and procedures as an integrated exercise. We’ve seen some customers back off of testing with the current economy, but the mature process and best practices deliver value only when you have verified that your plan will actually work within all the resource and time constraints you are tracking.