Skip Navigation

Thought leadership from SAI to accelerate your performance
 

Systems Alliance Blog

Opinion, advice and commentary on IT and business issues from SAI
Keyword: change management

NYSE

On Wednesday, the New York Stock Exchange was down for nearly four hours.  As soon as trading was halted, speculation began to fly that the outage was the result of the exchange being hacked. 

Reality turned out to be a little less interesting. NYSE realized that a botched software update was causing major glitches across its trading systems.  Although this was a very high profile outage, it is commendable that NYSE’s IT staff was able to recognize the problem and roll the change back.  This is a great example for how IT Change Management should be applied.

Not Every Outage Involves Hackers

With all the attention on cyber security, it’s easy to forget that human error and a lack of good IT governance are far more likely to cause an outage than malicious actors are.

Shooting yourself in the foot is a lot more embarrassing than getting hacked – especially since it can be avoided.

According to the Visible Ops Handbook from the IT Process Institute, "80% of unplanned outages are due to ill-planned changes made by administrators ("operations staff") or developers."  ITPI dives further into these self-inflicted & unplanned outages noting that the majority of the time to restore services is spent figuring out exactly what changed because of a lack of effective Change Management. 

Change Management Isn’t a Bad Thing

Many IT professionals have a very negative view of Change Management and ITSM frameworks like ITIL.  They see them as administrative and bureaucratic burdens that prevent “real work” from being done. 

Those true believers that feel like you have to implement every piece of the gospel according to ITIL aren’t helping the cause either.  It is unrealistic to go from an undisciplined environment to having every ITIL process fully realized overnight.

Always remember that the Change Management process is there to reduce risk and ensure changes are well thought out. It can be as simple as making everyone agree to write down and discuss their changes and preventing unauthorized changes.

IT “Cowboys” Are Symptoms of a Bigger Problem

Small IT shops without mature IT processes often have one key staffer that keeps all the lights on. They eschew documentation and fix things based on their gut feelings. They’ve always got a magic bullet ready to restore services when the worst case scenario happens.

“Cowboys” in IT have had a good run but it is past time to send them packing.  Not only do they often cause the very outages they’re fixing through human error, they tend to keep knowledge to themselves which prevents new staff from learning your systems and grinds troubleshooting to a halt when they’re unavailable.

It is an unacceptable risk to let critical production systems be run by cowboys who make changes outside of the Change Management process.  The presence of cowboys is a symptom of poor IT governance where the organization is operating without a plan.

Write it Down!

Documentation is one area where many IT shops struggle.  They don’t write down policies and procedures.  They don’t keep their configuration information readily available and up to date.  They find themselves flailing about when an outage happens because they don’t have any reference materials handy....Read More

I was surprised to see a reference to Fred Brooks’ 1975 classic, The Mythical Man Month(1), in a recent blog post(2); it was suggested as required reading for those involved with fixing the problems with healthcare.gov. The blog focused on the idea that throwing additional staff at a floundering project is likely to make things worse.

For those unfamiliar with Brooks’ classic work, it presents lessons learned from his time as manager of the OS/360 development effort. I’m probably telling more about myself than I should, but I have a dog-eared copy in my personal library and think it should be a must read for all IT project and program managers. The book covers a host of risk management topics, most of which are as relevant today as they were almost 40 years ago.

Estimation: Consistent application of an estimating discipline that considers performance on prior projects is key if the risk of delivering projects over budget and behind schedule is to be avoided. Brooks highlighted the importance of considering the amount of time likely to be spent on administrative and other non-technical tasks rather than focusing exclusively on technical activities. He also argued that testing involves a greater percentage of project effort than managers and developers typically estimate at the outset. While estimating discipline doesn’t change the level of effort or time required to deliver results, it does provide a more reasoned view of what you can expect.

Progress tracking: Actively tracking and reporting project progress not only provides a primary source of data for future project estimates but also reduces the risk of major project delays and overruns. Defining interim milestones and monitoring progress against them is critical to success.  To paraphrase Brooks, “major project delays don’t happen at once, they happen one day at a time”.

Communication: While software development methodologies have evolved significantly since Brooks’ days, the need for stakeholders to have a common understanding of project objectives, core requirements, deliverables to be produced and key roles and responsibilities has not. Minimizing project delivery risks requires communication and documentation. Yes, documentation, the bane of most IT staffers, has an important role to play in successful project delivery.

Staff skills and experience: One of the healthcare.gov development challenges that has been widely discussed in the press is the lack of a cohesive design; one that could accommodate the high number of users expected and enable delivery of a positive user experience. Some have suggested that the government would have done better by seeking assistance from top tier technology firms rather than relying on their traditional service providers. In The Mythical Man Month, Brooks cites the importance of conceptual integrity (i.e. design integrity) and the use of expert resources to lead both the overall design effort and the development of critical components.

Change management: On large projects where requirements changes are inevitable and design changes are a distinct possibility, a formal change management process is a critical requirement. On time, on budget delivery of a quality solution is jeopardized when the potential for change is not acknowledged and the process for handling changes is not clearly defined. Effectively planning for and managing change are themes that run throughout Brooks’ classic work.

While advancements in technology have enabled the delivery of many products and services that were unthinkable 30+ years ago, advances in project and program management discipline have lagged far behind. It seems that some of the lessons documented in The Mythical Man Month have yet to be learned.

Do you have any lessons learned from IT projects that aren’t listed here? Tell us about it in the comment section below. And as always, if you’re having difficulty with an IT project, we’re here to help. Get in touch.

References

(1)Brooks, Jr, F. P. (1975). The Mythical Man Month, Essays on Software Engineering. Massachusetts: Addison-Wesley.

(2)Yglesias, M. (2013). Obamacare's "Man Month" Problem. 

Calendar
Jul 2015
 1234
567891011
12131415161718
19202122232425
262728293031