Paying penalties to the client more than the value of the initial contract
You may be surprised to find out that sometimes, smaller companies provides services of building software to big corporations and own the infrastructure to run it at the same time.
You are accessing the web site of your bank and the web site is owned and operated by a separate company who is in charge of storing your data as well.
It happened often, due to the big number of projects these major corporation needs to implement, to give some of the smaller projects to various professional services companies to manage. The service company is bound by a service agreement to capture and deliver the customer's data to its client.
I was hired as a consultant by such a company: my assignment was to own the design and the development process for 2 web sites that performed business on behalf of two company's major clients.
Both of the clients were not satisfied with the services they have received: one client lost their patience on a request to modernize their application while the other client's motive was much more serious: critical information from their data collecting campaign of several months was actually lost.
To settle this blunder of loosing several months of their client's data, the company had to pay a sum that was greater than the initial value of the contract they have secured.
As a consequence of this, the full-time developer working on that project was fired.
The business ties between these 2 companies may have been strong since they were allowed to continue managing the campaign for the client.
And then, it happened again.
As it turned out, a client's request for saving a simple checkbox on a page was not captured by the requirement manager.
The reason why this situation continued to happen has to do with the way the company was running their business process.
The company was aggressively adopting the Open Agile methodology at that time. An external training company was hired and teams after teams of people were sent to several days of courses designed to switch the way they have conducted their daily business to a more interactive and collaborative approach.
Two project managers and a development director were attending the daily scrum meetings: the PMs were managing several projects at the same time at a high-level - having an interest in monitoring the daily project's status and not owning any project's plan - while the development director listened in and his involvement and guidance in any of the projects was minimal.
Every 2 weeks, the requirement manager and the BA were visiting the bank to demo the product and to gather the client's input. The BA and the development team were holding a retrospective meeting periodically, as well.
It seems as the Agile process was running smoothly and the teams were self reliable.
Except there was no-one in charge to run the project. And there was no project's plan.
The fluid nature of the Agile process implemented and the fact that the Requirement Manager and the BA were really driving the project's initiatives focusing on topics from their area of expertise, made several project's task to never be discussed and considered.
Note: A task could had been added to the project's plan and followed through as in: gather test data and verify with the client that it covers all of his requirements; another tasks could have been to save all the data and deliver to the client only the data he asked for.
You would think that the second time around, these tasks would have been a priority. They were not.
Takeaway: Always have a manager in charge of a project, who creates the project's strategy and follow through with its implementation. As far as the project's decisions and responsibilities are concerned, the bucks stops with this person!
As it turned out, a few years later, the company outsourced all their development.
They have fired the developers one more time.
Security testing by external party against a company's live environment
This example presented below happened in Toronto to one of the major investment corporation: this company business is to invests billions of dollars for their clients in the financial market and deliver on a good return.
The development process was decent, the development shop was conducted as in any knowledgeable software development shop.
I do not know the internal details of this decision so let me just present the facts: it has been decided that a security penetration test will have to be performed by an external partner: I think it was Microsoft doing it, if I am not mistaken.
The VPs decided to give the green light to their business partner to perform a series of tests to determine if their environments can be penetrated or not, and these tests were done against the Production servers which hosted the applications managing their clients accounts.
At the time when my direct VP was running around the company, everyone new there has been a fire. As it tuned out, one of the tests performed was able to encrypt the Production database and made the application unusable.
This situation is one of a few where it is hard for me to determine what exactly cause it: was it at play the managers utmost confidence in their product and servers protection? No other environment could be used for this test due to the unique nature of the set up involved?
What other good reasons can be for someone to take the risk of exposing their live servers and applications that constitutes the reason for their business to operate?
Your guess is as good as mine.
I will not try to give an answer to this question: enough to say that a similar environment as the Production one could have been built and the tests performed against it and this choice would have been a wise one. Building this environment would have taken some time but it would have been perfectly justified.
The takeaway from this example should be an easy one: do not use your live servers for un-predictable tests that were not tried on lower environments.
In addition, let's just add one more: always have a backup strategy in case the main one creates problems.
As I was saying before, common sense goes a long way.