In this article, I want to take a look at a few cases where costly mistakes were done during the process of developing software in major companies, some of these mistakes I have witnessed myself.
Beside presenting these cases as a curiosity, I will try to identify the root cause for these mistakes and propose common sense solutions to prevent them from happening again.
Common sense has a long way to go in my opinion: I will not present miracle solutions to fix the problems described in this article but gave you a short list of takeaways - simple advices backed by my years of experience in this field.
Enjoy the reading.
Citibank - UI disaster
You may be familiar with this first case that happened couple of years ago at Citibank and cost the company 500 millions U.S. $.
You can read more details about this case in the article from below link:
By using a financial software application - of which the main screen of interest is displayed in the image from a few paragraphs below - a transaction has been made by Citibank employees to send out interest payments totaling $7.8 million to Revlon's creditors, a company for whom Citibank was acting as an agent.
Revlon was in the process of refinancing its debt—paying off a few creditors while rolling the rest of its debt into a new loan.
To achieve this use case, the software had to be used in a particular way: the employee performing the transaction had to enter in the system as if paying off the loan in its entirety (thereby triggering accrued interest payments to all Lenders) but to direct the principal portion of the payment to a "wash account" - an internal Citibank account.
By entering the account number in the "Principal" text box and checking the corresponding checkbox located besides it, the Citibank employee thought that the principal portion of the payment will be deposited into a Citibank account.
This was not the case, as the application payed all the principal amount to the external parties - the Revlon's creditors.
To prevent this payment to the creditors to happen, the subcontractor actually needed to set the FRONT and FUND fields to the wash account as well as PRINCIPAL.
While Citibank tried to get the funds back, a federal judge ruled against it; the rational is presented as such: when a debtor accidentally wires money to a creditor, if the creditor doesn't have prior knowledge the payment was a mistake, he is free to treat it as a repayment of the loan.
An interesting fact is that Citibank's procedures requires that three Citibank's employees sign off on a transaction of this size.
In this case, the person initiating the transaction was a Citibank's subcontractor from India and the transaction has been reviewed and approved by a colleague of his in India, and a senior Citibank official in Delaware.
All three believed that setting the PRINCIPAL field to an internal wash account number, would prevent payment of the principal.
These are the facts and they have happened. As it is often the case life beats the movies, isn't it?
This shows how costly development mistakes can become for a company, in some cases.
Why did this situation has happened?
Let's start by saying that if 3 bank's employees who are doing regularly financial transaction using financial applications get wrong the meaning of a business process described by a software application then this application has not been designed well.
A software application needs to describe very precisely the operation it executes: if this operation needs to transfer the funds to an internal bank account then this and only this operation should have been made available by the UI.
The application should have built separate screens for separate operations: as it is for paying external accounts versus internal accounts. The users should know with clarity by the way the application looks and operates, what each screen allows them to do.
Additional information could have been displayed on the screen and inform the users how the payments will be broken down; the users could have been asked to confirm if they want to transfer all the amount to the external account, etc.
The intention of the application should have been made clear from the way it has been designed and no room for confusion and guessing should be left to the user.
Some of these scenarios (and a few others) could have been considered during the design phase of the application; test cases could have been created from that moment in time - and automated later on - with the intention to eliminate confusing situations to arise.
My professional experience taught me that only companies who build and sells software for a living performs well the design and development phase of building a software application.
In such a company - who builds applications that sells for hundred of thousands of dollars to external clients - the product does not go out of the door unless some testing conditions are met: one such condition may be to find one medium severity bug at 8 hours of testing.
Can you prevent such a scenario from happening?
Can you prevent such a scenario from happening for a company that builds hundreds of software applications for internal use in a whole year, in many cases by employing external contractors to perform the work?
In my opinion, it is always a trade-off between the cost of the project. its duration and the number of features you are building into the application.
As it is always the case, the natural tendency is to build more and more features into a software application: if the balance it is not well kept between the design, the implementation and the testing phases, it is easy to deliver poor quality software.
The software can be poor in many ways: in the way it looks and can be used, in the way it is modelling and describing the business, in the way it performs.
Takeaway - It is better to have less features built in a software product but well designed, coded and tested when being limited by time constraint and budget.
One Murphy's law for developing software I like to quote - because it shows the value of a good planning well before starting to code and not allowing disruptive changes to this process by constantly moving the target - is this one: designing based on specifications and walking on water are easy when both are frozen.
Note: I am well aware that my above statement can be seen as being against the agile methodology were teams are constantly adapting to changes to their requirements but an experienced manager will have to draw a hard line that cannot be crossed without serious consequences to the quality of the software implemented.
Takeaway - It is this person responsibility to be a clear and cool head in the mist of all the chaos.
Note about the UI design: Reviewing this article to correct and simplify some of the phrases, I have realized that there is so much more to say about this particular UI design.
It is for sure an old desktop application that most likely has been used for long time -I am wondering how many other mistakes were done with it?
The number of fields - 8 - are editable and data can be entered in each or all of them at one time. If the mathematics I have learnt 30 some years ago did not fail me, the number of possible combination for entering data on these fields is a sum of combination of 8 taken by 1 with combination of 8 taken by 2 and so on up to combination of 8 taken by 8. The checkboxes from the right side of each editable field added even more combination possible and it is not clear if there are additional rules to be set when performing to overwrite the default instructions. It is not clear if there are any fields validation either.
However, just by looking at the possible combination of entering data into the fields on this screen one should think about how many distinct use cases can be associated to all these combinations.
My initial impression was that all of these fields are part of a single transaction and there is a single meaning in which each field operates - but this has actually proven not to be the case from the way the application behaved when creating the problem in the first place.
It is very clear that the screen will create confusion for a user to differentiate between all these use cases; the name of the fields are not helping too much either in providing clarity nor any additional instructions are present on the UI.
And this confusion, we know by now, has indeed happened.
#
The next few cases I am presenting below, I have witness myself in various companies I have provided services to.
Millions of dollars spent in on-boarding costs
The first time I was on-boarding this large financial corporation with offices located worldwide, I was working into a room with another 20 to 30 contractors. At least one other similar room existed in the same location, at this client's site.
The process of approving various software requests by the company's VPs for each contractor took 4 weeks, therefore none of the contractors were doing any type of work during this time.
Several years later I was asked again by the same company to work on a different project for them.
The on-boarding period it took again 4 weeks, in spite of all the help I had from one of the company's directors I was reporting to at that time.
I remember his frustration when I have proposed him one way to mitigate this situation by creating a number of user profiles ahead of time with all the set up already completed that will be assigned to the new contractors: his frustration was coming from similar proposals he made and he could not advance in the company.
Let's make a very simple calculation: 10 new contractors hired at a given time with a rate payed to the agencies of 100$ an hour for 150 hours of work per month = 150,000 $ a month.
Imagine that 10 new contractors are hired / dismissed every month: this will bring the on-boarding cost to 1,8 million $ per year.
This is just a small example but the numbers involved could be significant bigger for a company with offices in the majority of the developed countries.
Why the situation kept repeating?
The following quote it is attributed to Winston Churchill: “Those that fail to learn from history are doomed to repeat it.”
I found during my years of work that companies who deal with managing huge amount of money do not care too much in saving them when used for their own processes and employees. These companies will rather let people go every few years to increase the values of their profit then to look at how they spend money for internal projects and try to optimize these costs.
In my first Canadian job I had 30 years ago, I was working in a small mechanical shop: for several months a faucet was dripping slowly in one of the shop's corners.
The company decided to move to Montreal after several months and in one of the last meetings, another immigrant worker who used to work as a manager for a car dealership in Middle East made the observation to the young engineers who were running the business process that they should have fixed the faucet not for a financial gain but because of the pride they are taking in their work.
Takeaway: Invest time, energy and passion in an idea you believe in; make it known to the decision people at the right time and be patient and follow on it until it came to a resolve.
Is the history doomed to repeat itself for my client?
In my opinion, I will say it is, if there is no one to ask for the faucet to be fixed.
Read the continuation of this topic on a separate blog's article located here:
##