A developer, architect or Project Manager working in the IT industry from almost 30 years has seen it all: he (or she) was involved in planning, designing, implementing, deploying and delivering a software project using either the Waterfall methodology or any flavor of the Agile methodologies known to date: Open Agile, Scrum, Kanban, etc.
The reason old methodologies as Waterfall were replaced with the newest Agile methodologies it is the same reason why the old is replaced with the new: there are advantages the new brings that ultimately translates in more profit being made by the companies adopting the new.
Some of the benefits are the following:
creates a more transparent and collaborative work environment.
includes much faster feedback from various stakeholders into the development process.
provides faster workflow by eliminating the blockers and limiting the work in progress performed by the workforce.
creates a visual representation of the overall process, from the conception to the delivery.
provides better customer satisfaction and more robust product, less downtime and increased productivity, better overall ROI.
It is easy for the IT professionals to understand the few points I am making above: it is common knowledge. Nowadays, every major IT company building software adopts an Agile methodology for their internal process.
Having wrote this, I still want to make a few points explaining the misconceptions many professionals falls into when pushing a new methodology forward into their company.
Know your process from within, separate hype from need and myth from reality.
In many organizations I have worked for, an important number of Project Managers, Service Deliver Managers and Development Directors comes from a different background and from a career path that has not originated and evolved in the environment of building software applications.
So many times, these people push the adoption of new technology based on a hype, proposing and adopting what is fashionable at a given time, without understanding the long term implications of their decisions.
I will give a quick example: in one of the first 4 major banks from Toronto I have provided services to, a P.M. convinced his Development Director to send to Hadoop courses the majority of the developers in his department; all of this while the processes developed inside the department were moving and processing a very small numbers of records every night: around 1000. In this organization a request for increasing the memory on a given desktop was taking more than 6 months; you can imagine the overhead of ordering and configuring a few thousands servers for a need that was inexistent. At the same time, the department was building Windows applications only and the developers expertise was limited to just supporting existing applications created by external consultants.
While the newly promoted manager wanted to make his mark, his strategic decision was not backed up by a real need nor by an understanding of what the technology he was proposing was indeed used for; this translated in wasted time and money for the bank.
We all understand that every new technology is a product that sells; there will always be companies interested to advertise and sell their products, The same is true for adopting a new methodology: there will always be companies and people interested for you to adopt it who will provide training for a cost. In many cases, the people providing the training lack the same experience of building software applications.
A major Canadian company I have provided services to, has decided to involve all of their developers and managers into Kanban training. Kanban methodology was originally adopted at Toyota and the first IT company that adopted it was Microsoft.
This methodology originally has been designed to provide a smooth flow of work that worked very well in an industrial environment, where people from the assembly line have similar qualifications.
During the training, a few exercises were set up: the trainer's intention is to show how the flow can be improved when several people allocates parts of their time to the same ticket in order to complete it faster and move it along in the process.
I have asked the trainer how probable this scenario is in real life: the majority of times people have different levels and areas of expertise making very difficult to really collaborate on the same work in this manner; the trainer was very quick to answer that peer programming is an established and common technique used everywhere. There is no surprise that this trainer, in spite of her good subject's knowledge and presentation skills never have worked on designing or implementing any feature in a complex software application or process.
While the trainer was pushing the myth of peer programming, I was in charge of mastering the whole company's server-side process that involves Azure functions, queues, table storage and integration with the Salesforce Marketing Cloud. This process has been designed by an external company couple of years prior and the knowledge was not retained in a team of around 50 people; enough to say that the code could not be started due to a multitude of errors, the development environments were incorrectly set-up and there were thousands of errors in Production when the queues' messages where processed every night. No new features have been added to this part of the application in the last 1 year and a half.
That was the reality.
Manage each stream of work in the same way, with the same attention to details and using the same tools your methodology employs.
There are cases when the company has a main product that is the main focus of their activity. This product is generating the bulk of revenue for the company and all the attention is focused on solving quickly any issue that arises daily.
In these cases, the scrum master maintains a board where tickets are created and worked on for this main activity. Due to the critical nature of this activity, the majority of developers and managers are involved in providing fixes, creating builds and deploying them with a quick turnaround that may vary in duration from a day to a week.
When I am writing this, I understand that this activity should have been separated and have a dedicated support team exclusively. However, with a single team in place, how is the scrum master dealing with the other stream of activities?
The recommended approach is to create a separate board for each separate activity that requires a significant effort to go to the final line. If these activities are dealing with serious work efforts that may involve several resources over a significant timeframe it is advisable that a separate meeting should have been held just for managing this activity. It is advisable as well that this board should contains tickets that breaks down the main activity into enough discreate pieces of work that covers all the aspects of the development process from assessing the requirements, perform the research and creates the POC, design, implement, test and deploy. Even though the company has a main activity of interest, the other stream of works have to be planned and managed following the same tenants provided by the methodology used and the same attention to details needs to be provided to the process.
My experience with the company I have provided services for it was different from the textbook ideas presented above that where well known to the scrum masters. The other stream of activities, while important in adding new features or re-designing major functionality that was not working properly for a long time (and yet not so visible to the end user and to the VPs of the company) where present on the main activity's board as single tickets. (The development work only for one of these tickets took 4 months and involved 2 expert consultants.)
The lack of separate boards and the level of breakdown of these activities that was missing makes hard for the scrum master and other managers attending the daily scrum meeting to understand what they were managing. It makes hard to understand the complexity of the issues, the daily work that was done by the developers, when this work will be completed and how it will integrate with other modules within the applications.
The lack of interest for the other stream of work was due to the fact that the main activity was taking the bulk of work for the whole team and the managers as well; the other aspect of this particular situation was that the daily scrum and board's review was managed by a scrum master that was not too technical while the senior development manager was focused on keeping the main activity afloat by coordinating and providing releases weekly (and sometimes daily).
In this way, some of the work fell behind for more than a year and major issues were presents for a long time in Production for processes that were not too visible to the end user.
The senior development manager knew about the other activities that were not covered properly; they were in the back of his mind but not having the same priority as the most present issues. However, the fact that they were not visible on a board that was maintained by the team and not well known and understood by the scrum master, it is the takeaway from this example.
Always keep visible all of your other stream of work and treat them with the same level of attention and using the same tools you are employing within your team.
Recognize the different level of expertise of your workforce and align the process to take this into account.
One of the characteristic of the agile process is that it is enhancing the collaboration between the team members. In some agile methodology, the team members used a card system to vote for the time required by another team member to complete his work. An average time it is used to be assigned to a ticket after eliminating the votes with the maximum and the minimum values.
The collaborative nature of the agile process makes less and less visible the lines that separates the team members due their different level of expertise.
In a waterfall model, a team will have a number of technical leads and a team lead working under the department P.M. The code completed by the developers were sent to review to the technical leads; only after the changes required by the tech leads were implemented by the developers the code was labeled and make it in the next day development build.
A recent experience I had in one of the Agile team I have been involved with, requires all the developers to ask for code review the other team members in a MS Team chat created specifically for the development team. The code requires 2 approvers before it can be merged into the main line. Any developer could jump in and provide his comments about the code and ask for changes.
This situation can easily evolve into long chats were the people involved into the code review needs to be coached about subjects they do no master and could delay the development process and creates friction between team members with conflicting personalities.
A similar situation happened when you are asked to investigate a critical issue that created serious problem for the company you are providing services to. One of my clients asked me to investigate a production issue that stopped their processes for more than one week due to a database timeout of more than 90 minutes. I have investigated the problem, found out the issues which were caused by a wrong design approach, provided a quick solution at the database level that makes the queries executed in milliseconds, created a thorough detailed report with the problem's assessment and solution and provided it to the team and the P.M.
During the meeting's review of the solution, the most inquisitive and unhappy person with the solution I had provided was the person who created the issue in the first place!
This situation it is a common occurrence in many work environments and can be addressed and alleviated just by the team manager in case he has enough experience to understand the issue.
The management should understand that an agile process does not eliminate the hierarchies that are established in a team by acknowledging the different level of expertise. Architecture work needs to be reviewed by architects, work created by senior people should be reviewed by senior people as well and so on.
There is no substitute to estimating and planning.
This topic it is the reason I have started to write this article in the first place.
In the last 20 years I have provided consulting services via my company to various other companies from the GTA and across Canada as well. In many cases I am asked to design not only a new product the company needs but as well build a development process that the company lack which sometimes involves as well the building of an infrastructure (servers) to support the process.
In every single assignment I was able to estimate correctly the overall work and deliver the project on time, with a project's duration that vary between 6 weeks to 1 year and different team's sizes. Upon the successfully completion of these projects, my clients became references to other clients interested in my work.
Having said this, this is the area where the majority of companies, their managers and scrum masters' experience lacks the most.
It is true that the agile methodologies are focused on the flow of work and in some cases the delivery milestone it is seen as "open" (hence open agile): Kanban it is focused on eliminating the blockers and set up the WIP limits to reduce the stress on the people performing the work.
Having said this, I have to note that I don't remember when it was the last time when a manager asked me for an estimate to the tasks I have been assigned with.
In one of the cases I was involved with, the team's P.M. was holding the daily scrum meetings with the team. In spite of daily interaction with the team, one week before the ending of the first 3 months of work, he did not knew what the team was delivering at the end of this period. This caused new requirements to be added to the project and an emergency release to be planned and created in another month timeframe and involving serious amount of overtime work by the team's members.
There is no substitute to estimating and planning. The proper way of doing this is first to ask for these estimates from the person doing the work. Once the estimate it is received, reviewed and understood other components needs to be added by the P.M. to complete the development timeframe i.e. deploying, testing and contingency time among others,
I have been working in companies where customers where scheduled to come in for a week of testing in order to test the new features of the product. In these companies, the project plan was created for long time in advance and specifies the work to be completed in each week (Teradata).
A case study: how Microsoft could not manage properly a team until they have adopted Kanban.
The present article grew very fast and I feel it is better to write a separate blog article about the next subject: the topic it is related with the lessons learned implementing Agile processes in major companies and describes the challenges encountered by the XIT Sustained Engineering team at Microsoft and the first implementation of a Kanban process in an IT department.
I have continued this topic in the blog post from below: