Agile transformation challenges

I was asked last year to do agile transformation in my organization. Before you continue reading, I would like you to read the following:

The 11 Laws (from the Fifth Discipline)
  1. Today’s problems come from yesterday’s “solutions.”
  2. The harder you push, the harder the system pushes back.
  3. Behavior will grow worse before it grows better.
  4. The easy way out usually leads back in.
  5. The cure can be worse than the disease.
  6. Faster is slower.
  7. Cause and effect are not closely related in time and space.
  8. Small changes can produce big results…but the areas of highest leverage are often the least obvious.
  9. You can have your cake and eat it too —but not all at once.
  10. Dividing an elephant in half does not produce two small elephants.
  11. There is no blame.
When we talk about agile transformation these days, that often comes as the top down initiative. In very beginning of agile, back in early years (2001-3), agile was either started by teams in small start up companies or by folks that are early adopters and innovators by their nature. Therefore, challenges that you face these days are different:
  • Since agile has become de-facto mainstream methodology, agile transformation is normally lead by people that are not agile by their nature (but are assigned to execute it)
  • People that should practice agile are not necessarily convinced that agile is a good idea
  • Stakeholders expect agile to fix all their problems (and that often has little to do with the development process).
To complicate story further, you should also remember that you will also be facing other problems that are related to the change management process. These forces are always there:
People who are currently in management positions will tend to resist the change as they can only potentially loose in that process and people who are going to be on your side are the ones who can potentially benefit from the change – so they don’t hold strong position in the organization. And if that was not enough, there is one problem more, which comes with agile. Agile is very transparent methodology, and it doesn’t bring excellence per se, but is very effective in pointing out to the incompetence in development. So even if you like to think that developers – people who should benefit from the change will be on your side, this is not necessarily true. This will also depend on their engineering skills. Therefore, you do need the strong support from stakeholders, otherwise you will never make it. You constantly need to work bottom up and provide training to dev. and PO organization, work on their skills and check the training needs, work with middle management and train them on agile and soft skills and finally provide honest and effective reporting to the stakeholders. Since reporting alone is another issue for stakeholders (“responding to change over following a plan”),  you will have a lot of fun there too, guaranteed.
So, here it how normally goes with agile transformation:
The 7 Phases of a Change Management Project and how it relates to agile transformation:
  1. Wild enthusiasm (everybody goes to scrum course, read few books about agile, looks that simple and obvious)
  2. Disillusionment (arguments and misunderstandings all over the place, people go loose and you start seeing real characters… it is about people before process – isn’t it?)
  3. Confusion (backlog is filled and flat, and folks just don’t know what to do first. Some people resist writing docs, other want them, PO is freaking, TPM hardly can keep things “under control”)
  4. Panic (stakeholders start hearing only problems, sales guys are coming back and asking for delivery date, and all that they can hear is epics, iterations, story points…)
  5. Search for the guilty (by now, agile coach is already either out, or like in my case, they still have trust that I know what I am doing)
  6. Punishment of the innocent (not there yet…)
  7. Promotion of non-participant (go back to waterfall, V model, or whatever existed before).

In my next blog post, I will be talking about how I tried to address all these challenges….

Agile development and people

It is always challenge for an engineer to talk, not to mention to put in writing, about subjects that have little to do with “doing stuff”. It is not that we haven’t read any books; we just read the ones that have nothing to do with other people, emotions and “non-scientific” topics. Engineer education is all about solving intriguing problems. It could be more than that, but frankly that is all I remember. That is fine, but there is one problem with that. You are coached how to solve problems ALONE.

So, here you have a young smart guy coming out of the University, with his diploma and great ideas, entering into a strange “jungle” called Corporation X, and suddenly surrendered by strange creatures called marketing, sales, managers, customers…. That is how it all started for so many of us.
After few miserable years and 2 project failures, I started reading some other books, not only on how to do things, but how to do things together. This time I was reading about people, about emotions, and about getting the job done. Luckily, about the same time, few great things happen to me, I became a project leader for 5530NA, the team was young and full of people eager to try things, and yes, agile arrived.

In this paper, if I may call it “essay” I will not talk about process and why I prefer X rather than Y, but only about things that people should be aware of if they want to try agile.

Process and people

Why process and people, why these things together?

Some people think that one person is simply either good or bad. Others might try to correlate education, parenthood, religion and things like that to how actually people behave in their working environment. I will call this a “floor spirit” (I really have no better name for this right now, I am low on coffee).  It is hard to define it, but I am sure everyone knows what I mean by this. For every novice at the floor, it takes 10 seconds to smell it, not more than that.
In my opinion, the most important factor in “floor spirit” is the way people are put together in a social context. “Common culture” that arises from a given process labels people and assigned them roles, which in the end make people to behave in predictable manner.  This is true for all processes, there is nothing wrong with this, and we just need to recognize existence of this invisible force that so often we forget about. Some processes can be very different from their predecessors, and any time someone wants to introduce new radically different approach, people need to be aware of forces that will eventually surface, whether they like it or not.

So, let’s first see what we have as processes in our Industry:

Below is the table that summarizes the main differences between traditional developments and agile methodology, from “Theoretical Reflections On AGILE DEVELOPMENT METHODOLOGIES” (one great thing I learned from agile, if there something already on the web, just grab it, that is fine as long as you are honest about it):

Ok, so let’s look closely some of these topics and see how I read it:

Design process

Traditional View of Design: Deliberate and formal, linear sequence of steps, separate formulation and implementation, rule-driven.

Naturally, due to the way the work is organized, you get “castes” in the group:
Project managers, core architects, module architects, feature analysis folks, spec writers, coders, testers ….
Sooner or later, coders will be perceived as “low caste”, the ones that are told (or given in writing) what to do, and as such after a while, moved to low cost country. “Coders are people that turn coffee to code”.

“Code for food”, USA beginning of 21st Century

“Smart” guys are the ones who have attributes in their role like architects or analysis, people with little or no exposure to actual code writing. Process of writing code becomes slow, painful and to some extend obscured activity.
People in such project tend to become either:
Frustrated people:

  • “Nothing moves, I have great idea, but who cares, and managers are idiots”
  • “These stupid coders only write buggy code, nothing actually works”
  • “You want to change this stuff now? That was not in the contract, so please make a new one and come back to me”
  • “You said stuff will be done 1 year ago!!”

Happy people:

  • “I have made a great architecture slide; I am master of the Universe”
  • “I have almost done with my spec and then I go for holiday”
  • “What ever I do it is not my fault; I am just doing what I was told to do”

Design process

Agile: Emergent, iterative and exploratory, knowing and action inseparable, beyond formal rules

Suddenly everything has changed. Rules of game have turned upside down.

If we go back to our castes, let’s see how they will react at first sight:
Project managers: “What do you mean, I have no control? Are you nuts?”
Core architects: “You mean I have to write a code?”
Module architects: “This is my stuff, it is too complex, and how you think that someone else can know all that stuff?
Feature analysis folks, spec writers: “Well I just don’t know any code, what I am going to do?
Coders: “yeah, I told you so …”

On top of that, other people within the company that are normally not visible can see this methodology as a threat to their job too. Agile preaches self-sustained groups, full of individuals that don’t wait for other people to act.


Traditional View of Design: Optimization

Since everything looks so predictable and convincing, project managers create a plan full of Gantt charts, put all dependencies in place, and optimize for cost. That typically means R&D (“coders”) in one low cost location and testers in the same or even another low cost site. Gurus, spec writers, project managers, PO’s are close to the marketing, sales and customers. That’s about it. It’s been like that for the best part of the last decade.

Agile: Adaptation, flexibility, responsiveness

Collaboration between manager, PO and developers is the key. There are no more coders and architects. In agile group, you have true developers in the team, people who think, act and change things on a daily basis.
You see new roles popping up: scrum master, tracker, developer, (true) architects, customer (did I say a customer? :)) . Also some roles are either disappearing or hard to define: project manager, spec writer, coder …

Key characteristics, “floor spirit”

Control and direction
Avoids conflict
Formalizes innovation
Manager is controller
Design precedes implementation
Collaboration and communication
Embraces conflict and dialectics
Encourages exploration and creativity

Manager is facilitator
Design and implementation are
inseparable and evolve iteratively

In agile, the team is everything. If you have a new fresh team that starts agile, you can nicely observed all phases of what they call “Forming – Storming – Norming – Performing” model. First you bring guys together, tell them “rules of engagement” and let them explore.
In the storming face, they don’t care that much about the project, neither on the task that is given to them. What people care about is how good they are perceived by others (everything is open field in agile), and who has better architecture idea. Brilliant guys tend to spend most of the time during this face in endless discussions in proving their points, locking their horns with others, during design phase of any feature.  Once that face is over, music starts and people start enjoying.  Everyone knows who can do what; norm of the team is created and maintained by the team alone. This is when you start hearing laugh at the floor and when you start wondering whether some people ever go home, or they just got kick out by a landlord and hide their slipping bag somewhere.  OK that was easy, you have tons of slides and books on the web about agile and there you go, sky is a limit….

If you really need to make transition with existing team, then things are not that easy. It is hard to generalize but I need to try.
In “Traditional teams” (I am really low on coffee, this sounds really terrible) “floor spirit”: is somehow settled. Nothing freaky happens. Spec guys stare at their screens and write some nice docs, project managers shift their gantt charts and spend all their time over the phone, PO’s fly from one customer to the other one and their collective “memoires” get synthesized by system analysis folks, and here or there you would here flies buzzing. And yes, there would be some meetings.  If you are still lucky, you would have few coders around. Otherwise, there would be in another place, coding the thing, and you would see tickets coming from another team, and you would then do trend analysis. Quality department would look into all kind of metrics; financial guys would look into R&D cost and try to squeeze on your coders, while process improvement guys would come with new magic formula every year that would fix all your troubles. In all that calm and smoothness of the atmosphere, can you imagine someone coming and screaming “Let’s do agile!”? I am not sure, I don’t think that people would ever come to this idea from within that environment.
Agile either needs to start clean, or it needs to break current culture. Change from within is hard, as process that defines cultures that defines roles that makes people to have sense of purpose are too far from each other in this case.  Change, if it is to happen, needs to be top down, and then bottom up. You find good coaches; you tell people to follow and at the same time find people bellow that are willing to take it. That is about change process, so let’s see about people

People that would need to change

  • “Old school” guys in their late 40s, 50s, decent C, C++ guys who are still good coders. They will have mixed thoughts on agile. First they like idea that coders are fancy guys again. They also like idea that they can get more freedom.  They might have few problems technical problem (need to compete with new kids on the floor in things that they are not good at, like new language …). In this case, agile is not nice, as it shows to everyone (via estimations for a given task) that they are not that good any more. They need training, they need time and their ego might hurt. If you have such guys on the floor, take your time, talk to them, and help these guys. Sooner or later they will get there. They also have process problem, they will disagree on idea that you don’t need to know everything upfront. They will struggle to split stories, work in iterations, always try to get full picture before they move. It is not bad to have such a people on the floor though; little bit of skepticism to counterpart young guns who see nothing beyond tomorrow is not that bad after all. Eventually they will see after while a benefit and slowly move in the right direction. After a year, these guys will be your soldiers, people who are always there, good developers, systematic in their work, and very loyal. You need them.
  • “Architects” Architects need to code. If they don’t code, then there are not product architects. So they either start coding or they can become solution architects. This is not a joke, I have seen some great guys who don’t code but there are good in writing solution documents, the ones you need in every serious project that involves many products, API’s, use cases etc. Let them go there, and they will do a great job. But if they want to be part of the agile team, they need to code. BTW, most of the time, they are not good in coding. So choice is hard, you need to talk and fix it right away. There is no repair later. Every good project starts with first good “skeleton” release. If you mess up first time, it is hard to recover later. If you have no architect in the team after chat with a guy, then find expensive consultant and pair him with your best young guy. I said young guy? Yes you need one freak, guy who likes more to code than to eat. So let’s find him
  • “Young guy” you need a guy that loves coding. Actually everyone should like it, but this one needs to be really smart, good coder, and would not care about his salary (otherwise would be probably somewhere else). Believe it or not, there are few people around like that, when you interview them, ask for favourite site, books and open source projects, and if you get an answer that sounds convincing, you probably got a guy.
  • “PO” Did I put this in quotes? Uh, let me correct it: PO, you need a good PO. OK, you have person with his MBA, smart guy that talks to customers and come back with requirements. Occasionally he would dress like he is getting married, and then you know he is giving a demo on the ground floor. That is not what you need. You need a guy who will show up every week on your demo and say yes that is what I want. That is the guy who sits together with you and makes tough calls when stuff is not all that right. That is the guy who wants to be involved; the one that sees that agile is a killer when time to market is important. If you don’t have PO who understands that, you are in trouble, you need to make calls yourself and that is not perfect.
  • “Spec writers” This is really hard call. On one side you need some documentation, if nothing else, other people will have questions. Good enough for coders, is not good enough for people who do integration, who look for impact if something is not there, and sooner or later you will discover that. So yes, I admit, you need one good guy who can put system architecture, use cases, flows, API, modules, all that stuff in one nice document, wiki and follow it up. Depending on complexity of the projects (now be careful, everything can be complex) you may need more.  But not too many, and they need to work in the same mode as any other person (iterations, estimations etc).
  • “Testers” they are in agile probably more valued than in any other methodology. They need to actively get involved from day 1, need to follow demos and know all about automatic tools and scripts.  Clicking on a GUI is not enough.  Best is to have at least one tester collocated with developers in case that the test team is on another site. He would act as a bridge, and follow up testers on the other side. He can as well quickly install and test new things.

Agile is not for everyone. Some people will like it, some people will hate it, and some people ill never know of its existence. Starting agile is more about conviction than rational decision. Even 2 biggest brains that ever walked this planet, Einstein and Heisenberg, couldn’t agree on basic principles of a Universe.  So, if you believe that time to market is a key for you, you believe that 1 guy can deliver 10 times more than the other one if trained and motivated, and you have a stomach to take a risk and learn something about people and their motivations, it might work for you too. After all, agile development is as much a social engineering as it is about engineering.