Realistic case study on agile development at large scale

A Practical Approach to Large-Scale Agile Development” by Gary Gruver, Mike Young and Pat Fulghum is a real-world example on how scaling of agile software development really works in a huge software producing organisation.

A Practical Approach to Large-Scale Agile Development

The authors describe in an easy-to-read language the journey of the HP firmware organisation starting in 2008 and taking around 3 years with a clear goal in mind: “10x developer productivity improvement”.

In the beginning they were stuck with a waterfall planning process with a huge planning organisation and not being able to move in software development as fast as business expected. A quick summary of activities showed that in the beginning the organization was spending 25% of developer time in planning sessions to plan the next years’ releases. Only 5% was spent on innovation. Nowadays, after acomplishing the 10x goal, 40% of developer’s time is spent on innovation.

The book highlights the relevance of a right mix of agile technologies with a good approach in software architecture and organisational measures to form a successful team of people striving for common goals. A fascinating read!

Most striving is the unemotional view on agile and how to apply it. They purposefully decided not to have self-organizing teams. So, agile is broken? Can’t be applied in such an environment? Not at all! The authors give good reason for not applying all agile patterns from the books – and it is working.

Agile at LEGO

Quote

I am interested in agile software development methods, LEAN management styles and the impact on a company’s organization. All these attributes are usually associated with software development companies – not hardware companies or toy companies.

I got a “must-read” recommendation for the book “brick by brick” by David Robertson. "Brick by Brick" by David RobertsonThe book proved to be a real “must-read”. Especially the portion where David Robertson describes the turnaround of the company. The methods applied to an almost bancrupt company reminded me to techniques nowadays well-known in software development teams. I mailed David Robertson and asked for permission to cite a portion of his book (page 170, Brick by Brick, David Robertson):

Before the rise of Bionicle, the LEGO Group’s product teams were soloed from one another and toys were for the most part developed sequentially: designers mocked up the models and then threw their creations over a metaphorical wall to the engineers, who prepared the prototypes for manufacture and then kicked them over to the marketers, and so on down the line. Rarely would one team venture onto another team’s turf to offer a suggestion or ask for feedback. If all went well, the team’s product would hit the market in two or three years.

The Bionicle team’s six-month deadlines forced a different way of working, one that was less sequential than parallel, and highly collaborative. Once the outline for the next chapter of the Bionicle saga was roughed out, the different functional groups would work side by side in real time, swapping ideas, critiquing models, and always pushing to simultaneously nail the deadline and build a better Bionicle.

“We had a massive project team,” recalled Farshtey. “It wasn’t just the creative people; it was also people from Advance and from marketing, sales, events, PR – all different parts of the company, all helping to steer the franchise.”

Because the marketing group worked directly with designers, Bionicle’s advertising campaign felt connected to the product. Promotional posters for Bionicle’s first-year run had to look and feel of movie posters, precisely because the toy featured the powerful visuals and narrative sweep of an epic film. “We wanted more communication in the product and more product in the communication,” said Faber. “That meant the marketing group needed to be involved at the very start of product development, so the story flowed out through the product. We wanted the product almost to tell the story by itself.”

“We had a kind of triangle, where the marketing, the story, and the product had to move ahead together,” he continued. “None of those could be the spearhead. Each needed to support and inspire the other.”

That single page really got to a point. I’m relentlessly telling that the core of the agile movement is about focus and collaboration (see Agile Defined). Usually, I refer to these values in software development environments, organizations and teams. But LEGO – a brick & mortar company – used the same principles in 2001 to turnaround their company. That was really mind-boggling news to me.

After doing some research on this topic, I found a SMB here in Germany working with agile principles: HEMA Bandsägetechnik. This (german) article talks about their experience.

 

Customer & Service Provider interface – the End of the Story

Customer & Service Provider interface … the End of the Story 

With the start of the-then-new job in October 2015 I started writing about my experience – and especially the shortfalls – in our customer and service provider relationship. It turned out to become a triptych. This is the final – the wrap-up.

To catch up you might want to read the first post: “Customer & Service Provider – Epic fails” and the second in this series: “Customer & Service Provider – silver lining on the horizon“.

You might ask yourself … why is that the end of the story? Improvements in processes usually don’t have an end. Well, let’s see.

The Modified Setting

In the meantime we upgraded from 2,5 software developer to even 3,5 software developers. Ambition behind is …

You might guess it? …

Come on …

YES! Right – on the first attempt.

Speed up the delivery of features. Our senior management is delighted by the achievements so far – however is far from being satisfied with the results, the outcome. Still, there is great belief that more people build more results. True! BUT we’re still having the issue of not well-defined requirements and still having long periods of wait for answers from our business leads.

On a daily basis we’re still in communication mode. Twice a week our project manager and the project manager of the service provider meet to exchange progress and start discussion on new topics. Again, communication proves to be a gold nugget!

The Now Situation

Missing product management – It became so obvious, so non-neglectable, so crystal-clear. Reason behind the low througput of our development cycle is not the lack of software developers – that’s a symptomatic solution. No, root cause is: we’re lacking a professional approach to manage our products – product management is missing. Usually, product management’s major duty is to define requirements on their products and harmonize the various needs and demands from all involved parties – customer, departments, management.

If we just had some people doing exactly this job. World would be much clearer!

  • Prioritization would be easier – since the product management would clearly include management in the overall prioritization process.
  • Definition of requirements would be clearer – only well-defined stories were handed over to our service provider.
  • Decision points would be clearer – product management is the definition point for detailed questions and processes.
  • Communication would be simpler – it’s simply clear whom to ask for guidance.
  • Development would be faster – no impediments any more … why should it be slow now?

In my last post, I raised a lot of points:

  • Multiple stakeholders talking to the service provider
  • No clear communication of priorities, requirements and goals
  • Fundamental changes in design after going live
  • Bad requirement quality
  • Silo thinking
  • Low quality – not well thought-out
  • No consistent prioritization

All of them – I really believe – all of them could be solved with the implementation of an organizational change – introduce product management.

So, you might ask yourself why is this the end of the story? Well, that’s an easy one. Despite my continued effort to convince my senior management about the absolute need to improve the organization by introducing a product management person (yes, one person!), I failed. “Too expensive”. I personally decided to quit.

Lesson learned?

I’m still convinced we changed a lot for the good in the customer / service provider interface. Agile patterns did a great difference for all of us. Furthermore, the service provider did invest in agile coaching (and architecture consultancy as well … not all is gold …) and showed great willingness to cooperate – to improve.

Go on, improve your interface. It’s well worth the effort! Ah, and don’t forget to secure your senior management’s backing in early stages 🙂

Customer & Service Provider – silver lining on the horizon

Customer & Service Provider interface … epic fails and how to overcome them

In a recent post I was talking about an interesting situation regarding the collaboration with our service provider (see https://www.agile-minds.com/customer-service-provider-epic-fails/). After some weeks I’d like to take a chance to draw some conclusions and report back some successes and learnings. I definitely see some silver lining on the horizon regarding the collaboration part – but also discovered some real black holes within our organisation. It’s always the same: process insufficiencies will always show up – latest – in the IT department …

The Modified Setting

Now in the development partnership with our service provider we pay for 2,5 software developers + 0,75 project management person. This means we added significant ressources to the aspect of project planning, communication and management.

Our project management person and the pendant at the service provider really met twice a week to discuss progress, issues and need for further planning on specific tasks. In addition to that it’s me and the general manager of the service provider meeting almost weekly to discuss issues and problems. This approach helped to improve the communication a lot. We’re now in a continuous dialog. Understanding of each other has raised enormously. A great step!

The New Situation

We are still not happy with the outcome – but the reasons have changed dramatically. Previously, we as an organization were not able to articulate our needs and get this information over to our service provider. We changed our communication paradigm and introduced the face-to-face communicaion twice a week. Now, the service provider has a ways better view of our requirements and is eager to start implementation. However, a lot of our stories lack quality from a subject matter view. How did we come to this point?

Multiple stakeholders talking to the service provider – We only have one person talking about new requirements – our project manager. He’s responsible to collect requirements in-house and communicate them back to our service provider. This works quite good. We’ve setup internal bi-weekly calls to collect and prioritize requirements. We’ve had our first quarterly face-to-face meeting where we started to build and organize the company backlog.

No clear communication of priorities, requirements and goals – Flag it as solved with the previous point.

Fundamental changes in design after going live – This is still an ongoing topic. But it’s a question of educating our internal people. Since we’ve one single person doing the communication towards the service provider it’s one of his duties to explain people that if there’s a change in requirements after going live we’ll treat this change as a new requirement – and this needs to get prioritized and implemented. Education.

The New Symptoms

No progress over all – Nope. Having progress. Or eager to go on. But …

Bad requirement quality – This was not transparent so far. But it came up when we introduced the new way of working. It became clearer and clearer that our subject matter experts – the stakeholder – didn’t exactly know what they wanted – not even at the point when they were talking to the service provider. Or they knew what they wanted – but didn’t consider the other business units in the design phase.

The Idea … did work out so far

The two aspects I’ve decided to implement

  • Setting up a stable and reliable single face of contact on working level
  • Introducing basic agile principles to get to stable requirements

worked out smoothly so far. As usual the introduction of agile aspects increased transparency of involved processes and especially the quality of the different artifacts a lot. And new issues turned up.

Quality of Requirements

As said previously the introduction of agile principles – well, I’d say of new communication patterns – produced process insufficiencies of our own organization.

Silo thinking – our business is separated in different business units. The different units have their own targets and try hard to reach them. There is no competition between the units – but everybody takes care of their own business. Two of the business units eventually use the same instance of the student information system. Especially between those units we have a lot of cases where e.g. the layout of the exam certificates differs, rule sets for course acceptance differ, specializations of courses differ and so on. Small differences – which might be needed … or not – which have a big, big impact on implementation capacities. It’s a difference to create different rule sets for courses – double the implementation work. And why? Ah, the business stakeholder didn’t have time to discuss their individual needs and combine them into mutual needs.

Low quality – not well thought-out – Another point we found is the quality and depth of the various business requirements. A lot of our business requirements arrive at our project manager at a stage where it says “We need a proper implementation of …”. Well, what does “proper” mean? Many times we experience developers from the service provider receiving last-minute changes – still. That’s one of the focus points for future improvements.

No consistent prioritization – Our exec management wants to review the implementation prioritization. Fair enough. We’ve agreed on having a call/meeting every second month. In this call we’ll run through the company backlog and listen carefully to what exec management sees as top priorities. In parallel we have our working calls with the business stakeholder every second week. Together, I believe, we’ll arrive at a prioritization which puts most important topics first.

My biggest learning out of this so far?

No matter how big the issue – start introducing agile patterns. Here especially the communication, company backlog and the face-to-face meeting helped a lot to unveil the real issues behind the “bad customer / service provider” interface.

The nice part about the agile pattern is the transparency. It automatically points you to the real pain points – through transparency.

Stay tuned and read on – to be continued.

Clear your mind using “The Thinking Day”

John Donahoe, at this time CEO & President at eBay, published a great post on his way to clear his mind, step back and rethink what’s important. He described three ways to focus and rethink. I found the most impressive one being “The Thinking Day”.

The Thinking Day

The thinking days are pre-scheduled, non-interrupted times with the intention to step away from daily business with the pure purpose of focus and rethink what’s most important right now.

Move away from your office, book a meeting room just for yourself and some paper. Start mapping out what’s going on around your company and identify the most pressing topics and issues in your organization. Think about what you need to focus on, where you need to put your attention, what you need to change.

Write your findings onto the papers and keep them as guiding principles for the coming period. Allign your daily calendar to focus on these principles.

To allow your direct reports to be able to follow your re-allignment, it’s worthwile sharing your findings with your mates. That’s the only way to work efficiently and effectively into the right direction – with a clear and shared goal.

Customer & Service Provider – Epic fails

Customer & Service Provider interface … epic fails and how to overcome them

Just recently I decided to change positions – moving away from a CTO role with a whole development department attached towards a CTO role with all software development tasks being outsourced to a service provider / product company. And I felt like being in a time-warp … not into the bright new and shiny future – no … directly into middle age – the darkest version of it.

See why – and how I plan to maneuver all of us out of this.

The Setting

The product company is very specialized on delivering a specific product aiming at university campus management software. We’re paying 2,5 developers to work full-time on our product branch – we call it a development partnership.

The product company develops in PHP, does deployments with puppet – however utilizes a full tomcat environment and PostgreSQL as database. They claim to work in an agile manner … and utilize JIRA (yeah!). Peeking into their JIRA unveils a per-person-association-to-tasks (well …). Visiting the development team produces clean white walls, no empty bottles or cans around, no architecture diagram scribbles, and no nothing. Suspect environment for full-heart agile developers. But let’s see.

The Situation

We are not happy with the performance of the service provider. Why? They don’t deliver on time, in quality and make promises all the time. Hmmm. After few visits and talking in-house and the service provider the situation became clearer to me.

Multiple stakeholders talking to the service provider – We had multiple parties of our business units talking to various people at the service provider. All of us spoke with different voices and everything was super-important. In the end not the feature with highest business impact made it live – it was the one of the person shouting loudest…

No clear communication of priorities, requirements and goals – In our discussions with the various people at the service provider we were not able to communicate a clear order of priorities nor were the requirements defined – not talking about documented or clear – and goals of the implementation tasks were sometimes not even clear to our internal stakeholders.

Fundamental changes in design after going live – I think that’s a real classic: The whole implementation is done, the product change is live and the stakeholder suddenly realizes that the color coding of a logo needed to change. So, the stakeholders didn’t even realize that changes in the live product are the most costly changes – compared to those done on powerpoint or photoshop level right in the beginning.

The Symptoms

No progress over all – Having not communicated a goal, a strategy or anything with a guiding function it’s no wonder there is no felt progress in the cooperation with our service provider. Sure, everybody was giving his/her best and we were moving. But nobody was satisfied at all. We didn’t get what we wanted and the service provider felt bashed at every move they made.

Communication went really personal – I personally attended the at-the-time two-weekly phone call with our service provider. We had the CEO, 3 business unit leads, one project manager and another 3 subject matter experts on the call – together with one of the managing directors of the service provider. As usually, I learnt afterwards, the call ended in real bad communication on a personal level. There was no common sense, every word was political, no commitment to anything – it was protect my a** everywhere.

The Idea

This needed to end. So, after my first week I had a meeting with the board of the service provider. Board means two people – the founder and investors. I presented them with an idea to improve the situation. At the core of the idea there were two main principles:

  • Setting up a stable and reliable single face of contact on working level
  • Introducing basic agile principles to get to stable requirements

They bought into this approach and we agreed to set this up.

Stable and reliable single face of contact on working level – I made my project manager to spend 80% of his time to manage the interface to our service provider. The service provider put 40% of a person into this task.

The two of them meet twice a week. Idea behind is to get closer communication flowing. The sooner we know about problems, the faster we can solve them. Anything not clear? Any issue to solve? Too less resources? Whatever. On Tuesday, they meet to discuss the current stories in implementation and issues. Furthermore, they discuss future stories to get feedback on expected implementation issues and too less specific requirements. On Friday, they meet to discuss the same topics and in addition I and the managing director meet to discuss and decide on topics to be resolved.

Internally, we made clear that it’s my person and the project manager who speaks to the service provider – nobody else. NOBODY! This is the only way to remove distracting conversations on non-important topics and puzzling requirements. Not to talk about setting other priorities on tasks. This all happens when the internal stakeholder have direct access to the service provider. Now, the project manager collects the user stories and requirements. We meet on a frequent basis – at least once a week, usually every day – and exchange our knowledge on user stories.

Introducing basic agile principles to get to stable requirements – Obviously, we had XLS-sheets of various formats and contents. All of them full with requirements and sorted to various priorities. We kindly removed all of them and started our Company Backlog.

We agreed with the service provider to have sprints of 2 weeks. The service provider still does releases on a 4 week basis – but we work in 2 week sprints. This allows us to communicate delays of features in terms of 2 weeks instead of month. “Well, it didn’t make it into this sprint – we will put it into the next sprint. This starts in 2 weeks.”

The Company Backlog is now organized in JIRA. We attach all relevant information to the user stories and hand them over to our service provider. Unfortunately, we couldn’t agree on using the same JIRA instance. Initially, I thought to use a Company Backlog view where we organize the next sprint which is then handed over to development and they start associating tasks to the user stories.

The daily standup is a bi-weekly get-together of the relevant people and starts to work out pretty good. During these meetings the two of them do a kind of backlog grooming and discuss the next stories to be implemented.

So, to summarize. Agile elements are:

  • The Company Backlog
  • The Product Owner (on our side – still named project manager)
  • The Scrum Master (on service provider side – also named project manager)
  • Sprints of two weeks duration
  • Backlog grooming
  • Planning 1

Does it work? Well, I’m sure it will be an improvement after all. Is it optimal? Let’s see!

 

Web frontend performance – distilled

Web frontend performance – distilled

Web performance used to be (in the good old server-only / server-rendering days) mainly dominated by the performance of your webservers delivering the dynamic content to the browser. Well, this changed quite a lot with application-like web frontends. Their main promise is to replace these annoying request/response pauses with one longer waiting period in the beginning of the session – but then have light-speed for subsequent requests.

Here are some really good links I just discovered today – they all deal with various aspects of frontend web-performance. Let’s start.

Comparing MV* frameworks? There is a great project – named TodoMVC – that compares various frontend-frameworks – amongst them are Backbone.js, AngularJS, Ember.js, KnockoutJS, Dojo, YUI, Agility.js, Knockback.js, CanJS, Maria, Polymer, React, Mithril, Ampersand, Flight, Vue.js, MarionetteJS, Vanilla JS, jQuery and a lot more.

Performance impact comparison by the filament group. A good effort on research was spent on the topic “Research: Performance Impact of Popular JavaScript MVC Frameworks” – focusing on e.g. Angular.js, Backbone.js and Ember.js. Performance testing was done with the previous mentioned implementation of TodoMVC. The raw data is accessible as well. Most interesting are the results (measuring avg. first render time):

Mobile 3G connection on Nexus 5

  • Ember averages about 5 seconds
  • Angular averages about 4 seconds
  • Backbone averages about 1 second

PC via LAN

  • Ember averages about 1.17 seconds
  • Angular averages about 0.88 seconds
  • Backbone averages about 0.29 seconds

Practical hints to accelerate responsive designed websites. In his post “How we make RWD sites load fast as heck” Scott Jehl (@scottjehl) gives some pracitcal hints on what to focus on:

  • Page wight isn’t the only measure; focus on perceived performance
  • Shortening the critical path
  • Going async
  • Inlining code
  • Shooting for 14kb
  • Taking advantage of caches
  • Using grunt in the deploy pipe

Angular 1.x and architecture problems. Another interesting blog article by Peter-Paul Koch (@ppk) focusses explicitly on Angular. “The problem with Angular” talks about severe performance problems with Angular 1.x versions. In his blog he notes

” … Angular 2.0, which would be a radical departure from 1.x. Angular users would have to re-code their sites in order to be able to deploy the new version …”

Wow. That’s interesting and a good indication for serious architecture issues with Angular 1.x …

Thought-leading companies and performance. Good articles / blog posts from leading companies on page speed performance:

Agile – criteria to look for in an Agile business

In February 2013, I made an attempt to define Agile on my blog. I ended in a quite high-level definition of Agile:

Agile is a collection of values and principles that encourage a certain type of behaviour: focus on value generation and collaboration.”

Just recently, I found an interesting blog by Rouan Wilsenach working with ThoughtWorks where he went one level deeper and explained “Four Attributes of an Agile Business

1. Feedback – “This is what my customer wants.”

In my definition this relates to focus on value generation. Companies need to put the customer in the center of all their efforts. Rouan refers to an interesting list / toolbox on experience design. Only if organizations take this attribute serious they will finally be successful applying principles of agility.

2. A responsive team – “Yes. We can do that.”

In my definition this very much relates to values and principles supporting collaboration. Responsiveness comes only with co-location. In my experience, true collaboration over a distance can only be achieved within a trusted environment. People need to trust and respect each other to become a distributed team. If they don’t know each other of follow slightly different goals, it’s very likely to fail. So, the team spirit can be created most easily when forming inter-disciplinary teams which are located ideally in the same room. Rouan refers to Conway’s law (comparison of code structures with organizational patterns)

3. A responsive code base – “It’s ready. Shall we release it?”

For me, this again a perfect example of value generation. Have a feature ready? Why not shipping it immediately? Great entry for principles like “Zero Bug Policy” or “Continuous Deployment”.

Furthermore, a responsive code base is a clean code base. People love to work on such a code base. It’s almost bug free, easy to read, simple to understand, documented. People are proud of what they’ve developed, created.

4. Continuous direction – “What do we do next?”

Again, this point goes as value generation. Why employing a bunch of people working hard on software development, building great products if there is no direction behind, no strategy?

Clearly one of the major tasks of managers in an agile environment is to set the strategy. Product people break this strategy down into epics and stories. So, what to do next? Pick the next story from the prioritized backlog.

So, for me Rouan picked some really great criteria and attributes for an agile business. They’re in line with my previous definition (which I like) and take it to a less abstract level.

Reducing staff leads to higher productivity – proof for Brooks’ law?

Recently, my teamlead agile methods pointed me to an interesting corelation. After a discussion we concluded with a really mind-boggling observation. A settled organization facing change will increase productivity when reducing staff headcount – in other words: “Reducing staff leads to higher productivity”.

Bang! But let me explain. For software development tracking and planing purpose we use JIRA – so we’re able to track the amount of created and closed software development related tasks quite easily. The headcount in software development, product management and UX/design can be obtained easily by – counting (we actually asked our HR department for support).

In our case we ended with this curve – impressively proving Brooks’s law.

Brooks Law in action

The graph shows a period from June 2013 (left) to July 2014 (right) and focuses only on one of our products at the time.

Green shows the number of software developers working on the product. Purple shows the number of product and UX/design people involved. The two curves are declining.

Red shows the amount of releases done in the period.The curve is trending towards a stable, flat curve.

Blue shows the amount of tickets closed and turquoise the amount of bugs closed. The two curves are decreasing trending towards a way higher plateau.

What happened? In June 2013 time frame we had a whole architecture team and quite some software developers organized in three teams working on our product. Around this point in time uncertainty hit the organization due to rumors of to-be-expected acquisition of our whole company. Uncertainty translates often to people leaving the organization. This can clearly be seen in the middle of the graphic. Surprisingly, the curves associated to work results don’t decline – as one would expect – but decrease even. So, productivity of the remaining staff increased! The most right date in the graph is July 2014 and reflects an organization which has one product team and one support team left. The architecture team disappeared completely, the team is ways lighter equipped with people.

What might we learn? Brooks’ law indicates that adding people to a project doesn’t automatically make the overall project finish faster. It might even end in a longer project run time. We’ve seen that removing people might result in better productivity. We think the fact of removing whole teams dedicated to special topics (in our case: the architecture team) helped to increase productivity in two ways. First, the responsibility of architecture decisions was pushed back into the teams and dead-lock situations where person A waits for a decision from person B just disappeared. Second, the time needed to communicate, agree and disagree simply vanished. Decisions had to be taken and nobody needed to be asked.

Is this a general pattern? I personally think this effect does only appear for a certain period of time. The drawback of such leaner organizations is obviously the lack of work on technical debt and architecture drawbacks. People in such situation focus on the obvious and postpone work on maintenance tasks for later. For me, it is definitely an observation I’d like to share with you. Perhaps you have similar experience? If so, please let me know.

Is there really a relation between the number of staff members and the productivity? Where is the limit? How can one push the limit?

Clean Code Developer Initiative – a structured learning way to skill improvement

“Clean Code – A Handbook of Agile Software Craftmanship” by Robert C. Martin is a classical reading for software developers. Great, but how do you get the spirit he described into the heads of your whole software development organization? The Clean Code Developer Initiative by Ralf Westphal and Stefan Lieser drafts an interesting way to improve the skills of your software developers with the spirit of Uncle Bob Martin’s book in mind.

The Clean Code Developer Initiative (CCDI) proposes a grade based system with color coding. It borrows concepts from game design by introducing a level concept. Entry level is marked with black and levels up via red, orange, yellow, green, blue to white.

For each level there are principles and practices listed which a software developer – or a whole department – should follow to get to the next level.

Urs Enzler from planetgeek.ch produced a nice Clean Code Cheat Sheet – it’s worth a look.

Clean Code Cheat sheet

There is a nice poster available by “unclassified software development“. It’s even possible to print it out in A0 format – huge!

Clean Code Developer Poster

The whole level of a developer can be shown through wristbands. Allow them to be proud on their achievements!

Clean Code Developer wristbands

I personally haven’t tried the effort to introduce the principles of Clean Code into a software development department – but it looks like it’s a feasible effort. Accompanied by trainings and some team events, it should be possible to appeal to the developer’s honor – and improve your software quality standards.