Just recently in Hamburg, I had the pleasure of talking to web site performance addicted people – on the 17th Web Performance Meetup. I talked about the way we at FriendScout24 got the importance of web page speed into the heads of our organization. The whole presentation is available publicly.
The architect role is sold as outstanding important in product development efforts – especially when IT is involved. But I’ve learned some lessons.
The secret.de case – skip the architect
Roughly 3 years ago we started a new business – a high class and exclusive casual dating site focusing exclusively at women. The technical decision was soon done by picking Ruby on Rails v3 as web framework and mongoDB as persistence layer. It was a radical shift away from our current technology stack – pure java and postreSQL.
When we started with Sprint 0 we hired external people to support us. One person acted as the Ruby on Rails trainer for 1 week – to get our people up to speed. At the time we started we had 1 skilled Ruby on Rails person focusing on frontend development and one not-so-experienced person with Ruby on Rails focusing on backend development. The remaining team were skilled java developers. The other external people were one Rails nerd and an architect. During the sprints, it turned out that the architect didn’t have any clue about Rails nor pragmatic architecture. He started to document our project with ARC42 templates … so, we decided to put him aside soon. Leaving the team – without lead. No architect, no direction, no guidance – no hope?
Not at all. What happened? The team started to accept the fact that there’s no over-brain available. No-one making decisions for them. No-one giving direction. And, magically, they took over the ownership for the overall project. Each and every design decision was discussed within the team. Planning II got a total new meaning to the team. Sure, quite some mistakes were made – but most of them due to non-experience with the new technology stack. The Rails nerd was out-phased as soon as the platform went live – after 3,5 month of development.
So, in self-empowered teams there is no need for an explicit architect role. Naturally in team configurations there are more experienced people and less experienced people. A good team will distribute the overall responsibility for good architecture work over all heads. Everybody will carry a piece fitting their experience and willingness to contribute. Plus – you will not run into knowledge distribution problems. Everybody is involved. Knowledge is flowing. New persons can be introduced without a lot effort. The teams credo “fix it if it breaks” led to a low-maintenance and up-to-date system. So, I’ve a fast, fast, fast running application and a real high-performing team.
For me – in the end – the best decision ever was to skip the architect.
At a point in time of your product it might turn out that the maintenance effort starts to increase, the people working with your product start asking questions like “Why does it take that long to achieve XYZ?”, the frontend / GUI doesn’t look that great anymore. Data points start to aggregate towards the obvious solution: Rewriting everything from scratch.
There are multiple posts out there in the community telling you why this is a really, really bad idea.
For me personally the biggest point not rewriting an existing product from scratch is the iceberg of undiscovered processes and dependencies. In a web company, the product actually forms the processes and hence forms the organization. It dictates how e-mail marketing should be done, how editors interact, how landing pages are optimized, how performance marketing is done, how accounting is done, and a lot more. So, in essence it’s the heart of your whole organization. You need to have good reasons to change this! Really good reasons!
This specific post on onstartups.com by Dan Milstein talks about “How To Survive a Ground-Up Rewrite Without Losing Your Sanity” which should be named “Why an Incremental Product Rewrite is superior to an Entire Rewrite”.
Why is the overall approach so tricky?
How could it be done?
Today, twitter (thank you) delivered to me one of the latest articles from HighScalability.com – an article on AOL.com’s architecture.
It’s really great reading and seeing that not only latest and greatest technologies have to be used to create a really good and stable software stack with well-thought-through processes and a great culture expressing proud and self-identification with the architecture.
Notably, AOL.com works with a distributed team of ~25 people on the 5th re-incarnation of the overall architecture. They re-implemented the whole architecture 5 times already! The latest instance is around 6 years old.
Read it – enjoy it! It unveils also some great insights on delivery and management of the software stack.
The really great blog highscalability.com contains a post from Ashwanth Fernando on “22 Recommendations for Building Effective High Traffic Web Software“.
He shares insights from his work experience. Some of them seem to be very Oracle biased – but others are really down to earth and worth considering! The blog post is really worth reading. My favorites are:
Why my favorites? Well, some of them turned out to be really bad design decisions within one of our products …
Page load time is crucial. Business and technology people as individuals start understanding the importance of this topic almost immediately and are willing to support any effort to get fast pages out of your service.
But how can you foster a culture of performance and make people aware of the importance of this single important topic – amongst hundred other important topics?
That was one of the challenges early 2013. Management and myself were convinced that 2013 one of our key focus topics is around web performance. 
The birthday of “T4T”. The acronym stands for …
So, early 2013 we started T4T as an initiative to bring our page load times down to good values. To measure the page load time we experimented with two tools: Compuware’s Gomez APM tool and New Relic’s APM tool. Gomez was used initially for our Java based platform and New Relic for our Ruby on Rails platform. But we were able to measure and track-down some really nasty code segments (i.e. blocking threads in Java or 900 database requests in Ruby where 2 finally did the same job).
How did we get the idea of T4T into the organization? Any gathering of people with presentation character was used to hammer the message of web performance to the people. Any insight on the importance, any tip, hint, workshop, conference, article, blog post, presentation, anything was shared with the team. Furthermore, T4T was physically visible everywhere in the product development department:
THE LOGO – visible … everywhere … creepy!
T4T logo and information on page load and web performance at the relax area for software developers and product owners …
T4T at the VP office door.
For me, especially the endless talking about the topic, raising the importance, questioning of e.g. JPG picture sizes, special topic discussions on CSS sprites vs. standalone images or the usage of web-fonts for navigation elements helped a lot to raise the curiosity of people. Furthermore, giving them some room and time for research work helped a lot.
What did we achieve? Well, one of our platforms – based on Ruby on Rails started with page load time of 2,96s in January 2013. End 2013, the platform was at an impressive 2,15s page load time. In the same time, the amount of page views increased by factor 1,5!
Page Load time over the year 2013
During the same time period, the App server response time dropped from 365ms to 275ms end of year – this time doubling the amount of requests in the same time.
App server response time over the year 2013
Most interesting, we had one single release with a simple reshuffling of our external tags. Some of them now load asynchronously – or even after the onLoad() event. This helped us drop the page load time from around 2,5s to 2,1s – 400ms saved!
Impact of one single release and the move of adtags after the onLoad() event.
So, my takeaways on how to foster such a performance culture?
For a technical person, page load time feels like being important. It’s a natural tendency, an instinct almost, to make everything perform best. Unfortunately, from a business perspective that’s not really a driver to impress people or make people being responsible for revenues to re-think the importance of page load time.
Why it might be of interest – also for business people?
There is a tight coupling between revenue and page load time in e-commerce businesses. The Infographic “How Loading Time Affects Your Bottom Line” shows slower page response time resulting in increased abandonment rates, a Forrester study shows that 14% of online shoppers go to another site if they have to wait for a page to load – 23% will stop shopping, Shopzilla redesigned their site to load 5 seconds faster resulting in 10% revenue increase, Bing reported from a trial that a 2-second slowdown of page load time resulted in reduced revenues per user by 4,3%, Amazon measured a relation of 1% sales decrease for every 100 millisecond lost in page speed, and Google reported a revenue decrease by 20% for every 500 millisecond page performance loss, the Mozilla corporation behind Firefox managed to reduce the page load time of their download pages by 2.2 seconds resulting in 60 million additional downloads per year,
Why is it important for people? Why should web sites simply be fast?
There is this article “Our Need For Web Speed: It’s about neuroscience, not entitlement” from radware / strangeloop. It gives deeper insights into human nature and why it is important to run fast websites. A really good motivation for technical and non-technical people to think about the nature of performance. On web performance today, there is this infographic “This is your brain on a slow website” which picks up some of the arguments of the article in a displayable way.
Jakob Nielsen wrote in 2010 already in “Website Response Times” about the impact of slow web pages on humans and gives good reasons why we should definitely try to avoid this bad user experience.
Also good source of information to get an impression of the current state of the union: “Ecommerce Page Speed & Web Performance“.
Furthermore, there is this article on “How Facebook satisfied a need for speed“. Robert Johnson, director of engineering explains how Facebook boosted their speed by factor 2.
Another poster “Visualizing Web Performance” by strangeloop.
Page load time is critical. A lot of people highlight the importance of fast web sites. Amongst them are Steve Souders, Patrick Meenan, Tammy Everts, Stoyan Stefanov and others.
What to do to make your site run fast? There are tons of pages, blogs, hints, tips, tricks and other stuff around in the web. Here’s my favorite collection:
At FriendScout24, we follow this idea of fast web pages as well. I talk in another post in greater details about our goals, achievements and how we actually did it.
On highscalability.com there is a great post on things you should do to prevent your web site from high scalability: “The 10 Deadly Sins Against Scalability“. The post points to Sean Hull who twitters and writes quite frequently on scalability topics (surprise, surprise).
Sean Hull wrote in his blog about “5 things toxic to scalability” (2011) and “Five More Things Deadly to Scalability” (2013). Definitely worth reading entries on high scalability – and common pitfalls.
In the context of this topic, Sean also recommends a book: “Scalability Rules for managers and startups”.
Very good reading to avoid all high-scalability pit-falls right from the beginning!
