Moving the News

Guardian News and Media (GNM) is at the forefront of the digital revolution. Its award winning network guardian.co.uk is the most popular news website in the UK. The company recognized that growing demands from users, advertisers and editorial staff were beginning to strain the capability of is technology.

Partnering with ThoughtWorks, The Guardian developed a new platform, taking advantage of innovative technology and adopting new ways of working as part of the process.

The Guardian

Guardian News and Media (GNM) publishes the Guardian and Observer newspapers and since 1996, guardian.co.uk, the UK’s most popular newspaper website. The Guardian is owned by the Scott Trust, a unique form of media ownership in the UK, which secures the continuity and editorial independence of the Guardian. Not owned by shareholders, dictated to by a press baron or influenced by a political party, Guardian journalists are free to present the truth as they see it.

Since its launch in 1996, guardian.co.uk has been the leader in online news in the UK, and has won 3 Webby awards for best news site. But ten years on, while still leading the market, GNM faced an environment in online journalism that was not only changed but was evolving faster than ever.

Fast growth needed a stronger foundation

GNM knew they were approaching the limits of what could be done in their existing technical environment. In 2006, much of the guardian.co.uk website was still manually created. Editors had to deal with thousands of templates, massive duplication of content, and multiple legacy Content Management Systems that mixed up content and presentation. They could only innovate around the edges, and it was taking weeks to make what should have been simple changes.

Some of these limits were already evident on the site, some were growing in seriousness behind the scenes. Executives specified a program to build a new and flexible digital media news and information site to gradually and seamlessly replace the old site, and they opted for a bespoke build rather than trying to fit a package solution to GNM needs – or worse, vice-versa.

Getting Agile, finding ThoughtWorks

As planning for the massive undertaking began, recent (bad) experience with a new registration system was fresh in the minds of GNM leaders, leading them to re-evaluate their approach to major IT projects. They were wary of repeating the up-front requirements process and then finding that the resulting system was not utilized as planned. GNM decided they needed an Agile approach for the new project. It would require a greater investment on the part of the business users – collaborating fully with the technical team at every step – but Guardian management knew it would reduce overall risk and was the only way to accommodate in-flight changes during what would be a multi-year, multi-release program. Agile projects by nature accommodate the unexpected.

Guardian carefully reviewed potential development partners, and chose ThoughtWorks based on a desire to learn from experts, and a recognition that the skills needed for the program did not exist in-house. Guardian’s own organisation would work as part of an integrated team with ThoughtWorks’ staff, learning Agile techniques and skills to allow them to support the platform and deliver future initiatives.

A journey of two years begins with two-weeks

A two-week Inception phase focusing on the pilot site, Travel, was led by ThoughtWorks, drawing out the Guardian’s domain expertise and knowledge of the current system. The joint team rapidly worked through high-level requirements, agreeing on the implementation roadmap and technical toolset - a lightweight Java / Open Source architecture. From there the team kicked straight off.

They had a lot to tackle. One of the main technical challenges was migration of content, ensuring that while both old and new sites used the core database there would be no breakages. The original system fostered considerable content overlap and repetition, which had to be eliminated in the migration. The Guardian site at that time held 800,000 articles. The old pages were relatively static, however the pages in the new site would be highly dynamic and contain complex logic. And with the transition from static to dynamic pages, the current site performance of 700 pages per second at peak needed to be maintained.

The new site would support multimedia content, visitor interactivity, social network linking and other community-building features. It would be at its core a digital content platform rather than a web version of the publication, representing a fundamental change in mindset for the entire business. What’s more the underlying architecture had to support audience growth of 30 percent or more per year.

And it all had to be transitioned seamlessly to replace, with minimal interruption, each section of the existing site.

Program Planning step-by-Agile-step

News being the main driver of readers to the site, the program was organized around the primary goal of launching the rebuilt UK and World News sites. The team devised a route to deliver value at regular points rather than as a ‘big bang’. Releases of other sites were staged in advance of the main News launch. The team was always working on requirements that were essential to the primary News goal, but through careful planning, other sites’ functionality could be completed on this journey, and so could be released to the public. Each release provided practice for the critical News launch, and also allowed requirements to be recalibrated.

Travel - First to launch

The first phase consisted of a 10-month project to rebuild the content management platform, while developing the first in a series of sites that would use the new platform and apply the new site design. The Travel section was selected as the pilot effort.

The Travel site pilot project went live in December 2006, eight months into the program, and quickly experienced a tenfold increase in the average number of pages viewed per visitor. The lessons from this phase laid the foundation for continuous improvement; each subsequent release was completed on time and with minimal disruption.

“We adopted an evolutionary approach to guardian.co.uk; occasionally we chose to expand our ambition, for example we made the decision to include video half way through the project. ThoughtWorks responded positively and flexibly - qualities which are characteristic of their approach”

Mike Bracken - Technology Director, Guardian News and Media

Additional releases converted the other 22 sections according to a schedule that accommodated commercial importance, editorial importance and other factors including risk, reputation, and technical difficulty. The project team regularly adjusted the release schedule to suit business needs. February 2008 saw the flawless launch of the News, Politics and Observer sites - the largest milestone in the collaboration. Since then, the team has progressed to sites requiring complex third-party integration, such as Sports, Football, Culture and Comment is Free - all now live.

Investing in people - building for the future

A major part of the project was to transfer skills to Guardian developers and analysts. This was achieved through pairing, “brown bag” training, and Agile coaching. Following the remarkable success of the program’s approach and its ability to change alongside the business, the blended ThoughtWorks Agile / Guardian cultural practices are at GNM to stay.

This major program, completed on time and within budget, has enabled Guardian to pursue innovations it couldn't with the prior platform, layering on community-building features and monetizing its large archive of news articles. The new platform provides the tools and ability to sustain their market leading position, and an opportunity to globalize the brand. The new platform is enabling Guardian to become the world’s leading liberal voice.

Benefits:

• Increased revenue: The new platform features flexible, contextual advertising and other enhancements which are significantly increasing page yields and sales
  

• Increased page views per visit: Content tagging and intuitive user navigation are already contributing to a remarkable increase in the average number of pages viewed per visit
  

• Improved staff productivity: The editorial staff benefit from the separation of content management from presentation, elimination of duplicated content, and new user features such as community-building, multimedia support and intuitive navigation across the site
  

• Future ready: The site is now a digital content platform, flexible and scalable, built to support new features even as audience growth continues at 50% per annum