We expedited the JLF CRO programme and team in-housing

Having been a long-time partner of John Lewis Finance, working with them to develop and run A/B tests and undertake conversion rate optimisation analysis and research projects, we were asked to provide further strategic CRO support to help them to overcome various challenges and grow the programme. Resource changes had slowed testing activity, but at the same time, new leadership brought an increased appetite for experimentation. Our CRO team were brought in to revamp the program, establishing a faster-paced experimentation programme and then aiding in recruiting a new in-house team.

We were already working with John Lewis Finance to implement A/B tests and set up on-site experiences via their testing tool, Monetate. During this new programme of work, we helped them further by using our CRO maturity model to advise the team on improving structure and governance and enhancing their working methodology to drive even more results.

Some of the challenges faced were:

• Lack of historical test documentation and documented process etc.
• Tracking issues and questions over the integrity of data
• Relatively low traffic volumes to test within certain areas of the website
• A tendency to use Monetate as a tool for quick and temporary fixes to the site that did not drive any value and prioritise this over A/B testing
• Struggle to get support for the programme in terms of internal development, design resource etc.

In October 2021, the opportunity arose for us to support and enable them to significantly change and grow their CRO programme. There was an appetite to accelerate testing and its learnings - the leadership team wanted to promote a culture of true data-led decision-making and embed a test-and-learn culture across the John Lewis Partnership. The investment was there to back it up, and the recruitment of a new expert team to lead the programme was in the process of being approved.

We were initially asked to provide interim senior CRO resource, seconded into the business whilst supporting the recruitment of a new in-house team. Our priorities were accelerating the pace of the CRO programme and the learnings to come from it whilst laying the foundations and processes that the newly recruited internal team would eventually own.

I believe a critical value-driver was Fresh Egg’s team of user researchers, whose qualitative insights helped identify key user objections and pain points. Through collaborative workshops with the John Lewis Finance team, we created over a year’s worth of individual test ideas that, due to low traffic volumes, we combined into numerous complex multi-change tests. This worked well because these bigger, bolder changes were often the ones that delivered significant performance swings.

Nicky Moody, Project Director 

Building on our strong existing relationship, we became engaged in activities to support John Lewis Finance’s CRO growth ambitions. Over 18 months, using our experience in embedding experimentation within large financial services companies, we helped them to establish best-in-class CRO processes, from research and ideation, to test implementation and analysis.

We supported the scaling up of their in-house capabilities through recruitment to grow the team, including creating job descriptions, CV reviews, candidate selection and interviews. We then conducted training and upskilling (Monetate, analysis, data support etc.) to onboard the new team, continuing this support as they grew into their respective roles.

Through improving research, test planning, test build and analysis practices, we enabled John Lewis Finance to build the foundations for a more successful CRO programme.

Our work included the following:

• Product-by-product analysis and discovery work, including user testing, which generated qualitative insights used to help identify critical user objections and pain points
• Creation of a large, strong backlog of individual test ideas, some of which were combined into complex multi-change tests to overcome issues with low traffic volumes
• Increasing the volume of tests overall, with a focus on value-driving testing
• Improving overall processes and test programme documentation, for example, test log (including status, archive, backlog etc.), production briefs, Trello, QA/UAT logs, prioritisation calculator, sample size calculators
• Implementing and following a new documented QA process to provide more thorough documentation, bringing it in line with compliance and regulatory testing specific to the financial services sector • Producing more detailed post-test analysis, leading to a deeper understanding of user behaviour and a healthier backlog of test ideas
• Deep dive and analysis training and consultancy, the format of business reporting, and aiding in the production of a one-pager “scorecard” to be shared with internal teams at John Lewis Finance
• Supporting the internal CRO team promote and communicate the success of the experimentation programme internally – giving the programme visibility.

When we picked up the CRO programme at John Lewis, we faced numerous challenges: resistance to challenging long-held beliefs, relatively low traffic volumes, tracking issues and a lack of historic test documentation. Thanks to unwavering support from Senior John Lewis Finance Stakeholders, we were empowered to challenge business norms and promote a culture of true data-led decision-making.

Nicky Moody, Project Director

We tripled the test velocity - during 2021/2022, 26 tests were run in 10 months (2.5 per month, up from just under 1 per month before the project).

A series of iterative redesign tests on the Home Insurance landing page increased the visibility of John Lewis Finance's value proposition and key selling points. They yielded a +1.6% increase in the 'Visit to quote' rate. This redesign was also rolled out across other financial products as tests.

A single test at the first step of the home insurance quote and buy journey to revamp their 'suitability page' improved start-to-quote conversion by +2.7%. A test further along the user journey to improve product presentation to mobile users achieved a 4.7% increase in progression from the quote summary page, resulting in a significant increase in sales. The business now has robust processes and frameworks for test ideation, prioritisation, briefing, development, QA, UAT, launch and analysis that underpin the experimentation programme. This has been integral in gaining the other internal teams' understanding, trust, and support.

The John Lewis Finance team grew by 300% (from 1 to 4), which has been vital for the success of the CRO programme and where it stands today. The test-and-learn culture we created has permeated through the Partnership, with internal teams embracing the potential of CRO to support their goals and taking an active interest in the test plans and results.

Moving forwards, the relationship is becoming more consultative and strategic, with Fresh Egg involved in ideation sessions, user research and any additional support needed whilst the team at John Lewis Finance continues to make in-roads to in-house even more of the activity, such as test development and QA.