Drawing from his experience at In The Pocket and background in UX design, Peter offers valuable insights for organisations navigating AI implementation. His piece aligns with our studio's mission to create impactful digital products, providing practical advice to overcome common AI project challenges. Peter's unique perspective makes this article a must-read for anyone looking to succeed in the complex world of AI adoption. 

Itʼs crunch time

2023 was the year you had to experiment with AI. 

This year is where you need to see your first returns.

But you know by now thatʼs easier said than done. And youʼre not the only one struggling. Gartner placed GenAI for the first time in the trough of disillusionment: the phase where interest wanes as experiments fail to deliver. Where are those big AI-powered productivity gains? Teams tasked with AI projects in organisations, small and large, are struggling to deliver GenAIʼs anticipated business value.

Everyone sees its potential, but getting there turns out to be incredibly difficult. Almost 90% of AI projects fail nowadays, according to Dan Saffer, Assistant Professor at Carnegie Mellonʼs Human-Computer Interaction Institute. Those are some pretty bad odds. But we shouldn’t just give up and quit here. Organisations like Klarna and Bunq are proving it can be done. In fact, we know why AI projects often fail, so we can do better. Letʼs make your winning strategy.

Do or die, GenAI?

Before we dive in head-first and create your winning strategy, itʼs important to understand why everyoneʼs struggling with making returns on their AI investment so we can learn from our mistakes.

1. The scope of AI projects is too large, and expectations of GenAI are simply too high. GenAI is not going to completely overhaul your business overnight.
2. The data required is either nonexistent or of very poor quality. Most companies sit on large amounts of data. True. But that pile is often useless or irrelevant rather than the promised gold.
3. AI is seen solely as a technical problem, whereas currently, itʼs more a UX problem than it is a technology problem. Too little attention and research goes to how users interact with your product.
4. AI is positioned as the product, instead of as a means to an end. Users donʼt care with which technology you will solve their problems. If it doesnʼt help users reach their goal, itʼs useless.
5. The focus is too much on reaching high model accuracy first, meaning “we need to reach 80% accuracy before we can ship thisˮ, rather than looking at the relative gains it brings compared to current benchmarks.

If youʼve recently built an AI product yourself, you can probably complete this list with several more items.

Your winning strategy

Now, letʼs look at how we can turn AI into ROI. Generative AI and more specifically, Large Language Models (LLMs) – is general purpose. It can tell you a whole lot about a whole lot of things. But to reap its benefits, you need to narrow its scope and point all its smarts to a specific challenge for which you can tailor and control its output.

Benedict Evans compares it with an electric motor:

Here we have a general-purpose technology, and yet the way to deploy is to unbundle it into single-purpose tools and experiences. However, seeing this as a paradox might just be misplacing the right level of abstraction. Electric motors are a general-purpose technology, but you donʼt buy a box of electric motors from Home Depot - you buy a drill, a washing machine and a blender.

So you need to look for single-purpose solutions. There are 4 key elements to a winning AI strategy:

1. Well-scoped challenge that offers business value 
2. A dedicated cross-functional team from start to finish 
3. Validate technical feasibility first
4. Keep your focus on the long-term

1. Pick the right challenge

Everything rises and falls on a challenge. Just like any organisation, youʼll have many problems to solve at any given time. But not all problems are equal. Start from a list of pain points you or your organisation are currently struggling with. Frustrating, time-consuming, difficult, costly and/or cumbersome processes that are a pain in the neck.

Keep in mind that automation is not the holy grail. Instead, explore how AI can augment human capabilities instead of trying to fully automate the process. Use our Levels of Automation framework to ideate how to best bridge the two types of intelligence.

Your ideal problem-to-solve scores well on all of the following characteristics:

1. Significant business value
   The use case should significantly help your organisation thrive. This excludes vanity AI projects just to jump on the bandwagon.
2. Overall strategic alignment
   Solving this problem aligns with your companyʼs overall, broader strategic objectives. This ensures keeping interest – and budgets – over time.
3. Part of a loosely-coupled process
   The problem youʼre trying to solve can be clearly defined and is easy to separate from other processes in your organisation. This allows you to get results quickly.
4. Direct access to users
   Go for a problem where you have direct access to (potential) users. To gain speed, you need to validate early and often.
5. LLM-friendly context
   Training models is hard and expensive. If you want to gain speed, aim for a problem that an LLM could help solve. Reap the benefits of the hard work of OpenAI, Google and Anthropic. 

💡List all your high-potential problems to solve and use the Data strategy canvas to pick out the one that overall scores the best.

‍

2. Gather a dedicated, human-centred team

Creating a successful AI product is not a challenge you can just drop on your IT team or expect your business units to do on their own. For AI products, a cross-functional team that closely collaborates is not preferred but is required. Next to subject-matter experts from the business, you need UX designers/researchers and ML engineers involved from start to finish. Great AI products are created by business people with a basic understanding of AI capabilities, together with AI experts with a basic understanding of the business. This crossover is where the proverbial magic happens.

If your challenge involves any ethical or legal sensitivities (cf. AI Act), you need to involve specialists in this field too. Ideally, your core team can look at the problem from all relevant perspectives and slowly but surely build towards a solution that mitigates the biggest risks.

AI products are just as much a UX problem as they are a technical problem.

Your team needs to be human-centred by focusing on the people youʼre solving the problem for. This isnʼt anything new, but for an AI product, this is even more make or break. Your team needs direct access to these people from the get-go. This goes for (external) customers but just as much when building for internal teams. You can have the most amazing and powerful model, however, if nobody understands it or trusts it, itʼs worthless

You need frequent and direct contact with target users to ideate and validate along the way. Let them try the solution so you can shape it together. This will also help you understand how to best position the product. Nobody cares if it uses AI. Which of their problems are you solving? The how doesnʼt matter. AI is here to add value, not to be the show pony.

At the same time, you need to know what you can promise. A human-centred team can focus on bringing the smarts of your AI-powered product across to the users in the best possible way. Set the right expectations from the start. Make it clear what users can expect from using your product and what they shouldnʼt expect so theyʼre not disappointed after trying it out.

💡Create a cross-functional, human-centred team that keeps a laser focus on the challenge at hand and the people for whom youʼre solving it.

3. Validate feasibility first, usability second

Once your team understands the problem and the challenge is defined, youʼll want to work on a technical proof of concept. You want to validate the technical side first before anything else. In typical (non-AI projects, we tend to look at the usability side first. You want to imagine the best possible experience and adapt the technology accordingly. Not so with an AI product. Itʼs perfectly possible your problem is still a hard nut that canʼt be cracked by the current state of AI. Or it could simply be too costly. Shiny new models come at a hefty price. You can figure out both things during the POC. Can it be done? And will it ruin us financially if this takes off?

An important part of this technical POC is gathering all required data and validating whether itʼs actually good. Only this way you can find out whether the pile of data you're sitting on is actual gold or rather useless.

Your team needs to do continuous user research to actively gather feedback from users. Make sure you have a process in place to use this feedback to improve the product. Let them know how youʼre going to use this and how it benefits them. You need to sweat the details of the user experience to get it right. Small details can have a huge impact on an AI-powered product. You can only find out which these are by frequent prototyping and testing.

💡 Start with validating the technical challenge first and – if successful – follow up with continuous user research throughout the project to constantly prototype and test with actual users.

4. Prepare everyone for a marathon, not a sprint

AI-enabled products are rarely a hit from the start. The potential is there, the tech works, and some first signs of success appear but youʼre not there yet. Stopping at this stage will set up your product for its final journey to the Elysian Fields of Tech. The excitement will wane quickly and so will your budget. Building an AI-enabled product is a marathon. It takes time to get the technology right, tailor the user experience, the copy... and thatʼs okay. If you picked the right problem, itʼs going to be worth it.

Regarding your time and budget allocation, you need to take this into account from the start. Too many promising solutions have been abandoned because the interest faded away when the first results were not as groundbreaking as hoped, or didnʼt deliver from the start. Prepare all your stakeholders for a long run instead of a sprint, to make sure all your stakeholders have the right expectations.

Another expectation to manage is the accuracy of your modelʼs predictions or output. Specific percentages are thrown around and big claims are made about it (ˮwe canʼt go live before we have an accuracy of 80%ˮ). The problem is this accuracy is always anchored by perfection: 100%. But thatʼs the wrong way to look at it. Rather benchmark it against the current process and see how much better you score in terms of output quality, speed and cost. Donʼt let yourself be pinned down on a certain accuracy number, rather focus on the improvements compared to the current way of working.

Make sure to communicate regularly with all involved about the progress youʼve made and the challenges youʼre still trying to tackle. Sharing user research findings is also a great way to keep people involved and interested in the process.

💡So what are you waiting for? Letʼs make 2024 the year of AIʼs first ROI at your organisation. Doing this all by yourself can be daunting. Let your team be surrounded by experts that can get you up to speed faster. 

‍