After a year with 350 developers and every AI tool available, here's what actually moved the needle.
Towards the end of 2024, we became convinced that AI would fundamentally change software engineering - and that it should be our number one priority to figure out how. For a full year at F-Secure, with 350 developers and every AI tool available, we lived and breathed the challenge of making AI actually deliver results.
In the end, it's all about the developers whose hands are on the keyboard. Managers might value productivity - developers value DX. Whatever IDE or CLI tooling they are using - if AI is not making their lives easier, it's not going to fly. And too often it doesn't.
The leaders of the organization need to show the way and get their hands dirty. This is not a theoretical exercise; it's a fundamental change to how software is built; it's an extremely fast-moving field, and if the leaders don't develop deep instincts for this new world, they will be lost.
AI is not a silver bullet on its own - you can't just sprinkle AI fairy dust across a team to get the benefits. Other things will need to change. Leadership need to be open to that change, and teams need to have the space to experiment and find what works.
If you want teams to experiment and navigate their way to an uncertain future, you need to justify that. Handwaving about urgency and productivity doesn't help: you need to paint a believable, developer-centric picture of what the future will be.
On the developer side, context management is the key differentiator between those who do well with AI and those who don't - but it's a tricky skill to learn.
On the leadership side, leading indicators are crucial. At AI speed it takes too long if you wait to track the right value-based KPIs: you need to zero in on something that changes on a daily basis.
We built (and threw away!) a lot of in-house solutions to solve these problems - and it worked. Over 12 months we saw:
4x
daily AI tool usage
40%
faster implementation
20%
shorter cycle time
4x improvement in daily AI coding tool usage. This is the earliest leading indicator. Developers won't use daily tools if they don't help: this shows developers found value in AI and were actually leaning into experimenting with new ways of working.
40% improvement in implementation time. Not from code generation - but from better coordination. From AI tools that got enough context that the developer didn't need to go and ask for help when they touch another system.
20% improvement in end-to-end cycle time.
Did it produce more business value? That's a great question, but in the end that's more than just an engineering team can manage - that's a bigger question across product and strategy as well. From an engineering perspective - did we build faster - unequivocally yes.
The crowning glory of this experiment was one particular project where we could look through the lens of business value. We did this project twice.
First attempt: 20 people for 12 months. They had all the AI tools - but we didn't understand context management, and AI coding velocity didn't translate to business outcomes. Result: did not launch.
Second attempt: 5 people for 6 months. This time, we focused hard on context management, we brought product, design, and engineering much closer together, and it paid off massively. Result: released.
The comparison
240 person-months (failure) vs 30 person-months (success)
More than 8x productivity
You have to align your developers, AI agents and AI assistants around the same context. Context divergence at AI speeds will destroy any productivity gains.
You have to get that context into the developer/AI hands with as little hassle as possible: no new interfaces, no install hassle. We guarantee less than 20 second installation in whatever IDE the developer is using.
You have to see what you're doing.
We'd love to hear about the challenges you're facing with AI in your engineering organization.
Get in touch