Why the four key metrics are essential in today’s software-driven automotive industry

Cars today are motorized computer networks, often containing over a hundred ECUs (electronic control units) and millions of lines of code. The rise of software-defined vehicles (SDVs) has made software a key differentiator for manufacturers, driving a significant shift in focus. Many companies are moving software development back in-house, recognizing its importance in achieving a competitive edge.

However, transitioning from hardware-first to software-driven development presents significant challenges. In automotive, integrating software and hardware typically happens late in development, often close to the start of production. This delay leads to costly consequences:

• Task forces to resolve late-stage defects
• Delayed production schedules
• Compromises in quality to meet deadlines

Additionally, development silos and limited test automation contribute to “big bang” integration, where problems surface too late to address efficiently. These issues are amplified by the lack of target hardware during early software development, making comprehensive testing nearly impossible.

Learning from other industries: Iterative development and feedback loops

Late integration challenges aren’t unique to automotive, though. Similar issues plagued software development in other domains until industries embraced iterative development and continuous feedback.

• Shifting left: (Automated) testing is conducted throughout development rather than reserved for the end.
• Continuous integration (CI): Automated systems verify every code change to ensure early detection of defects.
• Fast feedback loops: Rapid insights reduce surprises and enable timely corrections.

These practices enable high-quality software delivery, and their principles can be adapted to automotive development.

The four key metrics: A framework for measuring performance

The four key metrics — first introduced by Nicole Forsgren, Jez Humble and Gene Kim, can be very useful for measuring delivery performance. Specifically, these metrics are:

1. Change Lead Time: How quickly can changes be tested and validated?
2. Deployment Frequency: How often can teams deploy updates?
3. Change Failure Rate: What percentage of deployments fail?
4. Mean Time to Recovery: How quickly can teams resolve issues after a failure?

From an automotive perspective, these metrics may initially seem misaligned due to infrequent releases and a cultural mentality that things should be right first time. However, when reframed to measure feedback loops and incremental progress, they reveal areas for improvement in even the most hardware-constrained environments.

Applying the four key metrics in automotive development

Instead of tying these metrics strictly to final production, they can be adapted to reflect the best available environment at any given stage — one that most closely resembles production. Initially, this might involve software-based testing, which then progresses step-by-step toward the actual target environment:

• Change Lead Time becomes the time required to validate changes in available QA environments (e.g., Software in the Loop (SiL), Hardware in the Loop (HiL)).
• Deployment Frequency measures how often teams verify and validate their software in the best available environment that approximates production.
• Change Failure Rate highlights the percentage of failed deployments (incl. failed validation and verification in preceding environments
• Mean Time to Recovery tracks how quickly teams address issues identified in downstream validation.

This approach emphasizes rapid feedback loops regardless of release cadence and encourages accountability across all validation stages, from local development, SiL, HiL, all the way to the car.

Fast feedback hardware journey: From mockups to reality

Achieving rapid feedback in hardware-constrained environments requires a phased approach, as the target hardware often is only available late in the project:

• Start with software-only testing: Early in development, tests can run on developer machines or emulators like QEMU, validating foundational aspects such as compilation, static analysis, and unit tests.
• Adopt virtual ECUs: Virtual ECUs allow testing in cloud-based environments, enabling integration testing before physical hardware is available.
• Leverage evaluation boards: Use off-the-shelf hardware or existing resources to simulate target environments and uncover performance issues early.
• Build mockware: When evaluation boards aren’t sufficient, create custom mockups (e.g., CAN bus simulators, usb cameras and usb sensors) to test integration and gather meaningful insights.
• Transition to prototypes: Once prototypes are available, gradually replace mockups while benefiting from the earlier iterations.

This iterative progression ensures software quality and integration readiness improve well before the final hardware arrives. It allows you to validate core functionality early in the process. For instance, if video processing is a key feature, it may not initially matter whether the camera is connected via USB instead of LVDS — or even whether it’s the actual camera or just one that’s available and has roughly similar specs.

Walking skeletons and vertical slicing: Building for early integration

To avoid the pitfalls of late integration, start with a walking skeleton — a minimal version of the system connecting key components with basic functionality. This foundational step should take days or weeks, not months. It allows teams to:

• Ensure early integration across components
• Verify that parts work both individually and together
• Build incrementally by adding functionality

Paired with vertical slicing — breaking down problems into smaller, standalone pieces — this approach simplifies integration and debugging, reducing defect rates and overall costs.

A walking skeleton is not, it should be noted, a minimum viable product. If you find yourself still thinking about months of development, reduce the idea further down to its bones.

The value of early feedback and automation

Although early feedback may not come from series-production hardware, it’s invaluable for guiding development and reducing surprises. Automated testing, integrated alongside production code, ensures you can quickly validate:

• Code functionality and architecture alignment
• Integration with surrounding systems
• Non-functional requirements like performance and latency

Early validation builds confidence, uncovers issues before they snowball, and shifts quality improvements earlier in the process. Automation is the cornerstone of this approach, enabling faster iterations and tailoring architecture to overall needs.

Measuring on a system level

Focusing solely on team-level metrics risks local optimizations that obscure the big picture. In automotive, where rollouts affect multiple ECUs, it’s essential to measure system-level performance.

By emphasizing overall system readiness and integration quality, teams can align their efforts toward cohesive solutions rather than isolated successes. Measuring metrics like lead time and recovery time at the system level can reveal bottlenecks and help promote collaboration across teams.

Conclusion: A path toward high-performance development

The four key metrics provide a valuable framework for improving software delivery performance in the automotive industry. By adapting these metrics to hardware-constrained environments, emphasizing early feedback, and leveraging iterative practices like walking skeletons, teams can deliver high-quality systems more efficiently.

The journey to better delivery performance starts with small, incremental changes. Piloting practices like vertical slicing or early integration can create a foundation for long-term success. As automotive embraces software-first development, these principles can help organizations thrive in a rapidly evolving landscape.