What changed
HPE has introduced a new approach to measuring the cost-effectiveness of AI-assisted software development by focusing on the output rather than the input. The company suggests tracking the cost per merged pull request instead of per token used, which can provide a clearer picture of how AI investments translate into actual software delivery.
What this means in practice
The shift from token-based pricing to merged pull request-based metrics aims to better align AI spending with tangible outcomes. By evaluating the cost per merged pull request, organizations can assess how efficiently their AI tools are contributing to the development lifecycle. This method also encourages a more nuanced understanding of AI usage, distinguishing between necessary work and potential waste, such as retries or inefficient loops.
Limitations and availability
The proposed metrics are currently being tested within HPE's internal environment, where they have reported a reduction in token costs by 30 to 50%. While the approach is promising, it is not yet universally adopted and requires careful implementation to ensure it accurately reflects the value of AI contributions. Additionally, the effectiveness of these metrics may vary depending on the specific workflows and codebases within different organizations.
