Useful, yet not valuable

Since LLM offered the option to use them for programming tasks, benchmarks were created to see how good or bad the LLM can program 🔗:

Programming tasks as benchmark

The LLM benchmarks for progamming languages are useful. They show how good the AI can code. For most programming languages, it is enough to check if the LLM was trained on the syntax, best practices or even frameworks. The benchmark can now evaluate how good the AI performs when using the information stored in it to write code. Does it understand the concepts of object oriented programming? Does it know the standard library? Can it code unit tests? How good does it know the language specification and apply other best practices around coding. Such a benchmark shows if the LLM can be used for coding tasks.

There is a benchmark for ABAP available. The dataset 🔗 checks whether the LLM can handle various coding tasks. The coding examples are the typical tasks to solve: given input n, expected outcome is m. It is a little bit more SAP centric, as the tasks include some SFLIGHT related tests 🔗.

Knowing that an LLM can solve such tasks is good to know, but offers little value. That an LLM you want to use for SAP coding can write an ABAP class, knows the syntax, can write a unit test: that’s the bare minimum. The LLM must know the keywords, it must know what is available in a given SAP release. Converting a problem into “simple” code is the bare minimum requirement. A LLM for SAP development must know more: it must understand SAP ERP.

Understand SAP

What are the SAP tables, how do they work, how to access them? What BAPI, function, CDS view to use. What is the standard report for users to interact with data, the Fiori app, the OData service. What are the implicit dependencies? When to use an update function module, when is a commit work allowed? How to work with an info types, a document record, workflow, ABAP proxy, archiving, background task, BOM, sales order, customizing, … The LLM must also know how to work in complex landscapes: integration between SAP systems, non-SAP systems, on-premise, cloud, IDoc, EDI, RFC, …

These are challenges that any LLM that wants to deliver value must solve. An LLM that creates a new class to write data directly into the db instead of using an existing and mandatory function from SAP is close to useless. An LLM that doesn’t know that a user request is solvable to 90% by using a standard app and instead creates a new one adds (future) problems instead of solving them. An LLM that stores documents simply as a blob in the database instead of using DMS or doesn’t handle output management correctly is also adding technical depth. LLMs in the SAP context must first use what the system offers. LLMs have to be the greatest back to standard and clean core driver.

SAP benchmark

What is needed is an SAP benchmark for real world scenarios. Real world problems from customer / partner projects for the LLM to solve. A benchmark that tests how good an LLM understands SAP. Any LLM solution that wants to be valuable in the SAP world needs to understand SAP. How to create a business partner? How to enhance the business partner logic? How does the transaction work together with the Fiori app? How to create a sales order, variant configuration, keep track of stocks? There are so many SAP modules that demand domain specific knowledge. This is what an LLM needs to know.

This is more effort than validating the ABAP syntax or programming tasks. This gets into business knowledge. This kind of benchmark is too much for one person or even one company. Creating the use cases and the problems the LLM must solve is work. As the world is also very complex, it doesn’t help much to have such a benchmark for one region. In case the LLM knows how to EU legislation works, it still must be able to know this for Asia, Latam or Africa, including country specific and industry specific solutions. SAP should provide such a benchmark for their standard business processes implemented in SAP. With this, customers can validate if whatever LLM they are using works for them. But on top it is important that the customers provide their own benchmark. What is needed is that the SAP user groups around the world work together to offer an SAP benchmark. Input are the problems customers had to solve. Done right, a benchmark with thousands of real world problems can be used to validate an LLM for SAP coding.

A benchmark on how good an LLM can solve business problems. This would be a useful, and very valuable benchmark.