Training and Serving Models on SAP AI Core
As of 2026-09-25
What is Training and Serving Models on SAP AI Core?
SAP AI Core runs custom, non-generative models as two kinds of git-stored executables: workflow templates (Argo Workflows) that train, and serving templates (KServe InferenceService) that deploy. Both map into the same scenario → executable → configuration → execution/deployment vocabulary, but executables, scenarios and Docker registry secrets are shared tenant-wide — resource groups only isolate the runtime objects built from them.
Two workflows, one AI API vocabulary
SAP AI Core (C026) runs two separate families of custom-model work on the same Kubernetes-based runtime. Training and batch pipelines execute as Argo Workflows, an open-source, container-native workflow engine implemented as a Kubernetes Custom Resource Definition (CRD). Model serving deploys through KServe, using the serving.kserve.io/v1beta1 InferenceService CRD. Both are packaged the same way for the AI API: as executables — reusable templates stored and versioned in your git repository, with placeholders for input artifacts (datasets or models) and parameters (custom key-value pairs) that let one template run in different configurations. This card is the hands-on build guide for that packaging; C026 is the map of the tenant that hosts it, and C363 covers resource groups, scenarios and executables as a vocabulary in their own right.
Why it matters
- A consultant who only knows the generative AI hub cannot support a customer's custom fraud-detection or demand-forecasting model — that traffic runs through the training/serving path described here, not through orchestration.
- The 'templates are shared, runtime objects are isolated' split explains a whole class of deployment failures where a template built in one resource group silently works in every other one, while its running deployment does not.
- Coarse pod-level progress and the 730-hour baseline cap are the kind of detail that turns into an incorrect incident escalation or a wrong cost forecast if you have not seen the raw API behaviour once.
Key points
- Training = Argo `WorkflowTemplate`; serving = KServe `InferenceService` wrapped as a serving template. Both are packaged as `executables` for the AI API.
- Discovery relies on annotations (`scenarios.ai.sap.com/name`, `executables.ai.sap.com/name`, `artifacts.ai.sap.com/<name>.kind`) and labels (`scenarios.ai.sap.com/id`, `ai.sap.com/version`, `ai.sap.com/instanceType`).
- Only an output artifact carrying a `globalName` becomes the workflow's registered final output.
- Start a training run with `POST $AI_API_URL/v2/lm/executions`; progress is normalized to N/100 and coarse at the pod level unless you implement `ARGO_PROGRESS_FILE`.
- Training schedules add start/end RFC3339 timestamps; status is `ACTIVE` until expiry, then `INACTIVE`.
- KServe hard-codes `STORAGE_URI` as the model-download env var and `/mnt/models` as the mount path.
- Executables, scenarios and Docker registry secrets are tenant-wide; resource groups isolate only configurations, executions, deployments and artifacts.
- Custom-model billing (compute + storage + hourly baseline, capped at 730 h/month) is separate from the generative AI hub's token-based billing (C128, C333). Limit: 50 workflow templates and 50 serving templates per tenant.
Terms used on this page
- Executable
- A reusable, git-stored template (workflow or serving) with placeholders for input artifacts and parameters, registered under a scenario.
- Scenario
- A named grouping of related executables, discoverable via GET .../v2/lm/scenarios.
- Configuration
- An executable bound to concrete input artifacts and parameter values, ready to run.
- Execution
- A run of a configuration — a training or batch job in progress or completed.
- Deployment
- A running inference endpoint created by executing a serving-template configuration.
- globalName
- The label on a workflow's output artifact that makes it register as the executable's final output artifact.
- InferenceService
- The KServe Custom Resource that a serving template's spec wraps to define a model server.
Sources
- SAP AI Core docs (SAP-docs GitHub, Sep 2026) — Workflow Templates (Argo WorkflowTemplate, annotations, generic template YAML)
- SAP AI Core docs — Serving Templates (KServe mapping, generic serving template YAML, 50-template limit)
- SAP AI Core docs — Start Training (POST /v2/lm/executions, N/100 progress, ARGO_PROGRESS_FILE)
- SAP AI Core docs — Training Schedules (RFC3339 start/end, ACTIVE/INACTIVE)
- SAP AI Core docs — List Executables (executable definition, GET /v2/lm/scenarios)
- SAP AI Core docs — KServe Spec serving.kserve.io/v1beta1 (InferenceService, PredictorSpec)
- SAP AI Core docs — Resource Groups (tenant-wide scenarios/executables/Docker secrets vs resource-group scope)
- SAP AI Core docs — Metering and Pricing for SAP AI Core (baseline hourly charge, 730 h/month cap)
Full card available to members. What the full card adds: the full decision framework · the common pitfalls and their fix · the cheat sheet · the code blocks · the facts worth quoting.