Now and Next Java For AI -- JSpring 2026

Now and Next Java For AI
Ana-Maria Mihalceanu
Senior Java Developer Advocate @ Oracle
https://bsky.app/profile/ammbra1508.bsky.social
https://mastodon.social/@ammbra1508
https://x.com/ammbra1508

Shifts in Software Perspective
Copyright © 2026, Oracle and/or its affiliates
2
Desktop Applications
Client-Server Desktop
Applications
Web Applications Cloud
Applications/SaaS
Mobile Applications Generative AI
Applications
~1970s
~mid-1990s
2022
~2010s
~1990
late 2000s

“Change is the only constant”
Mark Reinhold, Devoxx Belgium 2018
https://www.youtube.com/watch?v=wHoRBvt3U6o
3 Copyright © 2026, Oracle and/or its affiliates

The AI/ML Model Journey
AI/ML Model
Stage
Description
Data (What?) Process (How?) Goal (Why?)
Train Build a model from
scratch.
Large, historical,
labeled datasets.
Iteratively learn from a
large dataset.
Obtain an accurate and
knowledgeable model.
Fine-Tune Adapt the pre-
trained model for a
specific purpose.
Smaller, task-specific
datasets.
Refine an existing
model with a smaller
dataset.
Customization to reduce
inaccuracies and attain
efficiency.
Inference Use a trained
model to make
predictions
(classifications).
Real, unlabelled data,
most likely not part of
the training dataset.
A single model “run”
of new data.
Achieve latency (speed),
throughput and scale.
Serve Improve model
accuracy and
capability.
Package the model
and expose it as a
library/API.
Reliably maintain and scale
the inference endpoint.

RUNTIME
Deep Learning Models
9
MODEL
• .pt
• .pb
• .onnx
• .gguf
• ...
Graph
(ops + layers)
Weights
IN (img, tokens, …) OUT (cat., tokens, …)
Loads Model
Dispatches Load
to HW
→ PyTorch
→ TensorFlow Runtime
→ ONNX Runtime
→ Llama.cpp

Two Control Surfaces from Application Perspective
Serving AI Models For Handling Inference Requests
Remote Inference Inference Control Layer
10
(Java) Client
Model Server
Native Runtime
CPU/GPU
HTTP/gRPC
Simplifies upgrades and isolation. Better control over data and more opportunity for predictable
specialized behavior.
(Java) Library

Examples of ONNX Java Inference Community Efforts
https://inference4j.github.io/inference4j/ https://github.com/langchain4j/langchain4j/tree/main/embeddings

Babylon Code Reflection
@Reflect
Talk Overview
Building blocks examples to perform inference through Java
Execution Provider
ONNX runtime (native)
Panama bindings for main
runtime methods
.onnx ML model
Execution Provider
Panama bindings for ML
operators
Java ML model
Now
Next

ONNX and Java
ONNX from Java Perspective
The Java platform knows nothing about ONNX.
Java considers ONNX runtime a foreign (native) library.
Java considers the ONNX programming model a foreign
programming model.

Foreign Function and Memory API
ONNX Native Library
(libonnxruntime.dylib | libonnxruntime.dll |
libonnxruntime.so)
Foreign Function & Memory (FFM) Java Bindings
Java Client
Memory Layouts Var handles
Function
Descriptors
Method Handles

MemorySegment as a Tensor Backing Store
OrtValue
(tensor handle)
MemorySegment
Java Data
(float[]/ByteBuffer)
ONNX Native Library
libonnxruntime.so)
Foreign Function & Memory (FFM) Java Bindings
Java Client

Deploy and Execute an ONNX Model
Image
(.png)
Java Client Classification/Probabilities
ONNX Model
(e.g. emotion-ferplus-8.onnx)
Execution Provider
libonnxruntime.so)
Panama bindings for main runtime
methods
runtime.createSession
inference.run
…

Demo: Run an ONNX Model
via FFM API

What’s Inside an ONNX Model
Model metadata
• version, description, ...
Graph structure
• operators and tensors
• protobuf (Protocol Buffers for serialization)
Initializers (weights)
• can be large chunks of binary data (float32, int64, etc.)
Inputs and outputs
• Example: input is float[1, 1, 64, 264], output is float[1, 8].
https://netron.app/

Project Babylon
__global__
void kernel(S32Array_t* a) {
int id = blockIdx.x * blockDim.x
+ threadIdx.x;
if (id < gridDim.x * blockDim.x) {
int value = a->array[id];
a->array[id] = square(value);
}
}
__device__ int square(int v) {
return v * v;
}

Java
programming
model
Foreign
programming
model
Reflect Translate
Code Reflection

@Reflect helps identify areas of Java source code to reflect over
and give access to as code models at compile time and runtime.
Extend Java Reach to Foreign Programming Models
func @"f" ()void -> {
%0 : java.io.PrintStream = field.load@"java.lang.System::out()java.io.PrintStream";
%1 : java.lang.String = constant @"Hello !";
invoke %0 %1 @"java.io.PrintStream::print(java.lang.String)void";
return;
};
@Reflect
static void f() {
System.out.println("Hello !");
}
Input Java Code
Java Code Model
public static void f();
Code:
0: getstatic #2 // Field java/lang/System.out:Ljava/io/PrintStream;
3: ldc #3 // String Hello !
5: invokevirtual #4 // Method java/io/PrintStream.print:(Ljava/lang/String;)V
8: return
JVM Bytecode
Reflect
Foreign Code Model
Translate
(eg. autodiff)
Lower

Project Panama is to
foreign
libraries
Project Babylon is to
foreign
programming models
as

Operators
Using ONNX Operators in Java
Pre-generated in Babylon Examples (come out-of-the-box with Code Reflection)
Conv (input, weights, ...)
Gemm (matrixA, matrixB, ...)
Relu (tensor)
...
Extract operators from
ONNX schema
(done already)
Operators are recognised
and transformed by Code
Reflection

Architecture of Java ONNX Prototype
ONNX runtime (ORT)
Foreign Function & Memory API
Panama ONNX binding
ONNX model authored using
Java ONNX API
JDK
Library
Application
onnxruntime_c_api.h
Java ONNX API &
Code Model Transformer
Java code
Native code
jextract

Demo: Execute a Java Model
through ONNX Runtime
github.com/openjdk/babylon/tree/code-reflection/cr-examples/onnx

How to run A Java Code Model on ONNX Runtime
Java Code Model
Code Reflection API
Java ONNX Script Library
Tensor…
ir.OnnxOp..
ir.OnnxType
compiler.OnnxTransformer..
OnnxRuntime…
FFM Bindings
foreign.OrtApi..
foreign.OrtGenApi
ONNX Runtime ONNX GenAI Runtime
OnnxOperators
ir.OnnxOps
proto.OnnxBuilder..
ONNX
Specs & Sources
OpGen
ProtoGen
jextract

More AI Layers
Agent
• The composition of inference
• Orchestrating LLM, tools, and information sources
Agentic
• The composition of agents, with dynamic context
• Workflow to autonomously achieve a desired outcome

Java Platform and AI Perspectives

Foundational
building blocks
in the JDK

Foundational
building blocks
in the JDK
Panama - Foreign Function & Memory (FFM) API
Deploy and execute ONNX models using native ONNX runtime
Panama - Vector API
Implement model runtimes on the CPU
Babylon - HAT
Implement model runtimes on the GPU
Valhalla - Express more numeric types arrays of which are flat in memory
Detroit - Deploy and execute PyTorch models

Thank you
32

Now and Next Java For AI -- JSpring 2026

Now and Next Java For AI -- JSpring 2026

More Related Content

Similar to Now and Next Java For AI -- JSpring 2026

More from Ana-Maria Mihalceanu

Recently uploaded

Now and Next Java For AI -- JSpring 2026