How to Implement AI into Java: Run a Model You Already Have

How to implement AI into Java, for a normal service, means loading a model that was trained somewhere else and scoring one input. It does not mean inventing a training loop inside a request thread. This note walks through that split. It is not a claim about accuracy, cost, or which vendor you should buy. Read the library pages linked below before you copy a snippet from memory.

If the thing you actually wanted was a chat box in front of a hosted model, that is a different job from local inference. The page on how to train an AI assistant in JavaScript separates prompting, browser inference, and real weight updates. The same split applies on the JVM.

Decide what how to implement AI into Java is asking for

Three jobs get the same sentence in a ticket. Calling a remote model over HTTP. Running an exported model in-process. Training or fine-tuning weights. Only the first two belong in a typical Java service. Weight updates belong in a training job whose own documentation you can point at. If the ticket says add AI and the data is a few example sentences, you are writing a client, not a trainer.

Write the job in one line before you add a dependency. Score this ONNX file on the JVM is implementable. Make the app intelligent is not. The line also tells the next person whether a GPU machine is in scope. Inference of a small exported model often is not a training cluster.

Run an ONNX model with the Java binding

The ONNX Runtime Java guide describes a Java binding for running inference on ONNX models on a JVM. It says Java 8 or newer is supported, and that release artifacts are published to Maven Central. The CPU artifact it names is com.microsoft.onnxruntime:onnxruntime, with platforms listed on that page. A separate GPU artifact is also listed there. Read the current platform table instead of assuming your server’s OS is on it.

The same guide shows how a scoring session starts: create an OrtEnvironment, then open an OrtSession with the path to the model file and session options. Inputs are a map of names to OnnxTensor values. The guide says those names have to match the input node names stored in the model, which you can read with getInputNames or getInputInfo on the session. Do not guess the names from a blog.

The run call returns a Result that the guide says is AutoCloseable. Use it in try-with-resources so the values are closed. Tensors can be built from a buffer plus dimensions, or from a multidimensional array that is not ragged. The guide’s sample, ScoreMNIST, expects a model path and a data path. Use a model you can name and a file you produced, not a random checkpoint from a chat.

GPU execution, when you need it, is an option on SessionOptions. The guide shows addCUDA with a device id, and says providers are prioritized in the order they are enabled. If you do not need that, leave it off. A CUDA flag on a machine without the matching build will not make the model smarter. It will fail the session.

Where Deep Java Library fits

Deep Java Library’s quick start describes DJL as a Java library for deep learning, and it points at a beginner tutorial that creates a model, trains it, and runs inference, plus a separate examples directory for training and inference. The prerequisites it lists are a JDK, with JDK 11 recommended and later versions also fine, and git if you clone the repository. Treat that page as the index. Do not treat this article as the tutorial.

Use DJL when you want its model zoo and its training examples, and you are willing to follow those examples rather than a paraphrase. Use ONNX Runtime when you already have an ONNX file and you want a session that scores it. Using both in one service, for the same model, is two runtimes. Pick one for the request path.

The DJL page also points at interactive toolkits for trying inference online and downloading a starter template. That is a way to see a result before you commit the dependency. It is not a production deployment. Production still needs the model file, the input names, and a test you wrote.

Keep the model out of the request thread’s imagination

Load the session when the process starts, or behind a lazy holder that loads once. Creating an environment and a session on every HTTP request repeats work the guide describes as opening a session on a file. Close the result of each run. Close the session when the process shuts down, following whatever close method the version you pinned documents. A leaked tensor is a support ticket, not an accuracy problem.

Pin the Maven version in the build file. A floating version will not match the snippet you tested. When you upgrade, run one known input through the old session and the new session and compare the output shape before you ship the row of endpoints.

Do not log the raw user payload next to the model output if that payload is personal. The inference library does not decide your retention rules. Your service does. If you cannot say how long the input is kept, do not add the endpoint yet.

What not to add

Do not paste a training loop into the Java service because a tutorial somewhere called fit. The ONNX Runtime guide is about scoring an existing model. DJL’s beginner tutorial is a separate path, and it lives on their site. Mixing the two from memory is how a service starts a long job inside a web request.

Do not invent input shapes. The guide’s MNIST sample talks about a specific model and a specific layout. Your model has the names and shapes stored in the file. Read them from the session. A shape you remembered from a screenshot will throw, or worse, it will score the wrong tensor quietly.

Do not put a secret in the jar. A local ONNX file does not need an API key. A remote model does, and that key stays on the server that holds it, not in a frontend. This Java path is the server. The browser sends the user text to you. You score or you call onward.

Tomorrow, take one ONNX file you are allowed to use, open a session the way the Java guide shows, print the input names, and run one tensor through run inside try-with-resources. If the result shape is what the model card describes, you have implemented the inference half. If you do not have a model file, you are not ready to add AI. Get the file, or call a service whose contract you have read. That is how to implement AI into Java without pretending the JVM is a training cluster.

When a teammate asks for a chatbot next, keep this session for scoring and write the chat client separately. A chat transcript is not an input tensor until you have defined the features. How to implement AI into Java stays honest when each endpoint names the model file or the remote contract, and the test input is in the repo next to the build.

Review the endpoint with the model file missing on purpose. The service should fail with the path it could not open, not with an empty 200. Then put the file back and run the one tensor again. The check is the output shape and a closed result, not a feeling that the answer was clever.

Store the model path in configuration, not in a string buried in a controller. The person on call needs to see which file the process opened. How to implement AI into Java includes that line in the config, the pinned dependency, and the one test input. Without those, the library is a jar you have not used yet.