A Model That Speaks Infrastructure: Fine-Tuning a Custom LLM While Shipping Product in Parallel

Start Your Project

Case Study

Two Requirements, One Team, No Sequencing Luxury

Autoptic came to us with a language problem, not a typical one. Their platform needed a Large Language Model that could understand a proprietary code language — one built specifically to query AWS services and Prometheus metrics. The model had to take a plain-language prompt, translate it into that internal syntax, and adjust the resulting code as the user refined their request.

Autoptic provisioned the AWS resources; Renaiss owned the infrastructure that would train, host, and iterate on the model. At the same time, and without pausing that track, the client needed new features shipped on the application itself. There was no phase one and phase two here — both fronts moved together, and the infrastructure decisions on one side had to hold up while the product kept changing on the other.

‍

Where It Stands Today

The first trained model is already producing accurate results against the client's own data. It's not the finished state — the next steps include tuning hyperparameters, refining the loss function, and a new deployment aimed at performance rather than just correctness. On the application side, new Svelte-based frontend features are in progress, each tied to backend calls that improve the day-to-day experience for the client's team.

This is, deliberately, a case study about a project mid-flight. The value Autoptic got wasn't a finished product on day one — it was an infrastructure and framework foundation solid enough that improving the model from here doesn't require rebuilding anything underneath it.

‍

Where the Judgment Calls Actually Mattered

From Axolotl to Unsloth

The client's own team suggested Axolotl to train the model. It looked sufficient on paper, but it gave us no reliable way to measure how good the model actually was. We moved to Unsloth instead, which added that measurement layer and, in initial testing, returned better responses out of the gate.

A staged EC2 setup that matched each phase of the work

A GPU-powered Training Instance carried the heavy lifting until the training sequence was stable. We then moved to a clean instance to strip unused libraries and reclaim EBS space, and finally introduced a smaller, cheaper Staging Instance once the model no longer needed training-grade compute.

Model transfer without the SCP overhead

Instead of moving model files between instances manually, we used Ollama's own transfer functions and its model repository as centralized, version-controlled storage. Retrieving a previous version stopped being a manual search-and-copy job.

Access locked down before it became a problem

An SSH security group restricted instance access to authorized users only — a basic control, but one that's easy to skip under delivery pressure and expensive to add retroactively.

Idle compute stopped billing on its own

An SSH security group restricted instance access to authorized users only — a basic control, but one that's easy to skip under delivery pressure and expensive to add retroactively.

Frontend and backend moved in step with the model

We modified OpenWebUI's frontend using React and Svelte, and built new backend functions in FastAPI, so the application's interface kept pace with what the model could actually do.

How the Pieces Fit Together

Framework Evaluation

Axolotl covered basic training but offered no way to benchmark model quality. Unsloth replaced it, adding the measurement layer the project needed and improving output quality from the first use case.

Infrastructure Staging

Training, cleanup, and staging each ran on a purpose-sized EC2 instance instead of one instance doing all three jobs at different levels of efficiency. Ollama served the model in a containerized OpenWebUI environment for staging.

Model Versioning

Cost and Access Controls

An SSH security group limited exposure, and an automated Lambda/EventBridge shutdown routine removed the recurring cost of idle instances — without requiring anyone to remember to do it manually.

Application Layer

React and Svelte on the frontend, FastAPI on the backend, both evolving alongside the model rather than waiting for it to be finished.

Frontend and backend moved in step with the model

We modified OpenWebUI's frontend using React and Svelte, and built new backend functions in FastAPI, so the application's interface kept pace with what the model could actually do.

FAQ

Let’s Collaborate