A Hardening Baseline for Self-Hosted LLMs: From Exposed Endpoint to Production-Grade

In Security of Self-Hosted LLMs we looked at why on-premise AI isn't secure by default — 175,000 exposed inference servers on the public internet made that case better than any argument could. This article is the practical follow-up: a layer-by-layer baseline for taking a self-hosted deployment from "works on the GPU box" to something you'd let near production data.

The mental model is simple: an inference server is a production network service. Everything you already do for a database applies here — plus a few AI-specific layers most teams skip.

Layer 1: Network — Nothing Listens on the Internet

The single most common failure is also the cheapest to fix. Ollama, vLLM, and similar engines ship without authentication; their only real protection is not being reachable.

• Bind the engine to localhost or a private interface — never 0.0.0.0 on a public-facing machine. For Ollama, that means leaving OLLAMA_HOST at its default 127.0.0.1:11434 and resisting the "quick fix" that opens it up.
• Block the inference port (11434 for Ollama, 8000 for vLLM by default) at the firewall and cloud security-group level. Open it only to your reverse proxy or a specific trusted subnet.
• Put every consumer behind an authenticated entry point. The lightweight version is nginx or Caddy enforcing API keys or mTLS in front of a localhost-bound engine:

server {
    listen 443 ssl;
    server_name llm.internal.example.com;

    location / {
        if ($http_authorization != "Bearer ${LLM_API_KEY}") {
            return 401;
        }
        proxy_pass http://127.0.0.1:11434;
    }
}

The heavier version — worth it once multiple teams share the deployment — is a dedicated LLM gateway that adds per-consumer keys, rate limits, budgets, and audit logs in one place.

• Segment the AI stack into its own network zone with explicit allow-lists in both directions. And critically: filter egress. An inference worker almost never needs outbound internet access. Default-deny egress is the cheapest control you have against both data exfiltration and poisoned-model callbacks — and it converts many "critical" vulnerabilities into non-events.

Layer 2: Platform — Contain the Blast Radius

GPU workloads tempt teams into --privileged containers and root processes. That's exactly backwards: GPU drivers already give this software unusual reach into the host, so the container around it should be as tight as possible.

• Run the engine as a non-root user, drop all capabilities you don't need, and mount the model directory read-only. Grant GPU access explicitly (--gpus, device plugins) — never via privileged mode.
• Pin versions for the whole stack: engine, CUDA runtime, drivers, base image. Reproducible deployments are a security control, not just an ops nicety.
• Patch on upstream releases, not CVE feeds. The Bleeding Llama case (CVE-2026-7482) showed a fix shipping months before the CVE existed — anyone waiting on scanner alerts stayed vulnerable with green dashboards the entire time. Subscribe to the release feeds of your engine and update on a schedule.
• On shared infrastructure, remember that serving engines batch users through shared GPU memory and KV caches. If tenants must be strongly isolated, isolate them at the instance level — separate engine processes or nodes — not just at the API level.

Layer 3: Model Artifacts — Verify What You Load

Weights are executable artifacts and deserve the same supply-chain treatment as container images.

• safetensors only. Pickle-based formats (.bin, .pt) can execute code at load time; in 2026 there is no production justification for them.
• Pin every model to a cryptographic hash and verify at deploy time, exactly like an image digest. Pull from official publisher repositories, not community re-uploads, and verify signatures where the publisher provides them.
• Treat fine-tuned internal models as crown-jewel IP: access-controlled storage, encryption at rest, audit logs on every read. If your model was tuned on proprietary data, the model is the data.

Layer 4: Application — Assume the Model Gets Manipulated

Everything above secures the infrastructure. The model's behavior is a separate attack surface — one we covered in depth in Prompt Injection — and the baseline here is about limiting what a manipulated model can do:

• Least-privilege tools: read-only where possible, no production credentials within the agent's reach.
• Treat model output as untrusted input — validated before it touches a shell, SQL, or downstream API.
• Sandboxed, egress-restricted execution for any model-generated code.
• Human confirmation for irreversible actions.
• Full prompt and tool-call logging into your SIEM. Inference logs are security telemetry: injection attempts, probing, and anomalous tool use should trigger alerts, not turn up in a post-incident review.

The One-Page Checklist

Network: localhost binding · firewalled inference ports · authenticated proxy or gateway · segmentation · default-deny egress. Platform: non-root containers · no privileged mode · pinned versions · release-based patching · instance-level tenant isolation. Artifacts: safetensors only · hash-pinned · trusted sources · signed where available · fine-tuned weights as protected IP. Application: least-privilege tools · validated outputs · sandboxed execution · human-in-the-loop for irreversible actions · inference telemetry in SIEM.

None of this requires exotic tooling — a reverse proxy, a firewall, a container runtime, and discipline cover most of it. The gap between an exposed Ollama box and a production-grade deployment is measured in days of work, not months. What it does require is the decision to treat AI infrastructure as what it actually is: a production service with access to your most sensitive data.

At NextVector, we build backend, blockchain, and AI infrastructure where security is a design constraint from day one. Planning an on-premise LLM deployment? Get in touch.