A self-hosted inference server is production infrastructure executing computation you cannot fully vet, fed by artifacts you did not produce, with an agent-shaped attack surface pointed at it all day. Securing it is five disciplines, and most deployments practice one and a half: verify what you load, harden the host, control the service surface, isolate and watch it, and operate it like the production system it is.
Verify the whole repository, not the weight file
Verification starts with an approved model registry: model, version, source, license, intended use, and the artifact hash. The hash is the artifact identifier, and a signed provenance record establishes source and approval; an aligned model and a stripped variant look operationally identical until you check both. Production pulls from an internal mirror, never directly from a public hub. Non-executable serialization formats such as safetensors are required and pickle-based formats rejected, and that alone is not enough: model repositories ship custom loader code, tokenizers, and configuration flags that can trigger remote code execution, so the first load happens in an isolated validation environment with remote-code flags disabled, and the unit of review is the complete repository. The threat is documented: in 2024, JFrog researchers found roughly 100 models on Hugging Face carrying malicious payloads, some executing code and opening a backdoor the moment the file was loaded.
Harden the host
The serving stack is software with CVEs like any other, and popular inference runtimes have shipped serious ones; patch it on a production cadence, not a lab one. The operating system or container gets the standard hardening treatment: minimal image, no unnecessary services, least-privilege service account. The administrative interface is its own exposure and gets its own isolation: management access rides a separate, authenticated path, never the inference port. The model cache and store carry restrictive permissions, because a writable model directory leaves the served artifacts open to tampering. And the host's secrets, registry pull credentials and any downstream keys, live in a secrets manager, not in the service environment. And the serving stack itself carries a signed manifest or SBOM: the runtime, loader, container base, drivers, and dependencies are inference-server supply chain exactly as the model repository is.
Control the service surface
An inference endpoint with no authentication is an open resource: anyone who reaches it can run inference on your hardware, enumerate models, and in some runtimes pull or delete them. Approved consumers are identified and authenticated, transport is encrypted, and everything else gets no answer. Resource quotas and rate limits bound what any one consumer can burn, which is denial-of-service protection and cost control in one move. Where hosts serve more than one tenant or project, GPU and memory isolation between them is part of the security boundary, not a performance nicety.
Isolate it, and watch it
The host sits inside its own boundary with inbound access limited to identified users and software and outbound access denied by default. Many malicious payloads depend on an outbound path; inside a default-deny boundary, that callback becomes a blocked, logged, alerting event rather than a quiet foothold. The boundary does not replace verification, because malicious code can still corrupt local state or abuse a destination the deployment intentionally permits. The three controls have distinct jobs: provenance and hash verification establish that the host loaded the approved bytes, scanning looks for known malicious content and unsafe loaders, and isolation limits the consequences when the approved artifact contains a threat neither control recognized. Run all three, and observe the result: served-model identity by hash, consumer identity per request, denials at the boundary, and resource consumption, flowing into the same evidence plane as everything else.
Operate it like production
Backups and rollback exist for the serving configuration and the model store, so a bad artifact or bad update reverses cleanly; the model recall drill depends on exactly this. Load-time validation repeats at every model swap, because the registry discipline is only as good as its most recent entry. And the incident response plan names the inference host explicitly: who takes it offline, what evidence is preserved first, and how consumers fail when it goes away.
The weights are an asset too
Base models, fine-tunes, and adapters trained on proprietary data are high-value derivative assets: a fine-tune can memorize and reproduce the data it was trained on. The same inner-boundary treatment given to a design database or clinical dataset applies to the model store and the host serving it.
Where the control plane fits
Ensage AI wraps the inference host and model store as protected assets with Virtual Chambers, enforces default-deny reachability and identified consumers through the enclave, and records served-artifact identity and access through zCenter.
