> ## Documentation Index
> Fetch the complete documentation index at: https://docs.sfcompute.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Inference fleet

> Run inference on spot compute with managed procurement

<Warning>
  This feature is in [public preview](/preview/roadmap).
</Warning>

A [deployment](/preview/deployments) maintains N instances on a [pool](/preview/pools). A
[procurement](/preview/procurements) buys and sells spot compute to keep them running. To scale,
update the deployment's target instance count. The procurement adjusts automatically.

## Prerequisites

* SF Compute CLI installed and authenticated (`sf login`)
* Credits on your account (`sf billing balance`)
* A sense of which hardware you want (`sf instance-skus list`)

## Create a pool

```bash theme={null}
sf pools create --name inference
```

## Create an instance template

Define the image and startup script for your inference instances.

```bash startup.sh theme={null}
#!/bin/bash

mkdir -p /root/.ssh
cat >>/root/.ssh/authorized_keys <<"EOF"
ssh-ed25519 AAAA... you@example.com
EOF

# Start your inference server here
```

```bash theme={null}
sf instance-templates create \
  --name inference-worker \
  --image ubuntu-22.04.5-cuda-12.7 \
  --cloud-init ./startup.sh
```

See [Instance templates](/preview/instance-templates) for details on cloud-init and image
configuration.

## Create a deployment

```bash theme={null}
sf deployments create \
  --name inference \
  --pool inference \
  --instance-template inference-worker \
  --target-instance-count 4
```

4 instances are created in `awaiting_allocation`. They start once the pool has compute time.

## Create a procurement

`node_count` as the target tells the procurement to match however many instances exist on the
pool. Pass `--instance-sku <id>` to pin the procurement to specific hardware; see
[instance SKUs](/preview/instance-skus) for the catalog.

```bash theme={null}
sf procurements create \
  --name inference \
  --pool inference \
  --target node_count \
  --max-buy-price 20.00 \
  --min-sell-price 10.00 \
  --window 2h \
  --instance-sku isku_4UpxzQw7A8N
```

The procurement sees 4 waiting instances and places buy orders. Within minutes, your instances move
to `running`.

## Scale up

```bash theme={null}
sf deployments set inference --target-instance-count 8
```

4 new instances are created. The procurement buys compute to cover them.

## Scale down

```bash theme={null}
sf deployments set inference --target-instance-count 2
```

Excess instances are removed. The procurement sells unneeded compute.

## Handling interruptions

<Warning>
  Spot compute is not guaranteed. Instances may shut down if the market price exceeds your buy limit
  or other buyers place reservations that consume the capacity. Design workloads to handle instances
  being replaced.
</Warning>

* **Stateless workers.** Download model weights on boot. Local disk does not persist between
  instances.
* **Health check your load balancer.** Route traffic only to instances that are ready.
* **Longer `--window`.** A higher value (e.g., `6h`) reduces gaps but commits more spend. See
  [Tuning the managed window](/preview/procurements#tuning-the-managed-window).

## Monitoring

```bash theme={null}
sf deployments get inference   # Deployment status and instance count
sf procurements get inference  # Procurement status and pricing
sf instances list              # Individual instance status
```

## Next steps

* Adjust `--max-buy-price` and `--min-sell-price` to control spend
* Buy a reserved block of compute into this pool when you need to guarantee availability:
  ```bash theme={null}
  sf orders create --pool inference --side buy --nodes 4 --start "in 5h" --duration 24h --max-rate 20.00
  ```