Inference fleet

​

Prerequisites

• SF Compute CLI installed and authenticated (sf login)
• Credits on your account (sf billing balance)
• A sense of which hardware you want (sf instance-skus list)

​

Create a pool

sf pools create --name inference

​

Create an instance template

#!/bin/bash

mkdir -p /root/.ssh
cat >>/root/.ssh/authorized_keys <<"EOF"
ssh-ed25519 AAAA... you@example.com
EOF

# Start your inference server here

sf instance-templates create \
  --name inference-worker \
  --image ubuntu-22.04.5-cuda-12.7 \
  --cloud-init ./startup.sh

​

Create a deployment

sf deployments create \
  --name inference \
  --pool inference \
  --instance-template inference-worker \
  --target-instance-count 4

​

Create a procurement

sf procurements create \
  --name inference \
  --pool inference \
  --target node_count \
  --max-buy-price 20.00 \
  --min-sell-price 10.00 \
  --window 2h \
  --instance-sku isku_4UpxzQw7A8N

​

Scale up

sf deployments set inference --target-instance-count 8

​

Scale down

sf deployments set inference --target-instance-count 2

​

Handling interruptions

• Stateless workers. Download model weights on boot. Local disk does not persist between instances.
• Health check your load balancer. Route traffic only to instances that are ready.
• Longer --window. A higher value (e.g., 6h) reduces gaps but commits more spend. See Tuning the managed window.

​

Monitoring

sf deployments get inference   # Deployment status and instance count
sf procurements get inference  # Procurement status and pricing
sf instances list              # Individual instance status

​

Next steps

• Adjust --max-buy-price and --min-sell-price to control spend
• Buy a reserved block of compute into this pool when you need to guarantee availability:
  sf orders create --pool inference --side buy --nodes 4 --start "in 5h" --duration 24h --max-rate 20.00