Run a PyTorch job across multiple nodes - San Francisco Compute Docs

This guide takes you from no instances to a torchrun job on 2 nodes with 8 GPUs each. The nodes find each other over a private subnet and exchange training data over an isolated InfiniBand partition. By the end, you’ll have trained nanoGPT on a small corpus of Shakespeare and generated text from it.

Before you start

Set up your account, a partition, a subnet, and a pool before you create any instances. A pool holds the allocation your instances run on. You need the sf CLI, logged in to your account. See the Quickstart to install it. The examples use landsend, a SKU with 8 H100 GPUs per node: ID sku_4EzdPrHcTps, region europe-north1, zone a, on the InfiniBand fabric europe-north1-a.fab1. A fabric is the physical InfiniBand network that a SKU’s nodes connect to. To use another SKU with InfiniBand, replace these values with its own.

Enroll in the InfiniBand preview

Enroll your account so the sf infiniband-partitions commands and the --infiniband-partition flag become available.

sf preview set infiniband-preview=true

Create a partition on your SKU's fabric

Create a partition named training on your SKU’s fabric.

sf infiniband-partitions create --name training --fabric europe-north1-a.fab1

Only instances in the same partition can reach each other over InfiniBand. See Create a partition for the other options.

Create a subnet in your SKU's zone

Create a subnet named training in your SKU’s region and zone.

sf subnets create --name training --region europe-north1 --zone a \
  --cidr 10.1.0.0/24

Note the subnet id in the output, which starts with snet_. You need it to delete the subnet later.Instances in the subnet reach each other over private IPv4 addresses. torchrun uses the subnet to start the job, and the NVIDIA Collective Communications Library (NCCL) sends the training traffic over InfiniBand.

Write a startup script

SF Compute public images ship with no SSH key. This script adds your public keys to the root user when the instance boots. The $(cat ...) lines run on your machine as you write the file. If you have none of these keys, create one with ssh-keygen -t ed25519 first.To use a key stored elsewhere, see SSH into an instance.

cat >startup.sh <<SCRIPT
#!/bin/bash

mkdir -p /root/.ssh
cat >>/root/.ssh/authorized_keys <<"EOF"
$(cat ~/.ssh/id_rsa.pub 2>/dev/null)
$(cat ~/.ssh/id_ecdsa.pub 2>/dev/null)
$(cat ~/.ssh/id_ed25519.pub 2>/dev/null)
EOF
SCRIPT

See Cloud-init for other ways to configure an instance.

Buy an allocation of 2 nodes

Create a pool named training and buy an allocation of 2 nodes of your SKU for 4 hours. The 4 hours start as soon as the order fills, so buy last and start the next section right after. Your organization needs enough credits to cover the price.The sf buy command quotes the price and asks you to confirm before it charges your organization for the full 4 hours. To skip the prompt in a script, add --max-rate with the highest price per node-hour you accept, and --yes.

sf pools create --name training
sf buy --pool training --sku sku_4EzdPrHcTps --count 2 --start now --duration 4h

Launch the job

Each instance runs torchrun with a node rank, a number from 0 that identifies it in the job. The rank 0 instance coordinates the job, and the job needs only that instance’s private subnet address.

Create one instance per node

Create 2 instances in the partition and the subnet. This loop names them training-0 and training-1 and gives them the private addresses 10.1.0.10 and 10.1.0.11. --public-ipv4 gives each instance a public address for sf instances ssh to connect to.

for i in 0 1; do
  sf instances create --name "training-$i" --pool training \
    --sku sku_4EzdPrHcTps \
    --image sfc:image:sfcompute:public:ubuntu-24.04-cuda-13.2 \
    --cloud-init ./startup.sh --infiniband-partition training \
    --subnet training --private-ip "10.1.0.1$i" --public-ipv4 --yes
done

Wait until both instances show running in this list.

sf infiniband-partitions instances training

SSH becomes available after the image downloads and the instance boots, which can take up to 10 minutes. See Instances. If sf instances ssh reports that SSH details aren’t available yet, wait a minute and retry.

Find the subnet interface

Open one terminal per instance and connect as root, the user the startup script added your key to. Start with training-0.

sf instances ssh training-0 --login root

On each instance, print the interface that holds the instance’s subnet address.

ip -br -4 addr show to 10.1.0.0/24 | tr -s ' '

ens1 UP 10.1.0.10/24

The first field, ens1 here, is the interface name, which can differ on each instance.

Set up nanoGPT on every instance

This guide trains nanoGPT, a small GPT model, on 1 MB of Shakespeare. Install uv, clone nanoGPT, install its packages, and prepare the dataset. Run these commands on each instance.

curl -LsSf https://astral.sh/uv/install.sh | sh
source ~/.local/bin/env
git clone https://github.com/karpathy/nanoGPT ~/nanoGPT
uv venv ~/venv
uv pip install --python ~/venv torch numpy tiktoken requests
cd ~/nanoGPT && ~/venv/bin/python data/shakespeare_char/prepare.py

prepare.py downloads the text and writes train.bin and val.bin to data/shakespeare_char/. Each instance reads its own copy.

Run torchrun on every instance

Start torchrun on each instance with its own node rank: 0 on training-0 and 1 on training-1. Replace ens1 with the instance’s subnet interface.

cd ~/nanoGPT
NCCL_SOCKET_IFNAME=ens1 NCCL_DEBUG=INFO ~/venv/bin/torchrun \
  --nnodes 2 --nproc-per-node 8 --node-rank 0 \
  --master-addr 10.1.0.10 --master-port 29500 \
  train.py config/train_shakespeare_char.py \
  --gradient_accumulation_steps=16 --max_iters=500 --lr_decay_iters=500 \
  --warmup_iters=50 --eval_interval=100 --eval_iters=20 --compile=False \
  2>&1 | grep -E "NET/IB|step "

NCCL_SOCKET_IFNAME sets the interface that NCCL uses to connect the instances at startup. The subnet interface is the one the other instance can reach. NCCL then sends the training data over InfiniBand.The flags after the config file change nanoGPT’s single-GPU defaults for 16 GPUs:

--gradient_accumulation_steps=16: nanoGPT splits this value across the GPUs, so it must be a multiple of the GPU count
--max_iters, --lr_decay_iters, --warmup_iters: shorten the run, since each step now covers 16 GPUs’ worth of data
--eval_interval, --eval_iters: evaluate every 100 steps on 20 batches, so evaluation fits the short run

Each instance waits until all --nnodes instances have joined, then training starts. Each GPU prints the NCCL network line, and training-0 prints the loss every 100 steps.

NCCL INFO NET/IB : Using [0]mlx5_4:1/IB … [RO]; OOB ens1:10.1.0.10<0>
…
step 0: train loss 4.2885, val loss 4.2831
step 100: train loss 2.3385, val loss 2.3798
…
step 500: train loss 1.2817, val loss 1.5370

The NET/IB line lists 1 InfiniBand device per GPU, so NCCL sends the job’s traffic over InfiniBand. Device names and their order vary by node. OOB (out-of-band) names the subnet interface that NCCL used to connect the instances.On 2 landsend nodes, the run takes about 40 seconds.If the job exits or hangs with no output, run the command again without the final grep to see the full log.

Generate text from the model

training-0 saves the checkpoint to out-shakespeare-char/. Generate 300 characters from it on training-0.

cd ~/nanoGPT
~/venv/bin/python sample.py --out_dir=out-shakespeare-char \
  --num_samples=1 --max_new_tokens=300

GLOUCESTER:
And will it there by the execution of your swell:
There is the mark of your grieves hew you love.

Clean up

Terminate the instances when the job ends, then delete the partition and the subnet. Run these commands on your machine, not on an instance.

for i in 0 1; do sf instances terminate "training-$i" --yes; done

Delete the partition once sf infiniband-partitions instances training no longer lists the instances.

sf infiniband-partitions delete training --yes

Deletion fails while an instance is still attached. See Delete a partition for the cases that block it. Then delete the subnet. The sf CLI can’t delete subnets, so open Networks in the console and delete training. To use the API instead, create an API token. Grant it a role that can delete subnets, such as admin. Then send a DELETE request with the subnet ID:

curl -X DELETE https://api.sfcompute.com/preview/v2/subnets/snet_1234567890123 \
  -H "Authorization: Bearer your_api_token_here"

If the delete fails because an instance is still attached, wait a minute for its host to finish teardown and retry. SF Compute reserves instance names for about 5 minutes after termination. To rerun the guide sooner, use different instance names. The training pool stays after cleanup, so skip sf pools create on a rerun. Terminating an instance doesn’t free its allocation. You paid for the 4 hours when the order filled, and the pool keeps the 2 nodes until they end. To get credits back for allocation you don’t use, place a sell order on the pool. This example offers the 2 nodes for the next 2 hours at $10.00 per node-hour or more:

sf orders create --side sell --pool training --sku sku_4EzdPrHcTps \
  --count 2 --start now --duration 2h --min-rate 10.00 --allow-standing

Next steps

Use these pages to manage the job’s resources:

InfiniBand partitions: fabrics, partitions, and link checks
Subnets: private networks and addresses
Instances: instance states, priority, and termination
Pools: the allocation for the nodes in your job

​Before you start

​Launch the job

​Clean up

​Next steps

Before you start

Launch the job

Clean up

Next steps