How to Run NanoGPT Locally: Step-by-Step Setup Guide

Running your own language model sounds intimidating, but NanoGPT makes it surprisingly accessible. This guide walks you through every step — from installation to generating your first private AI text.

Prerequisites

Before starting, ensure you have:

Hardware Requirements

Component	Minimum	Recommended
CPU	Any modern processor	4+ cores
RAM	8GB	16GB+
GPU	Integrated graphics	NVIDIA GPU with 4GB+ VRAM
Storage	2GB free space	10GB+ for datasets

Software Requirements

• Python 3.8+ (Python 3.10 or 3.11 recommended)
• Git for cloning the repository
• CUDA toolkit (if using NVIDIA GPU — optional but strongly recommended)

Step 1: Set Up Your Environment

Create an isolated Python environment to avoid dependency conflicts:

# Create project directory
mkdir nanogpt-project
cd nanogpt-project

# Create virtual environment
python3 -m venv nanogpt-env

# Activate the environment
# Linux/macOS:
source nanogpt-env/bin/activate
# Windows:
# nanogpt-env\Scripts\activate

Step 2: Install Dependencies

With your virtual environment active:

# Install PyTorch (CPU-only version)
pip install torch torchvision torchaudio

# For NVIDIA GPU support (much faster training):
# pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118

# Install additional dependencies
pip install numpy tqdm

Verifying GPU Access

If you have an NVIDIA GPU, verify PyTorch can see it:

python3 -c "import torch; print(f'CUDA available: {torch.cuda.is_available()}'); print(f'GPU: {torch.cuda.get_device_name(0) if torch.cuda.is_available() else \"None\"}')"

Expected output for GPU users:

CUDA available: True
GPU: NVIDIA GeForce RTX 3060

Step 3: Clone NanoGPT

git clone https://github.com/karpathy/nanoGPT.git
cd nanoGPT

Step 4: Prepare Training Data

NanoGPT needs text data to learn from. You have several options:

Option A: Shakespeare Dataset (Quick Start)

The included Shakespeare dataset is perfect for testing:

python data/shakespeare_char/prepare.py

This creates train.bin and val.bin files in the dataset directory.

Option B: Custom Dataset (Privacy Use Case)

For your own private data:

# save as prepare_custom.py
import os
import numpy as np

# Read your text files
text = ""
for filename in os.listdir("your_data_directory"):
    if filename.endswith(".txt"):
        with open(f"your_data_directory/{filename}", "r", encoding="utf-8") as f:
            text += f.read() + "\n"

# Create character-level encoding
chars = sorted(list(set(text)))
vocab_size = len(chars)
stoi = {ch: i for i, ch in enumerate(chars)}
itos = {i: ch for i, ch in enumerate(chars)}

# Encode the text
data = np.array([stoi[c] for c in text], dtype=np.uint16)

# Split into train and validation (90/10)
n = len(data)
train_data = data[:int(n * 0.9)]
val_data = data[int(n * 0.9):]

# Save as binary files
train_data.tofile("data/custom/train.bin")
val_data.tofile("data/custom/val.bin")

# Save vocabulary for decoding later
import pickle
with open("data/custom/meta.pkl", "wb") as f:
    pickle.dump((stoi, itos, vocab_size), f)

print(f"Vocabulary size: {vocab_size}")
print(f"Training set: {len(train_data):,} tokens")
print(f"Validation set: {len(val_data):,} tokens")

Option C: OpenWebText (Larger Dataset)

For more capable models, use the OpenWebText preparation script:

python data/openwebtext/prepare.py

Warning: This downloads several gigabytes and takes time to process.

Step 5: Configure Your Model

Edit config/train_shakespeare_char.py for a quick test run:

# Small model for quick training
out_dir = 'out-shakespeare-char'
eval_interval = 250
eval_iters = 200
log_interval = 10

# Model architecture
n_layer = 4      # Number of transformer layers
n_head = 4       # Number of attention heads
n_embd = 128     # Embedding dimension
dropout = 0.1    # Dropout rate
bias = False     # Use bias in linear layers

# Training parameters
batch_size = 64      # Sequences per batch
block_size = 256     # Context window size
learning_rate = 1e-3
max_iters = 5000
lr_decay_iters = 5000
min_lr = 1e-4
beta1 = 0.9
beta2 = 0.99
warmup_iters = 100

# System
device = 'cuda'  # Change to 'cpu' if no GPU available
compile = True   # Compile model for faster execution

Step 6: Train Your Model

python train.py config/train_shakespeare_char.py

Training progress displays in your terminal:

step 0: train loss 4.2984, val loss 4.2967
step 10: train loss 3.1456, val loss 3.1523
step 20: train loss 2.8901, val loss 2.9012
...
step 5000: train loss 0.8234, val loss 1.5678

Training Times (Approximate)

Hardware	Shakespeare	Custom (1MB)	OpenWebText
CPU only	2-4 hours	8-12 hours	Days
GTX 1060	15-30 min	1-2 hours	12-24 hours
RTX 3060	5-10 min	20-40 min	4-8 hours
RTX 4090	1-2 min	5-10 min	1-2 hours

Step 7: Generate Text

Once training completes, generate text:

python sample.py --out_dir=out-shakespeare-char

For custom models:

python sample.py \
    --out_dir=out-custom \
    --device=cpu \
    --num_samples=3 \
    --max_new_tokens=500

Troubleshooting Common Issues

"CUDA out of memory"

Reduce batch size and block size:

batch_size = 32    # Reduced from 64
block_size = 128   # Reduced from 256

Training is extremely slow on CPU

Use the smallest possible model configuration:

n_layer = 2
n_head = 2
n_embd = 64
batch_size = 32
block_size = 64

Poor quality output

• Increase training iterations (max_iters)
• Use more training data
• Increase model size (more layers, larger embeddings)
• Lower the learning rate if loss is unstable

Python version errors

Ensure you're using Python 3.8+. Check with:

python3 --version

Privacy Best Practices

When using NanoGPT for sensitive data:

1. Disable telemetry — NanoGPT doesn't include any by default, but verify PyTorch settings
2. Encrypt your training data — Use encrypted storage for sensitive documents
3. Network isolation — Consider training on an air-gapped machine for maximum security
4. Secure deletion — Properly wipe temporary files after training
5. Access control — Restrict who can access the trained model weights

# Disable PyTorch telemetry (optional extra step)
export DO_NOT_TRACK=1

What's Next?

You now have a working local language model. From here, you can:

• Experiment with different model sizes and architectures
• Train on domain-specific data for specialized applications
• Fine-tune pre-trained models for better performance
• Build applications that use your model via the Python API

Understanding how NanoGPT works under the hood gives you something cloud APIs never can: complete knowledge of and control over your AI system.

---

Related Articles:

• What Is NanoGPT? — Understanding the fundamentals
• NanoGPT vs GPT-4 — When to use each

Need help? Join our community or check the official NanoGPT repository for detailed documentation.