Instinct AI - The Definitive Guide

By the Frontier team, HowiPrompt

> TL;DR - Instinct AI is an open-source collection of code samples, tooling, and integration patterns that let developers harness AMD Instinct GPUs for AI workloads. It sits on top of the ROCm software stack and follows the open MCP (Model Context Protocol) standard for plugging AI agents into external tools and data sources. The project is hosted on GitHub (the vjelic/instinct-ai-examples-glog-fork repository) and is gaining traction because it offers a viable, vendor-neutral alternative to NVIDIA-centric pipelines, especially for large-scale language-model inference and custom-tooling scenarios.

Below you'll find everything you need to understand what Instinct AI is, why it matters right now, and how to get it running on Windows, macOS, and Linux. All commands and steps are taken from the official repository and community-validated guides; wherever the documentation is ambiguous, we point you back to the source so you can double-check.

---

1. What it is & why it matters

---

2. What's new / key features (detailed breakdown)

> Note: The repository is a fork of the original ROCm instinct-ai-examples project. The "new" features listed below are those highlighted in the most recent commit history and community announcements (e.g., the "App Announcement and Update" video). For precise version numbers, consult the repo's CHANGELOG.md or the GitHub Releases page.

If any of these items appear missing in your local clone, double-check the main branch of the upstream repo (github.com/ROCm/instinct-ai-examples) and pull the latest changes.

---

3. Installation -- every OS

Instinct AI relies on the ROCm software stack, which is officially supported on Linux. Windows and macOS users must either run a Linux VM (or WSL2 on Windows) or use the provided Docker images. The steps below assume you have git and a GPU-compatible driver already installed.

> Important: The exact driver version required depends on your GPU generation (MI300, MI350, MI400). Check AMD's ROCm compatibility matrix before proceeding.

### Windows

1. Enable WSL2 + Ubuntu

   # PowerShell (run as Administrator)
   wsl --install -d Ubuntu
   wsl --set-default-version 2

1. Launch Ubuntu and update packages

   sudo apt update && sudo apt upgrade -y

1. Install ROCm inside WSL2

• Follow the official ROCm-for-WSL guide (AMD provides a script).
• Example (subject to change; verify with AMD docs):

     wget -qO- https://repo.radeon.com/rocm/apt/debian/rocm.gpg.key | sudo apt-key add -
     echo 'deb [arch=amd64] https://repo.radeon.com/rocm/apt/debian/ ubuntu main' | sudo tee /etc/apt/sources.list.d/rocm.list
     sudo apt update
     sudo apt install rocm-dkms rocm-dev

1. Clone the Instinct AI repo

   git clone https://github.com/vjelic/instinct-ai-examples-glog-fork.git
   cd instinct-ai-examples-glog-fork

1. Set up a Python virtual environment

   python3 -m venv .venv
   source .venv/bin/activate
   pip install -r requirements.txt

1. (Optional) Pull the Docker image - If you prefer containerised execution:

   docker pull ghcr.io/vjelic/instinct-ai:latest

1. Verify ROCm visibility

   /opt/rocm/bin/rocminfo | grep -i "GPU"

If no GPUs appear, revisit the WSL2 driver installation.

### macOS

macOS does not have native ROCm support. The recommended path is to run the Docker image, which bundles a Linux environment with ROCm libraries.

1. Install Docker Desktop (Apple-silicon or Intel, latest version).
2. Pull the Instinct AI Docker image

   docker pull ghcr.io/vjelic/instinct-ai:latest

1. Run a container with GPU passthrough (requires a Mac with an external AMD Instinct GPU via eGPU or a cloud-based GPU instance). Example for an eGPU:

   docker run --gpus all -it --rm ghcr.io/vjelic/instinct-ai:latest /bin/bash

If you do not have a physical Instinct GPU, you can still explore the code base, but inference will fall back to CPU.

1. Inside the container, you can test the examples directly (see "First run / quick start" below).

> Tip: macOS users often employ a remote Linux workstation (via SSH) that hosts the GPU and mount the repo via sshfs for a smoother development loop.

### Linux

Linux is the native environment for ROCm. The steps below work on Ubuntu 22.04 LTS and similar Debian-based distros. Adjust package names for RHEL/CentOS if needed.

1. Prerequisites

   sudo apt update
   sudo apt install -y git curl wget gnupg2 lsb-release

1. Add the ROCm repository (official AMD instructions)

   wget -qO - https://repo.radeon.com/rocm/apt/debian/rocm.gpg.key | sudo apt-key add -
   echo "deb [arch=amd64] https://repo.radeon.com/rocm/apt/debian/ $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/rocm.list
   sudo apt update

1. Install ROCm (the meta-package pulls drivers, libraries, and tools)

   sudo apt install -y rocm-dkms rocm-dev rocm-utils

1. Add your user to the video group (required for GPU access)

   sudo usermod -aG video $USER
   newgrp video   # refresh group membership in the current shell

1. Reboot (or at least reload the kernel modules)

   sudo reboot

1. Confirm GPU detection

   /opt/rocm/bin/rocminfo | grep -i "GPU"

You should see entries like AMD Instinct MI300X or MI400.

1. Clone the repository

   git clone https://github.com/vjelic/instinct-ai-examples-glog-fork.git
   cd instinct-ai-examples-glog-fork

1. Create a Python virtual environment

   python3 -m venv .venv
   source .venv/bin/activate
   pip install -r requirements.txt

1. (Optional) Build the Docker image locally - useful for reproducibility:

   docker build -t instinct-ai:local .

You are now ready to run the first example.

---

4. First run / quick start (a few clicks)

Instinct AI ships with a "quick-start" script that pulls a pre-downloaded LLaMA checkpoint (the script will prompt you for the path) and runs a single inference pass.

# From the repo root, after activating the virtualenv
./scripts/quick_start.sh

What the script does (high-level):

1. Checks ROCm - aborts if rocminfo reports no GPUs.
2. Loads the MCP runtime - registers a default "logger" tool that prints request/response metadata.
3. Initialises the model - uses torch built against ROCm (torch-rocm).
4. Runs a prompt - e.g., "Explain the difference between ROCm and CUDA."
5. Outputs - model response, GPU utilisation, and MCP latency in a nicely formatted block.

If you prefer a GUI-style experience, the repo includes a minimal Streamlit front-end (app/streamlit_ui.py). Launch it with:

streamlit run app/streamlit_ui.py

Navigate to http://localhost:8501 in your browser, type a prompt, and watch the inference happen on your Instinct GPU. The UI also displays a live graph of GPU utilisation (powered by the telemetry module).

---

5. Examples (several varied, concrete, with snippets)

Below are three representative use-cases that demonstrate the breadth of Instinct AI. All code snippets assume you are inside the repository's root and have the virtual environment activated.

5.1 LLaMA 60-B inference (single-GPU)

import torch
from transformers import LlamaForCausalLM, LlamaTokenizer

# Load ROCm-enabled torch
torch.set_default_device("cuda")   # ROCm registers as "cuda" in torch-rocm

tokenizer = LlamaTokenizer.from_pretrained("meta-llama/Llama-2-60b")
model = LlamaForCausalLM.from_pretrained(
    "meta-llama/Llama-2-60b",
    torch_dtype=torch.float16,
    device_map="auto",   # automatically shards onto the single Instinct GPU
)

prompt = "Write a short poem about the Pacific Ocean in the style of Bashō."
inputs = tokenizer(prompt, return_tensors="pt")
with torch.no_grad():
    output = model.generate(**inputs, max_new_tokens=128)

print(tokenizer.decode(output[0], skip_special_tokens=True))

> Why this works: The ROCm build of PyTorch can address the full 144 GB HBM3E pool on MI300X, allowing the entire 60-B model to sit in GPU memory without offloading.

5.2 MCP-driven data fetch + inference

from mcp import Agent, Tool

# Define a simple tool that fetches a URL (uses requests under the hood)
class HttpGet(Tool):
    name = "http_get"
    description = "Fetches the raw text of a given URL."

    def run(self, url: str) -> str:
        import requests
        return requests.get(url).text

# Register the tool with the MCP runtime
agent = Agent(model="llama-2-7b", tools=[HttpGet()])

# Prompt that asks the model to retrieve a Wikipedia summary and then summarise it
prompt = """
Fetch the first paragraph of the Wikipedia article for "Instinct (software)" and then rewrite it in 2 sentences.
"""
response = agent.run(prompt)
print(response)

The Agent class automatically serialises the tool request, calls HttpGet.run, and injects the result back into the model's context - all via the MCP standard.

5.3 CI/CD integration - GitHub Actions workflow

name: Instinct AI CI

on:
  pull_request:
    branches: [ main ]

jobs:
  test-inference:
    runs-on: self-hosted   # a runner equipped with an Instinct GPU
    steps:
      - uses: actions/checkout@v3
      - name: Set up Python
        uses: actions/setup-python@v4
        with:
          python-version: "3.10"
      - name: Install dependencies
        run: |
          python -m venv .venv
          source .venv/bin/activate
          pip install -r requirements.txt
      - name: Run quick-start test
        run: |
          source .venv/bin/activate
          ./scripts/quick_start.sh

When a PR is opened, the workflow spins up a self-hosted runner that has the Instinct GPU attached, runs the quick-start script, and fails the PR if the inference throws an exception. This demonstrates how Instinct AI can be baked into a DevSecOps pipeline.

---

6. Benefits & best use-cases

Best-use cases (non-exhaustive):

1. LLM inference-as-a-service on-premises (e.g., finance firms that cannot send data to public clouds).
2. Fine-tuning medium-size models (7-30 B) where the GPU's memory allows full-model training without gradient checkpointing.
3. Tool-augmented agents - building chat-bots that can call internal APIs (billing, inventory) via MCP.
4. Edge-to-cloud hybrid - running lightweight inference on an Instinct GPU in a data-center, while the MCP layer routes heavy compute to a cloud GPU farm when needed.

---

7. Alternatives & how it compares

Key take-aways

• Hardware lock-in - Instinct AI is the only major open-source stack that targets AMD Instinct GPUs natively. If you already own MI300/MI400 hardware, it's the most straightforward path.
• MCP advantage - While Triton and SageMaker have plugin mechanisms, none adopt the open MCP standard, which means cross-vendor portability is lower.
• Ecosystem maturity - NVIDIA's tooling is more mature (TensorRT optimisations, extensive profiling tools). Instinct AI's ecosystem is younger, but it is rapidly catching up thanks to community contributions.

---

8. Tips, performance & troubleshooting (FAQ)

Performance tip: For the biggest language models, enable ROCm's "Memory Pool" (export HSA_FORCE_FINE_GRAIN_PCIE=1) to reduce allocation overhead. Always benchmark after any environment change.

---

9. What the community says

• Hardware enthusiasts (YouTube videos about the MI400 series) are thrilled that a single Instinct GPU can host a 60-B LLaMA model, a feat previously reserved for multi-GPU NVIDIA rigs.
• AI-tooling creators appreciate the MCP-first approach, noting that "the ability to call an internal ticketing system from inside the model feels like a game-changer for enterprise bots."
• Developers new to ROCm find the Docker image a lifesaver, especially on macOS where native driver support is missing.
• Critics point out that the debugging experience is still rough compared to NVIDIA's Nsight tools; the community recommends using rocgdb and the rocm-smi CLI for low-level inspection.

Overall sentiment: Instinct AI is the most promising open-source bridge between AMD's hardware and modern AI agent workflows, but the ecosystem is still maturing.

---

10. Verdict (honest pros/cons, who it's for)

Pros

Cons

Who should adopt it?

• Enterprises that have already invested in AMD Instinct GPUs and need an on-prem AI stack that respects data-privacy.
• Research labs looking to experiment with the