AI Toolkit for VS Code

The AI Toolkit extension (formerly Windows AI Studio) is Microsoft’s VS Code extension for browsing, downloading, running, fine-tuning, and evaluating local models - all from the editor.

1. What AI Toolkit Does

2. Installation

1. Open VS Code
2. Go to Extensions (Cmd+Shift+X)
3. Search for “AI Toolkit” (publisher: Microsoft)
4. Click Install

The extension adds an AI Toolkit icon to the Activity Bar (left sidebar).

3. Browsing and Downloading Models

From the Model Catalog

1. Click the AI Toolkit icon in the sidebar
2. Browse Popular Models or search by name
3. Click a model card to see details: size, architecture, quantization options
4. Click Download to pull the model locally

Where Models Are Stored

Models are downloaded to the AI Toolkit working directory:

Directory structure follows a 4-layer convention:

~/.aitk/models/{publisher}/{model-name}/{runtime}/{display-name}

Example:

~/.aitk/models/microsoft/Phi-4-mini/cpu/phi4-mini-int4

Model Formats

4. Using the Local Playground

Once a model is downloaded:

1. Select it in the AI Toolkit sidebar
2. Click Load in Playground
3. Chat with the model in the interactive panel

Playground Features

• System prompt: Set a custom system message
• Temperature / Top-p: Adjust generation parameters
• Token limit: Control response length
• Multi-turn: Maintains conversation context

Playground vs. Ollama

When to use each: Ollama if you want a local OpenAI-compatible API (see 04_llm_server_and_api.ipynb). AI Toolkit if you want a GUI-first experience with fine-tuning and evaluation built in.

5. Model Conversion to ONNX

To run a Hugging Face model in AI Toolkit, convert it to ONNX format:

Setup

conda create -n model_builder python==3.11 -y
conda activate model_builder
pip install onnx torch onnxruntime_genai transformers

Convert

python -m onnxruntime_genai.models.builder \
  -m /path/to/hf/model \
  -p int4 \
  -e cpu \
  -o ~/.aitk/models/publisher/model-name/cpu/display-name \
  -c /tmp/conversion-cache \
  --extra_options include_prompt_templates=1

Precision × Runtime Combinations

6. Fine-Tuning with QLoRA

AI Toolkit provides a guided fine-tuning workflow directly in VS Code.

Steps

1. Select a model in the sidebar → Fine-tune
2. Prepare dataset: Upload JSONL training data
3. Configure: Set LoRA rank, learning rate, epochs, batch size
4. Train: Runs locally using your GPU (or CPU with longer times)
5. Evaluate: Compare base model vs. fine-tuned model side by side

Dataset Format

{"messages": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "What is RAG?"}, {"role": "assistant", "content": "RAG stands for Retrieval-Augmented Generation..."}]}
{"messages": [{"role": "user", "content": "Explain embeddings"}, {"role": "assistant", "content": "Embeddings are dense vector representations..."}]}

When to Fine-Tune Locally

7. Evaluation

AI Toolkit integrates with promptflow-evals to evaluate model quality.

Built-in Evaluators

Running an Evaluation

1. Prepare a JSONL dataset with questions and ground truth
2. Select a model in AI Toolkit → Evaluate
3. Choose evaluators and configure the judge model
4. View results in the extension panel

Example Evaluation Script

from promptflow.evals.evaluators import SimilarityEvaluator, F1ScoreEvaluator
from promptflow.evals import evaluate
 
results = evaluate(
    data="test_data.jsonl",
    evaluators={
        "similarity": SimilarityEvaluator(model_config=model_config),
        "f1": F1ScoreEvaluator(),
    },
    target=my_model_function,
)

8. AI Toolkit vs. Other Local LLM Tools

9. Practical Workflow

Exploring a New Model

1. Browse catalog in AI Toolkit sidebar
2. Download a quantized ONNX variant
3. Test in playground with sample prompts
4. If quality is good → serve via Ollama or vLLM for your app
5. If quality is poor → fine-tune with QLoRA in AI Toolkit
6. Evaluate fine-tuned model against baseline
7. Export and serve the improved model

Connecting AI Toolkit to Copilot

AI Toolkit models don’t directly replace GitHub Copilot’s backend. However, you can:

• Use AI Toolkit to explore and evaluate models before deploying them as an API
• Serve a local model via Ollama or vLLM, then connect it as a custom endpoint
• Use fine-tuned models for domain-specific tasks (code review, doc generation) alongside Copilot for general coding

10. Troubleshooting

Next Steps

• Try 01_ollama_quickstart.ipynb for an API-first approach to local models
• See 03_local_rag_with_ollama.ipynb for building a local RAG system
• See 05_speculative_decoding.ipynb for inference optimization
• For connecting local models to your coding workflow, see 31-ai-powered-dev-tools/04_mcp_deep_dive.md