AI Agents - Assignment

User: "Show me the top 5 customers by revenue in 2024"

Agent:
1. Generates SQL: SELECT customer_id, SUM(revenue) ... 
2. Executes query
3. Returns: "Here are your top 5 customers:
   1. Acme Corp - $1.2M
   2. TechStart - $980K
   ..."
4. Creates bar chart visualization

User: "Research the latest developments in quantum computing"

Agent:
1. Searches for "quantum computing 2024 breakthroughs"
2. Scrapes top 5 articles
3. Summarizes each article
4. Compiles comprehensive report with citations

User: "Debug this Python function that's failing tests"

Agent:
1. Analyzes code structure
2. Runs test suite
3. Identifies: "Index out of bounds error on line 15"
4. Suggests fix: "Add bounds checking"
5. Applies fix
6. Re-runs tests → All pass ✅

User: "Schedule a meeting with John next Tuesday at 2pm to discuss Q1 planning"

Agent:
1. Checks calendar for conflicts
2. Finds available slot
3. Creates meeting event
4. Sends email invitation to John
5. Sets reminder for 1 hour before

User: "Evaluate my research agent on 20 test cases"

Pipeline:
1. Loads test suite from eval_cases.json
2. Runs agent on each case, records trajectory + tool calls
3. LLM-as-Judge scores each trajectory (0-5)
4. Computes pass@1 rate, avg tool calls, avg cost
5. Generates markdown report with failure analysis

class ConversationMemory:
    def __init__(self, max_messages=10):
        self.messages = []
        self.max_messages = max_messages
    
    def add_message(self, role, content):
        """Add message to history"""
        pass
    
    def get_context(self):
        """Return recent context for LLM"""
        pass
    
    def summarize_old_messages(self):
        """Compress old messages"""
        pass

class TaskMemory:
    def __init__(self):
        self.completed_steps = []
        self.pending_steps = []
    
    def record_step(self, step, result):
        """Record completed step"""
        pass
    
    def get_progress(self):
        """Return task progress"""
        pass

def test_tool_name():
    """Test tool with valid inputs"""
    result = my_tool(valid_input)
    assert result == expected_output
    
def test_tool_error_handling():
    """Test tool with invalid inputs"""
    with pytest.raises(ValueError):
        my_tool(invalid_input)

def test_agent_simple_query():
    """Test agent with straightforward query"""
    response = agent.run("simple query")
    assert "expected" in response.lower()
 
def test_agent_multi_step():
    """Test agent with complex multi-step task"""
    response = agent.run("complex task requiring multiple tools")
    assert agent.tools_used >= 2
    assert response.success == True

metrics = {
    "task_success_rate": 0.85,  # % of tasks completed successfully
    "avg_tool_calls": 3.2,       # Average tools used per task
    "avg_response_time": 5.4,    # Seconds
    "token_usage": 1500,         # Average tokens per interaction
    "error_rate": 0.05           # % of errors
}

# [Your Agent Name]
 
## Overview
Brief description of what your agent does
 
## Features
- Feature 1
- Feature 2
 
## Installation
```bash
pip install -r requirements.txt

from my_agent import Agent
agent = Agent()
result = agent.run("your query")

### 4.2 Code Documentation
- [ ] Docstrings for all functions
- [ ] Type hints
- [ ] Inline comments for complex logic
- [ ] API reference (auto-generated)

### 4.3 Demo
**Option 1: Video Demo (3-5 minutes)**
- Show agent handling 3+ different queries
- Explain tool selection decisions
- Demonstrate error handling

**Option 2: Live Demo + Gradio UI**
- Build web interface
- Demo during presentation
- Include example queries

---

## 🎁 Optional Extensions

### Optional Extension 1: Advanced Reasoning
Implement **ReAct** (Reasoning + Acting) pattern:

### Optional Extension 2: Parallel Tool Execution
- Execute multiple independent tools concurrently
- Aggregate results efficiently
- Handle parallel errors gracefully

### Optional Extension 3: Agent Optimization
- Cache frequent API calls
- Optimize token usage
- Reduce latency with streaming
- Smart tool selection (skip unnecessary tools)

### Optional Extension 4: Production Deployment
- Deploy as REST API (FastAPI/Flask)
- Add authentication
- Rate limiting
- Monitoring dashboard
- Docker containerization

### Optional Extension 5: MCP Integration
- Expose your agent's tools via MCP server
- Connect to your agent from an MCP-compatible client (Claude Desktop, Cursor)
- Demonstrate cross-runtime tool sharing

### Optional Extension 6: Agent Evaluation Pipeline
- Build an LLM-as-Judge scorer for trajectory quality
- Run eval suite of ≥10 test cases with pass/fail tracking
- Generate a markdown report with failure analysis
- See **Notebook 10: Agent Evaluation** for patterns

---

## 📊 Self-Review Guide

### Part 1: Agent Design & Implementation
| Criteria | Relative Emphasis | Description |
|----------|-------------------|-------------|
| **Architecture** | High | Clean code, separation of concerns, configurability |
| **Tools** | High | All tools work correctly, proper schemas, error handling |
| **Reasoning** | Medium | Intelligent tool selection, multi-step planning |

### Part 2: Memory & State
| Criteria | Relative Emphasis | Description |
|----------|-------------------|-------------|
| **Conversation History** | High | Properly stores and retrieves context |
| **Task Memory** | High | Tracks progress, resumes from failures |
| **Implementation** | Medium | Clean code, efficient storage |

### Part 3: Testing & Evaluation
| Criteria | Relative Emphasis | Description |
|----------|-------------------|-------------|
| **Unit Tests** | High | Comprehensive coverage, edge cases |
| **Integration Tests** | High | End-to-end scenarios, error cases |
| **Metrics** | Medium | Proper evaluation methodology |

### Part 4: Documentation & Demo
| Criteria | Relative Emphasis | Description |
|----------|-------------------|-------------|
| **README** | High | Clear, comprehensive, examples |
| **Code Docs** | Medium | Docstrings, type hints, comments |
| **Demo** | High | Shows key features, explains decisions |

### Optional Extensions
- ReAct pattern: +5
- Parallel execution: +5
- Optimization: +5
- Deployment: +5
- MCP integration: +5
- Agent evaluation: +5

---

## 💡 Hints & Tips

### Getting Started
1. **Start simple:** Build basic agent with 1-2 tools first
2. **Test early:** Write tests as you build tools
3. **Iterate:** Add features incrementally
4. **Use frameworks:** LangChain can simplify development

### Tool Design
- Keep tools focused (single responsibility)
- Validate inputs rigorously
- Return structured data (JSON)
- Include helpful error messages

### Debugging
- Log all LLM calls and tool executions
- Test tools independently before agent integration
- Use `print` statements liberally
- Check token usage to avoid context overflow

### Common Pitfalls
- ❌ Tools that do too much (break into smaller tools)
- ❌ Poor error handling (always validate inputs)
- ❌ No logging (impossible to debug)
- ❌ Ignoring context limits (manage tokens carefully)

---

## 📚 Resources

### Code Examples
- [OpenAI Function Calling Examples](https://cookbook.openai.com/examples/how_to_call_functions_with_chat_models)
- [LangChain Agent Templates](https://python.langchain.com/docs/modules/agents/agent_types/)
- [Agent Design Patterns](https://github.com/microsoft/ai-agents-for-beginners)

### Testing
- [Pytest Documentation](https://docs.pytest.org/)
- [Unit Testing Best Practices](https://realpython.com/python-testing/)

### Deployment
- [FastAPI Tutorial](https://fastapi.tiangolo.com/tutorial/)
- [Docker for Python](https://docs.docker.com/language/python/)

---

## 🤝 Collaboration Policy

- **Recommended default:** Build this project independently
- **Getting help:** GitHub Discussions, documentation, and targeted implementation questions
- **Code sharing:** Don't share solutions, but discuss approaches
- **AI assistance:** OK to use for debugging, not for writing entire agent

---

## 📅 Project Packaging

**Suggested GitHub workflow:**
1. Create repo: `ai-agent-[your-name]`
2. Include all deliverables
3. Add comprehensive README
4. Save the repo link with your project notes or portfolio materials

---

## ❓ FAQ

**Q: Can I use LangChain or must I build from scratch?**  
A: You can use frameworks, but you must understand and explain the code.

**Q: How many tools are required?**  
A: Minimum 4 tools. More is better if they're all useful.

**Q: Can I use mock/fake APIs for testing?**  
A: Yes for testing, but include at least one real API integration.

**Q: What if my agent makes mistakes?**  
A: That's OK! Document the failure cases and explain why they occur.

**Q: Can I work in a team?**  
A: No, this is individual. But you can discuss ideas with classmates.

---

**Good luck building your AI agent! 🚀🤖**