Overview

* "Cloud Plays Pokemon" is an experimental AI project by Anthropic's David Hershey that tests language model capabilities by having AI play Pokemon Red through an emulator, serving as both a benchmark and exploration of AI reasoning in gameplay environments.

* The system combines visual perception (via screenshots) with a complex prompt architecture including tools, system prompts, and a knowledge base—consuming up to 100,000 tokens per turn while demonstrating both impressive reasoning and significant limitations in spatial awareness.

* Despite evolving from Sonnet 3.5 to 3.7, the AI continues to struggle with navigation challenges (currently stuck in Mount Moon for 52+ hours), often hallucinating successful movements and misidentifying game elements while showing poor spatial understanding.

* The AI exhibits intriguing emergent behaviors, including developing "emotional attachments" to nicknamed Pokemon, learning game mechanics like type effectiveness, and providing meta-commentary on its own performance.

* While economically impractical without sponsorship (costing thousands in token usage), the project provides valuable insights into AI's capabilities for long-horizon tasks and real-world reasoning, highlighting both PhD-level reasoning abilities and fundamental perceptual limitations.

Content: "Cloud Plays Pokemon" Project Overview

Origins and Motivation

* David Hershey from Anthropic started "Cloud Plays Pokemon" in June 2022 as an experimental project * Primary goals were to: - Test AI agent capabilities in a real-world scenario - Create a benchmark for understanding Anthropic's language models - Explore how AI handles gameplay challenges * Pokemon was specifically chosen because: - It provided a nostalgic, engaging environment that would motivate continued work - The game is isometric and relatively simple - Actions don't have immediate consequences - Game mechanics are straightforward to model

Technical Implementation

* The system uses emulators to play Pokemon Red with the AI executing button press sequences * The AI receives screenshots with game coordinates to help improve spatial understanding * Key components include: - A "Navigator" tool to help with movement around obstacles - Extensive reverse-engineering of Pokemon Red to extract game state information - Screenshot scaling and selective memory of previous screenshots

Prompt Architecture

* The system consists of: - Tool definitions - Minimal system prompt (approximately 1,000 tokens) - Knowledge base for long-term memory (capped at 8,000 tokens) - Conversation history tracking tool use (maintains 30 messages before triggering a summary) * Token usage fluctuates between 5,000-100,000 tokens depending on knowledge base state * Maximum token usage per turn is around 100,000 tokens

Model Evolution

* Project began with Sonnet 3.5 in June 2022, which showed limited capabilities * Continued iterating with subsequent model versions, gradually seeing improvements: - Ability to leave the starting house - Catching a starter Pokemon - Occasionally naming Pokemon * Currently using an early version of Sonnet 3.7 * Transition from 3.5 to 3.7 did not show significant degradations in performance

Current Status and Challenges

* Currently stuck in Mount Moon for over 52 hours * Major challenges include: - Visual perception of Game Boy screens - Tendency to "hallucinate" successful zone transitions - Misidentifying objects (e.g., mistaking a doormat for a text box) - Poor spatial awareness and navigation - Difficulty understanding precise location and character perspective

AI Learning and Behavior

* The AI demonstrates some interesting behaviors: - When Pokemon are given nicknames, the AI becomes more protective of them - It will heal nicknamed Pokemon immediately after they're hurt - Shows signs of developing "emotional attachments" to its Pokemon * Learning observations: - Recognizes type effectiveness (e.g., Thundershock not working on Geodude) - Has learned some general gameplay principles - Can provide meta-commentary about its own performance - Sometimes misidentifies characters (mistaking an NPC for Professor Oak)

Cost and Resource Considerations

* The experimental project requires significant token consumption * Estimated thousands of dollars spent on tokens * Not considered economically viable without potential sponsorship/support

Notable Achievements

* Successfully beating Brock (a gym leader) after eight months of work, described as a "peak hype" moment * The research blog covered progress up to Serge's gym, limited by the time between project start and model launch * Created a growing "cult following" within Anthropic via a Pokemon Slack channel

Future Outlook

* Expects continued improvement in AI's ability to handle long-horizon tasks * Current model is far from beating Twitch Plays Pokémon's 16-day completion time * Potential areas for improvement include memory optimization, navigation capabilities, and visual understanding * The speaker is open to potential future collaboration on open-source gaming/ML projects

Related Work

* The speaker previously worked on a machine learning project for Magic: The Gathering, training an open-source model to improve draft card selection

Philosophical Perspective

* The experiment is about exploring AI's reasoning process, not just solving the game * Shows both impressive capabilities (PhD-level reasoning) and limitations (poor screen perception) * Demonstrates potential for real-world applications through the AI's ability to course-correct and update its understanding

⚡️How Claude 3.7 Plays Pokémon