I built this so an AI agent can work inside Unreal Engine 5.8. It gives Claude hands to drive the editor, eyes to capture and read the viewport, docs to learn from, and a loop to check and fix its own work. You point Claude Code at Unreal and it builds.
It's the setup behind my video on building cities with AI. Fair warning: the AI tooling in Unreal is real but still rough, so plan on spending time getting it connected. Start with the getting-started guide.
What it does
- Streams real-world cities like New York using Cesium and Google 3D Tiles, ready for a flythrough.
- Sets up an aircraft with a chase camera that you can fly over the city.
- Grows a procedural city on camera, stage by stage: shape, districts, blocks, buildings, then roads.
- Drops a playable third-person character into a city, including Epic's City Sample, so you can roam it.
- Turns big viewport captures into a small image and a short data file the agent can read cheaply.
How it works
- Act: the agent builds through the Unreal MCP tools, for example placing an asset in the scene.
- Capture: it takes a viewport shot from a set angle without moving your camera, with a grid and labels overlaid.
- Decode: ue_qa.py turns the capture into a PNG and a JSON file of the camera and labeled objects.
- Read and correct: the agent looks at the image and data, fixes problems, and repeats. It checks each step from top-down, eye-level and player-eye angles.
What you need
- macOS (Apple Silicon, M2+ for Nanite) or Windows
- Unreal Engine 5.8 with the MCP plugin enabled
- Claude Code
- Python 3 (plus imagemagick for refdiff)
- Blender (only for the modeling jobs)
Install it
git clone https://github.com/per-simmons/unreal-agent-harness.git- Install Unreal Engine 5.8, enable the MCP plugin with its server set to start automatically, and connect Claude Code. The full walkthrough is in docs/00-GETTING-STARTED.md.
- Check the connection: in Claude Code, SceneTools.get_current_level should return your level rather than “Unable to connect.”
The full instructions, and the skill itself, are on GitHub.
Try asking
- “Place a cube in the level and capture the viewport so I can see it.”
- “Stream New York with Cesium and set up a flythrough.”
- “Build a procedural city stage by stage, then drop in a playable character.”
Good to know
- The README warns the AI tooling is raw and says to budget time for setup and connection problems.
- Only one change to the scene can run at a time. Unreal has one editor and one game thread, so builders can't work in parallel.
- Captures must go through the agent's MCP tool layer. Raw HTTP requests come back empty for large results.
Written by
Pat Simmons. Pat tests AI models and tools on the AI for Mortals YouTube channel.
