Using Modal Sandboxes for Computer Use

Intro

The idea with this tutorial is to keep the code relatively simple and light on dependencies. Use it as inspiration for your own projects or as a reference with your coding agents. The only dependency is modal!

All the code for this post is on GitHub at DrChrisLevy/modal-computer-use.

Okay, let's get right into it.

Setup

git clone https://github.com/DrChrisLevy/modal-computer-use
cd modal-computer-use
uv sync
uv run modal setup
export MODAL_ENVIRONMENT=your-environment

The Idea

The code snippets here are slightly simplified versions of the code we will import and use later on, but the overall idea is the same.

import modal


class Computer:
    def __init__(self, sandbox: modal.Sandbox):
        self.sandbox = sandbox

    @classmethod
    def create(cls, **options):
        """Start a Sandbox running this computer and wait until its screen is usable."""
        app = modal.App.lookup("computers", create_if_missing=True)
        settings = {"image": cls.image, "timeout": 3600, **cls.options, **options}
        computer = cls(modal.Sandbox.create(*cls.command, app=app, **settings))
        computer.sandbox.wait_until_ready(timeout=180)
        return computer

    def run(self, *argv, binary=False):
        """Run one command inside the computer and return its output."""
        process = self.sandbox.exec(*map(str, argv), text=not binary, env=self.exec_env)
        output = process.stdout.read()
        if process.wait() != 0:
            raise RuntimeError(f"{argv[0]} failed: {process.stderr.read()}")
        return output

To create a Linux desktop from the above base class, we can do the following:

desktop_image = (
    modal.Image.debian_slim(python_version="3.12")
    .apt_install("xvfb", "xfce4", "dbus-x11")  # a virtual screen and a desktop to put on it
    .apt_install("x11vnc", "novnc", "websockify")  # the live viewer
    .apt_install("xdotool", "scrot")  # the hands and the eyes
    .apt_install("mousepad", "xfce4-terminal", "chromium")  # something to use
    .run_commands("useradd --create-home agent")  # the desktop runs as an ordinary user, not root
    .add_local_file("start-desktop.sh", "/opt/start-desktop.sh", copy=True)
)


class Desktop(Computer):
    image = desktop_image
    command = ("bash", "/opt/start-desktop.sh")
    options = {"cpu": 2, "memory": 4096, "readiness_probe": modal.Probe.with_tcp(6080)}  # ready when the viewer is up
    exec_env = {"DISPLAY": ":0"}

    def screenshot(self) -> bytes:
        return self.run("scrot", "--pointer", "-", binary=True)

    def click(self, x: int, y: int):
        self.run("xdotool", "mousemove", x, y, "click", 1)

    def type(self, text: str):
        self.run("xdotool", "type", "--clearmodifiers", "--delay", 12, "--", text)

    def key(self, name: str):
        self.run("xdotool", "key", "--clearmodifiers", "--", name)

The code for interacting with the browser or Android is similar; see the code.

A Quick Look at Each Use Case (Desktop, Browser, Android)

from computers import Android, Browser, Desktop

A Linux desktop

import time

with Desktop.create() as desktop:
    print("Watch it live:", desktop.live_view())  # a private link to the screen
    input("Open the link, then press Enter to start. ")

    desktop.launch("mousepad")  # open a text editor
    time.sleep(3)  # give the window a moment to appear
    desktop.type("Hello from a Modal Sandbox.")
    desktop.key("ctrl+a")  # select what we typed

    png = desktop.screenshot()  # the screen as PNG bytes
    input("Press Enter to shut the desktop down. ")

A browser

A Browser is that same desktop with Chromium running full screen on top:

with Browser.create() as browser:
    print("Watch it live:", browser.live_view())
    browser.goto("https://modal.com")
    input("Press Enter to shut the browser down. ")

An Android phone

An Android is Google's Android emulator running in a Sandbox, controlled with adb. The emulator is itself a virtual machine, so this one needs nested virtualization, which is available on Modal's Enterprise plan.

import time

with Android.create() as phone:
    print("Watch it live:", phone.live_view())
    input("Open the link, then press Enter to start. ")

    phone.open_app("Settings")
    time.sleep(3)
    phone.click(376, 660)  # tap the search bar
    time.sleep(2)
    phone.type("battery")
    time.sleep(3)
    phone.adb("shell", "cmd", "uimode", "night", "yes")  # any adb command works: this one turns on the dark theme
    time.sleep(3)
    phone.open_app("Clock")
    time.sleep(3)
    phone.key("HOME")
    time.sleep(2)
    phone.swipe("up")  # open the list of apps

    input("Press Enter to shut the phone down. ")

Skipping the phone's boot with a snapshot

The Sandbox itself starts in about a second, but the Android phone inside it takes about a minute to boot. To speed that up, we can boot one phone, take a memory snapshot of it, and start every later phone from that snapshot. Memory snapshots are in alpha at the time of writing.

import time

import modal

phone = Android.create(_experimental_enable_snapshot=True)
snapshot = phone.sandbox._experimental_snapshot()

copy = Android(modal.Sandbox._experimental_from_snapshot(snapshot))
print("Watch it live:", copy.live_view())
input("Open the link, then press Enter to start. ")

copy.open_app("Clock")
time.sleep(3)
copy.open_app("Settings")
time.sleep(3)
copy.swipe("up")  # scroll down the list

input("Press Enter to shut the copy down. ")
copy.terminate()

The first boot took about a minute, and each copy was ready in about 3 seconds.

Let an Agent Use the Tools

Now we hand those methods to an agent. The models here come from Modal's Model APIs (see docs). I used Kimi K3.1 Pro and also tried Qwen 3.8 Max VL, but the choice doesn't really matter: any model that can look at an image and speak the OpenAI-style chat API will do.

To follow along, create a kimi-k3-1-pro endpoint and a proxy token to call it with, as the Modal docs describe:

modal endpoint create --shared --name kimi-k3-1-pro --model moonshotai/Kimi-K3.1-Pro
modal workspace proxy-tokens create
export MODAL_PROXY_TOKEN_ID=...
export MODAL_PROXY_TOKEN_SECRET=...

How the tools reach the model

There is no framework here, just an LLM and a basic agent loop. In the code, each method with the @action decorator is available as a tool:

class Desktop(Computer):
    @action
    def type(self, text: str):
        """Type text into whatever has keyboard focus."""
        self.run("xdotool", "type", "--clearmodifiers", "--delay", 12, "--", text)

computer.actions("pixels") collects those into one line each: the signature plus the docstring. For the browser, it gives:

- click(x: int, y: int, button: str = 'left', count: int = 1) - Click at a point. button is left, middle or right; count=2 double-clicks.
- drag(x1: int, y1: int, x2: int, y2: int) - Press the left button at one point, move to another, and release.
- goto(url: str) - Load a URL in the browser.
- key(name: str) - Press a key or shortcut by its xdotool name: Return, Tab, Escape, BackSpace, ctrl+s, alt+F4.
- move(x: int, y: int) - Move the mouse to a point without clicking.
- scroll(x: int, y: int, dy: int) - Scroll the content under a point by about dy pixels. Positive scrolls down, negative up.
- type(text: str) - Type text into whatever has keyboard focus.

That text goes into the prompt. The model answers with one JSON object naming an action:

{"thought": "The page is blank. I'll open modal.com.", "action": "goto", "arguments": {"url": "https://modal.com"}}

Then computer.act("goto", "pixels", url="https://modal.com") looks the method up by name and calls it. So the tools are just the class's own methods, described by their own signatures.

The agent loop

import json
import time

SYSTEM = """You operate a computer to complete a task. Each turn you get a screenshot of its screen,
which is {width}x{height} pixels. Reply with one JSON object and nothing else:
{{"thought": "<what you see and what you will do>", "action": "<name>", "arguments": {{...}}}}

Actions:
{actions}
- done(answer: str) - Finish, once the task is complete."""


def run(computer, task, model, max_steps=30):
    width, height = computer.size
    actions = "\n".join(f"- {line}" for line in computer.actions("pixels").values())
    system = SYSTEM.format(width=width, height=height, actions=actions)
    notes = []  # the model's memory: what it thought and did on earlier steps

    for _ in range(max_steps):
        screenshot = computer.screenshot()  # 1. look at the screen
        prompt = f"Task: {task}\n\nYour steps so far:\n" + "\n".join(notes)
        decision = json.loads(model.complete(system, prompt, screenshot))  # 2. ask the model for one action
        if decision["action"] == "done":
            return decision["arguments"].get("answer")
        computer.act(decision["action"], "pixels", **decision["arguments"])  # 3. do it
        notes.append(f"{decision['thought']} -> {decision['action']}({decision['arguments']})")
        time.sleep(computer.settle)  # 4. give the screen a moment to change

Like the classes earlier, this is a simplified version of the real loop, which is about 190 lines. The real one can also cope with replies it can't read, show the model the page as text instead of a screenshot, and pause a task and pick it up later.

import agent
from computers import Browser

model = agent.modal_endpoint("kimi-k3-1-pro")

with Browser.create() as browser:
    result = agent.run(browser, "Go to modal.com and tell me which models are available on their Model APIs.", model)
    print(result.answer)

To try the real loop from your terminal, giving it one task after another:

uv run python -m examples.chat                      # a browser
uv run python -m examples.chat --computer desktop   # a Linux desktop
uv run python -m examples.chat --computer android   # an Android phone

Here it is with a browser:

And here is the Linux desktop version:

How Well Does It Work?

I ran the loop on four public benchmarks with Kimi K3.1 Pro. Every trial gets a fresh Sandbox, and the check reads the computer's real state (the page, a file, a phone setting).

The loop can show the model the screen in three ways. pixels is a screenshot, text is the screen as text with numbered controls, and both sends the two together. For the browser tasks I also ran Browser Use with the same model in the same Sandbox, as a baseline.

Benchmark Pixels Text Both Baseline
MiniWoB++, small browser tasks 117/125 76/125 117/125 Browser Use: 114/125
WebBench, long browser tasks 18/42 28/42 26/42 Browser Use: 33/42
AndroidWorld, phone tasks 82/116 58/116 90/116 -
OSWorld, desktop tasks 30/36 - - OSWorld's own agent: 32/36

Each cell is tasks passed out of the total. A few things stood out:

These are single runs with one trial per task, so a few tasks either way is noise. The commands to run each benchmark yourself are in the README.

Conclusion

I hope this inspires you to go build on Modal, check out Sandboxes, and try some computer use and browser use ideas of your own.