# Making the agent's computer feel local The technical path from a usable remote desktop to faster live control, more accurate agent browser use, and better recording UX. - Author: Connor Loi - Published: 2026-07-04 - Category: Engineering - Canonical: https://tryreplicas.com/blog/making-the-agents-computer-feel-local We want using the agent's computer to feel like using your own. That is a high bar because local development has almost no perceived distance between intent and feedback. You change code, open Chrome, click around, type into the form, drag the thing, scroll the page, and trust what you see. Cloud agents already give us the trade we want: background execution, parallel work, more compute, clean ports, and a real environment that does not depend on your laptop being awake. The part that still feels worse than local is the moment you need to touch the agent's machine yourself. So we have been pushing on computer use from three sides: - **Smooth desktop streaming:** make the desktop stream low-latency enough that testing through the agent's Chrome feels natural. - **Agent accuracy:** give agents a better computer-use interface so browser actions succeed more often. - **Recording UX:** make recordings smoother, faster to watch, and easier to follow than raw screen captures. This is the technical walkthrough of the first wave of that work. The desktop stream got smoother, agents got a more accurate browser interface, and recordings became easier to watch. Current desktop use can hold a stable 50-60fps path in normal usage. The bar we care about is 120fps, with input latency low enough that the desktop starts to disappear, so this is an early checkpoint in a much longer push. ## Smooth Desktop Streaming Remote desktop performance is easy to describe badly. You can make it sound like a video problem, or a networking problem, or a Linux desktop problem. In practice, it is an interaction loop. When you click in the dashboard, the input has to reach the workspace, Chrome has to react, the changed pixels have to be captured, and the next frame has to come back to your browser. If any part waits too long, the desktop feels fake. The useful model is simple: Chrome renders into a virtual screen, a capture process watches for changed regions, the dashboard viewer asks for updates, and the preview path carries frames back to you. The implementation names matter when you are tuning it. They matter less when you are trying to understand the problem. The important lesson from this PR was that the product path had to be measured directly. Synthetic fps can improve while scroll gets worse. A local test can look great while the public preview adds 100ms. We split the measurements into stream fps, click-to-visible latency, first frame after input, real scrolling, render throughput, and payload size. The first useful change was deleting invisible work. The workspace desktop had a compositor running, which is the piece that draws shadows, transparency, and other desktop effects. Those effects are nice on a laptop. They do not help someone watching Chrome through a stream. Turning it off moved local synthetic stream performance from about 42fps to 56.8fps, and local click-to-visible p50 dropped from 62ms to 33ms. Then we tuned how the viewer asks for frames. The desktop server does not just push an infinite stream of video. The browser viewer asks for changed rectangles, and the server replies. Before this work, the viewer was too passive, so it could average a reasonable fps number and still feel late after input. The setting that survived was a 120Hz request cadence with one update per request. Bigger batches looked attractive in synthetic tests and then made real scrolling worse. We also added short refresh bursts after pointer, wheel, and keyboard events, because the first frame after input is what people feel. The capture side needed its own pass. Capturing more often makes the stream look alive, but it can also compete with input and make clicks feel delayed. Capturing less often gives input more room, but then the desktop feels stale. The final tuning kept input eager, kept the capture path predictable, and judged every setting against real public-preview scrolling instead of localhost. We also tried replacing pieces of the transport. Some ideas improved one benchmark and lost the full loop. A raw framebuffer path had strong local numbers. A modern video-codec path decoded smoothly. A direct desktop server avoided some screen capture work. The version we shipped won because it survived the whole workload: dashboard viewer, public preview, input latency, scroll behavior, 1080p, and bandwidth. The follow-up PR shipped the tuning to the viewer with a 1080p floor, `REPLICAS_DESKTOP_RESOLUTION`, Fast / Balanced / Crisp presets, and better 6080 defaults. ## Agent Accuracy The second track was correctness. Before this, a lot of browser use was screenshot, click, sleep, screenshot, guess. That falls apart when the page animates, a modal opens, or the screenshot dimensions differ from the model's mental picture. `replicas computer observe` waits for the screen to settle, saves a 1:1 screenshot, and prints JSON with dimensions, stability, frame count, change count, mouse position, active window, and visible windows. The browser commands use Chrome's DevTools port. The agent can list tabs, read titles and URLs, snapshot visible text and controls, click by text, fill by label, and wait for text. Raw desktop commands still exist, and click / move / scroll / drag now accept percentage coordinates like `50% 50%`. We measured this on a small internal task set: launch an app, find a button, fill a form, wait for navigation, use a modal, scroll, verify state, and recover after the screen changes. GPT 5.5 in the Codex harness was the strongest run. Fable 5 and Opus 4.8 in Claude Code followed. The before/after was about +20 percentage points once the agent had observe, browser snapshots, text actions, waits, and percentage coordinates. ## Screen Recording Post-Processing The recording work was separate. Raw screen recordings are accurate and annoying: long idle gaps, tiny cursor movement, important clicks in one corner of a 1080p frame, slow page loads, and five minutes of video where ten seconds matter. `replicas computer record` now logs actions while ffmpeg captures the raw desktop. On stop, the post-processor builds an action-aware timeline. Actions stay at 1x. Idle time speeds up. Clicks can zoom. Typing and scrolling usually keep full-frame context. Nearby clicks preserve zoom so the camera does not bounce around. The renderer writes an ffmpeg `filter_complex_script`, trims raw segments, adjusts `setpts`, scales, crops back to 1080p, and concatenates the result. It also renders a synthetic cursor from the action log, dedupes noisy points, downsamples movement, gives clicks a dwell period, and applies the same zoom/crop math to the cursor. This PR also fixed drag. `computer drag` now sends a short stepped gesture, because a single mouse-down, jump, mouse-up can look valid at the X level and still fail browser drag handlers. ## Next This is the first wave of computer-use improvements, and it has already changed how we think about the product. The startup lesson from this work was blunt: users judge the system at the handoff. If the agent wrote the code but verifying it feels laggy, weird, or hard to trust, the job is not done. The infrastructure can be impressive and still lose to one bad minute in the browser. The last 10% of the loop becomes the whole experience. The agent space is moving from chat boxes that suggest work toward systems that actually operate software. They need browsers, terminals, local servers, OAuth flows, test accounts, screenshots, recordings, and a way for humans to jump in without breaking the run. That makes computer use less like a side feature and more like the interface between the agent, the environment, and the person supervising it. That is why we are spending time here. Cloud agents already win on parallelism, background work, compute, and isolation. Long term, we want to remove the tax on interaction: 120fps desktop streaming, lower public input latency, better browser observability, more reliable actions, and recordings that make long runs easy to review. If we get this right, using the agent's computer should stop feeling like remote control and start feeling like a normal part of development. You hand off work, let it run, inspect what happened, take over when needed, and trust the feedback loop enough to do it again.