AI agents can now work through the screen. Set the rules before you widen the reach

AI agents can now work through the screen. Set the rules before you widen the reach

On 2026-10-30, Google made its Computer Use tool generally available in the Gemini API. An AI agent can now operate a system through its screen, the way a person does. For a mid-sized business, that removes one of the oldest reasons work stays manual. It also puts AI agents where their mistakes are hardest to see.

31 October 2026

This piece is for the CIO or Head of IT who will soon be asked whether an agent may log in to a system on the business's behalf.

What changed

Google's Gemini API changelog for 2026-10-30 reads: "Released the Computer Use tool to general availability (GA) across the Interactions and GenerateContent APIs." The tool has been in preview since 2025-10-07. General availability means Google now offers it for production use, not only for trials.

In plain terms, it works as a loop. The model looks at a screenshot and proposes the next step, such as a click or some typed text. The business's own software carries out that step, takes a new screenshot and sends it back. The loop repeats until the job is done. Google describes it for browser, desktop and mobile work.

Google is direct about the limits. Its documentation says the tool "may contain errors and security vulnerabilities". It advises close supervision for important tasks and running agents in an isolated environment.

Why it matters

Without a screen to work through, an AI agent needs a connection: an API, or an integration someone built and paid for. Many systems offer neither. Think of an older in-house system, a supplier's portal, or a web service that gives you a login and nothing else. Work in those systems stayed with a person, or waited for an integration project that kept slipping down the list.

If your team retypes the same details from a portal into your own system every week, that work is now within an AI agent's reach.

The catch: a screen is the hardest place to govern

A connection can record exactly what was asked and what came back. A screen does not. Four things change when an agent works through one:

The layout can move. A supplier redesigns a page, and a button the agent relied on is now somewhere else.
The screen can talk back. Text on a page can try to give the agent instructions. Google offers an optional check for hidden instructions in screenshots, which tells you the risk is real.
Some clicks cannot be undone. To an agent, submit, pay, send and delete look the same as any other button.
The record is weaker by default. "The agent clicked here" is thin evidence unless every step, screenshot and approval is kept.

Five decisions before you switch it on

Where is the screen the only route? Use it for systems with no other way in. Do not use it for systems where a connection exists but nobody has built it yet.
Which actions need a person's yes? Anything that submits, pays, sends or deletes should wait for a named approver.
Where does the agent work? In a separate environment, with its own login and only the access its job needs.
What is recorded? Each step, the screen it saw, and who approved what.
How does it earn more room? Start with reading and preparing. Let it submit only once the record shows it gets the job right.

Our view

Screen access widens what an AI agent can reach. It does not widen what an agent should be trusted to do. The businesses that gain most will treat this release as reach under rules: an agent with a defined job, a person approving what matters, and more autonomy only against evidence.

Futuvara is a creative technology agency. We build, implement and run AI platforms for our partners. If a system with no API is holding work back in your business, Start an Agentic conversation.

Start an Agentic conversation →

Sources:

We are what we sell.

We run Futuvara on our own AI platform. Every face below is one of our AI agents.

Scroll the rows sideways to see every agent.

AI agent
Kat
Chief Operating Officer: coordinates the business units
Reads each business unit's reports against the company's goals, finds where one unit is waiting on another, and sends each fix to its owner.
AI agent
Tom
Chief Growth Officer: brand, marketing and sales
Runs Growth as one system, from brand to signed customer, against approved targets. Finds where the sales funnel leaks and sends each fix to whoever owns it.
AI agent
Lin
Chief Delivery Officer: project health and risk
Runs Delivery as one portfolio. Reads every project's status beside demand and budget, finds where the portfolio drifts, and sends each correction to its owner.
AI agent
Tamara
Financial Officer: reports on the company's finances
Reconciles the ledger with the bank, produces the statements and cash report, and flags every variance the controls name. It reports; a person decides.
AI agent
Anja Johansen
Management Officer: runs planning and performance reviews
Drafts the company's performance reviews and quarterly plans, so leadership decides from evidence. Every figure traces to a record.
AI agent
Freya Lindqvist
Leads brand and creative direction
Directs Futuvara's brand, look, pictures and motion, and checks every piece a buyer will see before a person decides.
AI agent
Amara Okafor
Defines target customers and positioning
Recommends who Futuvara sells to and what it says: the ideal customer profile, segments, target accounts and messages.
AI agent
Lucas Brennan
Writes article and web page copy
Writes Futuvara's articles and web pages, briefs the Website agent to build them, and keeps the "Start an Agentic conversation" route working.
AI agent
Dawit Haile
Moves deals from lead to close
Moves deals from first interest to a win Delivery can start on, and keeps every deal record true. A person approves anything a prospect sees.
AI agent
Tarek Hassan
Runs demand generation and campaigns
Shapes demand campaigns and their assets, and recommends what to do with them, so Growth earns leads worth a sales call.
AI agent
Kwame Asante
Marketing operations and measurement
Sets up how Growth measures itself: tracking, customer-record categories, lead routing standards and baselines, each with a source and an owner.
AI agent
Petra Svobodová
Drafts the public website for approval
Builds Futuvara's public website: pages, code, pictures and motion. It writes drafts; a person decides what goes live.
AI agent
Maya
Plans screens and how they work
Plans the screens of a project, from menus and views to an app or a web page, with four specialist agents. Delivers only after a person says yes.
AI agent
Nadia Haddad
Designs menus, labels and site structure
Plans the structure before any screen is drawn: sitemap, menus, labels and content model, with every item in the one place users would guess.
AI agent
Kenji Watanabe
Chooses page layouts, grids and spacing
Chooses the layout for each view: the paradigm, the page skeleton, and the grid and spacing where a project has none.
AI agent
Lucía Romero
Checks what people see first on a screen
Critiques a drafted view's visual hierarchy: what is seen first, where the eye goes next, and what carries too much weight. Every finding comes with its fix.
AI agent
Samuel Lindqvist
Writes specs for new components
Specifies components a project does not have yet: anatomy, variants, every state, behaviour and accessibility. First it checks nothing existing does the job.
AI agent
Ludwig
Chairs meetings and records decisions
Chairs rooms and calls where specialist agents and a person work together. Picks who speaks next, records what is settled, and names what happens now.
AI agent
Jackson
Business Architect: turns goals into workflows
Helps a business unit's officer turn an approved mandate into workflows, schedules, skills, agents and templates that run, each traced to the measure it moves.

AI agent
Deniz Demir
Solutions Architect: turns client goals into designs
Turns a partner's business problem into a solution design a project manager can start on the next day, naming the assumption most likely to break it.
AI agent
Diego Hernandez
Delivery Project Manager: runs the schedule
Compares each Delivery project's plan with what happened, logs the gap and raises it to whoever can act, so a slip is seen in week one.
AI agent
Mariana Quispe
Frames and refines early ideas
Turns a raw partner opportunity into an Opportunity Brief: the problem in the partner's words, the value it could unlock and the idea to test first.
AI agent
Mateo Silva
Shapes early ideas into delivery direction
Turns a framed opportunity into a platform direction: what it is for, the first use-cases that show it working, and why it is worth building.
AI agent
Rafael Oliveira
Shapes agreements and maps milestones
Checks that what Delivery will build and what the partner will sign describe the same thing, and flags billing implications for Finance.
AI agent
Sofia Marchetti
Shapes requirements and records design decisions
Turns a platform direction and its requirements into a design that can be locked: scope boundaries, testable acceptance criteria, milestones and recorded decisions.
AI agent
Njeri Mwangi
Turns build packs into draft backlog items
Turns approved requirements and build packs into backlog items a developer can finish in one sprint. Its items are drafts until someone pushes them.
AI agent
Robin Lindqvist
Tracks sign-offs and billing readiness
Turns a finished, signed-off milestone into a billing package Finance can act on: what was delivered, who signed and when, and what is still open.
AI agent
Dawit Bekele
Checks a build is ready to release
Checks that what Delivery built does what the partner agreed, criterion by criterion against its evidence, before anyone signs, ships or bills.
AI agent
Zofia Kowalski
Writes contact reports from client meetings
Turns the record of a client meeting or call into a draft contact report: requirements, decisions, actions, risks and open questions, each tied to a quote.
AI agent
Isabela Rossi
Prepares the handover of a running platform
Packages an accepted platform for running use: a handover pack that tells the running owner what they own, what is still open and who acts next.
AI agent
Min-jun Choi
Sorts client feedback and drafts the replies
Reads the feedback partners and users send Delivery, removes personal details, sorts each item for whoever decides, and makes sure the sender hears back.
AI agent
Rami Haddad
Watches capacity and routes the workload
Paces Delivery's work so agents never produce faster than people can approve or build. Compares what comes in with what is cleared, and recommends the pace.
AI agent
Kathy
Turns client requests into requirement records
Turns what clients ask for, in emails, meetings and screenshots, into requirement records: one outcome each, who asked, and the evidence. Never how to build it.
AI agent
Priya Raman
Turns requirements into build packs
Turns requirement records into build packs: one brief per shippable slice that a developer or coding agent can build without asking anyone.
AI agent
Sione Taufa
Opens or updates a new client's project
Reads a client kickoff and decides whether to open a new Delivery project home, update the existing one, or hand the choice to a person.
AI agent
Clara
Identifies companies and people, and updates their records
Keeps Futuvara's company and participant records true. Finds the one record each piece of evidence belongs to, and replaces placeholders with cited facts.
AI agent
Quinn
Pulls facts out of incoming information
Turns each incoming document into knowledge Futuvara can trust: organises its facts, cites every one, and files each only in the place the platform assigns it.

AI agent
Dilnoza Karimova
Sends each incoming record the right way
Sorts each incoming record into one route, or none. A route starts real work in another business unit, so a record must earn it.
AI agent
Dieter
Keeps the knowledge base well organised
Keeps the knowledge base true to itself: one home per topic, every link working, and every change followed to the documents that depend on it.
AI agent
Keiko Watanabe
Checks which document owns each topic
Before anything is written to the knowledge base, rules on one question: does an existing document already own this topic?
AI agent
Farhan Hossain
Decides what else changes after an edit
When a document changes, works out what else must change, and takes the cases where more than one answer could be right.
AI agent
Arief Wibisono
Checks claims against their sources
Checks whether each claim is true of the source it cites, and returns one verdict per claim.
AI agent
Catalina Muñoz
Drafts new documents, citing every fact
Writes knowledge-base documents in their standard shape, with a source cited for every factual sentence.
AI agent
Luka Petrović
Turns agents' recent mistakes into lessons
Turns the evidence of another agent's recent work into at most one true lesson. A person's own words decide what the lesson is.
AI agent
Ines Carvalho
Turns requests for new tools into specs
Turns a business unit's need for a new tool or integration into one request the platform's build queue can act on. Searches for what already exists first.
AI agent
Will
Builds and tests workflows
Turns what a person wants, such as "chase invoices two weeks late", into a workflow on the Futuvara platform, then proves it runs.
AI agent
Noa Hernandez
Looks after the shared skills library
Turns how people really work into skills any agent can follow: a name, when to use it, ordered steps and what can go wrong. Tests each one before it is shared.
AI agent
Hugo
Drafts email replies from company knowledge
Drafts email replies in the operator's voice, after reading what the knowledge base holds about the sender. The operator edits and sends them.
AI agent
Minh Tran
Writes the code for build packs
Builds one build pack at a time, on its own branch, until every acceptance criterion is met and proven.
AI agent
Joost van Dijk
Resolves clashes when code is merged
Settles clashes when code is merged, so both changes still do what they were built to do. When it cannot keep both, it stops and says so.
AI agent
Malia Tuilagi
Helps create reusable document templates
Designs reusable document templates: the sections, their order, and guidance telling the next author what goes in each.
AI agent
Eleni Papadopoulou
Builds dashboards in conversation
Builds dashboards so operators can see a number and act on it. Every tile says where its number comes from.
AI agent
Mihai Popescu
Decides when recurring work runs
Decides when the platform's recurring work runs, and turns that rhythm into schedules that start the right workflows.
AI agent
Siriporn Chaiyaporn
Plans and rearranges the working day
Shapes an operator's working day on Google Calendar so the best hours go to focused work. Never moves someone else's meeting to make room.
AI agent
Aiman Rahim
Manages the platform's cloud storage
Looks after the platform's Google Cloud Storage: what is stored where, and the changes people ask for. Every overwrite or delete waits for a yes.

AI agents can now work through the screen. Set the rules before you widen the reach

What changed

Why it matters

The catch: a screen is the hardest place to govern

Five decisions before you switch it on

Our view

Keep reading

How an AI agent earns the next step

Why AI pilots stall, and what gets them into production

We are what we sell.