What if there was a very patient tutor who guided you through almost any task on your PC screen? (No, I’m not talking about “Clippy”.) This became reality when e.g. Google launched Gemini Live with screen sharing. This (experimental) feature can significantly change how we learn new software, solve problems or brainstorm ideas in real-time. You can try it for free right now, without installing any software.
It can see (and keep track of) the content you share, hear the questions you ask and reply in a natural voice with useful feedback swiftly. But be aware that this is still early tech. AI can give wrong answers or misunderstand you, so always reflect its outputs critically.
Privacy deserves attention, too. Note that open windows, voice or text prompts etc. that you share go into that system. So, check Google’s terms first to make sure you’re okay with it. Personally, I’m only using it for tasks where I don’t mind if the AI/providers see such things. Stick to your organization’s rules, too.
Table of Contents
Optional: Short Setup Tutorial Before We Jump In
If you want to try it yourself, you can visit the Google AI Studio website – all you need is a Google account. My link brings you right into the area labeled “Playground”. There, you can adjust some settings (optionally) like the output voice or system instructions. If you want fact-checking, you can activate the “Grounding” setting, so it verifies responses via a quick Google search. (This may slow down replies a bit yet often improves accuracy.)
Then, you can directly start sharing your screen/mic (or cam) with Gemini. Choose which window or tab to share. Be careful if you have tabs with personal info open. After “opening its eyes”, Gemini starts watching your screen and you can talk to it (via your mic), type prompts etc. Gemini then talks back in a natural tone.
Note that this is only the “quick and dirty” way of accessing this tool directly with your browser. For more “advanced” users, you can use/connect it with Gemini’s Live API (also with Google’s Vertex AI platform). Without further ado, these are 7 of my favorite ways to put this feature to use:
Use Case 1: Live Coding or Data Analysis Partner
Whether you’re an experienced developer or just “vibing”, it takes time/energy to figure out the right code to make your programs work as intended. Gemini helps by inspecting your code editor in real time and suggesting fixes. (It doesn’t matter which editor you use). You may ask “What code can I use to create a function that does…?”
This also – to some degree – works for data analysis. Imagine you have a spreadsheet open with sales figures for the past quarter. You can ask “Do you see any unusual trends in these numbers?” The AI then comments on e.g. sudden spikes or dips. It may not be 100% perfect if your dataset is large or is highly specialized. It’s still helpful though to get some quick first pointers for deeper analysis.
That said, double-check its suggestions or code snippets. AI can mix up things e.g. library versions or outdated code. And if you’re working with confidential data, you maybe don’t want to put your intellectual property into a free public model.
Use Case 2: Smarter Online Shopping & Browsing
Web pages, e.g. product sites full of vague specs, can turn shopping into a headache. Gemini can step in by explaining everything in plain language. I was recently looking at docking stations for my laptop and saw multiple ports labeled with different icons. I asked “What does this symbol mean?” and it broke down each (based on the visible product pics).
Alternatively, when you use something like Google Maps for trip planning, ask it e.g. “Any interesting spots for sightseeing here?”. Or try touring through an “online museum/gallery” and let the AI explain various artistic works (historical facts etc.) on the fly.
For practical (and privacy) reasons, again, I suggest sharing only a specific tab or window, especially when you log into a shopping site incl. payment details. I usually try to keep this “sandboxed” in a separate browser window.
Use Case 3: Step-by-Step Software Tutorials
Few things are more frustrating than starting a new software and feeling lost between the menus and buttons. Gemini makes it easier by visually pointing out each step as you click around. If you need to create a chart in an unfamiliar analytics tool like Tableau, you can say “Show me how to insert a waterfall diagram from these data points.”
Even if a user is already familiar with a tool, I’m pretty sure it can still “teach an old dog a few new tricks”. When you can’t follow, just interrupt it (“Wait, that wasn’t clear. Which button do I press again?”) and it adjusts its instructions/pace. The best part is it never gets annoyed with repeated questions.
I recommend starting with smaller tasks to get a feel for how it guides you e.g. in a rather simple tool like Excel (or ChatGPT). If you notice anything off – like it refers to a button that doesn’t exist – just correct it and it will adapt. But again, it doesn’t know everything (only what it’s trained on), so be aware of its limitations, esp. for newer software versions.
Use Case 4: Co-Pilot for Research & Brainstorming
When I try to research or write new posts, I often hit a “creative block”. Gemini can help spark new ideas based on what’s on the screen. If you’re working on a short story, for instance, you can ask it “Do you notice any gaps in my plot?”. Using the mentioned “Grounding” feature, it can also support with online research on any topic (e.g. to give you a definition of an unfamiliar term on your screen).
This is also great for (group) brainstorming exercises. Try hosting a workshop and let Gemini see the shared whiteboard. As you scribble your notes Gemini can contibute extra angles or concepts. A similar use case is sharing your opened PowerPoint slide deck and asking it for feedback on the content or look and feel.
But always use your own judgment on the final decision. Even with “Grounding” AI can err and it won’t always capture the nuance of your style. (If you use it in a group setting, e.g. remote workshop, you can even consider making it the moderator/host. I haven’t tried that yet, but I find the idea “interesting”.)
Use Case 5: Real-Time (IT) Troubleshooting & Problem-Solving
We all have been stuck with “cryptic” error message or software glitches at some point… Gemini can be a lifesaver by interpreting what’s happening on your screen and suggesting possible solutions and next steps. For example, if your accounting tool fails to import a file, ask “Can you tell why this error popped up?”. It may find a particular format isn’t supported.
Gamers can share their screen during a tricky boss fight or quest. Or ask “Can you explain the game mechanics to me as a newbie?”. (Would have been a game changer in my old Ragnarok days…) That can be fun for onboarding – though in competitive scenarios, it’s probably not so fair. You can transfer this capability to various types of problem-solving situations: What ideas come to your mind?
Keep in mind that AI doesn’t replace thorough testing and support. If you’re dealing with business-critical issues, it’s best to validate any solution with a professional. Also, as always, make sure that your screen doesn’t reveal any sensitive (login) data.
Use Case 6: Design Aid for Usability & Accessibility (e.g. Apps)
Many website/app owners know the pain of guessing what their users want. Gemini brings “a second pair of eyes” viewing your product alongside you. It can, e.g., suggest layout improvements or simpler navigation paths. You could ask “Do you see anything confusing about my homepage?” and it points out an unintuitive menu.
There’s also potential for improving accessibility. For instance, it may flag poor text contrast or structures which screen readers struggle with. In many other scenarios, your “digital co-designer” can help too: You could ask it for suggested improvements (e.g. for your digital product) from the perspective of your target audience (i.e. to simulate the user persona with AI).
I’d still recommend not skipping feedback/user testing, because AI can’t always perfectly predict real-world behavior in all its complexity. But it’s a solid starting point, especially if your budget or team size is limited.
Use Case 7: Workflow Analysis & Productivity
Tabs, browser windows and tasks piling up… Been there, done that. Gemini, with its fresh outside-in perspective, brings order into your digital workspace. You can ask “Do you see a more direct way for me to do X?”. After scanning your screen, it may suggest batching certain tasks, using a tool you haven’t considered yet etc.
This is a relief for people (like me) who sometimes get scattered in flow. By pointing out inefficiencies or clutter, Gemini helps you reclaim some (mental) space. That may include highlighting where you’re getting lost in multi-tasking or teaching you handy shortcuts for repetitive tasks etc. Small changes add up if you do them consistently.
As always: mind privacy. If you have work folders open or personal docs, the AI can see them. Share only what’s needed/acceptable to reduce exposure. Make it a conscious decision.
Wrap-up: New Paradigm of Computer Use?
This tech carries great promise both for short-term experimentation and future maturation. It merges vision, reasoning and speech–to–speech interactions in an adaptive way that feels really refreshing to me.
I wonder for how long those companies will let us use these tools for “free”. (You know the saying “if you’re not paying for the product…“). For now, data security stays my core concern. I’m not too excited about letting an AI roam my screen that could capture private info or help train new models.
We’re seeing similar solutions from Anthropic, OpenAI etc. too – both for desktop sharing but also increasingly “agentic” functionality (i.e. where the AI controls your screen). Right now, Gemini’s reliable multi-modality, fluid flow and vast memory make it feel ahead to me.
Have you tried it already? What did you find it most useful for? Where did it disappoint you? Let me know your thoughts in the comments, get in touch and spread the word.
Cheers,
John
What do you think?