Screen sharing with AI

I wrote about sharing live video with ChatGPT last month. Today I’m exploring the ability to share your screen with AI assistants and what that’s set to enable.

ChatGPT currently enables paid subscribers to share their screen via its mobile app (instructions below).

Google hasn’t yet enabled screen sharing via the main Gemini website or app but you can try it out for free on Google AI Studio (instructions below).

Critiquing my website

The first scenario I tried was showing my website and inviting feedback. Here’s the resulting conversation:

Providing tech support

The second scenario I tried was tech support. Here’s the resulting conversation:

Observations

• Despite the pauses and somewhat long-winded replies, it felt surprisingly natural to chat with AI in this way.

• Gemini’s feedback on my website was all pretty sensible if a little generic and it managed to talk me through a resolution to my contrived tech support issue.

• Because Gemini is analysing snapshots rather than the whole video, it can miss important details.

• All the usual caveats about generative AI’s propensity to make things up apply here.

Conclusions

This is a first iteration of this capability and it’s pretty rough around the edges. However, it will improve and deliver greater utility.

In addition to the scenarios I tried above, it’s easy to imagine this getting good enough to have meaningful applications in education, business and accessibility.

I can imagine using it to get realtime feedback on my presentation run throughs (“I think you’re going to lose the audience with that slide, Dan”).

I can also imagine this having a profound impact on user testing, with the potential to test at much greater scale if every session doesn’t have to have a human facilitator for the duration (a significant limiting factor in my experience).

Google is already experimenting with using this capability to offer gamers a companion who can provide real time guidance informed by what’s happening on screen.

More generally, it should mean less copying and pasting and less downloading and uploading documents. Using AI tools currently involves quite a bit of this. Being able to share your screen will reduce the need to constantly manually transfer information from one application to another (I believe reducing rekeying will be a significant near-term impact of AI).

Instructions

Share your screen with ChatGPT (paid subscription required):

1. Open the ChatGPT mobile app

2. Start a voice conversation by tapping the soundwave icon (bottom right)

3. Tap the 3 dots icon and select ‘Share screen’

4. Tap ‘Start broadcast’ and then tap anywhere outside of the dialog box

5. Navigate to the app you want to discuss and start chatting

Share your screen with Gemini:

1. Open Google’s Chrome browser (download here if you don’t have it)

2. Go to Stream Realtime on the Google AI Studio website

3. Click ‘Share your screen’

4. Choose the Window you want so share (or entire screen if you want to show multiple windows)

5. Start chatting to Gemini about what’s on your screen