Turning portrait videos into landscape using AI
One of my AI predictions for 2024 was “video outpainting getting good enough to turn video shot in portrait into landscape (and vice versa), reducing the need to shoot in two aspect ratios or use cropping/pillarboxing”.
In November, Runway released a feature that does just that, called Expand Video.
I tried it out using a couple of portrait videos I shot of my daughter which I wanted to include in a montage of otherwise landscape clips.
The results were better than I expected - to the point that, if I didn't already know, I’m not sure I’d be able to tell that the videos had been expanded using AI. Even knowing, I’d struggle to pinpoint where the original footage ends and the AI-generated video begins.
However, they’re not perfect. The eagle-eyed will notice that the cyclist riding across the grass in the second clip loses their bike in the final frames (in order to match the final frames of the original video). This could likely be addressed by adding a text prompt, but it’s a reminder of generative AI’s propensity to deviate from real-world physics in visual output.
I used the Turbo model, which trades off quality for speed (the 10 second video took just 1 minute to generate). The non-Turbo (and more expensive) Gen-3 Alpha model promises higher fidelity generations.
If you want to try it yourself, you’ll need a paid Runway plan (which start at $15). It’s pretty intuitive but here’s step-by-step instructions:
Head to https://app.runwayml.com/ and register/login
Select ‘Start a new session’
In the left-hand nav bar select the ‘Expand video’ icon
Hit ‘Select video’ and upload the video you want to expand
Hit ‘Upgrade’ and select the $15 standard monthly plan (don’t forget to cancel!)
Describe what you want the AI to generate (optional)
Click ‘Generate’
That's pretty wild!