With Web Stories for WordPress, we want to make it easy and fun to create beautiful, immersive stories on the web. Videos contribute a large part to the immersive experience of the story format. Thus, we wanted to streamline the process of adding videos to stories as much as possible. For instance, the Web Stories editor automatically creates poster images for all videos to improve the user experience and accessibility for viewers. One key feature we recently introduced in this area is client-side video optimization — video transcoding and compression directly in the browser.
New to Web Stories? You can learn more about this exciting format in my recent lightning talk.
The Problem With Self-Hosting Videos
The Web Stories format does not currently support embedding videos from platforms like YouTube, which means one has to host videos themselves if they want to use any. And here’s where things get cumbersome, because you have to ensure the videos are in the correct file format, have the right dimensions and low file size to reduce bandwidth costs* and improve download speed.
A typical use case for a story creator is to record a video on their iPhone and upload it straight to WordPress for use in their next story. There’s just one problem: your iPhone records videos in the .mov format, which is not supported by most browsers. Once you realize that, you might find some online service to convert the .mov file into an .mp4 file. But that doesn’t address the video dimensions and file size concerns. So you try to find another online service or tutorial to help with that. Ugh.
We wanted to prevent you from having to go down the rabbit hole of figuring this all out.
* Aside: To reduce bandwidth costs, we are actually working on a solution to serve story videos directly from the Google CDN, which is pretty cool and will help a lot to reduce costs for creators!
Alternatives
Of course, there are some alternatives to this. For example services like Transcoder or Jetpack video hosting. These solutions will transcode videos on-the-fly during upload on their powerful servers. So you upload your .mov file, but you receive an optimized .mp4 video. However, that requires you to install yet another plugin. Plus, these services won’t optimize the video to the dimensions optimal for stories. So there’s still room for improvement.
We wanted a solution without having to rely on third-party plugins or services. Something that’s built into the Web Stories plugin and ready to go, requiring zero setup. And since hosting providers don’t typically offer any tools for server-side video optimization, we had to resort to the client.
Making Video Optimization Seamless
In our research, we quickly stumbled upon ffmpeg.wasm, a WebAssembly port of the powerful FFmpeg program, which enables video transcoding and compression right in the browser. Jonathan Harris and I did some extensive testing and prototyping with it until we were comfortable with the results.
The initial prototype was followed by multiple rounds of UX reviews and massive changes to media uploads in the Web Stories editor. In fact, I basically rewrote most of the upload logic so we could better cater for all possible edge cases and ensure consistent user experience regardless of what kind of files users try to upload.
The result is super smooth: just drop a video file into the editor and it will instantly get transcoded, compressed and ultimately uploaded to WordPress. Here’s a quick demo:
Technical Challenges
FFmpeg Configuration
A lot of our time fine-tuning the initial prototype was spent improving the FFmpeg configuration options. As you might know, there’s a ton of them and you can easily shoot yourself in the foot if you’re not familiar with them (which I personally wasn’t). We tried to find the sweet spot with the best tradeoff between video quality, encoding speed, and CPU consumption.
The FFmpeg options we currently use:
| Option | Description |
|---|---|
-vcodec libx264 | Use H.264 video codec. |
-vf scale='min(720,iw)':'min(1080,ih)':'force_original_aspect_ratio=decrease',pad='width=ceil(iw/2)*2:height=ceil(ih/2)*2' | Scale down (never up) dimensions to enforce maximum video dimensions of 1080×720 as per the Web Stories recommendations, while avoiding stretching. Adds 1px pad to width/height if they’re not divisible by 2, to prevent FFmpeg from crashing due to odd numbers. |
-pix_fmt yuv420p | Simpler color profile with best cross-player support |
-preset fast | Use the fast encoding preset (i.e. a collection of options).In our testing, veryfast didn’t work with ffmpeg.wasm in the browser; there were constant crashes. |
Cross-Origin Isolation
ffmpeg.wasm uses WebAssembly threads and thus requires SharedArrayBuffer support. For security reasons (remember Spectre?), Chrome and Firefox require so-called cross-origin isolation for SharedArrayBuffer to be available.
To opt in to a cross-origin isolated state, one needs to send the following HTTP headers on the main document:
Cross-Origin-Embedder-Policy: require-corp
Cross-Origin-Opener-Policy: same-originCode language: HTTP (http)These headers instruct the browser to block loading of resources which haven’t opted into being loaded by cross-origin documents, and prevent cross-origin windows from directly interacting with your document. This also means those resources being loaded cross-origin require opt-ins.
You can determine whether a web page is in a cross-origin isolated state by examining self.crossOriginIsolated.
In addition to setting these headers, one also has to ensure that all external resources on the page are loaded with Cross Origin Resource Policy or Cross Origin Resource Sharing HTTP headers. This usually means having to use the crossorigin HTML attribute (e.g. <img src="***" crossorigin>) and ensuring the resource sends Access-Control-Allow-Origin: * headers.
Now, if you have full control over your website, setting up cross-origin isolation is relatively easy. But the Web Stories editor runs on someone else’s WordPress site, with all sorts of plugins and server configurations at play, where we only control a small piece of it. Given these unknowns, it was not clear whether we could actually use cross-origin isolation in practice.
Luckily, Jonny was able to implement cross-origin isolation in WordPress admin by output buffering the whole page and adding crossorigin attributes to all images, styles, scripts, and iframes if they were served from a different host.
This won’t catch resources that are loaded later on using JavaScript, but that’s quite rare in our experience so far. And since we only do this on the editor screen and only when video optimization is enabled, there are less likely to be conflicts with other plugins.
Other Use Cases
Over time, we have expanded our usage of FFmpeg in the Web Stories editor beyond mere video optimization during upload. For example, users can now optimize existing videos as well and we also use it to quickly generate a poster image if the browser is unable to do so. But there are two other clever uses cases that I’d like to highlight:
Converting Animated GIFs to Videos
Did you know that the GIF image format is really bad? Animated GIFs can be massive in file size. Replacing them with actual videos is better in every way possible. So we tasked ourselves to do exactly this: convert animated GIFs to videos.
Today, in Web Stories for WordPress, if you upload a GIF, we detect whether it’s animated and silently convert it to a much smaller MP4 video. To the creator and the users viewing the story, this is completely visible. It still behaves like a GIF, but it’s actually a video under the hood. Instead of dozens of MB in size, the video is only a few KB, helping a lot with performance.
This feature was actually inspired by this issue my colleague Paul Bakaus filed for Gutenberg. It would be super cool to have this same feature in the block editor as well.
Muting Videos
Often times, creators upload videos to their stories that they want to use as a non-obtrusive background. For such cases, they’d like the video to be muted. But just adding the muted attribute on a <video> still sends the audio track over the wire, which is wasteful.
For this reason, when muting a video in the story editor, we actually remove any audio tracks behind the scenes. It’s one of the fastest applications of FFmpeg in our code base because the video is otherwise left untouched. So it usually takes only a few seconds.
What’s Next?
I am really glad we were able to solve cumbersome real-world issues for our users in such a smooth way. Even though it’s quite robust already, we’re still working on refining it and expanding it to other parts of the plugin. For example, we want to give users an option to trim their videos directly in the browser.
We can then use our learnings to bring this solution to other areas too. For example, it would be amazing to land this in Gutenberg so millions of WordPress users could take advantage of client-side video optimization. However, implementing it at this scale would be inherently more complex.
Comments
2 responses to “Client-Side Video Optimization”
Woah! This is pretty impressive 👏🏼👏🏼👏🏼
[…] With Web Stories for WordPress, we want to make it easy and fun to create beautiful, immersive stories on the web. Videos contribute a large part to the immersive experience of the story format. Thus, we wanted to streamline the process of adding videos to stories as much as possible. For instance, the Web Stories … Continue reading […]