Category: General

  • Automated AMP Validation using Jest and Puppeteer

    Automated AMP Validation using Jest and Puppeteer

    I’ve previously written about how I’ve implemented automated AMP validation using Jest and AMP Optimizer for the Web Stories WordPress plugin. The context there was that we’re writing unit tests using Jest and wanted a way to verify AMP validity of individual components. Recently, we were in a situation were we needed this AMP validation in a different context: Puppeteer.

    Using Puppeteer and Jest

    Puppeteer is a library which provides a high-level API to control a (headless) browser like Chromium or even Firefox. We use Puppeteer selectively to verify the Web Stories plugin’s behavior in end-to-end tests. One goal there was to ensure that the pages we generate in WordPress are 100% valid AMP. So, how can we do that in the browser?

    Puppeteer controls your browser like a puppet. Depicted: Puppets on strings
    Puppeteer controls your browser like a puppet. Photo by Sagar Dani on Unsplash

    Thankfully, Puppeteer provides a wide variety of APIs, such as for running arbitrary JavaScript on pages or taking the page’s source and do something with it (e.g. take a snapshot).

    A simple test/program using Puppeteer could do a task like this:

    1. Open example.com
    2. Click on link X
    3. Wait for navigation
    4. Do Y

    Or in the context of Web Stories for WordPress:

    1. Create new web story in WordPress
    2. Publish story
    3. Preview story on the frontend
    4. Run AMP validation

    In our case, we’re using Puppeteer together with Jest, which means we could implement the Puppeteer AMP validation in a way that was very similar to our pre-existing solution.

    Puppeteer AMP Validation

    And indeed the implementation was pretty straightforward. All that was needed is calling page.content() to get the full HTML contents of the page and then processing it further using our existing functions. The only caveat: we didn’t want to use the AMP Optimizer to accidentally skew results by transforming markup.

    Custom Jest Matchers

    Once again I leveraged Jest’s custom matchers API. Custom matchers allow writing tests like this:

    it('should produce valid AMP output', async () => {
      await expect(foo).toBeValidAMP();
    });Code language: JavaScript (javascript)

    I previously created such matcher for the unit tests, now I just needed the same for the e2e tests. The result is simple and easy to understand:

    async function toBeValidAMP(page) {
      const errors = await getAMPValidationErrors(await page.content(), false);
      const pass = errors.length === 0;
    
      return {
        pass,
        message: () =>
          pass
            ? `Expected page not to be valid AMP.`
            : `Expected page to be valid AMP. Errors:\n${errors.join('\n')}`,
      };
    }Code language: JavaScript (javascript)

    The only real difference to my previous article is the usage of page.content().

    That’s it! We now have a new Jest matcher that validates our web page and warns us when invalid markup is encountered!

    The Result

    Now, when using an assertion like expect(page).toBeValidAMP() in your test suite, you would warned in case of AMP validation errors.

    Curious to see the whole source code? Check out the Web Stories editor’s GitHub repository.

    Use Cases

    In this case, we’re using the Puppeteer AMP validation solution for end-to-end tests. But Puppeteer is much more powerful and can be used in a variety of ways.

    Another use case I could think of is quickly validating a whole website simply by opening each page in a headless browser and call the above function for the AMP validation. You could even do this on a schedule to get instantly notified if somehow your AMP pages become invalid.

  • AMP Fest 2020: Web Stories for WordPress

    AMP Fest 2020: Web Stories for WordPress

    Yesterday, on October 13th, AMP Fest 2020 took place — a free online event on all things AMP ⚡. There was a wide variety of sessions covering performance, Web Stories, AMP For Email and many more things.

    I had the opportunity to participate in AMP Fest with a talk on Web Stories for WordPress, a new visual editor I’ve been working on that brings first-class Web Stories support to WordPress. You can rewatch my AMP Fest session on YouTube to learn all about it:

    I was really excited to share my perspective on Web Stories for WordPress and its benefits for web creators. If you wanna check it out yourself, learn more about it on the plugin website or download it from the WordPress.org plugin directory.

    Check out the event playlist on the AMP YouTube channel for more great content from this year’s event!

  • Automated AMP Validation using Jest and AMP Optimizer

    Automated AMP Validation using Jest and AMP Optimizer

    At Google I am currently working on a new editor to create visual stories on the web. Under the hood, Web Stories are powered by the AMP story format.

    AMP is a simple and robust format for creating user-first web experiences. One of its benefits is the AMP HTML specification. The spec defines the markup requirements for a document to be considered AMP-valid. Tools like the AMP Validator allow developers to easily verify the validation status of a given web page.

    For our project, it is important that the resulting stories always adhere to the specification. In order to guarantee AMP-valid output and prevent regressions, we needed a way to automate the validation process and integrate it into the development workflow.

    Performing AMP Validation

    So how can we run the AMP Validator as part of our test suite? Luckily, the AMP project maintains the official amphtml-validator npm package. The package offers both a command line tool as well as a Node.js API. Using this API we can parse a given string and return a list of found errors:

    import amphtmlValidator from 'amphtml-validator';
    
    async function getAMPValidationErrors(string) {
      const validator = await amphtmlValidator.getInstance();
      const { errors } = validator.validateString(string);
    
      const errorMessages = [];
    
      for (const err of errors) {
        const { message, specUrl } = err;
    
        const msg = specUrl ? `${message} (see ${specUrl})` : message;
    
        errorMessages.push(msg);
      }
    
      return errorMessages;
    }
    Code language: JavaScript (javascript)

    Custom Jest Matchers

    The Web Stories editor is written in React, and Jest is our unit testing framework of choice. It seemed obvious that we’d want to try automated AMP Validation using Jest. For that, we can leverage Jest’s custom matchers API. Custom matchers allow writing tests like this:

    it('should produce valid AMP output', async () => {
      await expect(<MyComponent />).toBeValidAMP();
    });Code language: JavaScript (javascript)

    Next, we need to write our custom matcher that leverages the above helper function and displays potential errors to the developer. At this point it’s worth noting that the validator only parses static strings. Since we want to pass rendered components to the matcher, we need to process them accordingly using something like renderToStaticMarkup.

    import { renderToStaticMarkup } from 'react-dom/server';
    
    async function toBeValidAMP(component, ...args) {
      const string =  '<!DOCTYPE html>' + renderToStaticMarkup(component);
      const errors = await getAMPValidationErrors(string, ...args);
      const pass = errors.length === 0;
    
      return {
        pass,
        message: () =>
          pass
            ? `Expected ${string} not to be valid AMP.`
            : `Expected ${string} to be valid AMP. Errors:\n${errors.join('\n')}`,
      };
    }
    
    expect.extend({
      toBeValidAMP,
    });
    Code language: JavaScript (javascript)

    That’s it! We now have a new Jest matcher that validates our component’s output and warns us when invalid markup is encountered!

    However, the markup will inevitably be invalid, because it lacks the required HTML in the document head, like the AMP boilerplate code. Here is where the AMP Optimizer comes into play.

    Leveraging AMP Optimizer

    AMP Optimizer is a tool to simplify creating AMP pages. One of the many optimizations it performs is automatically importing missing AMP component scripts and adding any missing mandatary AMP tags. This is exactly what we need! Transforming our original markup using AMP Optimizer is as easy as follows:

    import AmpOptimizer from '@ampproject/toolbox-optimizer';
    
    const ampOptimizer = AmpOptimizer.create();
    const params = {
      canonical: 'https://example.com',
    };
    const string = await ampOptimizer.transformHtml(
      renderToStaticMarkup(component),
      params
    );
    
    // ...
    Code language: JavaScript (javascript)

    If we now run the resulting markup through the validator, it won’t complain about missing boilerplate code, but instead only about issues in our original component. One minor drawback there: if it does find issues, our custom Jest matcher will print the whole optimized AMP markup to the console (“Expected ${string} not to be valid AMP.“).

    We can circumvent this by validating the optimized markup, but only printing our original, unoptimized markup in the case of an error:

    const string = renderToStaticMarkup(stringOrComponent);
    const ampOptimizer = AmpOptimizer.create();
    const params = {
      canonical: 'https://example.com',
    };
    const optimized = await ampOptimizer.transformHtml(
      '<!DOCTYPE html>' + string,
      params
    );
    const errors = await getAMPValidationErrors(optimized, ...args);
    
    // ...
    Code language: JavaScript (javascript)

    Et voilà! Put all these pieces together and you have fully functioning and automated AMP validation using Jest and AMP Optimizer!

    The Final Result

    Now, when using an assertion like expect(output).toBeValidAMPStoryElement() in your test suite, you would get a message like follows in case of AMP validation errors:

    Expected <amp-video autoPlay="true" poster="https://example.com/poster.png" artwork="https://example.com/poster.png" alt="" layout="fill" loop="loop"><source type="video/mp4" src="https://example.com/image.mp4"/></amp-video> to be valid AMP. Errors:
        The attribute 'autoplay' in tag 'amp-story >> amp-video' is set to the invalid value 'true'. (see https://amp.dev/documentation/components/amp-video)Code language: PHP (php)

    Curious to see the whole source code? Check out the Web Stories editor’s GitHub repository.

  • Improving WordPress Internationalization with ESLint

    Improving WordPress Internationalization with ESLint

    Avid readers will already know that I am very passionate about internationalization (I18N). Some of my most popular blog posts are about that topic:

    Internationalization is an important aspect in WordPress development as it lays the foundation for a project’s global success. Unfortunately, it is often done wrong, but things get better over time thanks to simplified APIs, improved documentation, and tooling. For example, the WordPress Coding Standards for PHP_CodeSniffer has been detecting incorrect usage of I18N functions for years now. However, there was no equivalent for this kind of detection in JavaScript source files — until today.

    Being involved with the development of many JavaScript-heavy WordPress projects, I often see common mistakes when using the @wordpress/i18n package that could be easily caught by some kind of linter. To validate my thinking, I set out to fix this issue and contribute the solution to the WordPress community.

    Extending The WordPress ESLint Plugin

    First, I started writing down all the things that could possibly be developed to help improve the WordPress JavaScript I18N landscape. This includes things like detecting wrong usage of text domains, missing translator comments, and flagging usage of variables in translatable strings. I even thought about detecting strings that should probably be translatable, but currently aren’t. Tricky to do, but one can dream.

    Then, I was looking for the best place to implement this. Luckily, WordPress and also our own projects already use a handy tool for this: ESLint. ESLint is the JavaScript-equivalent of PHPCS, and the @wordpress/eslint-plugin package is the one that can be used to enforce WordPress coding standards. For me, that was the perfect place to start.

    By reading through ESLint’s great developer documentation I learned all about creating custom linter rules, and studying existing rules in the aforementioned package, as well as eslint-plugin-wpcalypso from WordPress.com, hel.

    Before I knew it, I was knee-deep in writing ESLint rules, tests, and fixes for the issues my rules discovered. Hundreds of lines of code later, you can now use these new features in your projects!

    The New I18N ESLint Ruleset

    In total, I ended up creating six new ESLint rules around internationalization, and improving one existing rule. If you’re already using the recommended ruleset from the WordPress ESLint plugin (version 5.0.0 or higher!), you automatically benefit from these enhancements. Alternatively, you can also only extend the I18N ruleset if wanted. For example:

    {
    	"extends": [ "plugin:@wordpress/eslint-plugin/i18n" ]
    }Code language: JSON / JSON with Comments (json)

    It includes the following rules:

    @wordpress/i18n-text-domain

    The 18n-text-domain rule enforces passing valid text domains to translation functions (e.g. only string literals). It flags things like __( 'Hello World' ), but allows __( 'Hello World', 'awesome-sauce' ) if your project’s text domain is awesome-sauce.

    Your desired project text domain can be specified in the ESLint config as follows:

    {
    	"@wordpress/i18n-text-domain": [ "error", {
     		"allowedTextDomain":  "awesome-sauce"
    	} ]
    }Code language: JSON / JSON with Comments (json)

    @wordpress/i18n-translator-comments

    If using translation functions with placeholders in them, they should have accompanying translator comments. The i18n-translator-comments rule flags the lack thereof.

    @wordpress/i18n-no-variables

    In WordPress development, you must call translation functions with valid string literals as arguments. They cannot be variables or functions for technical reasons. Use the i18n-no-variables rule to easily enforce this.

    @wordpress/i18n-no-placeholders-only

    Translatable strings that consist of nothing but a placeholder, e.g. __( '%s' ), cannot be translated. The i18n-no-placeholders-only rule prevents such usage.

    @wordpress/i18n-no-collapsible-whitespace

    With the i18n-no-collapsible-whitespace rule you can prevent using complex whitespace in translatable strings. Relying on HTML to collapse such whitespace can make translation more difficult and lead to unnecessary retranslation.

    @wordpress/i18n-ellipsis

    Lastly, the i18n-ellipsis rule disallows using three dots in translatable strings. Three dots for indicating an ellipsis should be replaced with the UTF-8 character (horizontal ellipsis, U+2026) as it has a more semantic meaning.

    @wordpress/valid-sprintf

    The existing valid-sprintf rule enforces valid usage of the sprintf function exposed by the @wordpress/i18n package. I’ve extended it to catch and prevent a mix of ordered and non-ordered placeholders. Multiple sprintf placeholders should be ordered so that strings can be better translated.

  • Saving the Romansh Language with WordPress

    Saving the Romansh Language with WordPress

    Tgi che sa rumantsch sa dapli — if you know Romansh, you know more

    (Deutsche Version)

    Switzerland has four official languages: German, Italian, French, and Romansh. Growing up in the canton of Grisons, I got in touch with the latter early on. Unfortunately, it is a dying language. To do something against this, I decided to translate WordPress into Romansh. And I don’t even speak the language!

    But WordPress would be the ideal platform for a Romansh translation. The world’s most popular content management system (CMS) has a market share of 35% and is also very common in Switzerland. That means many people are interacting with it on a daily basis.

    It all began with a simple idea a couple of years ago, I think it was around WordCamp Europe 2015. After talking about this with some people, many showed interest and also thought it would be a cool idea. However, nothing concrete happened yet.

    The First Steps

    In order to move things forward, I got in touch with the WordPress Polyglots team to properly set up Rumantsch on the translation management platform. I figured that this was the biggest hurdle to overcome. Once the translation platform was ready, interested people could just start translating and actually make this happen. I was able to do some basic translations myself thanks to an online dictionary. However, for the more complex strings I needed help from people who actually speak the language.

    Besides talking to friends and acquaintances who speak Romansh, I also got to know Gion-Andri Cantieni and his initiative Software rumantscha. I was pretty impressed when I learned that they have been successfully translating Firefox, Microsoft Office, and even the Contao CMS to Romansh for quite some time. This was even in the news, which showed me that it’s not a crazy idea at all to try to translate WordPress.

    Now that we were a group of people, we were quickly able to translate about a third of WordPress to Rumantsch. At WordCamp Europe 2017, I shared the story about how we got there with the global WordPress community:

    Getting Involved

    Efforts stagnated a bit after that, but now I want to take another attempt at translating WordPress into the Romansh language. It’s quite fitting that this year marks the 100-year anniversary of Lia Rumantscha, the local institution that promotes the Romansh language and culture.

    https://www.grheute.ch/2018/07/19/die-lia-rumantscha-wird-hundert-und-feiert-zuoz/

    As of today, the Rumantsch translation of WordPress is around 35% complete. This is what it looks like in the WordPress admin:

    WordPress in Rumantsch

    To get it to 100%, I need your help!

    First of all, if you’re interested in using WordPress in Rumantsch or want to support the translation efforts in any form, please let me know!

    If you want to jump right into the action and start translating WordPress, all you need is a WordPress.org user account. Once signed up you can head to translate.wordpress.org right away to find all the projects that can be translated.

    This includes WordPress core, but also the WordPress.org websites and even the WordPress mobile apps. The most important project to translate is certainly WordPress 5.0, the current WordPress release.

    We’ve collected some helpful resources for translators at roh.wordpress.org/translatar. Yes, that’s right — WordPress en Rumantsch has its own website! In addition to that page, the Polyglots handbook has some very useful information as well.

    Also make sure to join the WordPress Switzerland Slack workspace at wpch.slack.com using your WordPress.org email address (<username>@chat.wordpress.org). There we have a dedicated #polyglots channel for this purpose.

    Have you got any questions so far? Please leave a comment, send an e-mail, or ping me on Twitter.

  • Get Ready for WordCamp Zurich 2019

    Get Ready for WordCamp Zurich 2019

    It’s been a while since I have been involved with organizing a WordCamp. After a 4-year break, this is changing now. I am very excited that WordCamp Zurich is taking place on September 14, 2019.

    Back in 2014 and 2015, we already hosted two amazing WordPress conferences in Zurich. After a small local event in Switzerland in 2011, those were the first bigger WordCamps with a more international audience. We called them WordCamp Switzerland, as the Swiss community felt we were to small to host city-named events.

    In the following years, many things have changed. The Swiss WordPress community was — and still is — flourishing. This year’s WordCamp Europe in Berlin is excellent proof for that, as there were about 30 attendees from Switzerland present, which I think is amazing.

    The Swiss WordPress community at WordCamp Europe 2019 in Berlin, Germany (Photo: Florian Ziegler)

    Many new WordPress meetups throughout the country have been started since the last WordCamp Switzerland. There have been WordCamps in Geneva, Bern, and Lausanne in the last three years. As a community, we thought now is a good time to go back to Zurich for once.

    What to Expect at WordCamp Zurich

    WordCamp Zurich is not just a local WordPress conference. It is a collaboration between WordPress enthusiasts and friends from all over Switzerland, joining forces to make sure this event will be just as awesome as the previous conferences.

    Conference Day

    The main event is taking place on Saturday, September 14th in the heart of Zurich. This is gonna be one full day with talks from speakers from both local and international speakers, in German and English.

    The Call for Speakers for this conference day is going to open soon, so make sure to subscribe to any news updates.

    Contributor Day

    On September 13, the day before the actual conference, we are organizing a so-called Contributor Day. This event gives you a special opportunity to learn more about how you can contribute to the WordPress open source project.

    This will be a smaller gathering at a different venue. Once registration is open, we will communicate it on the WordCamp website and all social media channels.

    Call for Sponsors

    To make this event a success, the Swiss WordPress community needs your support! WordCamps are non-profit events, organized by people from within the community on a voluntary basis. We rely on companies and individual sponsors to support us, so that we are able to provide attendees with a great event at very affordable ticket prices.

    If you are interested in sponsoring WordCamp Zurich, please check out our Call for Sponsors post.

  • An Introduction to WP-CLI

    An Introduction to WP-CLI

    For the last two years I have been heavily contributing to WP-CLI. WP-CLI is the official command line tool for interacting with and managing WordPress sites. Especially through my work on the wp i18n command, which provides internationalization tools for WordPress projects, I learned more about how people interact with WP-CLI and command line tools in general. With this introductory blog post I intend to show you how easy it can be to use WP-CLI.

    Disclaimer: this post is basically the written version of my talk at this year’s WordCamp London. The recorded video should be available soon.

    The Command Line

    Before we dive right into WP-CLI, I want to introduce you to some general command line basics. This way you can get a better picture of how command line tools are meant to work and why they might respond in a certain way.

    Simply put, the command line is a text interface to interact with a computer. Before we had all these graphical user interfaces, the command line prompt is basically the only thing you got when booting up your computer. There you could type in some command that would execute a certain program.

    Nowadays the shiny UIs on our computers hide all the complexity underneath. However, the command line still gives you a powerful way to do basically anything on your computer. A big benefit of command line tools is that you can easily automate and even combine them together.

    On your computer, you can access the command line using a terminal application. It looks a bit like this:

    The terminal with its command line prompt.

    What you’ll usually see in the terminal is the command line prompt in the form of the $ (dollar) sign. That’s where you can then enter the name of the application that you want to run — in this case myprogram — and some arguments that should be passed to the program. After that, magic stuff will happen 🙂

    $ myprogram --foo="bar" --debug names.txtCode language: JavaScript (javascript)

    In all the upcoming examples, you’ll see this $ at the beginning of each line, indicating the command prompt where you would type the commands.

    There are essentially three types of command line arguments: named arguments, flags, and positional arguments. Here, we set the value of foo to bar. This is a named argument. Flags are like on/off switches. So passing --debug here would turn on debug mode. Last, names.txt is just a name of a file that we want to pass to the program. You could have many more of these so called positional arguments.

    Let’s use a more real-life example! Here’s how you could update literally all of the WordPress sites you manage using three simple commands offered by WP-CLI:

    # Update all your WordPress sites at once
    $ wp @all core update
    $ wp @all plugin update --all
    $ wp @all theme update --allCode language: PHP (php)

    Of course you could further tweak this. For example if you only want to do minor updates from let’s say WordPress 5.2 to 5.2.1 instead of 5.3. That would just require you to type a few more letters.

    Just imagine how long it would take you to update all your sites by manually clicking on some buttons.

    Command Line Building Blocks

    While WP-CLI is a command that you first need to install, there are already plenty of commands available on your system. Here’s a short list of some more common ones:

    • List directory contents: ls
    • Print working directory: pwd
    • Change directory: cd
    • Remove files: rm
    • Make new directory: mkdir

    There are tons of these little commands. And as you might notice from this list here, these commands all have a very simple job to do. Creating such small programs is actually one of the Unix philosophies: write programs that do one thing and one thing only, and write programs that work together.

    I often like to compare them to Lego bricks. A single command only gets you so far. However, when you combine them together, you can build some pretty cool stuff!

    In this post I only cover the basics of the command line. To learn more about it, I suggest checking out resources like Codecademy tutorials or perhaps LinuxCommand.org.

    Exit Codes

    Something that might be perceived as odd at first is that some commands don’t return anything. At first glance, you might think that they don’t work, since nothing is happening. One example for that is the WP-CLI command to check if your WordPress site has already been installed or if you have yet to set it up: wp core is-installed. Although nothing is being output to you directly, the command’s exit code will tell you the site’s installation status.

    To quickly see the exit code of your previously run command, you can use the dollar question mark variable:

    $ wp core is-installed
    $ echo $?
    0Code language: PHP (php)

    It’s good to know that every command has an exit code. On POSIX systems, an exit code of 0 means everything is OK (success), whereas any number from 1 to 255 is a non-success (or error, if you so will). Some commands only use 0 and 1 as exit code though, as they don’t have for more.

    Most of the time, you won’t need that $? variable to find out the exit code, as it’s mostly useful in combination with other commands.

    Command Chaining

    A very simple such combination would be command chaining. For example, instead of running commands like mkdir and cd on their own, you can write things like mkdir test && cd test to say “create a new directory and when that is successful, switch to that directory”. Or the other way around: “create a new directory or print a nice error message when something goes wrong” could be written as mkdir test || echo "Oops".

    So these && and || operators actually check for these exit codes:

    $ wp core is-installed && wp core update
    $ wp post exists 123 || echo "Post does not exist"Code language: PHP (php)

    Pipes

    Another way of combining programs are pipes, or pipelines. Simply put, pipes let you use the output of a program as the input of another one. Here are a few examples:

    • Filter lists using regex: ls -l | grep ".php$"
    • Get the word count: echo "These are four words" | wc -w
    • Delete temporary files: find . -name '*.tmp' | xargs rm

    The one command I like most there is the last one, xargs. In that example, the find command returns a list of all temporary files that one might want to clean up. xargs then takes this list and runs the rm command on each of the files to delete them individually.

    Here’s how you could use xargs in combination with WP-CLI:

    $ wp site list --field=url | xargs -n1 -I % wp --url=% option update my_option my_valueCode language: PHP (php)

    This will retrieve a list of all sites in a network, and then for each of the sites it adds a specific option to the database. Some more examples with WP-CLI and xargs can be found in the handbook. That is also a great place to look up the exact arguments needed for xargs.

    Scripting

    Many times, you need to run multiple commands in a row or run them very often. To make this easier, you can create a shell script for these tasks.

    A shell script is basically a text file with one or more commands that are executed in a linear order. You can also add some code comments to the script to make it easier to comprehend. Here’s a simple example:

    #!/bin/bash
    
    # Update all WordPress sites at once
    
    echo "Start updates..."
    wp @all core update
    wp @all plugin update --all
    wp @all theme update --all
    
    echo "Finished!"Code language: PHP (php)

    Now, we can just execute this single script instead of having to type all commands manually every time we need to use it:

    $ my-first-script.sh
    
    Start updates...
    # [...]
    Finished!Code language: PHP (php)

    This also makes it very easy to share, drop on a server, put on GitHub for collaboration, and so on.

    Meet WP-CLI

    With these fundamentals set, let’s add some WordPress to the mix and see what we can do with WP-CLI.

    First of all, many web hosts nowadays install WP-CLI on all of their servers by default. That means it is immediately available and you don’t have to worry about installing it first and setting everything up.

    Second, it is very intuitive to use and has extensive documentation for all the available commands and configuring WP-CLI. This way you can get started quickly, even if you are not a developer.

    Finally, the goal of WP-CLI is to provide the fastest way to perform any task in WordPress. So if you are ever in doubt about how to do something in WordPress, you might want to check the command line first.

    Getting Started

    To get started with WP-CLI, open the built-in documentation using wp help. This will give you a general help screen with a list of commands. You can also get a help screen for a specific command, e.g. by typing wp help post.

    For information about your WP-CLI environment, you can use wp cli info. And if that command tells you that your version of WP-CLI is out of date, you can simply update it using wp cli update.

    Bundled Commands

    When you install WP-CLI, it comes with a long list of useful commands. These are already built-in and cover pretty much all aspects of WordPress. You can manage things like posts, comments, and plugins all through the command line.

    But there are also some commands that don’t actually require WordPress, because they work independently of a specific WordPress site. One such command is wp i18n, which I’ve mentioned at the beginning of this article.

    Note: You can learn more about the wp i18n command in my blog post about internationalization in WordPress 5.0.

    To give you an idea of what you can do with WP-CLI, here’s a list of some more or less common examples:

    • Delete all products: wp post delete $(wp post list --post_type='product' --format=ids)
    • Generate some dummy users for testing: wp user generate --count=500
    • Show all active plugins: wp plugin list --status=active
    • Perform a search and replace operation on the database: wp search-replace 'http://example.test' 'http://example.com' --skip-columns=guid
    • Generate translation files: wp i18n make-pot

    Global Parameters

    There are some arguments that you can pass to all commands offered by WP-CLI. For example, if you like its output to be more quiet, you can suppress some of the informational messages WP-CLI usually prints using --quiet. Or for the other way around, you can use the --debug flag to get a little more extra information.

    Super helpful is also the ability to skip some plugins or even themes using --skip-plugins and --skip-themes. Some plugins might not always work well in combination with WP-CLI. This flag this allows you to disable a plugin or theme for just this one command.

    Learn more about global parameters.

    Common Use Cases

    Install a WordPress Site

    WordPress praises itself for its famous 5 minutes installation procedure. The truth is, it’s often a bit longer than that. But with WP-CLI, we can actually bring this time down to seconds.

    Using just three WP-CLI commands we can download WordPress, set up wp-config.php and run the whole installation procedure, without even having to open a browser.

    $ wp core download --locale=en_GB
    $ wp core config --dbname=mynewsite --dbuser=root
    $ wp core install --url=mynewsite.dev --title="My Site"Code language: JavaScript (javascript)

    Using the third-party wp login command you could even generate a link that automatically logs you in afterwards.

    Perform all Updates on a Site

    This is similar to the example I gave earlier, but now just for a single site:

    $ wp core update
    $ wp plugin update --all
    $ wp theme update --all

    Regenerate Thumbnails

    Another useful command that comes in handy when changing image sizes is wp media regenerate. There’s no need to install a plugin for this and performing this tedious task in the browser. With WP-CLI you can do it all on the command line and let it run in the background. You can even automate it using a cron job.

    $ wp media regenerate --yes --only-missing

    Site Migrations

    WP-CLI is also an ideal tool for site migrations. You can not only export and import your database for an easy backup of your site. You can also use it when changing your domain name or when going from HTTP to HTTPS.

    $ wp db export
    $ wp db import
    $ wp search-replace 'https://old.blog' 'https://new.blog' --skip-columns=guidCode language: JavaScript (javascript)

    Evaluate Code

    This one is more for developers I guess. wp eval allows you to quickly execute some PHP code, which is very useful for debugging. It allows you to quickly find out the value of a variable or run a function. This is especially useful if there is no WP-CLI command for a certain feature yet

    $ wp eval 'echo WP_CONTENT_DIR;'
    /var/www/wordpress/wp-contentCode language: JavaScript (javascript)

    Flush Rewrite Rules

    When you installed some plugins that messed with your permalinks in some way, and now your URLs aren’t working properly anymore, you can simply run wp rewrite flush to clean up and regenerate the permalinks.

    Configuration Files

    Many aspects of WP-CLI can be tweaked through configuration files. WP-CLI looks for these in various locations. This way you can have a global configuration file, as well as per-project configurations. The lookup order is like this:

    1. wp-cli.local.yml
    2. wp-cli.yml
    3. ~/.wp-cli/config.yml

    A simple configuration file could look like this. Here you just tell WP-CLI where your site is located and what the site URL is:

    path: wp-core
    url: https://pascalbirchler.com
    user: Pascal
    disabled_commands:
      - plugin installCode language: JavaScript (javascript)

    I think it’s really cool that it allows you to disable certain commands. This way you can prevent users from running commands that could potentially break your site if not executed with care.

    Aliases

    Configuration files can also contain defaults for any subcommand, as well as aliases to one or more WordPress installs. This way you can run WP-CLI on a server without having to memorize credentials and log into that server first.

    Aliases can even reference other aliases to create alias groups. Using just one alias you can simultaneously run a command against multiple sites on different servers.

    @staging:
      ssh:
      user:
      path:
    @production:
      ssh:
      user:
      path:
    
    @mywebsite:
      - @staging
      - @production

    This way you can use wp @mywebsite <command> to run something across both the staging and production environments of a site.

    Note: WP-CLI automatically creates the @all alias group you’ve seen in previous examples, which allows you to run a command across all your websites.

    Extending WP-CLI

    WP-CLI is very powerful and contains a lot of super helpful commands. However, if in any case these are not enough for you, you can also extend WP-CLI with third-party commands.

    WP-CLI is very modular. All the built-in commands are actually separate packages, and adding more commands just means adding another package to the mix. You could even install a package that overrides one of the built-in commands. Thanks to this modularity, the WordPress community is steadily creating new commands for WP-CLI.

    The commands you need to know:

    • List all installed packages: wp package list
    • Install a new package: wp package install <package>
    • Remove an existing package: wp package uninstall <package>
    • Update installed packages: wp package update

    In these examples, <package> refers to name of the GitHub repository the package is located at, or a fully-qualified URL.

    After adding a new package, you’re all set and you can immediately run it. No need to restart your computer or anything.

    Magic Login Links

    As mentioned above, the wp login command allows you to log into WordPress with secure passwordless magic links. These can be generated on the fly or even sent via email.

    $ wp package install aaemnnosttv/wp-cli-login-command
    $ wp login create <user>Code language: HTML, XML (xml)

    Vulnerability Scanner

    10up created a command that checks your installed plugins and themes against the WordPress vulnerability database. This way you can quickly check whether your site is potentially at risk for getting hacked.

    $ wp package install 10up/wp-vulnerability-scanner
    $ wp vuln status

    Image Optimization

    Another handy command I recently found allows you to do lossless image optimizations on all your media files in WordPress.

    Image optimization is resource and time intensive just by its nature. It makes sense to run this on the server at a convenient time. Plus, this way you don’t have to install the same image optimization plugin on all of your WordPress sites.

    $ wp package install typisttech/image-optimize-command
    $ wp media regenerate
    $ wp image-optimize batch

    Write Your Own Custom Command

    Of course, you can also create your very own WP-CLI command … using nothing less than WP-CLI itself!

    WP-CLI ships with commands to scaffold new plugins, themes, Gutenberg blocks, and even WP-CLI commands. All scaffolded commands will contain proper documentation, some initial boilerplate code, and even the complete testing setup.

    What’s Currently Missing?

    Need some inspiration for your first WP-CLI command? I recommend checking out the project’s ideas repository where people can suggest new features. Currently high on the list are commands for the built-in privacy management tools, as well as commands related to Gutenberg.

    Once you have found something you want to create a command for, you can use the powerful wp scaffold package command to bootstrap your new package. Yes, that’s a command to create another command! 🤯

    Note: The scaffold command is not yet fully updated for the new WP-CLI 2.0 infrastructure, so it currently is also worth checking out other existing commands like wp maintenance-mode to see how they’re constructed.

    Further Reading

    Wanna learn more about WP-CLI? I recommend checking out the project’s blog and handbook on make.wordpress.org/cli. There’s also a #cli Slack channel where you can ask questions and contribute back to WP-CLI. And of course, all the code of WP-CLI can be found on GitHub.


    Many thanks to Alain Schlesser for maintaining WP-CLI and striving to make it easier for people to use WP-CLI. His excellent presentation at WordCamp Berlin 2017 served as an inspiration for this post.

    Thanks to Alain and John Blackbourn for proofreading this post and giving valuable feedback.

  • CMS Security Summit

    A couple of weeks ago, I had the opportunity to attend the CMS Security Summit in Chicago. For this event, Google brought together content management systems, security researchers, and hosting providers to talk about security. WordPress, powering a third of the web, was represented by security team lead Barry.

    As a WordPress core committer and Noogler, this was a very insightful event for me. All the discussions with the attendees were super valuable—just the temperatures were a bit cold for my taste (-50 degrees, yikes!). If you wanna learn more about the event, some people published recap blog posts:

    I think the key takeaway is that most projects are dealing with the same issues and that they all benefit from working more closely together. Some examples include:

    • Automatic updates and package signing
    • Code reviews and static analysis
    • Collaborating with security researchers

    For this blog post, I want to dig a bit deeper on code analysis and what it means for WordPress.

    Static Code Analysis for WordPress Plugins

    WordPress is only as strong and secure as its ecosystem. Part of that ecosystem are the 60,000 plugins and themes that are available for download on WordPress.org. It’s impossible to manually scan all these projects for potential security vulnerabilities.

    At the summit, the RIPS code analysis platform was mentioned a few times. It’s a paid solution, but they also work together with open source projects. For example, Joomla uses RIPS to continuously scan their code base. At the moment WordPress doesn’t use that tool, but for RIPS the platform is of interest either way. The just recently demonstrated this via their WordPress Security Advent Calendar.

    Another example is their security risk analysis platform, CodeRisk. According to the website, CodeRisk “rates the security risk of WordPress plugins by analyzing them with the RIPS static code analyzer and combining the results into an easy to understand value”.

    I’m not sure how useful a plain number is, but I guess it works well for marketing. Anyway, I wanted to give the site a try to find out if there’s more behind that. It turns out that as a plugin developer you get free access to their static code analysis tool to scan all your plugins for security vulnerabilities.

    This is a really nice gesture! I wondered if other people use that feature too, so I posted a quick poll on Twitter:

    In that poll nobody said they use the CodeRisk platform, which was a bit of a surprise to me. Perhaps it’s not clear enough what the site does, or it’s just too complicated to set things up.

    Tools like this demonstrate that there are lots of possibilities to improve security in the wider WordPress ecosystem and in the overall CMS landscape. I’m curious to see how this area evolves in the next few years.

  • Internationalization in WordPress 5.0

    Internationalization in WordPress 5.0

    In my previous blog post I explained the importance of the text domain in WordPress internationalization. Today, I want to have a look at the bigger picture of the (new) internationalization features in WordPress 5.0 and beyond. This includes, but is not limited to, enhanced JavaScript internationalization.

    If you’re building a WordPress plugin or theme and want to make sure it can be fully localized, this post is for you.

    WordPress JavaScript Internationalization

    WordPress 5.0 shipped with a completely new editing experience called Gutenberg. This new editor is mainly written in JavaScript, which means a lot of internationalization now happens client-side instead of on the server. Although WordPress core has previously used functions like wp_localize_script() to make some of its more dynamic UIs translatable, a more robust solution was needed for such a complex addition like Gutenberg.

    JavaScript Localization Functions

    New JavaScript I18N Support in WordPress 5.0 brings the same capabilities to JavaScript development for WordPress that we’re already used to from PHP. This starts with a new wp-i18n JavaScript package that provides localization functions like __(), _x(), _n()_nx(), and even sprintf(). These functions mirror their PHP equivalents and can be used in the same ways.

    To use this package, you need to add the wp-i18n script as a dependency when registering your JavaScript:

    wp_register_script(
    	'my-plugin-script',
    	plugins_url( 'js/my-script.js', __FILE__ ),
    	array( 'wp-i18n' ),
    	'0.0.1'
    );Code language: PHP (php)

    After that, the localization functions are available under the wp.i18n global variable in JavaScript. You can use them like this:

    const { __, _x, _n, sprintf } = wp.i18n;
    
    __( 'Hello World', 'my-plugin' );
    
    _x( 'Glitter Box', 'block name', 'my-plugin' );
    
    // Get the comment count from somewhere else in our script.
    const commentCount = wp.data.select( 'my/data-store' ).getCommentCount();
    
    /* translators: %s: number of comments */
    sprintf( _n( 'There is %s comment', 'There are %s comments', commentCount, 'my-plugin' ), commentCount );Code language: JavaScript (javascript)

    That’s all you need to make your JavaScript components fully localizable.

    If you’re familiar with the PHP translation functions in WordPress core, you’ll notice the absence of something like esc_html() or esc_html__(). These aren’t needed in JavaScript because the browser is already capable of escaping unsafe characters.

    Note: although it’s discouraged to use HTML in translatable strings, sometimes this is necessary, e.g. for adding links (Check out this link to <a href="%s">my website</a>.). Right now, it’s not easily possible to do so, at least not without using innerHTML / dangerouslySetInnerHTML. However, this is actively being discussed on GitHub.

    Loading JavaScript Translations

    Keep in mind that just using the __() family of functions isn’t enough for a WordPress plugin or theme to be fully internationalized and localized. We also need to tell WordPress to load the translations for our scripts. This can be achieved by using the new wp_set_script_translations() function introduced in WordPress 5.0.

    That function takes three arguments: the registered script handle (my-plugin-script in the previous example), the text domain (my-plugin), and optionally a path to the directory containing translation files. The latter is only needed if your plugin or theme is not hosted on WordPress.org, which provides these translation files automatically.

    Note: If you’re registering multiple scripts that all use wp.i18n, you have to call wp_set_script_translations for each one of them.

    wp_register_script(
    	'my-plugin-script',
    	plugins_url( 'js/my-script.js', __FILE__ ),
    	array( 'wp-i18n' ),
    	'0.0.1'
    );
    
    wp_register_script(
    	'my-awesome-block',
    	plugins_url( 'js/my-block.js', __FILE__ ),
    	array( 'wp-i18n' ),
    	'0.0.1'
    );
    
    wp_set_script_translations( 'my-plugin-script', 'my-plugin' );
    wp_set_script_translations( 'my-awesome-block', 'my-plugin' );Code language: PHP (php)

    The reason for this is performance. Translations are only loaded when your script is actually enqueued. If that is the case, WordPress loads the translation files into memory and provides them to wp.i18n via inline JavaScript. That means WordPress requires one translation file per script handle with each file only containing strings relevant for that script.

    Imagine writing a JavaScript-heavy WordPress plugin with lots of different packages that can also be used independently of each other. You don’t want to load all translations if you just need the ones for a single package.

    JavaScript Translation Files

    We have now covered loading the JavaScript translation files, but what exactly is so special about them? Well, this time we’re not dealing with PO or MO files, but with JSON files instead. Since JSON can be read very easily in JavaScript, it’s a convenient format to store translations in.

    Also, the  wp-i18n package uses a library under the hood that is largely compatible with the Jed JavaScript gettext library, which requires Jed-style JSON translation data. As mentioned in the previous section, WordPress.org provides these translation files automatically. But if you want to ship your own, you need to create such JSON files yourself.

    A very simple Jed-style JSON translation file looks like this:

    {
       "domain": "messages",
       "locale_data": {
          "messages": {
             "": {
                "domain": "messages",
                "plural_forms": "nplurals=2; plural=(n != 1);",
                "lang": "de_DE"
             },
             "Source": [
                "Quelle"
             ],
             "Enter the information for this recommendation.": [
                "Gib die Informationen zu dieser Empfehlung ein."
             ],
             "%s comment": [
                "%s Kommentar",
                "%s Kommentare"
             ],
             "block name\u0004Recommendation": [
                "Empfehlung"
             ]
          }
       }
    }Code language: JSON / JSON with Comments (json)

    If you’re familiar with PO translation files already, this format contains similar information like information about the locale (de_DE) and its plural forms. All the strings are in the messages object, with the originals as keys, and the translations being the value. If a string has an additional context, it is prepended by it, with \u0004 acting as a delimiter.

    Note: An important detail here is the text domain, which right now needs to be messages and not the one you actually use in the code. There’s a WordPress Trac ticket for this though, so it might be supported in the future.

    JavaScript Translation File Names

    PO and MO translation files in WordPress usually have the format $textdomain-$locale.po, e.g. my-plugin-de_DE.po. For the JSON files things are a bit different now.

    You might remember that we need to pass the script handle name to wp_set_script_translations(). This handle needs to be in the file name as well, in the form $textdomain-$locale-$handle.json.

    So for our my-plugin-script script handle, the translation file name needs to be my-plugin-de_DE-my-plugin-script.json.

    For technical reasons, WordPress also looks for files in the form $textdomain-$locale-$md5.json, where $md5 is the MD5 hash of the JavaScript file name including the extension. In the earlier example, my-plugin-script points to js/my-script.js. The MD5 hash of my-script.js is 537607a1a008da40abcd98432295d39e. So the alternative file name for our translation file is my-plugin-de_DE-537607a1a008da40abcd98432295d39e.json.

    Generating JavaScript Translation Files

    Since WordPress requires one translation file per script handle, with each file only containing strings relevant for that script, this quickly means dealing with plenty of JSON files. Luckily, there’s no need to write these by hand.

    The recommended way to generate the JSON translation files is by using WP-CLI. The latest version, WP-CLI 2.1.0, provides a dedicated wp i18n make-json command for this.

    The wp i18n make-json command extracts all the JavaScript strings from your regular PO translation files and puts them into individual JSON files.

    Note: WP-CLI 2.1.0 been released on December 18. Make sure you’re using the latest version by running wp cli update. You can check your current version using wp cli version.

    Let’s say in your plugin folder my-plugin you have three source files: my-plugin.php, js/my-script.js and js/my-block.js. You use WP-CLI to extract the strings and generate the translation catalogue (POT) like this:

    wp i18n make-pot my-plugin my-plugin/languages/my-plugin.pot

    From there you can translate your plugin as usual and create the needed PO and MO files. Let’s say we add a German translation to my-plugin/languages/my-plugin-de_DE.po first. After that, you can simply run wp i18n make-json my-plugin/languages to generate the JavaScript translation files. The result will be as follows:

    • A new my-plugin/languages/my-plugin-de_DE-537607a1a008da40abcd98432295d39e.json file contains the translations for my-script.js.
    • A new my-plugin/languages/my-plugin-de_DE-dad939d0db25804f91959baeec56ea8a.json file contains the translations for my-block.js.
    • The my-plugin/languages/my-plugin-de_DE.po now only contains the translations that are needed on the server side.

    If you don’t want to modify the PO file, pass the --no-purge argument to the WP-CLI command, as explained in the documentation.

    Note: There are a few known issues in these WP-CLI commands with some edge cases. We’re continuously working on improving the tooling as we learn about how people use them.

    Tooling

    These new processes introduced with WordPress 5.0 and Gutenberg can feel a bit complex at the beginning. To make lives easier, I want to share some tips and tricks for your project’s configuration.

    Webpack Configuration

    If you reference the global variables like wp.i18n in your project everywhere, you don’t benefit from your code editor’s power to show things like type hints. To change that, I recommend installing the @wordpress/i18n package as a (development) dependency using npm / yarn. After that, you can use import { __ } from '@wordpress/i18n; throughout your project.

    Normally, this would make Webpack bundle the library with your code. Since WordPress already exposes the library via the wp.i18n global, there’s no need for code duplication. To prevent this, add the following to your Webpack configuration:

    externals: {
        '@wordpress/i18n': { this: [ 'wp', 'i18n' ] }
    }Code language: JavaScript (javascript)

    This way you’ll benefit from both your IDE’s powers as well as the already available wp.i18n global. Just make sure you add wp-i18n as a dependency when calling wp_register_script().

    Babel Integration

    In the previous section I mentioned using wp i18n make-pot to create the necessary translation catalogue from which you can create the actual localizations. Depending on your developer workflow, you might want to look into using a build tool for Babel called @wordpress/babel-plugin-makepot to create the POT file. The latter approach integrates with Babel to extract the I18N methods.

    To do so, run npm install --save-dev @wordpress/babel-plugin-makepot and add the following plugin to your Babel configuration:

    [
        '@wordpress/babel-plugin-makepot',
        {
            output: 'languages/my-plugin-js.pot',
        },
    ]Code language: JavaScript (javascript)

    Note: You still want to create a POT file for the rest of your PHP files, not just your JavaScript files. You can still do that using WP-CLI. Just skip the JavaScript string extraction and merge the resulting POT files like this:

    wp i18n make-pot my-plugin my-plugin/languages/my-plugin.pot --skip-js --merge=my-plugin/languages/my-plugin-js.pot

    In this scenario, languages/my-plugin-js.pot would only be of temporary nature, so you could remove it again afterwards.

    Available Hooks and Filters

    WordPress provides filters like load_textdomain and gettext to allow overriding the path to translation files or individual translations.

    In WordPress 5.0.2 we added the following filters to allow filtering the behavior of  wp_set_script_translations() so you can do the same for JavaScript translations. The following filters are available:

    • pre_load_script_translations: Pre-filters script translations for the given file, script handle and text domain. This way you can short-circuit the script translation logic to return your own translations.
    • load_script_translation_file: Filters the file path for loading script translations for the given script handle and text domain..
    • load_script_translations: Filters script translations for the given file, script handle and text domain. This way you can override translations after they have been loaded from the translation file.

    In addition to that, pull request #12517 to the Gutenberg project aims to add i18n.gettext, i18n.gettext_with_context, i18n.ngettext, and i18n.ngettext_with_context filters to the @wordpress/i18n package. To override an individual translation, you could use them like this:

    wp.hooks.addFilter(
        'i18n.gettext',
        'myplugin/filter-gettext',
        function( translation, text, domain ) {
            if ( 'Source' === text && 'foo-domain' === domain ) {
                return 'New translation';
            }
    
            return translation;
        }
    );Code language: JavaScript (javascript)

    WordPress PHP Internationalization

    With so many mentions of JavaScript in this post, you might be wondering if we also changed something on the PHP side of things. The answer to this is: no.

    However, now is a good time to do some sort of I18N spring cleaning for your plugin or theme. Here is some helpful information for that:

    • Make sure Text Domain is set in your main plugin file / theme stylesheet and that you use that very same text domain throughout the project.
    • If your WordPress plugin or theme is hosted on WordPress.org and requires WordPress 4.6 or higher (indicated via the Tested up to header in the readme), you don’t need to call load_plugin_textdomain() in it.
    • You can run wp i18n make-pot --debug to see which of your translatable strings should be improved.

    Further Reading


    Thanks to Omar Reiss, Gary Jones, and Dominik Schilling for their feedback and proofreading of this post.