Integration

Automating Visual Testing with Playwright, Argos and GitHub Actions

January 24, 2023

|

Jeremy Sfez

Visual testing allows developers to identify and fix visual bugs in applications before they become a problem. By automating visual testing, developers can ensure that their web applications look and function as intended when they make changes and updates.

In this guide, we will show how to automate visual testing with Playwright, Argos, and GitHub Actions. The goal is to be notified of visual changes on each pull-request directly within GitHub, and fix visual bugs before they make it into production.

Install the Argos GitHub App

To get started with Argos, you'll first need to install the Argos GitHub App. This will allow Argos to access your repositories, add test statuses on pull-request, and notify you when visual changes are detected.

To install Argos app, go to the GitHub Marketplace, subscribe to an Argos plan and complete the installation.

Playwright and Argos setup

Playwright is a powerful testing framework that provides a great developer experience and good out-of-the-box features for end-to-end testing. To install Playwright, execute the following command in your terminal and answer the questions in the prompt:

npm init playwright@latest

To use Playwright with Argos, you will also have to install the following packages:

npm install --save-dev @argos-ci/cli  @argos-ci/playwright

• @argos-ci/cli: easiest and fastest way to upload screenshots to the Argos.
• @argos-ci/playwright: improve screenshot stability and expose CSS to hide content.

Configure Argos Reporter

The Argos reporter seamlessly uploads test screenshots to Argos in real-time.

Install reporter in your Playwright config:

import { defineConfig } from "@playwright/test";

// "list" is the default reporter, if you use any other, don't forget to add it back
const defaultReporters: PlaywrightTestConfig["reporter"] = [["list"]];

export default defineConfig({
  // ... other configuration

  // Add Argos reporter only when it runs on CI
  reporter: process.env.CI
    ? [...defaultReporters, ["@argos-ci/playwright/reporter"]]
    : defaultReporters,
});

Capture the screenshots

Once Playwright and Argos app are installed, you can create test files to capture screenshots of your application. Use the argosScreenshot function provided by @argos-ci/playwright insure stable screenshots.

For example, the following test file will take a screenshot of your homepage:

// tests/homepage.spec.ts
import { argosScreenshot } from "@argos-ci/playwright";
import { test } from "@playwright/test";

test.describe("Homepage", () => {
  test("take screenshot", async ({ page }) => {
    await page.goto("http://localhost:3000/");
    await argosScreenshot(page, "homepage");
  });
});

To run the test, use the playwright test command:

npx playwright test

If everything is set up correctly, the test should pass, on the CI screenshots will be uploaded to Argos.

Integrate with GitHub Actions

With the screenshots captured, you can now integrate the Visual Testing into your GitHub Actions Continuous Integration. To do this, you have to create a new GitHub Actions workflow file that runs Playwright tests and uploads screenshots to Argos.

Create a new file called .github/workflows/main.yml in the root directory of your repository with the following content:

# file: .github/workflows/main.yml
name: CI
on:
  push:
    branches: [main]
  pull_request:
    branches: [main]
jobs:
  e2e:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout repository
        uses: actions/checkout@v4

      - name: Setup Node.js
        uses: actions/setup-node@v4

      - name: Install dependencies
        run: npm ci

      - name: Install Playwright browsers
        run: npm exec playwright install --with-deps

      - name: Run tests
        run: npm exec playwright test

This workflow defines a job that runs on pull requests to the main branch. The job does the following:

• Checkout the code from GitHub
• Setup Node.js
• Install dependencies
• Install Playwright browsers
• Run the tests with Playwright and upload screenshots to Argos

Once you have added the workflow file, you can commit and push the changes to your repository. GitHub will automatically start running the workflow on each pull request.

Monitor Visual Changes

You can now start monitoring visual changes in your web application. Argos will detect and report any visual changes between your pull request branch and your main branch.

GitHub check status By clicking on "Details" link, you can review the results of the comparison in Argos.

Argos example build

Read this documentation if you need more information about visual testing workflow.

Bonus: Capturing Screenshots on Test Failures

Playwright's recording options facilitate the automated capture of screenshots upon test failures. Notably, these captured screenshots are then automatically uploaded to Argos.

// playwright.config.{js,ts}
import { defineConfig } from "@playwright/test";

export default defineConfig({
  use: {
    // Setting to capture screenshot only when a test fails.
    screenshot: "only-on-failure",
  },
});

Conclusion

In this guide, we have shown you how to automate visual testing with Playwright, Argos, and GitHub Actions. By integrating visual testing into your CI workflow, you can catch and fix visual bugs before they make it into production. With Argos, you can easily configure baseline, monitor visual changes, and collaborate with your team to fix issues. By following the steps outlined in this guide, you can be sure that your web application looks and functions as intended, even as you make changes and updates.

---

We hope this post has been helpful. If you're interested in learning about how Argos can help you improve the quality of your web applications, take a look at our documentation. And if you're ready to start automating your visual testing, feel free to sign up for a free trial and see how Argos can help you catch bugs.

If you need more information about Visual Testing, it would be a pleasure to help you on our Discord community.