Automating Web Form Submissions with Playwright and JavaScript

Whether you need to gather data from multiple sources, test form functionality, or automate repetitive form submissions, learning how to automate web forms using Playwright and JavaScript can simplify these tasks. In this article, we’ll explore the benefits of automating forms and guide you through the process of automating form submissions with Playwright.

Why Automate Forms?

Before diving into the technical details, let’s consider a few scenarios where form automation can be beneficial:

Data Collection

Imagine needing to gather data from various online marketplaces, including product details and prices. The traditional manual approach is time-consuming and error-prone, requiring navigation to each product page and form filling. However, with automation tools like Playwright, you can automatically navigate these marketplaces, extract the necessary information, and populate a database, saving time and reducing errors.

Automating forms is easy with Playwright: follow our guide to learn why and how

Form Testing

When developing web applications, testing the functionality and behavior of forms is crucial. Manually testing each scenario can be repetitive and inefficient. By automating testing with form submissions, input validation, and error handling, you ensure the smooth operation of your applications.

11 best automated browser testing tools for developers

Repetitive Tasks

Many online services require filling in forms repeatedly, such as submitting support requests, job applications, or entering sweepstakes. Automating these repetitive tasks saves time and effort. Whether you’re an individual who wants to repeatedly enter your details or a company conducting web scraping projects, automation can streamline the process.

By the end of this article, you’ll have a solid understanding of how to automate forms using Playwright. You can then apply the techniques we’ll learn here to your own projects. So, let’s get coding!

What You’ll Need to Start Automating Forms

Before getting started, make sure you have the following prerequisites in place:

  1. Basic understanding of HTML forms and browser DevTools: It’s helpful to have a fundamental understanding of HTML forms and their structure, as well as being able to use your browser DevTools to inspect elements on a webpage.

  2. JavaScript knowledge: Since we’ll be using JavaScript and Node.js as the programming language for our project, it’s essential to have at least a basic understanding of concepts such as variables, functions, and asynchronous programming in JavaScript.

  3. Node.js installation: Ensure that Node.js is installed on your local machine. You can download and install the latest version of Node.js from the official Node.js website. Node.js allows us to run JavaScript code outside of the browser environment.

You don’t need to have any prior experience with Playwright, but it’s always beneficial to find out more about it before you start. ProgramMatek provides comprehensive information and resources on Playwright.

Setting Up Your Form Automation Project

Let’s begin by setting up our project for automating web forms:

Create a New Project Directory

Choose a suitable location on your computer and create a new directory for your project. You can name it anything you like, for example, “form-automation-project”. Open your terminal or command prompt, navigate to the desired location, and use the following command to create the directory:

mkdir form-automation-project

Initialize a New Node.js Project

Change into the newly created project directory and initialize a new Node.js project by running the following commands:

cd form-automation-project
npm init -y

This command generates a “package.json” file that will keep track of the project’s dependencies and configuration.

See also  JavaScript Classes in NYC

Install Playwright

With the project initialized, we can now install the Playwright library. In your terminal or command prompt, run the following command:

npm install playwright

Update “package.json” to Use Module Syntax

To enable the use of JavaScript module syntax when building our project, add "type": "module" to your “package.json” file. This syntax allows us to take advantage of the ES module system, which provides a more standardized and modern approach to organizing and importing JavaScript code.

Launching the Browser and Navigating to the Form

Now that our project is set up, we can begin automating the form by launching a browser instance and navigating to the target webpage containing the form.

Create a New JavaScript File

In the project directory, create a new JavaScript file to hold the logic for our bot. You can name it “bot.js” or choose any other suitable name.

Import Playwright Modules

Open the newly created “bot.js” file in your preferred code editor. At the top of the file, import the necessary Playwright modules using the following code:

import { chromium } from 'playwright';

This imports the “chromium” module from Playwright, which allows us to automate Chromium-based browsers like Google Chrome.

Launch a Browser Instance

Below the import statement, add the following code to launch a new browser instance:

async function submitForm() {
  const browser = await chromium.launch({ headless: false });
}

This code uses an asynchronous function to launch a browser instance with Playwright’s “launch()” method. The “browser” variable will hold the browser instance for further interactions. Note that we are also passing the parameter “headless: false” so we can see our bot in action.

Navigate to the Target Web Page

To navigate to the web page containing the form, add the following code after launching the browser:

async function submitForm() {
  const browser = await chromium.launch({ headless: false });
  const context = await browser.newContext();
  const page = await context.newPage();
  await page.goto('https://www.example.com/form'); // Replace with the actual URL of the form
}

In this code, we create a new context and a new page within that context. Then, we use the “goto()” method to navigate to the URL of the webpage containing the form. Make sure to replace 'https://www.example.com/form' with the actual URL of the form you want to automate.

Locating and Filling Form Fields

Now that we have successfully navigated to the webpage containing the form, we can proceed to locate the form fields and fill them in with our desired values.

Locate Form Fields

To interact with form fields, we need to locate them on the web page. Inspect the HTML structure of the form to identify the attributes or selectors we can use to locate each field. Common attributes include “name”, “id”, and “class”. For example, let’s assume we have an input field with the name attribute “firstName”. We can locate it using Playwright’s “page.locator()” method:

const firstNameField = page.locator('input[name="firstName"]');

Repeat this approach to target all the form fields you need to fill in. For example, these are the selectors for each of the fields present in our dummy Google form:

const firstNameField = page.locator('input[aria-labelledby="i1"]');
const emailField = page.locator('input[aria-labelledby="i5"]');
const addressField = page.locator('textarea[aria-labelledby="i9"]');
const phoneField = page.locator('input[aria-labelledby="i13"]');
const commentsField = page.locator('textarea[aria-labelledby="i17"]');

Fill Form Fields

Once we have located a form field, we can fill it with the desired value using Playwright’s “fill()” method. Add the following code after locating the field:

await firstNameField.fill('John');

Replace 'John' with the value you want to fill in the field.

Handle Different Field Types

Form fields can vary in type, such as text inputs, checkboxes, radio buttons, dropdown menus, and file upload fields. Use Playwright’s appropriate methods to interact with each field type. For example, to check a checkbox, use the “check()” method:

const checkboxField = page.locator('input[name="acceptTerms"]');
await checkboxField.check();

Remember to adjust the code based on the specific field types and actions you want to perform.

For instance, let’s fill in the details for a fictitious “John” in our dummy form:

// Fill the form fields
await firstNameField.fill('John');
await emailField.fill('john@gmail.com');
await addressField.fill("John's Street");
await phoneField.fill('11111111');
await commentsField.fill('This form was submitted automatically.');
await checkboxField.check();

Submit the Form

Once we have filled in all the necessary form fields, it’s time to submit our form using Playwright’s “submit()” method. Locate the submit button and add the following code:

const submitButton = page.locator('button[type="submit"]');
await submitButton.click();

Close the Browser

After we successfully submit our form, it’s important to explicitly tell Playwright to close the browser instance; otherwise, it would continue to stay open even after the form is submitted.

// Close the browser
await browser.close();

Final Code for Automating the Form

And here is what the final code for automating our dummy form looks like:

import { chromium } from 'playwright';

async function submitForm() {
  const browser = await chromium.launch({ headless: false });
  const context = await browser.newContext();
  const page = await context.newPage();

  await page.goto('https://forms.gle/7rhchFiZF2faMQxz7'); // Replace with the actual URL of the form

  // Wait for form element to be visible on the page before proceeding
  await page.waitForSelector('div.lRwqcd > div', { state: 'visible' });

  // Select the form fields we want to target
  const firstNameField = page.locator('input[aria-labelledby="i1"]');
  const emailField = page.locator('input[aria-labelledby="i5"]');
  const addressField = page.locator('textarea[aria-labelledby="i9"]');
  const phoneField = page.locator('input[aria-labelledby="i13"]');
  const commentsField = page.locator('textarea[aria-labelledby="i17"]');
  const checkboxField = page.locator('div#i26');
  const submitButton = page.locator('div.lRwqcd > div');

  // Fill the form fields
  await firstNameField.fill('John');
  await emailField.fill('john@gmail.com');
  await addressField.fill("John's Street");
  await phoneField.fill('11111111');
  await commentsField.fill('This form was submitted automatically.');
  await checkboxField.check();

  // Submit the form
  await submitButton.click();

  // Close the browser
  await browser.close();
}

submitForm();

Note that in the final version of our bot, I added an extra line of code to explicitly wait for a specific element on the page to load before proceeding with the rest of the code.

await page.waitForSelector('div.lRwqcd > div', { state: 'visible' });

This step is not always necessary, but it is possible for the bot to attempt to “act” before the page is fully loaded, leading to an error due to the bot’s inability to interact with the target element.

See also  Workflow Builder in JavaScript: Simplify Your Workflow Visualization

Automating Multiple Form Submissions

In some cases, you may need to automate multiple form submissions, such as when you have a batch of data to process or want to simulate user interactions. Here’s how you can automate multiple form submissions using Playwright:

Encapsulate Form Submission Logic

To automate the submission of multiple forms, it’s beneficial to encapsulate the form submission logic into a reusable function. We have already done this in the previous section, so we can continue using the form function we have created.

async function submitForm() {
  // Locate and fill form fields
  // Submit the form
}

Use a Loop

Determine the number of times you want to submit the form and use a loop to automate the process. For example, if we want to submit the form five times, we can use a for loop as follows:

for (let i = 0; i < 5; i++) {
  await submitForm();
}

Adjust the loop conditions and the number of iterations based on your requirements.

Optional: Add Delays Between Submissions

In some cases, it may be necessary to introduce delays between form submissions to mimic user behavior or account for server response times. We can use Playwright’s waitForTimeout method to add delays between submissions within the loop.

For example, we could add a 2-second delay right before we tell Playwright to close:

// Wait before closing the browser
await page.waitForTimeout(2000); // Wait for 2 seconds before the next submission

// Close the browser
await browser.close();

Adjust the delay duration as needed.

By encapsulating the form submission logic in a function and using a loop, we can automate as many form submissions as we want!

Error Handling and Debugging

When automating forms, it’s essential to handle potential errors and have mechanisms in place for debugging. Here are some techniques we can use to do that:

Try-Catch Blocks

We can wrap our code inside try-catch blocks to catch and handle any errors that may occur during form automation. This allows us to gracefully handle exceptions and prevent our automation script from crashing.

try {
  // Form automation code
} catch (error) {
  console.error('An error occurred:', error);
}

By logging the error to the console or reporting it in some other way, you can quickly identify and diagnose issues.

Logging and Debugging Statements

Use console.log() statements strategically throughout the code to output useful information. These statements can help us track the execution flow, inspect variable values, and identify potential issues.

console.log('Filling in the first name field...');
// Your code for filling in the first name field
console.log('First name field filled successfully.');

By logging key steps or variables, you can gain insights into what’s happening during the automation process.

See also  Bot Detection with JavaScript

Taking Screenshots

Playwright allows us to take screenshots of the browser at any point during the automation process. Capture screenshots to visualize the state of the page and potentially identify issues or unexpected behavior.

await page.screenshot({ path: 'screenshot.png' });

Save the screenshot to a file for later examination.

Inspecting Network Traffic

Playwright provides tools for inspecting network traffic, which can be useful for debugging. We can intercept network requests, analyze responses, and verify the data being sent and received.

page.on('response', (response) => {
  console.log('Received response:', response.url());
});

Use the page.on() method to listen for specific events related to network traffic.

Code Including Error Handling and Debugging

Now, let’s update our previous code to include the error handling and debugging techniques we discussed:

import { chromium } from 'playwright';

async function submitForm() {
  const browser = await chromium.launch({ headless: false });
  const context = await browser.newContext();
  const page = await context.newPage();

  await page.goto('https://forms.gle/7rhchFiZF2faMQxz7'); // Replace with the actual URL of the form

  // Wait for form element to be visible on the page before proceeding
  await page.waitForSelector('div.lRwqcd > div', { state: 'visible' });

  try {
    // Select the form fields we want to target
    const firstNameField = page.locator('input[aria-labelledby="i1"]');
    const emailField = page.locator('input[aria-labelledby="i5"]');
    const addressField = page.locator('textarea[aria-labelledby="i9"]');
    const phoneField = page.locator('input[aria-labelledby="i13"]');
    const commentsField = page.locator('textarea[aria-labelledby="i17"]');
    const checkboxField = page.locator('div#i26');
    const submitButton = page.locator('div.lRwqcd > div');

    // Fill the form fields
    console.log('Filling in the first name field...');
    await firstNameField.fill('John');
    console.log('Filling in the email field...');
    await emailField.fill('john@gmail.com');
    console.log('Filling in the address field...');
    await addressField.fill("John's Street");
    console.log('Filling in the phone number field...');
    await phoneField.fill('11111111');
    console.log('Filling in the comments field...');
    await commentsField.fill('This form was submitted automatically.');
    console.log('Checking the box ✅...');
    await checkboxField.check();

    // Take a screenshot of the completed form
    await page.screenshot({ path: 'screenshot.png' });

    // Submit the form
    await submitButton.click();

    // Wait before closing the browser
    await page.waitForTimeout(2000); // Wait for 2 seconds before the next submission

    // Close the browser
    await browser.close();
  } catch (error) {
    console.error('Oops, something went wrong:', error);
  }
}

for (let i = 0; i < 5; i++) {
  await submitForm();
}

Note that in the final version of our bot, I added an extra line of code to explicitly wait for a specific element on the page to load before proceeding with the rest of the code.

await page.waitForSelector('div.lRwqcd > div', { state: 'visible' });

This step is not always necessary, but it is possible for the bot to attempt to “act” before the page is fully loaded, leading to an error due to the bot’s inability to interact with the target element.

Advanced Techniques

In this section, we’ll explore some advanced techniques for form automation with Playwright. These techniques can help us handle more complex scenarios and overcome common challenges.

Handling Dynamic Forms

Some forms may have fields that appear or disappear dynamically based on user interactions or other factors. We can handle such forms by employing techniques like waiting for specific elements to appear or disappear using Playwright’s waitForSelector() or waitForFunction() methods.

Working with CAPTCHAs and Anti-bot Measures

Websites often employ CAPTCHAs or other anti-bot measures to prevent automated interactions. Automating forms that include CAPTCHAs can be challenging. Consider using third-party services or libraries specifically designed to bypass CAPTCHAs or explore browser automation techniques like mouse movements or human-like delays to mimic user behavior.

Handling File Uploads

If the form includes file upload fields, we can automate file uploads using Playwright’s setInputFiles() method. Specify the path to the file you want to upload, and Playwright will handle the file selection process for you.

Navigating Between Pages

Sometimes, form automation may require navigating between multiple pages or steps. Use Playwright’s page navigation methods like goto() or click() to move between pages and perform interactions on each page as needed.

Parallelization and Performance Optimization

To improve performance and reduce execution time, we can employ parallelization techniques. For example, we can use multiple browser contexts or instances of Playwright to run form automation in parallel, especially when dealing with a large number of forms or submitting forms with a time-consuming process.

Remember, advanced techniques depend on the specific requirements and challenges of the forms you’re automating. It’s essential to understand the unique aspects of each form and apply the appropriate techniques accordingly.

That’s a Wrap! Now You Know How to Automate Forms with Playwright

In this tutorial, we explored how to automate forms using Playwright and JavaScript. We covered the essential steps to build a form-filling bot and provided explanations and code examples along the way. By now, you should have a solid understanding of automating forms using Playwright!

We also discussed the importance of form automation in various real-world scenarios, including data collection, form testing, and automating repetitive tasks. By automating form submissions, you can save time, reduce errors, and increase efficiency.

Now that you’ve learned how to automate forms using Playwright and JavaScript, feel free to apply these techniques to your own projects and explore further possibilities for automation.

And remember to always respect the terms of service and usage policies of the websites you are automating. Use form automation responsibly and ethically, ensuring that your actions comply with legal and ethical standards. In other words, take Uncle Ben’s advice to heart: “With great power comes great responsibility” 🕷️

Playwright vs. Puppeteer: which is better?