Introduction

Let’s start with a practical example of removing elements from DOM. Imagine scrapping a website with Selenium and taking a screenshot of the page. However, the timed modal window will unannounced pop up when Selenium’s driver spin the browser window. So it is probably a good idea to remove the window before taking a picture in such a case.

When scrapping sites, removing excessive advertisement and disturbing elements is hard enough. But pop-up windows in the wrong moments are an overall user experience nowadays on many sites and blogs.

For example, we want to scrap stock time-series from famous sites like finviz.com. Unfortunately, after the first few seconds on the page, the cookies modal window pop-up and block the stock graph. Therefore, I will need to remove the cookie allowance modal window to take a clear screenshot.

Finviz.com cookie popup window

Let me show you two different ways to remove elements from Selenium’s HTML DOM. One way will be about eliminating parts that load with the page load and can be removed immediately. The second way will require calling WebDriverWait instance, and the way will be used for elements that pop up or load to page letter.

However, both options do not remove the element from the DOM. Instead, they hide it and make it invisible in the Selenium web driver.

Remove elements from Selenium’s DOM without waiting

FirefoxOptions options = new FirefoxOptions();
options.addArguments("-width=1920");
options.addArguments("-height=1080");
FirefoxDriver driver = new FirefoxDriver(options);

String stock = "T"; // AT&T Inc.
driver.get("https://finviz.com/quote.ashx?t=" + stock + "&ty=c&ta=1&p=d");

driver.executeScript("return document.getElementsByClassName('snapshot-table2')[0].remove();");

If we look at the code, it might look straightforward. But I will go through it briefly. Initially, we set options for our Selenium Driver. Then, we pick Firefox as Selenium driver and put options (browser windows dimensions) into the driver itself.

After setting drivers options and instantiating Firefox drivers, we are clear to the call page in Firefox. We call finviz.com stock page for AT&T Inc. company. The page which is loaded is immediately available for changes. If we want to remove any element we do not like, we can call script execution upon the loaded HTML DOM. We call executed script, which will remove the first (0-th) element of the specific class name. It will be the first data table under the stock time-series graph.

Note : If you have better idea for elements removal from Selenium’s HTML DOM, let me know in comments below.

Remove elements from Selenium’s DOM with waiting

FirefoxOptions options = new FirefoxOptions();
options.addArguments("-width=1920");
options.addArguments("-height=1080");
FirefoxDriver driver = new FirefoxDriver(options);

String stock = "T"; // AT&T Inc.
driver.get("https://finviz.com/quote.ashx?t=" + stock + "&ty=c&ta=1&p=d");

WebDriverWait wait = new WebDriverWait(driver, 10);

WebElement cookieWindow = wait.until(ExpectedConditions.visibilityOfElementLocated(By.className("ConsentManager__Overlay-np32r2-0")));
driver.executeScript("return arguments[0].remove();", cookieWindow);

The code above looks straightforward also. Initially, we set options for our Selenium Driver and picked Firefox as Selenium driver.

After setting drivers options and instantiating Firefox driver, we are clear to the call page in Firefox. We call the finviz.com stock page for AT&T Inc. company and wait for 10 seconds or until HTML DOM does not contain an element with a specific class name. If it does, we will execute the script upon it, which will remove the first (the 0-th) element when it is found. Action means removing the whole pop-up modal window that appeared in the first 10 seconds of finviz.com page duration.

Class name “ConsentManager__Overlay-np32r2-0” is class name of pop-up cookie modal window. However, this class name might and surely will change in the future. So I would not rely on it for 100%. But for now, it serves our purpose.

Conclusion

This article has shown how remove elements from HTML DOM when using automated web-browser framework Selenium. Articles provide two different ways how to remove element on the page. Elements which load with page load can be removed immediately, without calling WebDriverWait instance. However, elements which pop-up or load to page letter needs to be removed with help of WebDriverWait instance.

However, if you look better, code snippets in the article can help you to build your Selenium web-scrapper 😉

Did you find element removal easy? Do you have your trick or know another way how to remove elements from HTML DOM in Selenium? Let us know in the comments below the article. We would like to hear your ideas and stories, and you might help others as well.

This is a small guide explaining how to start with Codeception PHP test framework and how to integrate it with Selenium for automated tests.

What is Codeception and what is good for?

Codeception is PHP framework devoted to test PHP code. Installation is very simple and requires only a few steps. It is actually well known for its easy code readability of test units. Codeception PHP framework is used for acceptance testing. Codeception emphasize usage of the variable name $I, so than many lines of code are very descriptive exp. $I want to do this and that, $I expect to see something. etc.

Codeception helps with creating test suites for 3 types of testing:

• Accaptance
• Unit
• Functional

What is acceptance testing?

Acceptance tests are emulating behavior of a real user visiting website.

Acceptance testing is formal testing with respect to user needs, requirements, and business processes conducted to determine whether a system satisfies the acceptance criteria and to enable the user, customers or other authorized entity to determine whether or not to accept the system. Acceptance testing is also known as user acceptance testing (UAT), end-user testing, operational acceptance testing or field (acceptance) testing.

Source: Wikipedia

First of all I have to recommend to watch first video by Jeffrey Way from EnvatoTuts+ for Codeception series Modern Testing in PHP with Codeception. Although the first video is very well narrated and instructive for beginner, rest of the serie is not and I would not recommend to pay for premium account to watch the rest. Jeffrey expect knowledge of many related topics where you can use Codeception (DB connection, Laravel, …) but do not create simple code snippets for your first test and does not talk much about usage of elemental Codeception functions. Serie is aimed for advanced users.

1. Download

You can download Codeception framework out of the box via composer by CLI command $ composer require "codeception/codeception" .

You can also implement it into already existing project updating a composer.json file placing following:

{
    "require": {
        "codeception/codeception": "^2.2",
        ...
    }
}

For newest version of Codeception please visit and search packagist.org . Over CLI just hit composer command in folder of your project: composer update (or composer install).

2. Create codeception.yml

By executing codecept bootstrap Codeception creates the codeception.yml file and bunch of new files for test suites.

What is test suite?

The test suite is a collection of test cases that are intended to be used to test a software program to show that it has some specified set of behaviours. A test suite often contains detailed instructions or goals for each collection of test cases and information on the system configuration to be used during testing. A group of test cases may also contain prerequisite states or steps, and descriptions of the following tests.

Collections of test cases are sometimes incorrectly termed a test plan, a test script, or even a test scenario.

Source: Wikipedia

3. Create your first test

Create your first test by Codeception command:

codecept generate:cept acceptance FirstTest

Note: Codeception scenario tests are called Cepts.

4. Setting Acceptance test method

Basically there are 2 method which I managed to try to in short time – PHP Browser and Selenium WebDriver:

1. PHP Browser – works on idea of web scrapper. Code does not emulate real browser but just download HTML code and apply test scenarios on it.
2. Selenium WebDriver – works as “artificial user”. PHP test case code emulate real browser and perform actions upon real website in browser.

I would recommended to first start with PHP Browser and than try Selenium WebDriver. PHP Browser can check HTML page contents, click links, fill forms, and submit POST and GET requests. Unfortunately PHP Browser have few drawbacks (you can click only on links with valid urls or form submit buttons,  you can’t fill fields that are not inside a form, you can’t work with JavaScript interactions like modal windows or date-pickers) and for more complex tests is necessary to use Selenium WebDriver.

4.1. PHP Browser

For setting up PHP Browser just change acceptance.suite.yml to:

class_name: AcceptanceTester
modules:
    enabled:
        - PhpBrowser:
            url: "http://localhost/app"          
        - \Helper\Acceptance

4.2. Selenium WebDriver

For setting Selenium WebDriver you need to make 3 steps.

1) Change acceptance.suite.yml to:

class_name: AcceptanceTester
modules:
    enabled:
        - WebDriver:
            url: "http://localhost:8000/app"
            browser: firefox            
        - \Helper\Acceptance

2) In your .bash_profile you must also set GeckoDriver (drivers emulating Firefox) which you can download from mozilla/geckodriver GitHub page.

## Selenium GeckoDriver
export PATH=$PATH:/PATH_TO_GECKODRIVER_FOLDER/

3) To set Selenium WebDriver is also necessary download Selenium Server and run it prior the test. To run Selenium Server change directory to folder where is file located and type server run command java -jar selenium-server-standalone-YOUR.V.NO.jar .

5 .Writing first PHP test

Under acceptance folder we will create first test in FirstTest.php file. We will try to check “Hello World!” text at homepage:

<?php
  $I = new AcceptanceTester($scenario);
  $I->wantTo('Test Frontpage');
  $I->amOnPage('/');
  $I->see('Hello World!', 'h1');
?>

6. Running the test

Running the test case is possible by simple command codecept run . You will write this command often so you can create temporary (alias in your folder) or permanent alias (in .bash_profile) by creating alias exp. alias runtest=vendor/bin/codecept run and than just type your alias.

If you have done everything right, with Selenium you should new see Firefox window executing your first test. After finishing it, browser will close down and terminal shows announcement about Success. PHP Webdriver execute all test just over the terminal info panel.

6.1 Further use

To show the list of all available commands you need to type vendor/bin/codecept . It will also handy to create alias for it ( but just for vendor/bin/ and than type codecept ).

Codecept will always by run execute every single test you have wrote all together. It helps to nerrow down only the test suite which we want by selecting test suit in command: codecept acceptance/functional/unit run .