Merge pull request #307 from seleniumbase/refactor-visual-testing

Refactor visual/layout testing

Author: mdmintz

examples/visual_testing/ReadMe.md

@@ -0,0 +1,48 @@
+### Automated Visual/Layout Testing
+
+The purpose of automated visual/layout testing is to help you determine when something has changed the layout of a web page. Rather than comparing screenshots, a more effective way is to compare HTML tags at different points in time on the same web page. If a change is detected, it could mean that something broke on the webpage, or it might be something harmless like a website redesign or dynamic content loading on the web page. To handle automated visual/layout testing, SeleniumBase uses the ``self.check_window()`` method, which can set visual baselines for comparison and then compare the latest versions of web pages to the existing baseline.
+
+The first time a test calls ``self.check_window()`` for a unique "name" parameter provided, it will set a visual baseline, meaning that it creates a folder, saves the URL to a file, saves the current window screenshot to a file, and creates the following three files with the listed data saved:
+* tags_level1.txt  ->  HTML tags from the window
+* tags_level2.txt  ->  HTML tags + attributes from the window
+* tags_level3.txt  ->  HTML tags + attributes/values from the window
+
+Baseline folders are named based on the test name and the name parameter passed to ``self.check_window()``. The same test can store multiple baseline folders.
+
+If the baseline is being set/reset, the "level" doesn't matter.
+
+After the first run of ``self.check_window()``, it will compare the HTML tags of the latest window to the one from the initial run.
+Here's how the level system works:
+* level=0 ->
+    DRY RUN ONLY - Will perform a comparison to the baseline, and
+                   print out any differences that are found, but
+                   won't fail the test even if differences exist.
+* level=1 ->
+    HTML tags are compared to tags_level1.txt
+* level=2 ->
+    HTML tags are compared to tags_level1.txt and
+    HTML tags/attributes are compared to tags_level2.txt
+* level=3 ->
+    HTML tags are compared to tags_level1.txt and
+    HTML tags + attributes are compared to tags_level2.txt and
+    HTML tags + attributes/values are compared to tags_level3.txt
+
+As shown, Level-3 is the most strict, Level-1 is the least strict. If the comparisons from the latest window to the existing baseline don't match, the current test will fail, except for Level-0 tests.
+
+You can reset the visual baseline on the command line by using:
+    ``--visual_baseline``
+As long as ``--visual_baseline`` is used on the command line while running tests, the ``self.check_window()`` method cannot fail because it will rebuild the visual baseline rather than comparing the html tags of the latest run to the existing baseline. If there are any expected layout changes to a website that you're testing, you'll need to reset the baseline to prevent unnecessary failures.
+
+``self.check_window()`` will fail with "Page Domain Mismatch Failure" if the page domain doesn't match the domain of the baseline.
+
+If you want to use ``self.check_window()`` to compare a web page to a later version of itself from within the same test run, you can add the parameter ``baseline=True`` to the first time you call ``self.check_window()`` in a test to use that as the baseline. This only makes sense if you're calling ``self.check_window()`` more than once with the same name parameter in the same test.
+
+Automated Visual/Layout Testing with ``self.check_window()`` is not very effective for websites that have dynamic content that changes the layout and structure of web pages. For those, you're much better off using regular SeleniumBase functional testing.
+
+Example usage:
+```
+    self.check_window(name="testing", level=0)
+    self.check_window(name="xkcd_home", level=1)
+    self.check_window(name="github_page", level=2)
+    self.check_window(name="wikipedia_page", level=3)
+```

examples/visual_testing/__init__.py

@@ -1,7 +1,7 @@
 from seleniumbase import BaseCase
 
 
-class AutomatedVisualTest(BaseCase):
+class VisualLayoutTest(BaseCase):
 
     def test_applitools_helloworld(self):
         self.open('https://applitools.com/helloworld?diff1')

examples/visual_test.py renamed to examples/visual_testing/layout_test.py

@@ -17,9 +17,4 @@ If you're installing SeleniumBase directly [from GitHub](https://github.com/sele
 pip install -U -e git+https://github.com/seleniumbase/SeleniumBase.git@master#egg=seleniumbase
 ```
 
-If you're installing SeleniumBase directly [from Anaconda Cloud](https://anaconda.org/seleniumbase/seleniumbase), use:
-```bash
-pip install -U -i https://pypi.anaconda.org/seleniumbase/simple seleniumbase
-```
-
 (If you're not using a virtual environment, you may need to add ``--user`` to your ``pip`` command if you're getting errors during installation.)

help_docs/install.md

@@ -13,7 +13,7 @@ pytest-cov>=2.6.1
 pytest-html>=1.20.0
 pytest-metadata>=1.8.0
 pytest-rerunfailures>=7.0
-pytest-xdist>=1.27.0
+pytest-xdist>=1.28.0
 parameterized>=0.7.0
 beautifulsoup4>=4.6.0
 colorama==0.4.1

requirements.txt

@@ -20,6 +20,7 @@
 
 PROXY_LIST = {
     "example1": "159.122.164.163:8080",  # (Example) - set your own proxy here
+    "example2": "158.69.138.8:1080",  # (Example) - set your own proxy here
     "proxy1": None,
     "proxy2": None,
     "proxy3": None,

seleniumbase/config/proxy_list.py

@@ -17,7 +17,7 @@
 
 setup(
     name='seleniumbase',
-    version='1.22.3',
+    version='1.22.4',
     description='Reliable Browser Automation & Testing Framework',
     long_description=long_description,
     long_description_content_type='text/markdown',
@@ -66,7 +66,7 @@
         'pytest-html>=1.20.0',
         'pytest-metadata>=1.8.0',
         'pytest-rerunfailures>=7.0',
-        'pytest-xdist>=1.27.0',
+        'pytest-xdist>=1.28.0',
         'parameterized>=0.7.0',
         'beautifulsoup4>=4.6.0',  # Keep at >=4.6.0 while using bs4
         'colorama==0.4.1',