Python Selenium Learning Series for Beginner-Intermediate Developers

1. How to Install Selenium Python into a Virtual Environment (venv) and setup VSCode / ChromeDriver

This video tutorial shows how to:

  • Install Python 3.10 (the latest version at the time of recording).
  • Create a virtual Python environment with venv. A virtual environment allows us to isolate libraries, like Selenium, depending on project needs. This makes future upgrading of Python and libraries much easier.
  • Install the Selenium Python library into the new virtual environment.
  • Install the Chrome Web Driver. The driver allows Selenium to control the browser. Note that the version of Chrome Web Driver must match the version of Chrome installed on your computer. Keep this in mind, because if your Chrome is set to auto-update, and it should be, your copy of the Chrome Web Driver will fall out of sync and your code will not run. It might feel like your Selenium code mysteriously stopped working, but this is the reason. All you have to do is sync the versions again.
  • Install Python extensions in VSCode. Particularly, the Jupyter Notebook extension. Jupyter Notebook allows Python code to run in blocks, rather than having to rerun the whole script after every code edit.
  • Validate that everything is working by running a quick test in VSCode to launch a browser and go to a test URL.

With the above steps complete, we are prepared for browser automation, web application testing automation, and web scraping.

Watch on YouTube

2. Scrape a DoorDash-like Site with 50 Lines of Code | Web Scraping Tutorial with Selenium Python

In this video tutorial, we’ll scrape data off a real website with Selenium & Python. You just need basic Python understanding to follow along. You’ll learn how to:

  • Examine the target page’s structure with the Chrome Dev Tools’ Inspector tool to determine the best way to perform the web scraping.
  • Locate single web page Elements and list of Elements by CSS Class Name and ID with Selenium.
  • Use Conditional Breakpoints in VSCode to find optional elements within a loop.
  • Simulate clicking on elements and “wait” for elements like pop-up modal windows to appear.
  • Distinguish duplicate elements.

After this tutorial, you should be equipped with enough basics to scrape much more complicated sites.

Watch on YouTube

3. Resolve “ElementClickInterceptedException: Element is not clickable at point” in Selenium Python

We’ll work thru an example of where Selenium is not able to click on an element and throws the dreaded ElementClickInterceptedException. We’ll go over how to:

  • Check if we are targeting the right element
  • Double-check that the element really clickable
  • Check if there is a delay before the element is clickable
  • Check if there are other elements overlapping or floating on top of the spot we’re trying to click on
  • Check where on the webpage Selenium is actually clicking on (see code below)
  • Show a almost guaranteed-to-work way of clicking element via JavaScript
//Code to check where on the webpage Selenium is actually clicking on. Paste this into the Chrome Dev Tool console. Replace the ??? with the coordinates.
let div = document.createElement("div");
div.append('<==CLICKED HERE');
div.style.position="absolute";
div.style.left='???px';
div.style.top='???px';
div.style.color='red';
document.body.appendChild(div);
div.scrollIntoView(true);
Watch on YouTube
4. How to use ChatGPT to write Selenium Python code

I used ChatGPT’s help to complete a small selenium task. It definitely saved me an hour of work, but I’d also wasted an hour trying to get thru some of ChatGPT’s frustrations. All in all, I broke even. However, like any tool, practice is required to gain efficiency. In the video below, I go over a few tips to help you get started and work more efficiently with ChatGPT and hopefully minimize the frustrating parts.

  • Let ChatGPT take the first cut. Maybe you have a task to do, but haven’t thought about how to do it yet. Now would the best time to let ChatGPT do the initial thinking for you.
  • How do you get ChatGPT to write Selenium code if it can’t go to your webpage? Using Chrome Dev Tools, you can copy the HTML of the target element and feed it to ChatGPT. If the element has an ID, Selenium can find the element without needing to know its parent or ancestor elements. You could feed the whole HTML page to ChatGPT, but there is a character limitation on the input box and the more information that you feed it, the more likely that it will get confused. So it’s best to limit the scope as precise as possible.
  • ChatGPT has outdated knowledge. ChatGPT has knowledge up until 2021. It just so happens that the Selenium library depreciated and stopped supporting some commons functions, like find_element_by_id, after 2021. Be prepared to make some corrections on the code it generated for you.
  • Assume the code is wrong. We should be aware that, at its current state, ChatGPT is wrong a lot, so it’s still up to us to make sure the code is correct. Just because it runs and the output looks good doesn’t necessarily mean there are no potential bugs.
  • Don’t get lazy or overly reliant on ChatGPT. While using ChatGPT to generate Selenium code, I found myself asking it to generate code for simple things that I could have and should have done myself.
  • Know when to cut your losses: I would have liked ChatGPT to continue building upon the code that it initially generated for me, but after a few exchanges, it started spitting out code that looks drastically different from before. At some point, you have to accept that ChatGPT isn’t going to produce what you need. So instead of wasting anymore time, it’s best to move on.
  • Take opportunities to learn from ChatGPT. It generated code that uses the Python zip() function, which I was not familiar with. I asked “what is zip” and it gave me a nice explanation.
Watch on YouTube
Scroll to Top