Getting ready for the next UC Mode video tutorial

[mdmintz]

Getting ready for the next UC Mode video tutorial

If you haven't already heard the news, the last one was quite popular:
Thousands of views, hundreds of likes, and people want to see more.

(Here's the GitHub ticket for the previous video: #2213)

---

Now comes the time to plan the next one. There are lot of things that I didn't get to cover last time. I'll start a list:

📗 People wanted more information about the renamed Chrome console variables that appear when chromedriver launches Chrome. This is quite easy to show in a screenshot:

If you use regular Selenium/WebDriver to launch Chrome, those variables appear in the console. It's one of the easiest ways for websites to detect Selenium. In UC Mode, those variables are renamed/removed.

📗 Since the previous tutorial, more examples have been added:

• SeleniumBase/examples/raw_turnstile.py
• SeleniumBase/examples/raw_form_turnstile.py

In particular, those example deal with a different type of CloudFlare CAPTCHA: One where all users must click the box (generally after filling out a form) in order to proceed. The first video covered the CAPTCHA that only appears when you initially load a page. Only traffic detected as bot-traffic would be stopped by it. With the form CAPTCHA, everyone has to click the box (even if Cloudflare thinks you're human) so it has to be handled in a different/special way.

📗 The previous tutorial did not cover all the differences between the Driver(uc=True) and SB(uc=True) formats. People need to know when they should use SB() instead of Driver(). In particular, the SB() format does a lot more, and has a lot more methods. For instance, the SB() format will automatically close the driver for you at the end of the test or code block, whereas in the Driver() format, the user needs to call driver.quit() in the script. Not doing that would cause a memory leak, which gets you detected. Also, the SB() format also automatically handles virtual displays for environments with no GUI. With the Driver() format, users may need to add in something like sbvirtualdisplay themselves if they want to run on certain Linux environments while remaining undetected.

📗 Since the previous tutorial, a new way has been added in so that you can easily combine automation with manual intervention in UC Mode in such a way that it keeps your browser undetected. Here are a few examples of that:

sb.driver.uc_open_with_reconnect("https://nowsecure.nl/#relax", reconnect_time="breakpoint")
sb.driver.uc_open_with_reconnect("https://nowsecure.nl/#relax", "breakpoint")
sb.driver.reconnect(timeout="breakpoint")
sb.driver.reconnect("breakpoint")

It's important to note that while in that special breakpoint(), websites can't detect you, but it also means that you can't call any Selenium commands until after you've left the breakpoint(). (More info on that here: #2384)

📗 People want to see more live demos using sites with non-Cloudflare protection. (The previous live demos mainly covered Cloudflare-protected sites.)

📗 People also commented on the video clips (mainly of Star Trek actors) that were embedded in the full video for entertainment value and highlighting important points. There seemed to be a love/hate reaction to that, where some people really enjoyed them, while others saw it as a distraction. Based on YouTube's apparent algorithm, adding small clips into a video like that seems to make videos far more likely to appear in the Suggested Videos section of other videos, which greatly increases the number of people who find out about a video and watch it. At least for the time being, my channel and videos can really benefit from any marketing assistance that they can get... which means the video clips will continue for now, but I'll try using fewer of them in total. (To the person who said Christopher Lloyd wasn't in Star Trek... Please watch Star Trek III: The Search for Spock... Lloyd was the main villain!)

📗 People were wondering how to combine UC Mode with Wire Mode so that the browser remains undetected while still being able to see the requests being made. (The two modes cannot be used together.) Wire Mode uses an unmodified selenium-wire for that, so there's not much that can be done there unless selenium-wire makes changes to their repo to support that. To make matters worse, "Selenium Wire is no longer being maintained" according to their repo. (They announced that in January 2024). The good news is that alternatives exist, eg: SeleniumBase/examples/uc_cdp_events.py, which demonstrates how to collect CDP events from the browser. Additionally, I'm working on a selenium-wire alternative that allows you to see browser requests (and still be compatible with UC Mode).

📗 People also want to know how to record automation scripts while using UC Mode. Just start the Recorder like this:

sbase recorder --uc

Additionally, you can activate the Recorder in the middle of a script like this: self.activate_recorder()

📗 There's also a new UC Mode section in the SeleniumBase Docs: SeleniumBase/help_docs/uc_mode.md

Looks like there are lots of things to cover in the next UC Mode video tutorial. Get ready to be ready!

[mdmintz]

More things to cover:

📗 What is Rosetta 2 and why might macOS users care? Rosetta 2 lets you run an Intel-based chromedriver on an ARM system. (That script that patches chromedriver into uc_driver currently requires an Intel-based chromedriver.)

For macOS users, it's easy to get it if you don't already have it:
(https://discussions.apple.com/thread/254439437?sortBy=best)

softwareupdate --install-rosetta

📗 In the previous tutorial, only Chrome was supported in UC Mode. Now, two more browsers are supported!
Learn more as information gets revealed...

📗 Make sure that people realize that if the same IP Address hits a website too many times, then that IP Address could be blocked, even if that site can't detect that you're using a bot. (This is one reason why some people use proxies / rotating proxies.)

[mdmintz]

📗 If there's time, also cover how to properly do multithreaded UC Mode without pytest.

(Hint: Need to include sys.argv.append("-n") so that the current thread-locking system activates. The built-in thread-locking system is what prevents multiple threads from updating the same resource at the same time, such as multiple threads trying to patch the same driver at the same time.)

Sample script using concurrent.futures for spinning up multiple processes:

import sys
from concurrent.futures import ThreadPoolExecutor
from seleniumbase import Driver
sys.argv.append("-n")  # Tell SeleniumBase to do thread-locking as needed

def launch_driver(url):
    driver = Driver(uc=True)
    try:
        driver.get(url=url)
        driver.sleep(2)
    finally:
        driver.quit()

urls = ['https://seleniumbase.io/demo_page' for i in range(3)]
with ThreadPoolExecutor(max_workers=len(urls)) as executor:
    for url in urls:
        executor.submit(launch_driver, url)

---

Here's the script from above with randomized window placement so that the windows don't overlap:

import sys
from concurrent.futures import ThreadPoolExecutor
from random import randint
from seleniumbase import Driver
sys.argv.append("-n")  # Tell SeleniumBase to do thread-locking as needed

def launch_driver(url):
    driver = Driver(uc=True)
    try:
        x, y, w, h = randint(0, 755), randint(38, 403), 900, 600
        driver.set_window_rect(x, y, w, h)
        driver.get(url=url)
        driver.sleep(2)
    finally:
        driver.quit()

urls = ['https://seleniumbase.io/demo_page' for i in range(3)]
with ThreadPoolExecutor(max_workers=len(urls)) as executor:
    for url in urls:
        executor.submit(launch_driver, url)

[Joyen12]

Hi,
Is it possible to use ProcessPoolExecutor instead of the Thread one ?
And use the pytest function directly in the launch function , by doing for example os.system('pytest test.py -proxy=.....')

I'm getting so many error and so many failed test that should all pass :s idk why...
Many INTERNALERROR kind of stuff

Maybe it's just not possible to do multiprocessing that way ?

Thanks a lot for your reply I'm really list on this, I spent a lot of time figuring it out but now I really don't know what to do :s

[Joyen12]

Hello, my comment was marked as off-topic but I don't think that is off-topic.

The SB multi-threading/processing is not working with multiple proxy (one for each test) using pytest...I'm not sure what is the problem as I tried everything in the docs and stakeoverflow and so on...There is no info about this.
The first error I get is this one from chrome, the sessions are not starting until I click "OK" on this box attached. (Sorry it's in french but it's just saying : "Failed to Load Extension From Manifest File Is Missing or Unreadable"

Is there a way to remove that box or to just click on ok automatically ?

Thanks a lot

[mdmintz]

Note: This thread is specifically for documenting steps for the next UC Mode video tutorial.
Anything not related to video preparation will be marked off-topic.

Update: Repeat offenders will be blocked.

[Joyen12]

Note: This thread is specifically for documenting steps for the next UC Mode video tutorial. Anything not related to video preparation will be marked off-topic.

Ok sorry, I wasn't sure where to post that as it is exactly related to this part of code and I haven't seen any thread talking about advanced multithreading or/and multiprocessing using proxys and complex stuff and options. If there is one, could you give me the link I will repost there, otherwise maybe it would be good to have one because for very advanced workflow there are so many bugs happening but it's hard to know exactly why. Thanks a lot

[mdmintz]

Video is ready!

https://www.youtube.com/watch?v=2pTpBtaE7SQ