Undetectable automation with SeleniumBase UC Mode and undetected-chromedriver. (Python)
Ғылым және технология
Undetectable automation with SeleniumBase UC Mode and undetected-chromedriver. (Python)
(Part 2: • Undetectable Automatio... )
0:00 - Welcome / Introduction
0:35 - Live Demos of "UC Mode" and multithreaded "UC Mode"
1:32 - Who am I? (Learn about the creator of SeleniumBase / "UC Mode")
3:24 - A brief general history of bots (with some entertaining video clips)
5:40 - A brief history of Google reCAPTCHA v1 / v2 / v3
6:34 - A brief history of Cloudflare Turnstile
7:07 - Introduction to undetected-chromedriver
10:03 - Introduction to SeleniumBase "UC Mode"
11:58 - Basic "UC Mode" code example
13:50 - More code and more methods
15:45 - Live Demo of "Demo Mode" (highlights actions with JavaScript)
16:25 - Differences between undetected-chromedriver and "UC Mode"
18:34 - More "UC Mode" code, followed by a Live Demo
19:43 - Things you need to do to remain undetected ("UC Mode" does this)
23:25 - New "driver" methods included with "UC Mode"
27:38 - Links to more "UC Mode" code on GitHub
29:05 - General information (with lots of entertaining video clips)
35:32 - Conclusion (Visit SeleniumBase on GitHub for more)
SeleniumBase on GitHub:
github.com/seleniumbase/Selen...
This video includes short video clips from various places to make things more interesting:
* Star Trek (TNG, LD, SNW, and a blooper/outtake from The Voyage Home)
* Duck Tales
* Deadly Games (Christopher Lloyd, Brent Spiner, LeVar Burton)
* Ghostbusters
* Back to the Future
* Battlestar Galactica
* Princess Bride
* The Orville
* MIT Musical Theater Guild's "Star Wars: The Musical"
* And a few video clips filmed in person from my phone
The 2nd UC Mode video is now on KZread: • Undetectable Automatio...
Пікірлер: 165
The next UC Mode video is ready: kzread.info/dash/bejne/ZKSI0qStkaidg7Q.html - More live demos (with code), more browsers, changing geolocation, bypassing login screens with cookies, and more.
Micheal thank you so so much for great work and time you put in to Seleniumbase. You saved my life!!!!!
Wow, Mike, once again you continue to amaze me with your additional great ideas. I've been following you on the net.. keep up these very interesting and informative articles Thanks 😊
I've been trapped in UC hell waiting on updates for over a year, trying to keep the battleship afloat with duct tape and bubble gum. This one demo showed solutions for half of the cannon-ball sized holes we've been plugging, and after looking through the docs it looks like we'll be building a whole new battle(star)ship using SeleniumBase. I'll name the next version Enterprise in honor of this masterpiece. Live long and prosper.
@MichaelMintz
8 ай бұрын
The Cylon/Borg battle will be marvelous. Get those FTL/Warp engines online!
The community really appreciate this job so far. Thanks you some much for your time sharing this knowledge.
Saved me a lot of time with SeleniumBase, I used to only use undetected-chromedriver itself. thank you man
I wrote a similar library internally about 4-5 years back using Selenium and Capybara for Ruby. It patched chromedriver and did the UC tricks like modifying window.navigator and such. But now that I dusted it off it needs too much maintenance to justify, so SeleniumBase is perfect as a replacement. Boom, quick pip install and I have a working turnstile-passing browser. You're a legend Michael.
This is an awesome video! Thank you for sharing! 🎉
You are a genius! I truly enjoyed your fascinating ideas and your expertise is truly impressive.
Great work brother, keep doing videos like this about selenium, ill shared with my friend and liked, thanks again !
Very capacious and educational video. Thank you for your work. SeleniumBase is simply great.
You are doing a great job! You cannot image how much I suffer from undetected-chromedriver😂
That's was fun and extremely useful. SeleniumBase is amazing!
Great work my friend, congrats from Brazil !!
Excellent! I really appreciate your good work. Thank you.
O MY GOD! I spent a lot of time to trying solve CF problem ) and with you it was super easily! just few lines of code ))) THANK YOU )
Much more interesting than lectures at the university. Thank you for the work done. I really appreciate it. Already tested uc in the case
Cool video & Also very helpful. Thanks!
Great video and great software. Thank you.
Best Video i see about selenium thnx ❤
really appreciate your work dedicated... the best father of automation i think....
Wow, it works for me perfectly!!!
Thank you for this! 🙏🏻
absolute gold!! Thanks Micheal
thank you so much Michael
Thanks for great library !
Great job bro, thank you from Turkey
Very impressed!
Damn, man its amazing!!! I am stuck for 5 days on this captcha, and thanks to you work, now i got bypass!!! Page just loading with captcha, ckecking and going directly to the website, i can't believe in that!!
@MichaelMintz
6 ай бұрын
Awesome!
This is amazing thank you
this video is so funny and informative. Thanks for this wonderful content. :D
this guy is simply awesome🖤🖤🖤
Great work!
Lay of the cutscenes
great video!
you saved me bro. I love u
Nice Video Pls more
You are brilliant.
Absolute Legend.
All the memes and videos are killing me lol
Mike you the goat fr 🐐
Thank you for the seleniumbase it is a great tool
You great, I love seleniumbase.
Ty so much!
Great Video! Helped me so much!! However when i use Driver(uc=True, proxy=myproxy) i get a Notification in the browser: "you are using an unsupported command line flag: --ignore-certificate-errors. I also cant get passed on nowsecure when i use this proxy line (My Proxy works fine). " How do i use proxy in uc mode?
@MichaelMintz
8 ай бұрын
That warning shouldn't be an issue. There's another selenium-specific command-line option to remove it, but then you would get detected. Maybe your proxy was already blacklisted, as it is working on a proxy I tried with the latest version of SeleniumBase (4.20.8), which actually fixed a bug with proxies.
Hi, do you mind elaborating on the part where it said "if using a custom user_data_dir, don't let that be used with non-UC-mode chrome"? How do I prevent this from happening? Sorry I'm a beginner.
@MichaelMintz
2 ай бұрын
It means don’t reuse an existing user_data_dir for UC Mode scripts unless it was created by UC Mode and only used by UC Mode. Keep track of how each user_data_dir was created and used.
Is it possible in any way to just exchange the driver and use all the old function that are for normal selenium? If yes, how? I found this raw driver part in the docs but i didnt got it entirely. This will make it much easier to integrate into exisitng algorithms to have a comparison between normal selenium and seleniumbase
@MichaelMintz
8 ай бұрын
Yes, see github.com/seleniumbase/SeleniumBase/blob/master/help_docs/syntax_formats.md#sb_sf_23 with the "driver" format, number 23.
@leonwinkel6084
7 ай бұрын
awesome, thanks! @@MichaelMintz
Hey, SeleniumBase seems awesome. I've been using UC for a while and am looking into migrating some things over and am probably doing something wrong, but perhaps you can help. When I copy code from the docs my autocomplete stops working. The way I create a driver now: web = uc.Chrome(options=chrome_options, driver_executable_path=path) Based on the docs it seems I should do : web = Driver(browser="chrome", uc=True) or: web = SB(browser="chrome", uc=True) When I use a class-based syntax autocomplete seems to work, but when I use a more functional programming syntax it does not. Any idea what I'm doing wrong?
@MichaelMintz
8 ай бұрын
Are you referring to auto-completion from a text editor / IDE such as VS Code? That may be up to the editor, but tab-autocompletion should work from within a Python breakpoint. Try `breakpoint()` in your code to open up Pdb+ (SeleniumBase's default debugger). Then use tab to autocomplete. To leave the breakpoint, type `c` and press `Return`.
@csea148
8 ай бұрын
@@MichaelMintz I will create a github issue so i can give you some decent details to work with. Thank you for the great work on Seleniumbase!
Please how did run your multiple drivers at the same time because i used multithreading and GIL didn't allow it all run at the same time
@MichaelMintz
5 ай бұрын
github.com/seleniumbase/SeleniumBase/blob/master/examples/presenter/multi_uc.py
Thank you Michael, the SeleniumBase is a great package I have ever used for web automation. I just wonder can you provide more detail to show how to bypass bot detection system from DataDome with UC mode?
@MichaelMintz
4 ай бұрын
Thank you. See the DataDome thread here: github.com/seleniumbase/SeleniumBase/issues/2365#issuecomment-1858065260
Hey maybe not the right place for this question but have been using SB for a few days now and reading through the docs. I was just curious the point of find_elements. I was originally trying to get text from a dynamic number of questions, find_elements worked with the right selector and got all the elements I expected. But get_text does not work with elements only selectors. Most methods seem to use selectors. So to get my text I will have to iterate by the length of the element list and use get_text('selector[i]') or something like that. Sorry for the length and over-specificity but this seems redundant and I think there is a better way to do it that I am missing in the docs. Tyia for any help
@MichaelMintz
6 ай бұрын
Most SeleniumBase methods expect a selector (not a WebElement, which has `.text` as a property to get the text from the WebElement directly). The find_elements(selector) method will return a list of WebElements. From each of those WebElements, you can call element.text for the text. I hope that answers your question.
@joewilliams9783
6 ай бұрын
Ohh got it, thank you so much. Seems I need to look into WebElements more 😅. Also, really wasn't expecting a reply that fast on a holiday weekend appreciate it king. @@MichaelMintz
Can you do Something similar with playwright?
Hello, the particular website I am going to requires location permissions in the browser to fully load. If there anyway I can modify seleniumbase UC mode to either, 1.) give the site location permissions with a fake location, or 2.) give the site my actual location permissions while allowing my selenium webscraper to go otherwise undetected. Thank you for your time, I appreciate you for sharing your creation and knowledge.
@MichaelMintz
4 ай бұрын
Geolocation example: github.com/seleniumbase/SeleniumBase/blob/master/examples/test_geolocation.py
Thanks man
@MichaelMintz I use a scraper to open 'hits' in new tabs so that I can review them once the scrape is completed. Unfortanately, I must keep my browsed page count rather low, so I must pop new sessions fairly often. My old selenium/firefox based setup would leave all windows intact when opening a new browser session. How do I configure seleniumbase in UC mode to leave the previous windows open when starting a new session? (self.driver = Driver(uc=True) closes the previous browser window) Brilliant work btw. Thanks!
@MichaelMintz
5 ай бұрын
If you're using the `Driver()` format, then the browser only quits after calling `driver.quit()`. You can choose to include the `driver.quit()` (or not) like in github.com/seleniumbase/SeleniumBase/blob/master/examples/raw_browser_launcher.py where the `quit()` happens after the block. For keeping the tab open, use `driver.uc_open(url)` when opening URLs in UC Mode.
how to attach to the opening browser ? or I understand something that you recommended launch chrome before attaching chromedriver
@MichaelMintz
4 ай бұрын
That's partly how UC Mode works: By attaching the driver to an already-running Chrome browser.
Firstly, congratulations on your incredible work. This tool has helped me a lot. I really liked the editing of this video. I would like to take the opportunity and ask a question. I have a script that opens a new driver "self.get_new_driver()", however, I am having difficulty closing the other driver and keeping just one. I've already tried using "self.quit_extra_driver()", but this command ends up closing all open drivers. one by one. Thank you very much for the answer. Hugs.
@MichaelMintz
3 ай бұрын
Thank you. "self.quit_extra_driver()" is for closing any browser window that was opened with "self.get_new_driver()". You can also use `driver.quit()` to close a browser window.
@morroderir6624
3 ай бұрын
@@MichaelMintz It's works. thanks man
Hello, I made a script with UC mode that works on Windows. But when I tried to put it on a Raspberry Pi 4 with Ubuntu 23.10, the UC mode doesn't work. The website detects me as a bot. I tried headless and with a virtual display from pyvirtualdisplay. Do you have any ideas on what I need to change to make it work?
@MichaelMintz
4 ай бұрын
Hello. See the discussion about UC Mode on Raspberry Pi here: github.com/seleniumbase/SeleniumBase/discussions/2286
Hi great content, a shame though that you only have less than 1000 subs. Why not set UC mode as the default mode? Is there a disadvantage? Also, which example scripts if any use the multithread ?
@MichaelMintz
6 ай бұрын
Changing default settings can break backwards compatibility, so UC Mode is activated by setting it directly. Regular mode is a bit faster, as it has less to do. For a multithreaded example, see github.com/seleniumbase/SeleniumBase/issues/2006#issuecomment-1679789785. There are more examples in Github.
@MichaelMintz
17 күн бұрын
Now there are more than “1000 subs”… 😉
Hi, it seems like the most suitable things for me. However, i'm very beginner. When i start scripts browser window doesn't appear. Is it an bug on my side, or it is intended behavior ?
@MichaelMintz
7 ай бұрын
Are you using Linux? By default, tests on Linux run in headless mode. Use `headed=True` or `pytest --headed` to see the browser on a headed Linux machine.
@artemiynoskov7827
6 ай бұрын
thanks for help, i've found --gui flag for tests@@MichaelMintz
Has anyone managed to use the uc configuration in Selenium Grid? When I'm starting it is using normal selenium.
@MichaelMintz
6 ай бұрын
UC Mode isn’t compatible with the Selenium Grid.
Thanks a lot for sharing your knowledge! This seems truely like an amazing framework and smart though through extension of the functions that are executed in the webautomation. Will start to test it for my applications, Thanks a lot again!
@leonwinkel6084
8 ай бұрын
I was thinking about a way, where its possible to actually move a mouse in between the elements that are clicked. In theory a caluclation of the coordinates and then following a vector from one element to the other, with a bit off set + correction movments should create a very much human like behaviour. Since there are tools as Hotjar, that tracks the cursor movement, mainly for landigpage or UI optimization, it shouldnt be so difficuilt to use it for bot detection if the "guards" would like to do that
Can I ask one more thing? Your code at 12:21 won't open a chromium, even with the --gui option. However your my_first_test.py file works fine, with --gui option. This is important because I am adapting the my_first_test.py to my use case, the site detects it as a bot. I wanna use the code at 12:21, but that doesn't do anything sadly :/
@MichaelMintz
2 ай бұрын
Use `Driver(uc=True, headed=True)` to override the default headless mode on Linux. Although I may add the `--gui` command-line shortcut in the next release. The pytest formats already have all those shortcuts.
@MichaelMintz
2 ай бұрын
Upgrade to seleniumbase 4.26.2 (or newer).
Can uc mode work with geckodriver or brave driver? Or only chromedriver.
@MichaelMintz
2 ай бұрын
Works with other Chromium browsers, such as Brave. For details, see the 2nd UC Mode video: kzread.info/dash/bejne/ZKSI0qStkaidg7Q.html
@RedPillSpill101
2 ай бұрын
@@MichaelMintz thank you
is there any way to use selenium base with seleniumwire, seems like both use different driver: from seleniumbase import Driver from seleniumwire import webdriver
@guocity
2 ай бұрын
ok, I find out wire=true, how to pass in wire_options and chrome_options?
@MichaelMintz
2 ай бұрын
github.com/seleniumbase/SeleniumBase/issues/2145
Hey, awesome work and thanks for sharing it, it's really helpful to have this tutorial. I have 3 questions and will ask them here in case you're still reading the comments: - How will Google Ads clicks perform with the current version ? - Is there any way to donate for your work on this ? - Do you have any email where I can reach you ? Thanks and keep up the good work ! :)
So how is this different from selenium-wire's UC mode? Sounds pretty much the same...
@MichaelMintz
5 ай бұрын
selenium-wire integrates with undetected-chromedriver for all of its anti-detection abilities. SeleniumBase has a separate integration with selenium-wire (but not for avoiding detection): github.com/seleniumbase/SeleniumBase/issues/2145
Great video! I have few questions. 1) How to rotate user agents and IP ? 2) Is cookie management necessary?
@MichaelMintz
3 ай бұрын
1. See the documentation in the SeleniumBase GitHub repo for that information. 2. No cookie management is necessary.
Lol serious though..My script runs and once its finished, i navigate to do a pixel scan and it is detected as a bot. Wonder if you could look at the Frankenstein code I put together and see why. I made sure im using the latest seleniumbase as well 🤔
@MichaelMintz
8 ай бұрын
GitHub is the place for opening tickets.
@web3twon
8 ай бұрын
@@MichaelMintz I'm actually on there heavily reading and digging in. I can hardly get a basic version running without detection I must be doing something wrong fundamentally. Pretty new to GitHub is there a certain ticket I need to put in? Appreciate your work on this, I'm going back to watch your presentation you gave a little over 6 months ago to get a better idea of how it all works
I tried the scripts today, didn't worked for me, is UC mode not working anymore or I'm doing something wrong
@MichaelMintz
5 ай бұрын
This sample script worked for me: github.com/seleniumbase/SeleniumBase/blob/master/examples/raw_uc_mode.py
forgive but iam new to this how can i set proxy in uc mode ?
@MichaelMintz
2 ай бұрын
Pass the “proxy” option with “ip:port” or “user:pass@ip:port”.
Thank alot for your UC mode, but can you please kindly tell us how to use : user_data_dir
@MichaelMintz
23 күн бұрын
Set the "user_data_dir" arg when you want to create and use a custom Chrome User Data Dir. (And don't mix your UC Mode User Data Dir with non UC Mode Chrome.)
will it work with headless. or can we schedule it in baremetal server?
@MichaelMintz
Ай бұрын
For maximum stealth, avoid headless mode.
@hammadzaki1624
Ай бұрын
@@MichaelMintz But without making it headless is there anyway to run it on server
@MichaelMintz
Ай бұрын
@@hammadzaki1624 Use `xvfb=True` with the `SB()` format.
@hammadzaki1624
Ай бұрын
@@MichaelMintz with sb(uc=True, xvfb = True) like that??
@MichaelMintz
Ай бұрын
@@hammadzaki1624 `with SB(uc=True, xvfb=True) as sb:`
can this be used in Jupyter Notebooks?
@MichaelMintz
6 ай бұрын
I haven’t tried. Maybe… maybe not.
@joewilliams9783
6 ай бұрын
yes it works fine in ipynb files
Mate it doesn't open a chromium window unlike selenium itself. How do we do this?
@MichaelMintz
2 ай бұрын
What do you mean? UC Mode opens Chrome.
@EmreYavuzalp
2 ай бұрын
@@MichaelMintz Sorry Linux doesn't open that by default. I solved it by adding --gui option. Like "pytest main.py --gui" But how do you add sleep, what is the best way? I use time.sleep but is this optimal?
@MichaelMintz
2 ай бұрын
@@EmreYavuzalp If you're referring to the sleep during the reconnect, you can modify the reconnect_time. Details in the second video: kzread.info/dash/bejne/ZKSI0qStkaidg7Q.html (For a regular sleep, you can do `sb.sleep(seconds)`, and `driver.sleep(seconds)` for the modified driver.
Michael, how can I add options like "--mute audio" from browser using the driver manager?
@MichaelMintz
7 ай бұрын
Pass extra options into `chromium_arg` - Eg. "ARG=N,ARG2" (comma-separated; initial dashes not needed)
@gustavoconsoni2750
7 ай бұрын
@@MichaelMintz Thank you very much, it worked! Please answer one last question, what are the new features that headless2 has compared to normal headless?
@MichaelMintz
7 ай бұрын
@@gustavoconsoni2750 `headless2` triggers the newer Chrome headless mode: stackoverflow.com/a/73840130/7058266 - It improves on the old headless mode by allowing extensions, etc. `headless2` is automatically used in SeleniumBase as needed, such as when using headless UC Mode.
hello! how to implement click on Enter
@MichaelMintz
5 ай бұрын
UC Mode special clicks are done with `driver.uc_click(selector)`
Great and very informative. kindly make a video on the following developing a web brower that has following functions URL insertion open the inserted URL into diffrent New Tabs randomly and assign a random user agent (such as : chrome , opera , safari , edge , firefox, and other known browsers also ) open every new tab with new user agent. Proxy Insertion 500 proxies adding options available and randomly pick only one then don't use this proxy which is alreday used pick a new one for all the open Tabs Number of Tab (randomly between 1-20) Random user agent switcher and manager (such as : chrome , opera , safari , edge , firefox, and other known browsers also ) automatically scroll the whole open page up and within 60seconds scroll down then up then down then till the end of the page most important stay undetectable. thank you for your time 😌❤
How to add cookies before getting url
@MichaelMintz
6 ай бұрын
stackoverflow.com/a/63220249/7058266
what is difference between basecase and sb?
@MichaelMintz
2 ай бұрын
github.com/seleniumbase/SeleniumBase/blob/master/help_docs/syntax_formats.md
@guocity
2 ай бұрын
@@MichaelMintz Thanks, that's really helpful, is there anything in the web you can't automate?
Remove the annoying video clips its more annoying than KZread ads 😂
@MichaelMintz
6 ай бұрын
The Star Trek, Ghostbusters, Orville, or other clips?
@benh1163
5 ай бұрын
@@MichaelMintzall of them, those are good in KZread videos if it’s maybe 1 or 2 for the whole video but it felt like u put one every 10 seconds lol
@MichaelMintz
5 ай бұрын
@@benh1163 KZread's "Suggested videos" algorithm seems to favor videos that contain clips from other videos. This one is currently my most-watched video. I'll try fewer clips in the next one.
@utkendi
5 ай бұрын
All@@MichaelMintz
@snusmumriken232
4 ай бұрын
Oh my god please remove the annoying joke clips. Why would you take an informative talk and make it annoying?
Hoping you’re available! I needs asssitance creating a python based web app. Was wondering if you’d help. Don’t mind to pay!
Video still be informative without using unnecessary random video clips
Anyone have a bot to skip those annoy video clips
Dude, good video but when using these cutaway b-roll clips, please practice less is more. 1,2,3 is fine but one after every single sentense gets annoying, fast
Guys can someone build a bot using seleniumBase just for skip all the memes on this video !!
Please don't add so many clips, I am here to learn, not to laugh lmao
Honestly, while I appreciate the technicalities, I found the over-use of meme-clips obnoxious and disruptive.
Your use of old film clips, nah you need to be stopped (not really) . I'm cracking up -- it's funny how they slow down and accelerate the learning.
Ok, I literally had to stop watching this video because of the sheer number of ridiculous and obnoxious clip inserts from Star Trek, Star Wars, BSG and other irrelevant crap. If you' re going to talk about a topic, stay on topic. Don't bombard your audience with unrelated video clips.
@MichaelMintz
5 ай бұрын
With KZread’s algorithm, a few video clips added in greatly increases the number of times a video appears under suggested videos. My previous videos didn’t include clips, and didn’t get anywhere close to this many views. Plus, this is gold for Star Trek fans. :)
@DerickMasai
2 ай бұрын
@@MichaelMintz Aaah perfectly sums it up then. I thought you were new to editing and got overzealous with the memes but adding all that just for engagement as the main motivation is not something I could ever torture myself with and this simply isn't worth my time.
@RedPillSpill101
2 ай бұрын
@@DerickMasai We shall all mourn your presence but i expect we will get over it.
This abrupt movie clips make this borderline unwatchable, keep the "humor" to minimum
is there a way disconect driver right after clicking an element?
@MichaelMintz
2 ай бұрын
Use “sb.driver.uc_click(selector)”
@blacksundarkskies
2 ай бұрын
@@MichaelMintz cool , but it don't see to work for my shenanigans