Wordpress Robots.txt Best Practices - How to Optimize WordPress Robots.txt [Still true in 2023!]

Improve your site management with these WordPress Robots.txt best practices. The robots txt file, if one exists on your site, tells bots what they are and are not allowed to access. These bots include search engine crawlers, SEO software and even malicious software.
Discover how to create and optimize the WordPress robots.txt file in this tutorial. This tutorial is the robots.txt file explained for beginners.
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: www.example.com/site_map.xml
Grab your free 17-Point WordPress Launch Checklist PDF: wplearninglab.com/get/wordpre...
Robots.txt Google Search Console link: support.google.com/webmasters...
Black Hole for Bad Bots tutorial: • [Real Results] Black H...
Always backup your site just in case (or use a staging site), here's how: • WordPress Backup, Rest...
Join our private Facebook group today! / wplearninglab
Here are 20+ reasons why I host all my sites with SiteGround: wplearninglab.com/siteground-...
Post videos of your WordPress success using the hashtag #WPLLCommunity!
Here's the link for Elementor Pro (aff): wplearninglab.com/recommends/...
I hope this information helps you! If you have any questions leave a comment below or ping me @WPLearningLab on Twitter.
WP Learning Lab Channel: kzread.info_c...

Пікірлер: 37

  • @wplearninglab
    @wplearninglab2 жыл бұрын

    If you enjoyed this video don't forget to the like it and subscribe! Then check out this playlist about speeding up WordPress: kzread.info/dash/bejne/lXZ4u498iLrTeZM.html

  • @franshescomviran5741
    @franshescomviran57417 ай бұрын

    Bloody brilliant mate! Cheers from Australia! 🎉

  • @weldingempire
    @weldingempire Жыл бұрын

    Thank you so much. I was having a lot of trouble trying to have ahrefs crawl my site and adsense. wonderful and clear tutorial.

  • @17aig
    @17aig2 жыл бұрын

    As always thank you so much for this tutorial

  • @wplearninglab

    @wplearninglab

    2 жыл бұрын

    You're welcome Nissim, thanks for watching! Let me know if you have any questions :)

  • @personaldronerepair6141
    @personaldronerepair61412 жыл бұрын

    You Rock !! Thank you for your efforts and time in!!

  • @wplearninglab

    @wplearninglab

    2 жыл бұрын

    Thanks, you rock too!

  • @worldtraveler303
    @worldtraveler3032 жыл бұрын

    Thanks so much!!!! 🙏🏿🙏🏿🙏🏿🙏🏿🙏🏿🙏🏿

  • @MH-jp1fv
    @MH-jp1fv Жыл бұрын

    awesome thank you!

  • @Sarahsplantbites
    @Sarahsplantbites Жыл бұрын

    Thank you!!!!

  • @WilliamFernandesvideos
    @WilliamFernandesvideos11 ай бұрын

    Congrats and thanx !

  • @wplearninglab

    @wplearninglab

    11 ай бұрын

    Thanks and thanks for watching!

  • @mahmoudgh2030
    @mahmoudgh20302 жыл бұрын

    Thanks

  • @Electronzap
    @Electronzap2 жыл бұрын

    Good stuff :)

  • @wplearninglab

    @wplearninglab

    2 жыл бұрын

    Thanks Paul and thanks for watching!

  • @TheBlades2012
    @TheBlades2012 Жыл бұрын

    Hi, When I put my sitemap on just like you did it comes up with the red X on the left?

  • @rgkh1503
    @rgkh1503 Жыл бұрын

    Hi there, thanks for the video, but this seems not to be the best practice anymore (cf usage of meta tags)..any update of this video planned?

  • @tevinkiprop1883
    @tevinkiprop18832 жыл бұрын

    Hey so I have a large website and when I click on to coverage I have the discovered- but not indexed yet message also when I try to use the URL inspection it displays noindex. kindly help since only a few of my pages are being indexed

  • @victoragburum1858
    @victoragburum18582 жыл бұрын

    Every time I crawl my site with semrush, it keeps detecting errors that my some pages could not be crawled, pls does anyone have an idea how to fix that issue

  • @frankstone
    @frankstone2 жыл бұрын

    Thanks to your previous tutorial, I have installed IThemes Security and activated the "hide backend" feature. Will the robots.txt file provide additional site security that is not already installed?

  • @wplearninglab

    @wplearninglab

    2 жыл бұрын

    Hi Frank, If you use the "hide backend" feature of iThemes Security you wouldn't need the user agent rules in the robots.txt file. The only thing that may benefit you is having the link to the sitemap in the robots.txt file. But you can also add your sitemap directly to the webmaster tools in major search engines instead. I hope that helps :) Let me know if you have any further questions. Thanks for watching!

  • @nachogalvezantunez1915
    @nachogalvezantunez1915 Жыл бұрын

    Great Bjorn! Really thankfull! My site got broken with the "500 Internal Server Error" after adding the "Really Simple SSL" Plugin to my Wordpress and Google Search Console is telling me that's because my server (at Google Cloud Platform) bloqued Google search engines (Googlebot) so they can not find my domain. Google Search Console though the robots.txt taster tool is prompting me to go back to the last usefull "robots.txt" downloading it to my computer and uploading it to my server. I can´t find any "robots.txt" file at my "html" directory; acording to your video I just have to upload the one downloaded from "robots.txt taster" tool, is this right? I really apreciate very much in advanced and hope you can help me. Best regards.

  • @17aig
    @17aig2 жыл бұрын

    Could you please add links to the Robert text tester and the blackhole Thanks in advance

  • @wplearninglab

    @wplearninglab

    2 жыл бұрын

    For sure, here you go: Robots.txt Google Search Console link: support.google.com/webmasters/answer/6062598?hl=en Black Hole for Bad Bots tutorial: kzread.info/dash/bejne/Yn913Kl9eNSTYbA.html

  • @shelbythomas7744
    @shelbythomas77442 жыл бұрын

    What is your favorite contact form and spam blocker for that form?

  • @wplearninglab

    @wplearninglab

    2 жыл бұрын

    Happy forms is pretty awesome and the free version is quite powerful. For free forms I also like Contact Form 7. I know it's old and a bit clunky, but you can build out lots of simple form types and there are plenty of addons for more functionality. If the form needs to have calculations done on data entered in the fields I like Gravity Forms. I like Recaptcha v3 and/or honeypot fields to reduce spam. ..I hope that helps, let me know if you have any further questions. Thanks for watching!

  • @pahtashow
    @pahtashow2 жыл бұрын

    How to make a sitemap in 2022 wordpress ?

  • @GBFragrances
    @GBFragrances Жыл бұрын

    Hi! How do I edit the robot txt file that's blocking Google to crawl my pages?

  • @Cognitoman

    @Cognitoman

    Жыл бұрын

    Should be in the public part of the website

  • @mustafatankus9343
    @mustafatankus93432 жыл бұрын

    hello, only the homepage of my site comes up in google searches. None of the articles on my site appear in google searches. Do you have any idea about the reason? I registered the property on google search console, submitted the sitemap to google. I checked the robots.txt file but I can't seem to solve this problem in any way.

  • @budsgrower9970

    @budsgrower9970

    2 жыл бұрын

    I have the same problem. I hope someone will answer

  • @starsproduction9676
    @starsproduction96762 жыл бұрын

    It's the robot.txt make your blog post show in search results

  • @guadadis
    @guadadis2 жыл бұрын

    Hi! Do you know what could be happening with my file robots.txt? Every so often it automatically switches to user-agent: * Disallow:

  • @mauwiks

    @mauwiks

    Жыл бұрын

    Mine too

  • @Cognitoman

    @Cognitoman

    Жыл бұрын

    Prolly something with your server from your hosting company

  • @siddhant2002
    @siddhant20022 жыл бұрын

    Do you have any video to how to make money as wordpress developer

  • @wplearninglab

    @wplearninglab

    2 жыл бұрын

    Hi Sidhant, I don't at the moment. From my experience, you'll need to (1) have the development skills, which it sounds like you have and then (2) you have to go out and find clients. Part 2 is the hardest for most people. You have to go where your ideal clients meet up (conferences, meet ups, etc.) or you have to go door to door with an offer they can't refuse.