Wordpress Robots.txt Best Practices - How to Optimize WordPress Robots.txt [Still true in 2023!]
Improve your site management with these WordPress Robots.txt best practices. The robots txt file, if one exists on your site, tells bots what they are and are not allowed to access. These bots include search engine crawlers, SEO software and even malicious software.
Discover how to create and optimize the WordPress robots.txt file in this tutorial. This tutorial is the robots.txt file explained for beginners.
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: www.example.com/site_map.xml
Grab your free 17-Point WordPress Launch Checklist PDF: wplearninglab.com/get/wordpre...
Robots.txt Google Search Console link: support.google.com/webmasters...
Black Hole for Bad Bots tutorial: • [Real Results] Black H...
Always backup your site just in case (or use a staging site), here's how: • WordPress Backup, Rest...
Join our private Facebook group today! / wplearninglab
Here are 20+ reasons why I host all my sites with SiteGround: wplearninglab.com/siteground-...
Post videos of your WordPress success using the hashtag #WPLLCommunity!
Here's the link for Elementor Pro (aff): wplearninglab.com/recommends/...
I hope this information helps you! If you have any questions leave a comment below or ping me @WPLearningLab on Twitter.
WP Learning Lab Channel: kzread.info_c...
Пікірлер: 37
If you enjoyed this video don't forget to the like it and subscribe! Then check out this playlist about speeding up WordPress: kzread.info/dash/bejne/lXZ4u498iLrTeZM.html
Bloody brilliant mate! Cheers from Australia! 🎉
Thank you so much. I was having a lot of trouble trying to have ahrefs crawl my site and adsense. wonderful and clear tutorial.
As always thank you so much for this tutorial
@wplearninglab
2 жыл бұрын
You're welcome Nissim, thanks for watching! Let me know if you have any questions :)
You Rock !! Thank you for your efforts and time in!!
@wplearninglab
2 жыл бұрын
Thanks, you rock too!
Thanks so much!!!! 🙏🏿🙏🏿🙏🏿🙏🏿🙏🏿🙏🏿
awesome thank you!
Thank you!!!!
Congrats and thanx !
@wplearninglab
11 ай бұрын
Thanks and thanks for watching!
Thanks
Good stuff :)
@wplearninglab
2 жыл бұрын
Thanks Paul and thanks for watching!
Hi, When I put my sitemap on just like you did it comes up with the red X on the left?
Hi there, thanks for the video, but this seems not to be the best practice anymore (cf usage of meta tags)..any update of this video planned?
Hey so I have a large website and when I click on to coverage I have the discovered- but not indexed yet message also when I try to use the URL inspection it displays noindex. kindly help since only a few of my pages are being indexed
Every time I crawl my site with semrush, it keeps detecting errors that my some pages could not be crawled, pls does anyone have an idea how to fix that issue
Thanks to your previous tutorial, I have installed IThemes Security and activated the "hide backend" feature. Will the robots.txt file provide additional site security that is not already installed?
@wplearninglab
2 жыл бұрын
Hi Frank, If you use the "hide backend" feature of iThemes Security you wouldn't need the user agent rules in the robots.txt file. The only thing that may benefit you is having the link to the sitemap in the robots.txt file. But you can also add your sitemap directly to the webmaster tools in major search engines instead. I hope that helps :) Let me know if you have any further questions. Thanks for watching!
Great Bjorn! Really thankfull! My site got broken with the "500 Internal Server Error" after adding the "Really Simple SSL" Plugin to my Wordpress and Google Search Console is telling me that's because my server (at Google Cloud Platform) bloqued Google search engines (Googlebot) so they can not find my domain. Google Search Console though the robots.txt taster tool is prompting me to go back to the last usefull "robots.txt" downloading it to my computer and uploading it to my server. I can´t find any "robots.txt" file at my "html" directory; acording to your video I just have to upload the one downloaded from "robots.txt taster" tool, is this right? I really apreciate very much in advanced and hope you can help me. Best regards.
Could you please add links to the Robert text tester and the blackhole Thanks in advance
@wplearninglab
2 жыл бұрын
For sure, here you go: Robots.txt Google Search Console link: support.google.com/webmasters/answer/6062598?hl=en Black Hole for Bad Bots tutorial: kzread.info/dash/bejne/Yn913Kl9eNSTYbA.html
What is your favorite contact form and spam blocker for that form?
@wplearninglab
2 жыл бұрын
Happy forms is pretty awesome and the free version is quite powerful. For free forms I also like Contact Form 7. I know it's old and a bit clunky, but you can build out lots of simple form types and there are plenty of addons for more functionality. If the form needs to have calculations done on data entered in the fields I like Gravity Forms. I like Recaptcha v3 and/or honeypot fields to reduce spam. ..I hope that helps, let me know if you have any further questions. Thanks for watching!
How to make a sitemap in 2022 wordpress ?
Hi! How do I edit the robot txt file that's blocking Google to crawl my pages?
@Cognitoman
Жыл бұрын
Should be in the public part of the website
hello, only the homepage of my site comes up in google searches. None of the articles on my site appear in google searches. Do you have any idea about the reason? I registered the property on google search console, submitted the sitemap to google. I checked the robots.txt file but I can't seem to solve this problem in any way.
@budsgrower9970
2 жыл бұрын
I have the same problem. I hope someone will answer
It's the robot.txt make your blog post show in search results
Hi! Do you know what could be happening with my file robots.txt? Every so often it automatically switches to user-agent: * Disallow:
@mauwiks
Жыл бұрын
Mine too
@Cognitoman
Жыл бұрын
Prolly something with your server from your hosting company
Do you have any video to how to make money as wordpress developer
@wplearninglab
2 жыл бұрын
Hi Sidhant, I don't at the moment. From my experience, you'll need to (1) have the development skills, which it sounds like you have and then (2) you have to go out and find clients. Part 2 is the hardest for most people. You have to go where your ideal clients meet up (conferences, meet ups, etc.) or you have to go door to door with an offer they can't refuse.