What is the ROBOTS.txt file and how to create one easily?

There are many factors that affect your website’s SEO and robots.txt is one of them. Today we are going to share why robots.txt file is important for your website and how to create one easily?

Robots.txt file plays a very important role in your SEO. This file is created to tell the search engines like Google on how to crawl the pages of the website or which pages to crawl and which not to and this is how your website pages are listed in the search engine.

But before we create a robots.txt file, it would be better to check if your site is having this file or not because if you have got your website developed from a professional company which is good at SEO than the file should be there in the root directory of your website but still it is good to check if you have that file or not.

How to check Robots.txt file exists or not?

There is a simple method to check if your site has that file or not? Just type in the browser

www.example.com/robots.txt

Change example.com with the name of your website … like mine is starcindia.in so it would be www.starcindia.in/robots.txt

If you see a 404 error that means the file doesn’t exist on your website so you need to create one.

How to create a robots.txt file?

Creating a robots.txt file is not that tough unless you know which pages you want the search engine to crawl and which not to and if you are using a WordPress you can include posts and pages both. Also if you are using WordPress, the robots.txt file is created automatically with the following credentials which you can change as per your requirement

User-agent: *

Disallow: /wp-admin/

Allow: /wp-admin/admin-ajax.php

Now let’s understand the lines…

User-Agent – means web crawlers or we can say search engines…

*  – Means everything … it is like a *.* command of dos

Disallow – means you don’t want web crawlers to crawl the specific page or post. Like in the above case, we don’t want /wp-admin/ page to be crawled.

Allow – definitely means allowing crawlers to crawl

Now if you want to create robots.txt file yourself than just create a simple blank text file and add the contents as mentioned above and upload the same on your website’s root directory using FTP