What is Robot.txt File?
Robot.txt file is a text file which instructs the search engines robots/crawlers/spiders which folder or page of any a website should not be accessed by them. You can disallow any search robot from some of your website’s specific folders or pages. They will not be able to access those pages which you’ve specified in the robots.txt file.
How to create a robots.txt file?
The syntax of creating this file is quite simple, just open notepad & save the file, give it a name as robots.txt and write
following two lines:
User-agent: *
Disallow: /
The above example is the simplest syntax for creating a robots.txt file, and the the first line “User-agent*” means that all crawlers/robots are allowed to access all the pages/folders of the website, however you can specify a folder/page in the below line “Disallow:/” you just need to write that page/folder after the / backslash.
if you wanna disallow search robots from any of your pages or folders then just add that page address after the back slash of Disallow line. Like this: Disallow: /kaka.html. Upload that txt file to your root directory of your website and you’re all done! in Blogger you can find this file by default by visiting setting page & then search preferences. Also if you want to disallow many pages or folders in the txt file then just do like this :
User-agent:*
Disallow:/images
Disallow:/plugins
Disallow:/wp-admin.php
Disallow:/login.php
In above txt file we disallowed some pages and folders of our site, so now Search robots/crawlers can not access them, and they will neither be indexed nor be crawled.
Video is also here in Urdu to help you better understanding of it.
User-agent: *
Disallow: /
Disallow: /
User-agent:*
Disallow:/images
Disallow:/plugins
Disallow:/wp-admin.php
Disallow:/login.php
