As a practicing SEO consultant I heard one doubt from the webmasters as why should they insert the robots.txt file on the root directory of their site. Their point is they don’t need to restrict spiders from any part of their website and why should they insert a file which gives full permission to spiders. […]