Restricting web spiders - Joomla! Forum - community, help and support
hi - developing new sites online , don;t want them spidered - need add robots.txt stop them being spidered until ready?
just add these line below robots.txt file plus others want.
cheers
paul
code: select all
user-agent: *
disallow: /administrator/
disallow: /cache/
disallow: /components/
disallow: /editor/
disallow: /help/
disallow: /images/
disallow: /includes/
disallow: /language/
disallow: /mambots/
disallow: /media/
disallow: /modules/
disallow: /templates/
cheers
paul
Comments
Post a Comment