Restricting web spiders - Joomla! Forum - community, help and support


hi - developing new sites online , don;t want them spidered - need add robots.txt stop them being spidered until ready?

just add these line below robots.txt file plus others want.

code: select all

user-agent: *
disallow: /administrator/
disallow: /cache/
disallow: /components/
disallow: /editor/
disallow: /help/
disallow: /images/
disallow: /includes/
disallow: /language/
disallow: /mambots/
disallow: /media/
disallow: /modules/
disallow: /templates/


cheers
paul





Comments

Popular posts from this blog

Flip address is out of range arduino uno r3

Arduino DUE ADC to DAC Piezo 30 Khz-100Khz

Indesign and MathType fonts