The entire site with a forward slash (/):
Disallow: /
A directory and its contents by following the directory name with a forward slash:
Disallow: /sample-directory/
A webpage by listing the page after the slash:
Disallow: /private_file.html
A specific image from Google Images:
User-agent: Googlebot-Image
Disallow: /images/dogs.jpg
All images on your site from Google Images:
User-agent: Googlebot-Image
Disallow: /
Files of a specific file type (for example, .gif):
User-agent: Googlebot
Disallow: /*.gif$
Pages on your site, but show AdSense ads on those pages, disallow all web crawlers other than Mediapartners-Google. This implementation hides your pages from search results, but the Mediapartners-Google web crawler can still analyze them to decide what ads to show visitors to your site.
User-agent: *
Disallow: /
User-agent: Mediapartners-Google
Allow: /
Disallow: /
A directory and its contents by following the directory name with a forward slash:
Disallow: /sample-directory/
A webpage by listing the page after the slash:
Disallow: /private_file.html
A specific image from Google Images:
User-agent: Googlebot-Image
Disallow: /images/dogs.jpg
All images on your site from Google Images:
User-agent: Googlebot-Image
Disallow: /
Files of a specific file type (for example, .gif):
User-agent: Googlebot
Disallow: /*.gif$
Pages on your site, but show AdSense ads on those pages, disallow all web crawlers other than Mediapartners-Google. This implementation hides your pages from search results, but the Mediapartners-Google web crawler can still analyze them to decide what ads to show visitors to your site.
User-agent: *
Disallow: /
User-agent: Mediapartners-Google
Allow: /