User-agent: * Allow: / #deny non-content folders Disallow: /images/ Disallow: /search/ Disallow: /sitemap/ #deny code folders Disallow: /_errors/ Disallow: /_includes/ Disallow: /_logs/ Disallow: /_scripts/ Disallow: /_styles/