# Disallow all crawlers access to all pages except media files. User-agent: * Disallow: / Allow: /arquivos Allow: /Arquivos Allow: /scripts Allow: /Scripts Allow: /js Allow: /assets Allow: /_v/public/assets