# All robots will spider the domain User-agent: * Disallow: # Disallow directory /xnet/ User-agent: * Disallow: /xnet/ # Disallow directory /resources/ User-agent: * Disallow: /resources/