Add a X-Robots-Tag: noindex in the response headers of the content sources
Request: Currently, on multiple websites, a rule is added in the robots.txt file to avoid indexing the content sources paths: Disallow: /pf/api/ However, there is a fundamental difference between authorizing crawling (access) and authorizing index...