Block Search Engine Indexing of CloudFront Content with a Custom Response Headers Policy
If, like me, you use AWS CloudFront as a CDN to host content stored in an S3 bucket, you might not necessarily want search engines to index that content. When researching a solution to this problem for myself, I found plenty of forum discussions and blog posts suggesting you can accomplish this with a simple robots.txt file stored at the root of your S3 bucket. Google’s robots.txt documentation warns against doing this, however:
• 2 min read