I have a mobile version of a site, which is accessed through auto-detection of mobile user-agents.
Since the content is virtually the same as the regular site, I didn’t want it indexed by search engines, both to prevent site visitors from landing on a mobile site, and to avoid SEO penalties for duplicate content.
I used the following RewriteCond/RewriteRule to deliver an alternate robots.txt file for the mobile site.
The alternate robots.txt file is:
In this case, the site is run by eZ publish, but the same strategy should work with any other mobile site that is run as a subdomain.