Feature #14798
Updated by Mathias Schreiber over 9 years ago
I've got a handful of those pdf files that I don't want indexed. So, to satisfy google, and other search engines, I use a robots.txt file so they aren't indexed. My requested feature is, getting typo to honor robots.txt files, and skip indexing files listed in there. (issue imported from #M1170)