>0 ant @ 2025/07/20 00:49
Many tilde.club members have placed robots.txt files in their ~/public_html/ directories, unaware that they have no effect, because crawlers consult robots.txt only at domain root[1]. Therefore, I have written a script for the consoliation of the individual users's robots.txt files into a unified server-level robots.txt.
This script is now installed and reguarly executed on the tilde.club machine, and everyone is welcome to verify whether their personal robots.txt is properly incorporated:
My latest version of the script is always available at:
Bug requests and feature reports are welcome in the form of diff patches, English prose (or verse, for that matter) to my e-mail (ant [at] tilde.club) or in the #club channel on Newnet IRC[2].
1. :
> The rules MUST be accessible in a file named "/robots.txt" (all lowercase) in the top-level path of the service.
2.
>1 fenris @ 2025/07/21 16:49
>>OP
Wow, thanks a lot for the effort.