mirrors/nitter
1
0
Fork 0
mirror of https://github.com/zedeus/nitter.git synced 2025-03-28 13:16:51 +00:00
Code Issues Packages Projects Releases Wiki Activity
1105 commits 4 branches 0 tags 7.6 MiB
Commit graph

4 commits

Author SHA1 Message Date
Zed 6c4c691fe7 Add Crawl-delay to robots.txt 2023-01-09 00:12:07 +01:00
Zed 138826fb4f Fix Twitterbot rule in robots.txt 2022-06-04 17:55:35 +02:00
Zed dfb2519870 Explicitly allow Twitterbot to generate previews 2022-06-04 17:52:47 +02:00
minus c543a1df8c
Block search engines via robots.txt () 
Prevents instances from being rate limited due to being senselessly
crawled by search engines. Since there is no reason to index Nitter
instances, simply block all robots. Notably, this does *not* affect link
previews (e.g. in various chat software).
 2022-06-04 17:48:25 +02:00