全球主机交流论坛

标题: 我的buyvm小机机要被蜘蛛们抓死了，怎么都一起来了 [打印本页]

作者: fwt0125 时间: 2011-4-7 22:54
标题: 我的buyvm小机机要被蜘蛛们抓死了，怎么都一起来了
220.181.94.216 - - [07/Apr/2011:22:51:47 +0800] "GET /play/?28384-0-0.html HTTP/1.1" 200 14451 "-" "Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)"
66.249.67.178 - - [07/Apr/2011:22:49:54 +0800] "GET /good-2807514066.jsp HTTP/1.1" 200 72 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.67.155 - - [07/Apr/2011:22:51:55 +0800] "GET /uploads/allimg/1/1a/1282367566.jpg HTTP/1.1" 200 14612 "-" "Googlebot-Image/1.0"
220.181.94.216 - - [07/Apr/2011:22:51:57 +0800] "GET /play/?28383-0-0.html HTTP/1.1" 200 14825 "-" "Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)"
220.181.94.216 - - [07/Apr/2011:22:52:07 +0800] "GET /play/?2494-0-60.html HTTP/1.1" 200 38724 "-" "Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)"
61.135.249.210 - - [07/Apr/2011:22:52:08 +0800] "GET /content/?24009.html HTTP/1.1" 200 12868 "-" "Mozilla/5.0 (compatible; YoudaoBot/1.0; http://www.youdao.com/help/webmaster/spider/; )"
220.181.94.216 - - [07/Apr/2011:22:52:18 +0800] "GET /play/?28726-0-0.html HTTP/1.1" 200 15690 "-" "Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)"
220.181.108.156 - - [07/Apr/2011:22:52:18 +0800] "GET /good-4700735893.jsp HTTP/1.1" 200 50206 "-" "Baiduspider+(+http://www.baidu.com/search/spider.htm)"
220.181.94.216 - - [07/Apr/2011:22:52:27 +0800] "GET /play/?28577-0-0.html HTTP/1.1" 200 18282 "-" "Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)"
220.181.94.216 - - [07/Apr/2011:22:52:37 +0800] "GET /play/?28356-0-0.html HTTP/1.1" 200 13461 "-" "Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)"
66.249.67.155 - - [07/Apr/2011:22:52:39 +0800] "GET /uploads/allimg/f/f2/1301745628.jpg HTTP/1.1" 200 6542 "-" "Googlebot-Image/1.0"
220.181.94.216 - - [07/Apr/2011:22:52:47 +0800] "GET /play/?28360-0-0.html HTTP/1.1" 200 13506 "-" "Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)"
123.122.94.119 - - [07/Apr/2011:22:52:22 +0800] "GET /attachments/lyh/%E8%80%81%E7%BD%97%E8%AF%AD%E5%BD%95_%E9%AB%98%E5%B1%B1%E6%97%8F.wma HTTP/1.1" 200 2926202 "-" "NSPlayer/11.00.6002.18392 WMFSDK/11.00.6002.18392"
220.181.108.158 - - [07/Apr/2011:22:52:51 +0800] "GET /content/?1930.html HTTP/1.1" 301 328 "-" "Baiduspider+(+http://www.baidu.com/search/spider.htm)"
220.181.108.106 - - [07/Apr/2011:22:52:51 +0800] "GET /content/?1838.html HTTP/1.1" 200 15626 "-" "Baiduspider+(+http://www.baidu.com/search/spider.htm)"
220.181.108.86 - - [07/Apr/2011:22:52:51 +0800] "GET /content/?1930.html HTTP/1.1" 200 13976 "-" "Baiduspider+(+http://www.baidu.com/search/spider.htm)"
220.181.108.85 - - [07/Apr/2011:22:52:55 +0800] "GET /content/?1893.html HTTP/1.1" 301 328 "-" "Baiduspider+(+http://www.baidu.com/search/spider.htm)"
220.181.94.216 - - [07/Apr/2011:22:52:57 +0800] "GET /play/?28361-0-0.html HTTP/1.1" 200 13756 "-" "Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)"
220.181.108.111 - - [07/Apr/2011:22:52:59 +0800] "GET /content/?1893.html HTTP/1.1" 200 14633 "-" "Baiduspider+(+http://www.baidu.com/search/spider.htm)"

作者: 刹那芳华 时间: 2011-4-7 22:54
屏蔽啊

作者: fwt0125 时间: 2011-4-7 22:56

原帖由 刹那芳华 于 2011-4-7 22:54 发表
屏蔽啊

不知道说什么好

作者: xtmp 时间: 2011-4-7 22:56
sougou的spider屏蔽没用，它根本不遵守robot

作者: walkman660 时间: 2011-4-7 23:03
直接把搜狗IP拉黑

作者: l12ab 时间: 2011-4-7 23:07
在user-agent里面屏蔽spider

作者: Host.HowPick 时间: 2011-4-7 23:09
搜狗是谁家的？我只知道 gg和bd

作者: mslxd 时间: 2011-4-7 23:11
sogou是这样的呃，你要给它吃别的。。。

作者: 糕手 时间: 2011-4-7 23:49
提示: 作者被禁止或删除内容自动屏蔽

作者: wdlth 时间: 2011-4-7 23:59
上缓存吧

作者: ypandjp 时间: 2011-4-8 00:58
怎么禁用蜘蛛。。

作者: walkman660 时间: 2011-4-8 01:01

原帖由 ypandjp 于 2011-4-8 00:58 发表
怎么禁用蜘蛛。。

http://***/robots.txt
User-agent: *
Disallow: /

欢迎光临全球主机交流论坛 (https://sunk.eu.org/)