Google is very rude.
Nov. 13th, 2003 11:15 am![[personal profile]](https://www.dreamwidth.org/img/silk/identity/user.png){kind=small}
thistlechaserThere is such a thing as too much of a good thing. In the last 13 days, Google's spider has been to my site (Catlove) 473 times, and has used 41.64 MB of bandwidth. The next most-visiting spider came 29 times and used 1.11 MB bandwidth.
As far as I can tell, robots.txt will only keep a spider out or let it in. I thought I recalled that you could restrict them to like "once a month" or something like that, but I can't find any reference to that now...
Can you tell I'm really bored?
As far as I can tell, robots.txt will only keep a spider out or let it in. I thought I recalled that you could restrict them to like "once a month" or something like that, but I can't find any reference to that now...
Can you tell I'm really bored?
