General Category > Meta Forum

This year, September 70.000 more hits to Joses's Forum then last year

<< < (2/3) > >>

Theo Gottwald:
Of course there are spiders, looking at the absolute numbers these are to some percentage spiders.
But the increasing part are users, not spiders. As the spiders tend not to double in such a short time.
Else there would have to be a nest of them somewhere :-).

Eros Olmi:
Well, not so sure.
Last year for example (during the time Microsoft / Yahoo dispute) there was a really massive number of connections from Yahoo spiders. In some days I had more than 100 simultaneous Yahoo spiders connected for hours. I had to write to Yahoo a mail asking to stop such a massive connection because they where stealing my server bandwidth.
Quite surprisingly they replied to me. This is the reply I got:


--- Quote ---Subject: Reporting a Problem

Hi there. I manage a little web site at
www.thinbasic.com and community.thinbasic.com
Problem is that your spiders are visiting my site so much frequently
that a lot of my limited bandwidth is consumed by your spiders. Since
about two weeks I can see even more that 1 hundred spiders present
almost at the same time scanning all posts in our forum.
I have a limited band with my provider and my users have to support some
delays when using our sites.
I can understand your needs but please understand mine.

Hope someone will read this post and reply to me soon.

Thanks
Eros

--- End quote ---


--- Quote ---Hello Eros,

Thank you for writing to Yahoo! Search.

We apologize for the excess traffic to your site. In a continuing effort
to improve our index freshness and comprehensiveness we are bringing
online additional crawler machines to prepare them for production.

To limit the amount of traffic your site receives, you can place a
crawl-delay in your robots.txt, specifying the number of seconds each
crawler will wait between each request. Please check out the following
pages for detailed information about this:

   http://help.yahoo.com/help/us/ysearch/slurp/slurp-03.html

   http://www.ysearchblog.com/archives/000078.html

Again, my apologies for the inconvenience and we appreciate your
patience as we're improving Yahoo! Search.

Thank you again for contacting Yahoo! Search.

Regards,

Yasmen

Yahoo! Search Customer Care
--- End quote ---

Of course "robots.txt" was not the solution because in a forum usually you do not limit the search of a spider (while in a standard web site yes).
In any case after few days they stopped but in the meantime they consumed many GB of my precious and payed bandwidth.

Google is also surfing/scanning all the day but with very few simultaneous spiders. The same for MSN

Well, the conclusion is that you have to pay attention to numbers before giving them a meaning.
Another conclusion can be that the popularity of José forum is for sure increasing because my problem was last year so if also this site was suffering by the same problem it was last year and not this year so figures of this year are more precise than the last year.

Eros

Patrice Terrier:
There are also a few more redirection, at least from my new web site  ;D

On that aspect that would be interresting to know where the people are coming from (i mean which country).

...

José Roca:
 
Last year, there was an inordinate increase of pages read because of the "crawler's war". Then, in January of this year it dropped to 77,133, but since then it has been increasing steadily (I guess that many of them will be French, since Patrice has been doing an extraordinary promotion). However, what I hope is that more PBers lose their fear to the Windows API and begin to use it. With the upcoming release of FireFly 3 the number of users of the WinAPI headers will increase. Regarding users of other programming languages, the stuff that more has interested them is the one related with GDI+ because my reference guide for it was the first one fully documenting the GDI+ Flat API (M$ documentation only document the C++ wrapper classes). A PureBasic user adapted my reference guide to French and did a very good job providing tons of examples written in PureBasic.

Theo Gottwald:
Purebasic?
Can you post a link to this, Jose. I need to use Purebasic64 for a soon coming Project because PB can actually only generate 32bit Executables while Purebasic is a step ahead about this.

Navigation

[0] Message Index

[#] Next page

[*] Previous page

Go to full version