Search Engines Eat Resources
I love it when search engines finally decide to index the site. More listings, greater chance of people finding us! But just today I checked the server logs and discovered that about 20 minutes of processor usage was used for just ONE search engine!
That's a lot of time when you think about it. So, being the curious type I am, I checked the logs and discovered (hopefully) why this was: the search engine had indexed the exact same 5MB 30 second load page 82 times!
So for now, a change was made, the page is only available from the index page. If the high processor usage hits again, the large page will be altered into many smaller pages. There are times I wish there was a way to tell a seach engine "it's the same page, you don't need to re-examine it"
That's a lot of time when you think about it. So, being the curious type I am, I checked the logs and discovered (hopefully) why this was: the search engine had indexed the exact same 5MB 30 second load page 82 times!
So for now, a change was made, the page is only available from the index page. If the high processor usage hits again, the large page will be altered into many smaller pages. There are times I wish there was a way to tell a seach engine "it's the same page, you don't need to re-examine it"
<< Home