J

Your analogy is slightly incorrect.

Putting your files on a webserver without any form of access restriction, more closely resembles leaving your car in an airport or major metropolitan parking lot WITHOUT LOCKING THE CAR DOORS, while you go on vacation. And if you don’t want your car stolen, then you definately should NOT do that.

I run alot of tutorial and article directory sites so I deal with people scraping my content on an hourly basis.

When I put something on any of my webservers that I do not want being publicly used, or scraped, then I use any number of well-known methods to restrict access to the data. Htaccess files would be one general example.

When I see what I know are certainly scraper bots (besides the search engines, who are generally the biggest resource hogs I deal with) I allow them to scrape anything to which I already allow public access (aka UNRESTRICTED data). In the event that a particular bot begins to use too many resources, I simply ban the bot. I choose to do this manually, but there are quite a few extremely good and free pieces of software that will ban overzealous bots with a great deal of automation.

If you don’t want unrestricted access to your data, you can “lock your doors”. So, that being said…

“Greg – 1/6/2004 10:46:00 PM

If you dont want people to use your html, dont put it on a web server. You yanks are truly amazing”

And for those who still insist on having all their data publicly accessible but wish to continue to get upset when they find their servers being visited by scraper bots, I advise that you add a T.O.S. section to your website clearly forbidding automated access to your website (this is what google,yahoo,msn,etc do). It won’t stop the scrapers, but it will give you a much more definitive legal standing. Of course if you do that, then I guess Google and the other two main engines will be breaching your T.O.S. everytime they come along to perform some more automated spidering and scraping. It’s a mad mad world…

Now that I’ve said all that… don’t add a scraper to newsgator. It would be bloat more than anything else.

Its kinda like putting a drinking fountain in a public park using public water supply and then getting mad if anybody tries to take a drink.

1. How are you supposed to create a search engine without screen scraping part of the content?

2. I use Google’s cached pages feature all the time. Does anyone else?

With that said, perhaps there is a way to advertise for them within our applications; possibly an extra meta tag with some ad info.

Right on. And if you don’t want people to steal your car, don’t go leaving it in a parking lot.

http://dictionary.reference.com/search?q=literally