Comments on: Referrer abuse?

By: Greg Reinacker

Greg Reinacker — Thu, 06 Feb 2003 16:14:07 +0000

Now here's an interesting incentive to remove the referrer althogether. This RSS link (http://www26.brinkster.com/hch946cn/weblog/rss.xml) will return a 403 Forbidden error if the referrer is set to anything other than blank or something on its own site, with an error that says "This file cannot be directly accessed from a remote site, but must be linked through the Brinkster Member's site." Wow.

By: Joe Madia

Joe Madia — Sat, 01 Feb 2003 17:47:08 +0000

Just to clarify… my vote is for the referrer to be left blank in all cases.

By: Mike Gunderloy

Mike Gunderloy — Sat, 01 Feb 2003 16:52:27 +0000

Well, here’s a vote for letting the user set. Given the general overall state of RSS, I find getting on a “standards compliance” crusade in connection with RSS absurd. As someone who runs an RSS feed, I’m interested in who’s reading it as well as in who’s linking. That information I can get from customized referrers.

If you don’t want to see RSS aggregators in your stats, just tell your stats program to ignore hits on your RSS feed.

By: Dare Obasanjo

Dare Obasanjo — Sat, 01 Feb 2003 16:52:18 +0000

Yup, that’s my vote.

By: Greg Reinacker

Greg Reinacker — Sat, 01 Feb 2003 16:25:37 +0000

Ok…so are you guys (other than Joe) voting that the referrer field should be blank in all cases?

By: Steve Makofsky

Steve Makofsky — Sat, 01 Feb 2003 15:46:05 +0000

I agree with Dare – Now when I analyze my January stats, I have 4033 hits from “http://radio.userland.com/newsAggregator” and 2052 from “http://www.rassoc.com/newsgator”. Doesnt really help me discern any information about who’s linking to me.

By: Dare Obasanjo

Dare Obasanjo — Sat, 01 Feb 2003 09:41:50 +0000

I personally can’t stand the fact that people fill my referrer logs with bogus referrer links. I now have to figure out how to configure Webalizer to calculate referrer stats while ignoring RSS aggregators.

By: Joe Madia

Joe Madia — Sat, 01 Feb 2003 09:40:08 +0000

Whoops… my comment was written in regard to “retrieving pages from the target site” and not “retrieving the RSS itself” even though your original post was very clear. Sorry about that!

After adjusting my brain back to “retrieving the RSS feed itself”, I now think you should omit the referrer entirely. The spec seems pretty strong on the point that Referrer fields should only be used for sources that have valid URIs. Since that’s not the case with aggregators (at least not yet), I would omit the referrer. Also… I would vote that giving users explicit control over the Referrer field should be avoided for the same reasons.

By: Greg Reinacker

Greg Reinacker — Sat, 01 Feb 2003 07:23:59 +0000

Joe, which RSS feed are you referring to? If I’m reading news, do you mean I should use my own RSS feed as the referrer? That doesn’t seem to make sense.

And if I were retrieving pages from the target site, I can see how their RSS URI might make sense…but we’re talking about the referrer for when we’re retrieving the RSS itself.

Or did I just completely misunderstand what you were saying?

By: Joe Madia

Joe Madia — Sat, 01 Feb 2003 06:44:13 +0000

I believe that sending the URI of the RSS feed as the referrer would be the most consistent with the spec. The RSS URI seems to be a perfect match for “the address (URI) of the resource from which the Request-URI was obtained” as stated in Section 14.36 of RFC 2616.

One benefit of referrers mentioned by the spec explicitly is the ability to trace broken links back to their source. Using the RSS URI as the referrer would preserve this benefit. No other option really would.

The spec also says, “The Referer field MUST NOT be sent if the Request-URI was obtained from a source that does not have its own URI, such as input from the user keyboard.” I would avoid user-defined referrers since there would be no guarantee that they represent valid resources. More importantly, I would avoid them because they don’t seem to be consistent with the intent of the spec.

Just my two cents, of course!