Blank HTTP_REFERER

[Philip M]

I hope I have posted this in the right forum.

On my site I have an on-line order form which transmits using
a modified version of Formmail.

For security reasons, if a form is submitted with the HTTP_REFERER blank (or not my domain) then access is denied and an error message is returned. I also record details of submissions (successful or otherwise) in an error file. (Some failed submissions arise because the form is not correctly completed).

Here is a fragment of the PERL script:-

open (ERRORFILE, '>>Errors.fil' );
print ERRORFILE ("$date\n");
print ERRORFILE ("$ENV{'HTTP_REFERER'}\n");
print ERRORFILE ("$ENV{'REMOTE_ADDR'}\n");
print ERRORFILE ("$ENV{'REMOTE_HOST'}\n");
print ERRORFILE ("$ENV{'HTTP_USER_AGENT'}\n\n");
close (ERRORFILE);

Examining the Errorfile shows that sometimes customers fill in the
form properly but access is denied them as the HTTP-REFERER is blank.

Here is an example:-

Sunday, October 6, 2002 at 19:41:58

62.252.224.8
cache5-leed.server.ntli.net
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; Q312461; Hotbar 4.1.4.0; .NET CLR 1.0.3705)

Access was denied and an error message screen was displayed.

The customer then did something to correct the situation and re-submitted his
form successfully:-

Sunday, October 6, 2002 at 19:42:24
http://www.mydomain.co.uk/orderform.html
62.252.224.8
cache5-leed.server.ntli.net
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; Q312461; Hotbar 4.1.4.0; .NET CLR 1.0.3705)

Note that it took only 26 seconds for the customer to read the error message screen (several lines of text), and do whatever he did to reinstate the HTTP_REFERER.

My question is:-

a) How does it arise that the HTTP_REFERER in the browser
address box can be a blank?

b) What did my customer do to re-instate this in his browser?

Is it something to do with the browser configuration, a firewall, or
anti-virus program?

Any comments or advice would be appreciated.

[Feyd]

It is probably a firewall or browser privacy standards. HTTP_REFERER is contained within the browser and you shouldn't ever really rely on it in order to perform an action, at the very least you should have the ability to detect if there is no refererer and use 'not referred' or something similiar instead of erroring out.

As for 're-instating' it, that seems a bit odd...does your error page alter the referer variable with other information onerror? Is a form re-presented to the user that does not contain this check? Maybe the user turned off their firewall in order to process the request (unlikely). It might help to get a little more information...

[Philip M]

Feyd - I now learn that there are browsers such as Opera and "privacy tools" such as Webwasher where the user can elect not to transmit the referer URL (why should he wish to do this???) but the user can quickly switch this feature off. I guess this is what has happened here.

On my site I found that someone was stealing my bandwith and using my formmail.pl to send spam. I therefore altered the script so as to only accept form submissions from a defined referer page (not a blank), and send mail only to one address (mine!). this has cured the problem. My ISP has made all users upgrade to formmail.pl version 1.92 as the previous version had security flaws in it.

[Fataqui]

Hi


REFERER = a embeded string sent by the browser, stating the last page the browser viewed!


If you trusting [REFERER] for any reason that is important, you should not, because it can be faked very easy!



1. Some browsers limit access to not allow REFERER to be passed!

2. Type a address in the address bar will not pass the REFERER

3. open a new browser window will not pass the REFERER, because REFERER = NULL


F!

[realisis]

Philip, you're right about Opera... I believe recent Mozilla builds can also withhold the Referer but don't quote me on that (going by memory).

As Feyd pointed out, firewalls can also suppress the Referer, as does Norton's Personal Firewall.

"(why should he wish to do this???)" - because some users do not want to be tracked across domains.

[toolkit]

Philip M,

It's VERY easy to spoof the HTTP_REFERER var and therefore fool your script.

1 easy way:

open telnet, and try the following:

telnet www.yoursite.com 80 (press return)
GET /index.htm HTTP/1.0 (press return)
Referer: http://www.anyurlyouwant.com (press return)
(press return again)

You'll see the referrer you inputted in your access_log.

It's much more secure to hard code the email address you want to send the form to into the script, and don't bother about checking the referrer.

And yes, I know this from experience

[Philip M]

Originally posted by toolkit
Philip M,

It's VERY easy to spoof the HTTP_REFERER var and therefore fool your script.

1 easy way:

open telnet, and try the following:

telnet www.yoursite.com 80 (press return)
GET /index.htm HTTP/1.0 (press return)
Referer: http://www.anyurlyouwant.com (press return)
(press return again)

You'll see the referrer you inputted in your access_log.

It's much more secure to hard code the email address you want to send the form to into the script, and don't bother about checking the referrer.

And yes, I know this from experience

Thanks, Toolkit. You have explained how to spoof the referer, (which I did not know about) and have to say that I am well aware of the need to hard code the recipient of the email into the perl script.

But I am still not entirely clear why my perfectly ordinary customer(s) who are not computer experts or Telnet users, nor are they trying to spoof anything, can end up with a blank referer (and hence have their form submissions rejected).

Yes, Opera offers the option of masking the referer, but they are not using Opera! Likewise Webwasher or similar 'privacy tools' as far as I can see.

[Mouldy_Goat]

Hi Philip M,

If the referrer variable is blank, it's probably because a user wasn't referred.

If a user was just to type in the address of your form without being referred to it via a link then the browser would not have any HTTP_REFERER header to send.

Also, as pointed out, there are some browsers which will not provide the header, and some firewalls that will block it.

Also, don't assume that your user wasn'ta computer expert.. he may have known enough to realise what was going wrong and sort it out.

[Philip M]

Originally posted by Mouldy_Goat
Hi Philip M,

If the referrer variable is blank, it's probably because a user wasn't referred.

If a user was just to type in the address of your form without being referred to it via a link then the browser would not have any HTTP_REFERER header to send.

Thanks, Mouldy_Goat! I am now in an area where I am woefully ignorant, but I have to say that in fact my form submits fine when I simply type in the form's URL as opposed to reach it via the index or another page.

Surely the referer is the page (URL) of the form itself?