And indeed you can't even guarantee that a *normal* client will give you a valid HTTP_REFERER value. Some people sit behind proxies or firewalls that are so paranoid that they won't send the referer value. You have to treat referer as a "friendly thing to have when it works" but you should *never* rely upon it.
And don't forget search engines: They don't provide referer info, but you surely don't want to cut them off.
But I would say that, if the referrer is blank, you should allow the access. Just as a for-instance, it will be blank if the user clicked on an entry in his/her "favorites".
Now, that does mean that it's trivial for hackers to give you a blank referrer, but for the reasons I already noted there's not much you can do about that, anyway.