Need to get information about visitors

[agent186]

Hi everyone,
I need help getting information about the users who visit my site. The idea seems very simple, but I can't find any help on it anywhere.

What I need to do is write a script (prefer cgi in Perl, but could also use JavaScript and cgi combined) that will run as soon as any page on the site is hit, and capture the referrer, the browser type, the current URL, and if possible the operating system of the user, and put those values into a database.

Getting the data into the database is the easy part. The part I can't figure out is how to get the information about the user's system directly through cgi. I know how to get it through JavaScript, but can't get the JavaScript to pass the data to the cgi script to get it in the database.

I wrote a simple script that I thought would work, but all it does is produce an Internal Server Error. For now it just writes to a file, I'll have it write to the database once it works.


!#/usr/bin/perl;
use strict;
use CGI qw(:standard);

open (LOG, '>>log.txt' );
print LOG ("$ENV{'HTTP_REFERER'}\n");
print LOG ("$ENV{'REMOTE_ADDR'}\n");
print LOG ("$ENV{'REMOTE_HOST'}\n");
print LOG ("$ENV{'HTTP_USER_AGENT'}\n\n");
close (LOG);



I know this can be done because there are a thousand companies out there that want to sell you this service. I need something far simpler, so I don't want to pay the price.

Any advice is appreciated, even if you tell me a whole differant way to do it. Thanks in advance.

[KevinADC]

this line is not correct:

!#/usr/bin/perl

should be:

#!/usr/bin/perl

the shebang line starts with !# and does not end with a semi-colon, there is also ne need to use () in your print commands:

!#/usr/bin/perl
use strict;
use CGI qw(:standard);

open (LOG, '>>log.txt' );
print LOG "$ENV{'HTTP_REFERER'}\n";
print LOG "$ENV{'REMOTE_ADDR'}\n";
print LOG "$ENV{'REMOTE_HOST'}\n";
print LOG "$ENV{'HTTP_USER_AGENT'}\n\n";
close (LOG);

but if this is really running as a CGI you need to print something back to the browser otherwise you will still get an error, so:

!#/usr/bin/perl
use strict;
use CGI qw(:standard);
print header,start_html;

open (LOG, '>>log.txt' ) or die "Can't open log: $!";
print LOG "$ENV{'HTTP_REFERER'}\n";
print LOG "$ENV{'REMOTE_ADDR'}\n";
print LOG "$ENV{'REMOTE_HOST'}\n";
print LOG "$ENV{'HTTP_USER_AGENT'}\n\n";
close (LOG);

print 'finished',end_html;

[agent186]

Thanks Kevin,
The !# thing was a mistyping on my part. I didn't cut that part of the script by mistake, and so I typed it in.

I tried your script, but still got an Internal Server Error. When I looked at the Apache log I found that it had written the output from the script to the Apache log file instead of the browser. The log.txt file was created, but there's nothing in it. Another weird thing is that the script keeps writing to it every three seconds even after the browser is closed. So now I have file with 2000 some blank lines in it. (it finally stopped after I deleted the cgi script). None of this makes any sense. Should I be approaching this an entirely differant way?

[KevinADC]

well, how are you invoking the script? What makes it start to run?

[agent186]

I'm invoking it directly. Typing in www.myurl.com/my.cgi. Is there another way to invoke it as soon as the page loads (i.e. No form submits or clicks required)?

[KevinADC]

When I looked at the Apache log I found that it had written the output from the script to the Apache log file instead of the browser. The log.txt file was created, but there's nothing in it. Another weird thing is that the script keeps writing to it every three seconds even after the browser is closed. So now I have file with 2000 some blank lines in it. (it finally stopped after I deleted the cgi script). None of this makes any sense.

What gets printed to the Apache log? The environment variables or something else? I don't know why that script would just keep running over and over if it was invoked once from the URL (in a browser), it shouldn't. If the blank lines are in the file it's possible that the ENV variables are not getting defined. Try this script:

#!/usr/bin/perl
print "Content-type: text/html\n\n";
foreach my $var (sort keys %ENV) {
print "<b>$var</b> = $ENV{$var} <br>\n";
}
exit;

and see if it prints the ENV variables to the browser.

[agent186]

I am trying a slightly differant approach now. I wrote a JavaScript file to gather the information and then write a transparant gif. The transparent gif is acctually written by the cgi script, and passed the parameters that way. So, the js code is as follows... (the variable setting part is not included here)

document.write("<img src='http://www.mysite.com/scripts/collect.cgi?width="+visitor_width+"&height="+visitor_height+"&doc_location="+location.pathname+location.search+"&doc_title="+doc_title+"&doc_refer="+doc_refer+"' border=0 height=1 width=1>");


The cgi code for now just attempts to write each parameter to a file like so ...

#!/usr/bin/perl

use CGI::Fast;
use strict;

my $q = new CGI;
#set transparent gif
print $q->header(-type=>'image/gif',-expires=>'Tue, 01 Jan 1980 11:11:11 GMT');
printf("%c%c%c%c%c%c%c%c%c%c%c%c%c%c%c%c%c%c%c%c%c%c%c%c%c%c%c%c%c%c%c%c%c%c%c%c%c%c%c%c%c%c%c",71,73,70,56,57,97,1,0,1,0,128,255,0,192,192,192,0,0,0,33,249,4,1,0,0,0,0,44,0,0,0,0,1,0,1,0,0,2,2,6 8,1,0,59);

#get parameters
my @params = $q->param();

#open file and write all parameters to it
open(LOG, "> output.txt");
print LOG scalar(@params) . "\n";
foreach my $parameter (@params)
{
print LOG "$parameter \n";
}
close(LOG);
exit;


I am getting close with this one. I get an error in the Apache log that says ...

FastCGI: incomplete headers (0 bytes) received from server "/var/www/mysite/scripts/collect.cgi", referer: http://www.mysite.com/index.html

It also may be worth noting that the page hangs for 30 seconds before loading completely.
The output.txt file just says "0" (the number of parameters recieved).

Am I not setting the headers correctly in the cgi script?

[KevinADC]

you are using FastCGI, that might explain the looping of the script in the first place, you may have to do an explicit exit command on scripts invoked though FastCGI, but I am not familiar with how that works so you will need to check if that is true or not.

I'm confused by your new code. Are you trying to print the gif and gather the query-string data from the just printed gif at the same time?

[KevinADC]

did you try the little script I posted to see what ENV variables get printed? Did it work?

[agent186]

Kevin,
Thanks for all of your help. I finally found that the problem lied with FastCGI. When I first tried your script with the envionment variables it didn't work at all. That's when I knew there was something bigger going on.
I disabled FastCGI in the Apache config file, and it worked fine (both your script and mine). So I re-enabled it and used only Fast::CGI in my script not both CGI and Fast::CGI. I guess if you're running fastCGI, you can't run just plain CGI scripts.
It appears to be working, and I'm getting the data I need. Thanks for all of your help!

[KevinADC]

Ah, makes sense now.