parsing string..

[shreyaranga]

Hi There,
I am trying to get a value from a file .

file data....
216.148.244.91 - - [11/Mar/2005:11:05:27 -0500] "GET /tracker.html?retailer=Test+Retailer&order=xyz123&uniqueId=12345&sku=sku-00005&prodName=the%20greatest%20product%20in%20all%20the%20world&qty=1&unitPrice=15.25 HTTP/1.0" 200 - "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"


From this file I have to get 1)retailer 2)order these 2 string with data.
I am abel to get data from file and store it an a array.
After that what i have to do. What is best wat to achive these results.
I appreciate anybody's help.
Thanks

[mlseim]

I am abel to get data from file and store it an a array.
Thanks

Please show us what you have so far ... paste your
script on here. I'm curious how you got the data from
a file and stored it into an array.

Without seeing how you created your array and without
actually testing it ... I'm envisioning something like this:
===============================================

#!/usr/bin/perl

#You've done something like this?
#Read the lines in from a text file to an array.
#
$database = "mytextfile.txt";
open (DB,"<$database");
@ODB=<DB>;
close (DB);

print "Content-type: text/html\n\n";

#Read each line and separate by "&" symbols.
foreach $rec (@ODB){
chomp($rec);
($one,$two,$three,$four,$five,$six)=split(/\&/,$rec);

#so now, $one = 216.148.244.91 - - [11/Mar/2005:11:05:27 -0500] "GET /tracker.html?retailer=Test+Retailer

#then, split $one by question mark
($one_one,$one_two)=split(/\?/,$one);

#so now, $one_two = retailer=Test+Retailer

#then, split $one_two by "=" equal sign.
($temp,$retailer)=split(/\=/,$one_two);

#so now, $retailer = Test+Retailer
$retailer =~ tr/+/ /; #remove the plus symbol

#do the same for the other parts....
}

[shreyaranga]

Thanks for your help,
this is my code>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
#!/bin/perl
print STDOUT "Enterlogfilename\n";
$name = <STDIN>;
print "$name";
$README = $name;
open(README);
#print "$README";

@lines =<README>;
#print @lines;
for($i=0;$i < $#lines;$i++){
$lineText = $lines[i];
($junk,$retailer,$junk2) = split("&" ,$lineText);
print "$retailer\n";
#print "Retailer value:$junk\n";
($j1,$ret) = split("GET /tracker.html?",$junk);
print "Retailer : $ret\n";
print;


}

print "\n no of lines",$#lines + 1;
close(README);

[shreyaranga]

how can we append this to a file...
thanks in before...

[mlseim]

So I'm assuming your script didn't work (that's why you posted it)?

I know this script below works.
I created a data file with three of your lines (made them all unique).
Here is my working example using the script below: http://www.catpin.com/cgi-bin/parsetest.pl

====================================================

#!/usr/bin/perl

$database = "parsetest.txt";
open (DB,"<$database");
@ODB=<DB>;
close (DB);

print "Content-type: text/html\n\n";

#Read each line and separate by "&" symbols.
foreach $rec (@ODB){
chomp($rec);
($one,$two,$three,$four,$five,$six)=split(/\&/,$rec);

#This is the retailer ...
($one_one,$one_two)=split(/\?/,$one);
($temp,$retailer)=split(/\=/,$one_two);
$retailer =~ tr/+/ /; #remove the plus symbol

#This is the order ...
($one_one,$one_two)=split(/\=/,$two);
$order = $one_two;

print "Retailer: $retailer &nbsp;&nbsp; ... &nbsp;&nbsp; Order: $order<br>\n";

}

========================================================

[shreyaranga]

Hi Thanks for your reply...
I got htat i a abel to get the data ina na array and am abel to split it .. finally i ahve to write iti an another file.. how to do that..
I want to store that data in a file like this.... in a two dimension form ... how is it possibel
data
retailerid date orderid
data data data
data1 data1 data1

[mlseim]

What exactly is the stuff in red?

data
retailerid date orderid
data data data
data1 data1 data1

And, it everything in one file, or each line
has it's own file?

Take three of your lines and create what
output you would like to see.

[shreyaranga]

I appreciate your help. I am abel to append a file with data....
this is my log ifle.....

216.148.244.91 - - [11/Mar/2005:11:05:27 -0500] "GET /tracker.html?retailer=Test+Retailer&order=xyz123&uniqueId=12345&sku=sku-00005&prodName=the%20greatest%20product%20in%20all%20the%20world&qty=1&unitPrice=15.25 HTTP/1.0" 200 - "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"

216.148.244.86 - - [11/Mar/2005:11:05:27 -0500] "GET /tracker.html?retailer=Test+Retailer&order=xyz123&uniqueId=12345&numRecords=1&productsTotal=15.25&grandTotal=16 HTTP/1.0" 200 - "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"

216.148.244.90 - - [11/Mar/2005:11:05:29 -0500] "GET /tracker.html?retailer=Test+Retailer&order=xyz123&uniqueId=67890&sku=sku-00005&prodName=the%20greatest%20product%20in%20all%20the%20world&qty=1&unitPrice=15.25 HTTP/1.0" 200 - "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"

216.148.244.85 - - [11/Mar/2005:11:05:29 -0500] "GET /tracker.html?retailer=Test+Retailer&order=xyz123&uniqueId=67890&numRecords=1&productsTotal=15.25&grandTotal=16 HTTP/1.0" 200 - "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"

170.224.224.119 - - [11/Mar/2005:11:47:44 -0500] "GET /tracker.html?retailer=Adasa+Hawaii&order=xyz123&uniqueId=67890&sku=sku-00005&prodName=the%20greatest%20product%20in%20all%20the%20world&qty=1&unitPrice=15.25 HTTP/1.0" 200 - "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"

170.224.224.122 - - [11/Mar/2005:11:47:44 -0500] "GET /tracker.html?retailer=Adasa+Hawaii&order=xyz123&uniqueId=67890&numRecords=1&productsTotal=15.25&grandTotal=16 HTTP/1.0" 200 - "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"

170.224.224.92 - - [11/Mar/2005:11:50:14 -0500] "GET /tracker.html?retailer=Raw+Clothing&order=xyz123&uniqueId=67890&sku=sku-00005&prodName=the%20greatest%20product%20in%20all%20the%20world&qty=1&unitPrice=15.25 HTTP/1.0" 200 - "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"

170.224.224.87 - - [11/Mar/2005:11:50:14 -0500] "GET /tracker.html?retailer=Raw+Clothing&order=xyz123&uniqueId=67890&numRecords=1&productsTotal=15.25&grandTotal=16 HTTP/1.0" 200 - "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"





this is my log file I am abel to parse it...
now for each retailer i have to write different files...
like for retailer=Raw+Clothing
I should be abel to show his arderid ,date,price in differnt file...
like that ohter retailer........



what is the best way to do this.... and how can we send paramenter automatically without any man help... if i want to send logfiles automatically to my perl program... how is it possible?

[mlseim]

The part about doing it automatically ....

You need to find out how these entries are being written
into the file. Within that script, you'll have to have it also
write into the separate retailer files.

If you don't have control over the script that is doing
the logging, you'll need to somehow execute another
script periodically and keep track of where you left off,
so you don't keep writing the same data over and over
again.

You need to see if your webhost had CRON services.
This is where you schedule a Perl script to run automatically
several times each day. If you don't have CRON, you would
need to subscribe to a CRON service that would run your
script for you.

The alternate method would be to imbed a javascript into
an existing website you have that gets viewed many times
each day. Whenever the site is viewed, your Perl script
would run. This is a little bit more advanced.

--------------

How are going to name the files for each retailer?
like this (each retailer has a unique ID code) ...
ID67890.txt
ID75643.txt
??

How many unique files to you think you'll end up with,
and is it a problem if you have 500 files?

This is going to grow into a rather involved project.

Maybe you could save a lot of time if you mention what
the final result should be ... what is the purpose of the
script?

[shreyaranga]

the filenameshould be like this retailer1.txt....
retailer1.txt file contains :-
orderid date price retailername
retailer's will be "n " number.. this is a growing company . so we can''t predict...
Ultimately i have to load these files data in data base that's different part....................



The alternate method would be to imbed a javascript into
an existing website you have that gets viewed many times
each day. Whenever the site is viewed, your Perl script
would run. This is a little bit more advanced.

this is the way I want it...

[mlseim]

... a few more questions ...

Sorry, but how the script is written will be affected by your
future plans. You might be doing way more work than you
need to. It's possible that this could all be done with the
same script that is "logging" the lines to begin with.

I'm not trying to be a "smart-alec", but you're asking for help
in writing a script. It takes me (and others) time to do this.
I'd hate to spend a lot of time when the whole thing could be
done in a much easier way.

------------

Question 1:

How is the log created ... the one that has the lines:

216.148.244.91 - - [11/Mar/2005:11:05:27 -0500] "GET /tracker.html?retailer=Test+Retailer&order=xyz123&uniqueId=12345&sku=sku-00005&prodName=the%20greatest%20product%20in%20all%20the%20world&qty=1&unitPrice=15.25 HTTP/1.0" 200 - "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"

Is that created by your webhost, or is that created by a "counter" or
"stats" program? Something is writing these lines into the file ...
what is it? If you are able to access the script that is writing these
lines, you could save yourself a ton of time.

-------------

Question 2:

you say , "the filenameshould be like this retailer1.txt...."

It looks like the log is hits on a webpage or something.
You might have 20 lines in one file, 10 lines in another, etc.

How do you determine which lines go into which file?

------------

Question 3:

Why are the lines put into text files instead of right into a database?
You mentioned that you are going to put them in a database later?
What kind of database?

----------

Question 4:

You have a web page on your site that gets regular "hits" that
can be used to execute your Perl script? Do you know how to,
or have the ability to alter that page (insert a javascript)?

[shreyaranga]

Hi There,
Thanks for your concern.
1) The log file is coming form apache....
2) The file names should be unique.. there is no lines limit for the time being..............
3)Actually they are wiriting them in files because they want to show the status to the retailers.. but now
1)I will write the data from the log file to the data base. in table (I will take care of this part)
2)then parse the log file store the dta indifferent files according to the retailer...
3)they wanted to call this perl script in CRON service...

[mlseim]

Here's the first shot at it ... see attached.

It currently only writes to one file for now "retailer1.txt"

Just to test the part where it saves the last timestamp in
a separate file so that you don't re-write old lines every
time the script runs. It only add lines when there are new
entries in the log.

[shreyaranga]

Hi ,
I am trying to store data from log file into database..I can split the data store it in a file.. and then i can call it in my javaprogram to store that data in database..
other than this is there any way to do this perl script writng in database

[mlseim]

What kind of database ... MySQL, Access ???

[shreyaranga]

database is postgres...

[shreyaranga]

postgres database...

[mlseim]

I've never used postgres, but here's some tutorials on
using it with Perl:
http://www.google.com/search?hl=en&lr=&c2coff=1&q=perl+postgres+database&btnG=Search

So, it would be good to know what the format should be
for the file that we are creating (retailer1.txt).
Should the fields be comma delimited, pipe deliminated, spaces?
Have the format preset would make it easier to write the data
to the database.

[shreyaranga]

postgres database...

[shreyaranga]

Thanks for your help.. If you wouldn't have done such a help i might have gone crazy... I am new to this perls so it's very hard to get ideas.. Thanks 1 more time...

[mlseim]

Here's the latest .... see attached.

It has the following fields saved into text files:
$Uniqueid|$order|$price|$retailer|$date|pending|

It creates a filename based on Retailer Name with a dash "-"
in place of spaces and extension .txt

[shreyaranga]

Hi ,
I am writing a shell script that will run a perl and loader java program.
In the java program deals with database.If I get any exception from the database what should i need to do in the shell script.. If you have any idea please let me know