pjsulin
04-06-2006, 10:32 PM
I have quite a bit of web automation with perl scripts but cannot seem to get this one working and I'm not sure if the server has some more sophisticated method for preventing this type of automation or I am missing something relatively simple. This post will seem quite long but it is really just three parts and the lengthy parts are just long cookies and a parameter list:
1.The headers from a successful POST from Mozilla's Live Headers add-in
2. The script
3. the results from running the script.
1. Successful POST Headers
http://oas.zacks.com/pls/screnn/awi_report
POST /pls/screnn/awi_report HTTP/1.1
Host: oas.zacks.com
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.8.0.1) Gecko/20060111 Firefox/1.5.0.1
Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Proxy-Connection: keep-alive
Referer: http://oas.zacks.com/pls/screnn/screening.show_categories?p_site_id=3&p_i=&p_o=&p_v=&p_comp_id=zcom2003&p_customer_id=
Cookie: ora_100m1.5k=6\20,21,24,25,115,30|select master_table.ticker,stock_data.EXCHANGE,stock_data.SECTOR_NAME,stock_data.X_IND,master_table.comp_na me,prices.CLOSING from master_table,prices,stock_data where master_table.m_ticker=stock_data.m_ticker and master_table.m_ticker=prices.m_ticker AND upper(stock_data.MKT_VALUE) >= 100 AND upper(stock_data.VOLUME) >= 150000; TOTAL_POP=3; user_session=7a792c8b829032e1fc0d1a491b1e1927
Content-Type: application/x-www-form-urlencoded
Content-Length: 487
p_site_id=3&p_i=1&p_o=1&p_v=1&p_columns=1&p_comp_id=zcom2003&p_customer_id=&p_save_string=6%5C20%2C21%2C24%2C25%2C115%2C30%7Cselect+master_table.ticker%2Cstock_data.EXCHANGE%2C stock_data.SECTOR_NAME%2Cstock_data.X_IND%2Cmaster_table.comp_name%2Cprices.CLOSING+from+master_tabl e%2Cprices%2Cstock_data+where+master_table.m_ticker%3Dstock_data.m_ticker+and+master_table.m_ticker% 3Dprices.m_ticker+AND+upper%28stock_data.MKT_VALUE%29+%3E%3D+100+AND+upper%28stock_data.VOLUME%29+%3 E%3D+150000
2. The Script
#\usr\bin\perl -w
use LWP::UserAgent;
use HTTP::Cookies;
use utf8;
my $url = 'http://oas.zacks.com/pls/screnn/awi_report HTTP/1.1';
my $browser = LWP::UserAgent->new;
$browser->agent('User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.8.0.1) Gecko/20060111 Firefox/1.
.0.1');
#$browser->default_headers->push_header('Cookie'=>'Cookie: ora_100m1.5k=6\20,21,24,25,115,30|select master_table.t
cker,stock_data.EXCHANGE,stock_data.SECTOR_NAME,stock_data.X_IND,master_table.comp_name,prices.CLOSI NG from master
table,prices,stock_data where master_table.m_ticker=stock_data.m_ticker and master_table.m_ticker=prices.m_ticker
ND upper(stock_data.MKT_VALUE) >= 100 AND upper(stock_data.VOLUME) >= 150000; TOTAL_POP=3; user_session=7a792c8b82
032e1fc0d1a491b1e1927');
$browser->default_headers->push_header('Host'=>'oas.zacks.com');
$browser->default_headers->push_header('Keep-Alive'=>'300');
$browser->default_headers->push_header('Proxy-Connection'=>'keep-alive');
$browser->default_headers->push_header('Content-Type'=>'application/x-www-form-urlencoded');
$browser->default_headers->push_header('Accept-Charset'=>'Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7');
$response = $browser->post($url,'p_site_id'=>'3','p_i'=>'1','p_o'=>'1','p_v'=>'1','p_columns'=>'1','p_comp_id'=>'z
om2003','p_comp_id'=>'','p_save_string'=>'6%5C20%2C21%2C24%2C25%2C115%2C30%7Cselect+master_table.ticker%2Cstock_da
a.EXCHANGE%2Cstock_data.SECTOR_NAME%2Cstock_data.X_IND%2Cmaster_table.comp_name%2Cprices.CLOSING+fro m+master_table
2Cprices%2Cstock_data+where+master_table.m_ticker%3Dstock_data.m_ticker+and+master_table.m_ticker%3D prices.m_ticke
+AND+upper%28stock_data.MKT_VALUE%29+%3E%3D+100+AND+upper%28stock_data.VOLUME%29+%3E%3D+150000');
print "HTTP status: ", $response->code(), "\n";
print "HTTP message: ", $response->message(), "\n";
print "HTTP content: ", $response->content(), "\n";
3. The $response->code
No DAD configuration Found
DAD name:
PROCEDURE : 1.1
URL : http://oas.zacks.com:80/pls/screnn/awi_report HTTP/1.1
PARAMETERS :
===========
ENVIRONMENT:
============
PLSQL_GATEWAY=WebDb
GATEWAY_IVERSION=2
SERVER_SOFTWARE=Apache/1.3.12 (Unix) ApacheJServ/1.1 mod_ssl/2.6.6 OpenSSL/0
.9.7d mod_perl/1.22
GATEWAY_INTERFACE=CGI/1.1
SERVER_PORT=80
SERVER_NAME=oas.zacks.com
REQUEST_METHOD=POST
QUERY_STRING=
PATH_INFO=/screnn/awi_report HTTP/1.1
SCRIPT_NAME=/pls
REMOTE_HOST=
REMOTE_ADDR=69.36.182.22
SERVER_PROTOCOL=HTTP/1.1
REQUEST_PROTOCOL=HTTP
REMOTE_USER=
HTTP_CONTENT_LENGTH=0
HTTP_CONTENT_TYPE=application/x-www-form-urlencoded
HTTP_USER_AGENT=User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US;
rv:1.8.0.1) Gecko/20060111 Firefox/1.5.0.1
HTTP_HOST=oas.zacks.com
HTTP_ACCEPT=
HTTP_ACCEPT_ENCODING=
HTTP_ACCEPT_LANGUAGE=
HTTP_ACCEPT_CHARSET=Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
HTTP_COOKIE=
Authorization=
HTTP_IF_MODIFIED_SINCE=
Maybe someone recognizes the response and that points to the problem or there is some other glaring issue that is blinding me with it's obviousness.
Thanks.
1.The headers from a successful POST from Mozilla's Live Headers add-in
2. The script
3. the results from running the script.
1. Successful POST Headers
http://oas.zacks.com/pls/screnn/awi_report
POST /pls/screnn/awi_report HTTP/1.1
Host: oas.zacks.com
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.8.0.1) Gecko/20060111 Firefox/1.5.0.1
Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Proxy-Connection: keep-alive
Referer: http://oas.zacks.com/pls/screnn/screening.show_categories?p_site_id=3&p_i=&p_o=&p_v=&p_comp_id=zcom2003&p_customer_id=
Cookie: ora_100m1.5k=6\20,21,24,25,115,30|select master_table.ticker,stock_data.EXCHANGE,stock_data.SECTOR_NAME,stock_data.X_IND,master_table.comp_na me,prices.CLOSING from master_table,prices,stock_data where master_table.m_ticker=stock_data.m_ticker and master_table.m_ticker=prices.m_ticker AND upper(stock_data.MKT_VALUE) >= 100 AND upper(stock_data.VOLUME) >= 150000; TOTAL_POP=3; user_session=7a792c8b829032e1fc0d1a491b1e1927
Content-Type: application/x-www-form-urlencoded
Content-Length: 487
p_site_id=3&p_i=1&p_o=1&p_v=1&p_columns=1&p_comp_id=zcom2003&p_customer_id=&p_save_string=6%5C20%2C21%2C24%2C25%2C115%2C30%7Cselect+master_table.ticker%2Cstock_data.EXCHANGE%2C stock_data.SECTOR_NAME%2Cstock_data.X_IND%2Cmaster_table.comp_name%2Cprices.CLOSING+from+master_tabl e%2Cprices%2Cstock_data+where+master_table.m_ticker%3Dstock_data.m_ticker+and+master_table.m_ticker% 3Dprices.m_ticker+AND+upper%28stock_data.MKT_VALUE%29+%3E%3D+100+AND+upper%28stock_data.VOLUME%29+%3 E%3D+150000
2. The Script
#\usr\bin\perl -w
use LWP::UserAgent;
use HTTP::Cookies;
use utf8;
my $url = 'http://oas.zacks.com/pls/screnn/awi_report HTTP/1.1';
my $browser = LWP::UserAgent->new;
$browser->agent('User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.8.0.1) Gecko/20060111 Firefox/1.
.0.1');
#$browser->default_headers->push_header('Cookie'=>'Cookie: ora_100m1.5k=6\20,21,24,25,115,30|select master_table.t
cker,stock_data.EXCHANGE,stock_data.SECTOR_NAME,stock_data.X_IND,master_table.comp_name,prices.CLOSI NG from master
table,prices,stock_data where master_table.m_ticker=stock_data.m_ticker and master_table.m_ticker=prices.m_ticker
ND upper(stock_data.MKT_VALUE) >= 100 AND upper(stock_data.VOLUME) >= 150000; TOTAL_POP=3; user_session=7a792c8b82
032e1fc0d1a491b1e1927');
$browser->default_headers->push_header('Host'=>'oas.zacks.com');
$browser->default_headers->push_header('Keep-Alive'=>'300');
$browser->default_headers->push_header('Proxy-Connection'=>'keep-alive');
$browser->default_headers->push_header('Content-Type'=>'application/x-www-form-urlencoded');
$browser->default_headers->push_header('Accept-Charset'=>'Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7');
$response = $browser->post($url,'p_site_id'=>'3','p_i'=>'1','p_o'=>'1','p_v'=>'1','p_columns'=>'1','p_comp_id'=>'z
om2003','p_comp_id'=>'','p_save_string'=>'6%5C20%2C21%2C24%2C25%2C115%2C30%7Cselect+master_table.ticker%2Cstock_da
a.EXCHANGE%2Cstock_data.SECTOR_NAME%2Cstock_data.X_IND%2Cmaster_table.comp_name%2Cprices.CLOSING+fro m+master_table
2Cprices%2Cstock_data+where+master_table.m_ticker%3Dstock_data.m_ticker+and+master_table.m_ticker%3D prices.m_ticke
+AND+upper%28stock_data.MKT_VALUE%29+%3E%3D+100+AND+upper%28stock_data.VOLUME%29+%3E%3D+150000');
print "HTTP status: ", $response->code(), "\n";
print "HTTP message: ", $response->message(), "\n";
print "HTTP content: ", $response->content(), "\n";
3. The $response->code
No DAD configuration Found
DAD name:
PROCEDURE : 1.1
URL : http://oas.zacks.com:80/pls/screnn/awi_report HTTP/1.1
PARAMETERS :
===========
ENVIRONMENT:
============
PLSQL_GATEWAY=WebDb
GATEWAY_IVERSION=2
SERVER_SOFTWARE=Apache/1.3.12 (Unix) ApacheJServ/1.1 mod_ssl/2.6.6 OpenSSL/0
.9.7d mod_perl/1.22
GATEWAY_INTERFACE=CGI/1.1
SERVER_PORT=80
SERVER_NAME=oas.zacks.com
REQUEST_METHOD=POST
QUERY_STRING=
PATH_INFO=/screnn/awi_report HTTP/1.1
SCRIPT_NAME=/pls
REMOTE_HOST=
REMOTE_ADDR=69.36.182.22
SERVER_PROTOCOL=HTTP/1.1
REQUEST_PROTOCOL=HTTP
REMOTE_USER=
HTTP_CONTENT_LENGTH=0
HTTP_CONTENT_TYPE=application/x-www-form-urlencoded
HTTP_USER_AGENT=User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US;
rv:1.8.0.1) Gecko/20060111 Firefox/1.5.0.1
HTTP_HOST=oas.zacks.com
HTTP_ACCEPT=
HTTP_ACCEPT_ENCODING=
HTTP_ACCEPT_LANGUAGE=
HTTP_ACCEPT_CHARSET=Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
HTTP_COOKIE=
Authorization=
HTTP_IF_MODIFIED_SINCE=
Maybe someone recognizes the response and that points to the problem or there is some other glaring issue that is blinding me with it's obviousness.
Thanks.