Using Storable to save large data structures

[naqvia]

I am using Storable and saving four hashes. However, when I try to retrieve the hashes from another script, a few of the values do not produce the proper results and possibly blank. Here is where I make the hashes:

use strict;
use Storable;

my %family_hashh = ();
my %family_hashhh = ();
my %salks;
my %family_hash = &family_hash();
my $path = "ammarsum.txt";

store(\%family_hash,"family_hash_store");
store(\%family_hashh,"family_hashh_store");
store(\%family_hashh,"family_hashhh_store");

my %sfamily_hash = %{ retrieve("family_hash_store") }; # direct to hash
my %sfamily_hashh = %{ retrieve("family_hashh_store") }; # direct to hash
my %sfamily_hashhh = %{ retrieve("family_hashhh_store") }; # direct to hash

while (my ($key, $value) = each %sfamily_hash) {
print "\nQQQ",$key,"QQQ",$value,"QQQ";
foreach my $gene (@{$sfamily_hashh{$key}}){
print "WWW",$key,"WWW",$gene,"WWW";
}
}

open (SUM,"ammarsum.txt") or die "Cannot Open";
while (<SUM>){
chomp;
my ($gene,$salk_num, $sent) = split " ";
$gene = "\U$gene";
$gene =~s/\s+//;
$gene =~s/\n//;
$salk_num =~s/\s+//;
$salk_num =~s/\n//;
push (@{$salks{$gene}}, $salk_num.":".$sent);
}
store(\%salks,"salks_store");

sub family_hash{
open (FAM, "gene_family_tair.txt");
my %family_hash;
while (<FAM>){
my ($superfam, $subfam, $gene) = split "\t";
$gene ="\U$gene";
$gene =~s/\s+//;
$gene =~s/\n//;
$superfam = $subfam if $superfam=~/NULL/;
$subfam = $superfam if $subfam=~/NULL/;
push @{$family_hash{$superfam}}, $subfam;
push @{$family_hashh{$superfam}}, $gene;
push @{$family_hashhh{$subfam}}, $gene;

}

## remove duplicates from hashes
foreach my $key (keys %family_hash) {
my %seen = ();
@{$family_hash{$key}} = grep {!$seen{$_}++} @{$family_hash{$key}};
}
foreach my $key (keys %family_hashh) {
my %seen = ();
@{$family_hashh{$key}} = grep {!$seen{$_}++} @{$family_hashh{$key}};
}

foreach my $key (keys %family_hashhh) {
my %seen = ();
@{$family_hashhh{$key}} = grep {!$seen{$_}++} @{$family_hashhh{$key}};
}
close (FAM);
return %family_hash;
}



And here is how I retrieve them (in a separate cgi script)...


my %family_hash = %{ retrieve("family_hash_store") }; # direct to hash
my %family_hashh = %{ retrieve("family_hashh_store") }; # direct to hash
my %family_hashhh = %{ retrieve("family_hashhh_store") }; # direct to hash
my %salks = %{ retrieve("salks_store") }; # direct to hash



Whats wrong?

[KevinADC]

Check your suspect hashes with Data::Dumper before storing them to see if they appear correct at that time.

[naqvia]

They are correct because I have another version of the script that does the same thing but does not store the hashes and makes them on the fly. Something wierd thats been happening is when I go in the loop, some keys and values don't show up, but if I hard-code/manually access they appear!

[naqvia]

anyone? is there a different way to store the hashes?

[KevinADC]

You could try Data::Dumper to store the hash or the MLDBM module

[naqvia]

How can I use it? Sample? Example?

[KevinADC]

Read the module documentation

[naqvia]

Ok, I am using Data::Dumper now. I am saving the hashes properly, because when I do a "do "filename" on the script where dump/create the hashes they print corectly. However when I do a "do" on a different program (cgi) it does not print out anything. I tried declaring the hashes before hand and openin the file that the dumper wrote out to but still no luck. What am I doing wrong? I saved 4 hashes (like i did above using storable) to a filename using Data Dumper. How do I retrieve that!

[KevinADC]

Post your code.

[naqvia]

Its same as above but I do

open GENE_FAM, ">gene_family_tair2.txt";
print GENE_FAM Data::Dumper->Dump([\%family_hash],[qw(family_hash)]);
print GENE_FAM Data::Dumper->Dump([\%family_hashh],[qw(family_hashh)]);
print GENE_FAM Data::Dumper->Dump([\%family_hashhh],[qw(family_hashhh)]);
print GENE_FAM Data::Dumper->Dump([\%salks],[qw(salks)]);
close GENE_FAM;


Then to retrieve it... do "gene_family_tair2.txt";

[KevinADC]

You probably want to do something more like this:

open GENE_FAM, ">gene_family_tair2.txt";
print GENE_FAM Data::Dumper ([\%family_hash,\%family_hashh,\%family_hashhh,\%salks]);
close GENE_FAM;

that should create an array of the hashes in that order. To read them back into a perl script:

my $array_of_hashes_ref = do "gene_family_tair2.txt";

then each element of the array will be one of the hashes you printed to the file. You could deference them into hashes or whatever it is you need to do with them.

[naqvia]

I am trying to retrieve each hash by

%family_hash = $array_of_hashes_ref[0];
%family_hashh = $array_of_hashes_ref[1];
.
.
.

But that is not working. It not printing any content?

[KevinADC]

try:

my %family_hash = %{ $array_of_hashes_ref->[0] };
etc
etc

I thought you understood about dereferencing because you use it in your original code:

my %sfamily_hash = %{ retrieve("family_hash_store") }; # direct to hash
my %sfamily_hashh = %{ retrieve("family_hashh_store") }; # direct to hash
my %sfamily_hashhh = %{ retrieve("family_hashhh_store") }; # direct to hash

You need to start posting more descriptive explanations too, saying "it's not working" or "I get an error" is useless information most of the time. Describing what happens in detail or posting actual error messages is much more useful.

[naqvia]

Kevin,

This is still not working. None of the content of the hashes are printing. In fact, they seem to be empty. Here is the exact code from the sub-routine that tries to print the hashes (and I know the hashes are correctly generated because of the above explanation.

my $array_of_hashes_ref = do "gene_family_tair2.txt";
my %family_hash = %{$array_of_hashes_ref[0]};
my %family_hashh = %{$array_of_hashes_ref[1]};
my %family_hashhh = %{$array_of_hashes_ref[2]};
my %salks = %{$array_of_hashes_ref[3]};

print "<ul id=\"designtree\">";# print "@{$salks{'AT5G67620'}}";

while (my ($key, $value) = each %family_hash) {
my $superfam_num = $#{$family_hashh{$key}} + 1;
my $salk_num = 0;
my $num_count = 0;
my $salk_sent = 0;

foreach my $gene (@{$family_hashh{$key}}){
$salk_num = $#{$salks{$gene}} + 1;
$num_count = $salk_num + $num_count;
foreach my $salk (@{$salks{$gene}}){
if ($salk=~/SentToABRC/){
$salk_sent++;
}
}
}
my $array_of_hashes_ref = do "gene_family_tair2.txt";
my %family_hash = %{$array_of_hashes_ref[0]};
my %family_hashh = %{$array_of_hashes_ref[1]};
my %family_hashhh = %{$array_of_hashes_ref[2]};
my %salks = %{$array_of_hashes_ref[3]};

print "<ul id=\"designtree\">";# print "@{$salks{'AT5G67620'}}";

while (my ($key, $value) = each %family_hash) {
my $superfam_num = $#{$family_hashh{$key}} + 1;
my $salk_num = 0;
my $num_count = 0;
my $salk_sent = 0;

foreach my $gene (@{$family_hashh{$key}}){
$salk_num = $#{$salks{$gene}} + 1;
$num_count = $salk_num + $num_count;
foreach my $salk (@{$salks{$gene}}){
if ($salk=~/SentToABRC/){
$salk_sent++;
}
}
}

my $key_m=();
foreach my $gene( @{$family_hashh{$key}} ){
# print "$input TTT $gene<br>";
if($input=~/^$gene$/){

$key_m = "<li><font color=\"red\">".$key."</font>";
# print "key:$key_m";
last;

}
else{
$key_m = $key;
}
}
#print "key: $key_m ";
print "<li>$key(<font color=\"darkslategray\">$superfam_num</font><font color=\"lightseagreen\">, $num_count,</font><strong><font color=\"green\">$salk_sent</strong></font> )<ul>";

foreach my $item_sub (@{$family_hash{$key}}){
if ($item_sub=~/(\D{2}\dD\d{5})/){
# &null_fam($item_sub);
}
else {
my $subfam_num = $#{$family_hashhh{$item_sub}} + 1;
my $salk_num = 0;
my $num_count = 0;
my $salk_sentsub = 0;
foreach my $gene (@{$family_hashhh{$item_sub}}){
$salk_num = $#{$salks{$gene}} + 1;
$num_count = $salk_num + $num_count;
foreach my $salk (@{$salks{$gene}}){
if ($salk=~/SentToABRC/){
$salk_sentsub++;
}
}
}
my $item_sub_m = ();
foreach my $gene (@{$family_hashhh{$item_sub}}){
if($input=~/^$gene$/ ){
$item_sub_m = "<li><font color=\"red\">".$item_sub."</font>";
# print "key:$key_m";
last;

}
else{
$item_sub_m = $item_sub;
}
}
#print "$item_sub:";
print "<li>$item_sub(<font color=\"darkslategray\">$subfam_num, </font><font color=\"lightseagreen\">$num_count,</font><strong><font color=\"green\">$salk_sentsub</strong></font>)<ul>";

foreach my $item_gene(@{$family_hashhh{$item_sub}}){

if ($item_gene eq $input){
print "<li><font color=\"red\">$item_gene</font><fontcolor=\"red\"><strong></strong></font> ";
}
else{
print "<li><font color=\"darkslategray\">$item_gene</font> ";
}

foreach my $salk_id (@{$salks{$item_gene}}){
my ($salk_id, $sent) = split ":", $salk_id;
if ($sent=~/SentToABRC/){
$salk_id = $salk_id."c";
print "<input type=\"checkbox\" id=\"stock_number\", name=\"stock_number\", value=\"$salk_id\", label=\"\">";
print "<font color=\"green\"><strong>$salk_id</font></strong>" if @{salks{$item_gene}};
}
else {
print "<input type=\'checkbox\' id=\"stock_number\", name=\"stock_number\", value=\"$salk_id\", label=\"\">";
print "<font color=\"lightseagreen\">$salk_id</font>" if @{salks{$item_gene}};
}
}

print " </li>";
}

print "</ul></li>";

}

}
print "</ul></li>";
}
print "</ul>";
&end_html();
}

[naqvia]

The code you gave me to store the hashes is producing an error actually. I get this error: Undefined subroutine &Data::Dumper called at store_data2.pl line 23, <SUM> line 26958. And it points to this print GENE_FAM Data::Dumper ([\%family_hash,\%family_hashh,\%family_hashhh,\%salks]);

[KevinADC]

at the beginning of your code:

use Data::Dumper;


later to print the data structure to a file:

open GENE_FAM, ">gene_family_tair2.txt";
print GENE_FAM Dumper ([\%family_hash,\%family_hashh,\%family_hashhh,\%salks]);
close GENE_FAM;


see how that works. It might really be better to store each hash in it's own file instead of lumping them all together. It should make working with each hash individually much easier.

[naqvia]

It doesn't produce that error but I am back to square one! In the script where I am doing the printing, there is no content being printed.. :( I pasted the code in my previous post.

[KevinADC]

not this:

my $array_of_hashes_ref = do "gene_family_tair2.txt";
my %family_hash = %{$array_of_hashes_ref[0]};
my %family_hashh = %{$array_of_hashes_ref[1]};
my %family_hashhh = %{$array_of_hashes_ref[2]};
my %salks = %{$array_of_hashes_ref[3]};

but this:

my $array_of_hashes_ref = do "gene_family_tair2.txt";
my %family_hash = %{$array_of_hashes_ref->[0]};
my %family_hashh = %{$array_of_hashes_ref->[1]};
my %family_hashhh = %{$array_of_hashes_ref->[2]};
my %salks = %{$array_of_hashes_ref->[3]};

It wrks when I test with some simple hashes of arrays:


use DATA::Dumper;
my %hash1 = (foo=> [qw(foo1 bar1 baz1)], bar => [qw(this1 that1 theohter1)]);
my %hash2 = (foo=> [qw(foo2 bar2 baz2)], bar => [qw(this2 that2 theohter2)]);
open(FH,">dumper.txt");
print FH Dumper ([\%hash1,\%hash2]);
close FH;

my $h_ref = do "dumper.txt";
my $h1 = %{$h_ref->[0]};
my %h2 = %{$h_ref->[1]};
foreach $array (keys %h2) {
print "@{$h2{$array}}\n";
}