Seems like you just need to animate the margin-top as well as the height of the elements. Something like:
Code:
<script type="text/javascript">
$(document).ready(function(){
//When mouse rolls over
$("li").mouseover(function(){
$(this).stop().animate({height:'150px',marginTop:'-100px'},{queue:false, duration:200, easing: 'easeOutBounce'})
});
//When mouse is removed
$("li").mouseout(function(){
$(this).stop().animate({height:'50px',marginTop:0},{queue:false, duration:200, easing: 'easeOutBounce'})
});
}); </script>
You might need to give your menu a bit more space at the top to fit it in.