I've never ran a forum before so I don't really know what you can or what plugins/modules you can add to minimize spam. Maybe a mod (or the people at vbulletin) can help shed some light on that.
I just know that people build tools (XRumer) specifically to spam forums, and every forum I've ever been a part of has a problem with spam. The bigger forums have more money to spend on killing off spam, but they struggle even more with it...getting thousands of new accounts that spam everyday. I've seen forums that decided to start charging money to register because of spam.
As for Google, I'd say stop worrying about it, but I know that won't happen.
There are a lot of things that can happen. I don't think these spam bots can cause you to be blacklisted unless they were linking to some really shady sites. Some people might also call this a "Google dance", but I don't agree with the typical definition of it. It's a dance if you jump around 1, 2, or 3 spots. It's a different story if you see the site go from the 2nd page to nowhere.
If you did any backlinking, then Google may have penalized you. Even if you don't do any link building, Google is fickle so they may penalize you. Your competition can also be spamming links to you so that can penalize you. A new site is always prone to penalties. I'd listen to probytes and use the disavow tool and get rid of any low quality links.
Google also does this thing where they penalize you and watch you. These penalties vary in severity and duration. They can be 1 month, 3 months, or even 6 months. They can be -5 (in rankings), -10, or -100. After your site falls, it can't get back up until it passes this "duration" period that they put on you. They watch to see if your site continues doing what it has been doing so just continue doing what you have been doing.
Long story short, I'd say just stick with the forum and be patient.