Site running slow transferring to HTTPS

I have an intermittent issue with my site in that, at times, it runs really, really slow when it transfers from http to https.
The server administrators says there's no problem their end so I'm kinda stumped, as it runs fine most of the time, then for a couple of hours on some evening or weekend, any request has about a 10% chance of getting onto the next (secure) page. The page takes forever and might eventually load after a few minutes.
I'm not running any cron jobs at the time, I think I've ruled out network attacks. This happens with the public https pages. The password protected, back-end https pages seems slow, but still get there.
So really what I'm asking is this: Are there any inherent pitfalls to look out for at that point of transferring from http to https?
Otherwise its basically a question of optimising everything! Not a bad thing, but don't want to do it in a hurry.

Are you able to run the same stuff unencrypted on the same server to test? At least that could help isolate the problem. I'd try running something like ethereal to see what the network is doing. You should be able to reconstruct the request fairly easily and tell by the packet frequency and timestamps where things are getting blocked. If negotiation itself is taking a long time your host almost certainly made a poor configuration choice.

Google for "slow ssl" has a lot of people troubleshooting this, my favorite response of the few I read has been "I'll bet you are using /dev/random instead of /dev/urandom for your randomness." Call your host and ask them to plug a mouse into the server and wiggle it around and see if it works faster :]

Good idea i should have already thought of. Tried the same page under http and it still didn't work. Very strange.

As for network analysis, my boss was on the phone for 30 minutes with the network people while they checked stuff. I presume that they would have checked this, and also that they would be able to notice a problem better than I would.

The page itself is one of the more lightweight ones on the site, in terms of scripting and weight of html output.

So if it's not an https problem, not a problem of slow processing or download, occurs only on 1 or 2 pages, only occasionally, and only, it seems, on some computers (on 1 pc for sure, but some stats suggest there's a prob on more) then i'm not sure what I'm left with. Something specific to the ISP? I'm pretty stumped :confused: