Trusting image links in emails.

When I open my email images aren't displayed. I understand that some can be external references. That's cool and everything, but it gets incredibly annoying.

Do most email clients "trust" email images based on the sender or based on the root address of the image reference?
If I'm correct, "trust" is built based on the sender, which to me is a tad bit annoying. Correct me if I'm wrong... please.

Edit:
What are the trust issues? The most another website could do is get someone's IP and store data on them. (which would allow them to know when they got the email... hrm...)

You and only you set which sites are allowed to display images in every email reader I have ever used.

I know iBall already said this, but wanted to make it doubly clear to you. It is you and not the email client that allows images.

There are two ways images can be supplied to emails - either linking them from a web site (where the web site logs your email address when you display the image) - or embedded directly in the email itself. Emails in this second group do not represent any security risk and so they will be displayed irrespective of the setting that controls whether those that are linked display or not.