I have the same problem, there are many libraries, namely
OpenGL and DirectX are really efficient but they are so low level that it is not realistic to use them for big projects for most people.
SDL Allegro and SFML are fast and efficient and very convenient, but they are a pain to set up at times, SDL is the simplest to set up.
I have only succeeded setting up DirectX and SDL, everything else is spitting in my face
every time I try to set it up... it is very discouraging...
hopefully this helped...