At least you should LEO. I prefer to use gamelib on top of that, since I don't want to write the same code each time again. But since gamelib is still in a early stage (not even 0.1 yet), you could as well just copy parts of it and make your own library for now (I think that's what many people do, from what I've seen and heard).
Gamelib needs to be modularized more, so that it can be used in parts. I'm trying to split it up into more and more seperate classes, which can be difficult at times. Today I added a Window class, which handles stuff like window control buttons, window icons, etc...
Documentation for gamelib is available in 4 places:
1) wiki
2) gamelib demo
3) power tutorial
4) game distribution tutorial (not published yet)