You don't do anything to generate a random number to start with. The only time you create one is when you successfully guess the random number. So you need to generate that elsewhere during construction.
The colour changing is working properly, just not the way you want it. You are attempting to set the background of the JFrame, not the panel. Change those to panel.setBackground and it will work fine.
You should do some more evaluation of the structure here. GuessGame is a JFrame, but doesn't make any use of being as such. Instead it invokes its own JFrame and shows that, so this means you cannot call any component specific calls on the nested classes without providing an object to work with since the JFrame that would resolve to the implicit "this" of the parent wouldn't have any impact on the existing gui. Also, these properties are all static. If you leave them as such, then it will impact on the new game functionality which spawns a new frame. Both of these then share the same random number and all the components will be the same. It'll likely break the event handlers too I'd expect since the focusing will become. . . bizarre.
Move the control of the guessing to the GuessGame. Let the trybuttonhandler simply call the GuessGame methods.