When considering how a game is perceived by its player(s), it is also interesting to consider how a game is perceived by pure observers. How engaging is it for those merely watching? There is probably little financial incentive to explore this aspect of a game, but it's a curious design experiment if nothing else.
Thinking about this, it seems there are a few qualities that significantly affect how watchable a game is.
1) Nature of Challenge
Platformers have obvious challenges that are usually approached at the player's pace. Shooters are largely reflex challenges with a high degree of surprise. Puzzle-based games rarely require manual dexterity of any sort. How readable the goals are to an observer affects how enjoyable it is to watch. Having little understanding of what's going on, or being unable to follow the player's rapidly changing focus, makes for poor watching. But being able to gasp at whether the player will make that jump, or better yet being able to actively assist the player ("What if you put the red block there instead?") hold the attention far better.
2) Camera Perspective
Something of a corollary to the above, first-person games are probably less watchable than third-person ones. First-person allows, and often necessitates, changing the orientation of the viewport rapidly and dramatically. If you're not the one doing that adjustment, it's easier to become disoriented. Third-person tends to change camera alignment less often and more slowly, with all movement being respective to a fixed object (the player's avatar). Without the ability to mentally map controller input with the camera's output, having a representation that can be understood purely visually is much easier to watch.
3) Limited Systemic Information
Games that take place largely in the player's head, e.g. strategy games, are likely difficult to enjoy watching. When a great deal of gameplay takes place in the player's mind, there is little to watch. For the player, the game is really about internalizing and reacting to the game's state. Learning to parse that state from the game's visual representation takes time. If an observer doesn't have this systemic knowledge, there is little to observe.
Now there are some games that could be very engaging to watch for those familiar with it. Arcade fighting games, e.g. the Street Fighter series, represent this distinction well. As the crowds at Evo will attest, watching two masterful players going head-to-head can hold attention rapt ... if you're familiar with the game. If you aren't, well, it's a lot of confusing sound and fury that ends in 90 seconds or less. Starcraft is nationally televised in South Korea, but it's also so ubiquitous that just about any viewer is at least passingly familiar with the gameplay. I'm not sure I'd ever find watching someone play Team Fortress 2 or Civilization to be especially engaging, but seeing someone pull off a near-impossible stunt in Burnout is only enhanced by knowing just how hard that stunt was to accomplish.
There are other obvious things too, e.g. single player or local multiplayer is more watchable than online multiplayer, just due to physicality, console is better than PC or handheld, etc. Ultimately, it's about being able to understand the game's drama. Action tends to be better at this, since character-based drama obviously requires familiarity with said characters. Without that familiarity, character-based drama has to hang on the quality of writing and acting ... yeah. That one probably isn't to be relied up.
If there's any functional design goal here, it would be encouraging observers to become players. As you may have been thinking already, Rock Band does this better than almost any other game. It has a simple, understandable goal, creates a context where play feeds directly into the experience and it has an intuitive interface. Plus, given that you'll likely be playing with others, all someone has to do is push that plastic guitar in your hand and say, "Take a turn." No single player is put on the spot either. Many of Rock Band's lessons are design specific, but it's still interesting to see how they make their game so damn inviting.
As an aside, the co-op portion of DeathSpank was designed with these principles in mind as well. It's too long to get into here, but I believe I'll be writing something for DeathSpank.com about it next week. Look for it and I'll try to remember to put up a link too.
Are there games you found fascinating just to watch? Have others watched you play games? I'm curious as to what qualities other folks find engaging when someone else has the reins.