I think the equivalent to what Rob is describing–this lack of fidelity actually providing a space for fear to grow–can be found in a lot of games with pixel art. Obviously, the sound design and contextualization of the assets in the game also do a lot. The hybrids go from spooky to terrifying when they can take off some of your precious health points (though, I will say I still think System Shock 2 would have benefitted from a less abusable save system). Harris Brewer also mentioned that the distortion of VHS tapes, which is also why I think found footage horror films are so damn suspenseful.
What comes to mind first is Yume Nikki (Dream Diary), specifically, this thing called Uboa:
There’s a small chance this thing will appear in a specific room when you turn off the lights, its appearance accompanied by static. The lack of detail makes this entity far more unnerving. What is it? Is it a face? What does it want? Is it happy? Is it angry? I don’t know. But it creeps me out.
In a remake of Yume Nikki, everything was made in 3D. And I’ve heard it became a lot less creepy. As for Uboa, it suddenly became less of a mystery, less of aberration of pixels. It became a shape who’s dimensions and size are easier to visualize and understand. And that took away a lot of the unease present there.