I tend to call them “first person exploration games”, but that’s a bit of a mouthful and reminds me of when people tried to get people to call roguelikes “procedural death labyrinths” (lmao)
I think the thing is that a lot of these games don’t really have a ton in common other than being in first person and not involving murdering anyone. Kitty Horrorshow’s games aren’t really much like Fullbright’s games, which aren’t really like Connor Sherlock’s games, but they’re all grouped under “walking simulator” because we have no frame of reference to talk about first person games that aren’t about murder. People had trouble classifying Portal, for christ’s sake, because the fact that it was a puzzle game that was in first person and wasn’t an adventure game was too much for them.
In the taxonomy of animals they tend to call groups like this “paraphyletic”, because they include all things with a certain common ancestor (in this case, the first first-person game), but exclude some that would otherwise be included (in this case, every first person game where you shoot people or stab people or blow people up with your mind or whatever it is we do in video games.)
IMO trying to jam these games into a category like “narrative games” (not all of them have a strong narrative focus) or “exploration games” (not all of them are really about exploration) or “wandering games” (not all of them are about or encourage wandering) misses the point and restricts the genre arbitrarily. That said, I still hate the term “walking simulator” and am desperate for an alternative. I think “first-person adventure game” is closest, but people tend to think “Myst” when you say that and not “Gone Home”.