So I was working on my own little bit of writing and tossed it out to a friend who’s alpha reading and got a message not long after of: “I don’t think your allowed to use Owlbears...” This brought up the initial thought, What creatures can’t we use for our LitRPG/GAMELit books? and the further question of what would you change to make them useable?
Anything based in mythology should be fine. Wizards of the Coast uses their open gaming license and covers some monsters. From memory they cover Beholders and mind flayers specifically but there are probably others. Things like owlbears that don't have a solid mythological bases should probably be avoided but there are TONS of cool creatures from random mythologies all over the world.
Mind flayers and Beholders are also off limits, I believe. As is Dungeon Master (altho someone named their series Dungeon Master and it's still up...)