the AI you are referring to is comprised of many systems operating together, you have various path finding and steering algorithms for creating 'intelligent' movement from place to place in the environment, this usually operates as a service to higher level code such as state machines, priority lists, scripts etc

Interaction between entities happens at different levels, to calculate the path for the player to take to walk around an enemy, avoiding intersection of geometry is a lower level from game like interaction such as instructing an enemy to attack the player.

Also you'd use/have different approaches/restrictions depending on how many AI entities you plan on processing and the kind of environment that they inhabit.

IE a dynamic destructible environment requires more work for AI to manage than a static environment. Some situations allow you to cache paths across maps, others not so much.

At the highest level it's all game specific, imagine skyrim and enemy AI casting appropriate spells for the situation etc