Why Your Enemy AI is a Mess (And How Scriptable Finite State Machine Unity Can Fix It)

Here's the thing about enemy AI—it starts simple. You write a basic patrol script, maybe add a chase behavior, and boom, you've got a working enemy. But then the feature requests start rolling in: "Can the enemy flee when low on health?" "What if it should investigate sounds?" "Can we add an attack animation?" Before you know it, your once-clean AI script has ballooned into a thousand-line monster with nested if-statements that even you can't debug anymore.

Been there. Spent a good chunk of my early Unity days wrestling with bloated AI controllers that broke every time I added a new behavior. The solution? A scriptable finite state machine unity architecture that actually scales. Let me show you the exact system I use now—one that lets designers create complex AI behaviors without writing a single line of code.

What Makes AI Code Turn Into Spaghetti

Let me tell you what happens when you don't use a proper unity ai state machine. You start with one behavior, so you write it directly in your enemy's Update method. Clean enough. Then you add another behavior. Now you've got an enum and a switch statement. Still manageable. But by the time you're on your fifth or sixth behavior, your Update method looks like this nightmare:

void Update()
{
    switch(_state)
    {
        case AIState.Patrol:
            // 50 lines of patrol logic
            break;
        case AIState.Chase:
            // 40 lines of chase logic
            break;
        case AIState.Attack:
            // 60 lines of attack logic
            break;
        // ... and it keeps growing
    }
}

This is what I call the "mega-script" problem. Everything lives in one file, behaviors are tangled together, and debugging becomes a guessing game. When your enemy starts acting weird, good luck figuring out which of those nested conditions is causing the issue.

A Finite State Machine (FSM) solves this by enforcing one simple rule: your AI can only be in one state at a time. It's a behavioral design pattern that allows an object to change its behavior when its internal state changes. Think of it like your phone—it can be in "silent mode," "ring mode," or "vibrate mode," but never two at once.

The Traffic Light Revelation (How FSMs Actually Work)

The best way I've found to explain scriptableobject fsm is with a traffic light. A traffic light has three states: green, yellow, and red. It can only be in one state at any given moment. It transitions between states based on a timer. Green → Yellow → Red → Green, and the cycle repeats.

Your enemy AI works exactly the same way. It has distinct states like "Patrol," "Chase," or "Attack." It can only be in one state at a time. And it transitions between them based on specific conditions—like "Is the player within sight?" or "Is my health below 30%?"

The genius of using Unity's ScriptableObjects to build this is that each state becomes a reusable asset. You create a "PatrolState" asset once, and you can use it on ten different enemy types, each with their own unique parameters (like patrol speed or waypoints). Game designers can drag and drop states in the Inspector to create entirely new AI behaviors without touching code. This is what turns your AI system from a programmer bottleneck into a designer playground.

Breaking Down the Scriptable Finite State Machine Unity Architecture

Before we dive into code, let's make sure we're on the same page about terminology. When I'm talking about enemy ai behavior unity systems, these are the core pieces:

• State: A ScriptableObject that encapsulates one specific behavior. Your "PatrolState" contains all the logic for moving between waypoints. Your "AttackState" handles playing attack animations and dealing damage. Each state is isolated and focused.
• Action: The specific logic that runs continuously while in a state. For example, in the Patrol state, the action might be "move towards the next waypoint."
• Transition: The condition that triggers a state change. This is your "if the player gets close, switch to Chase" logic. When a transition condition becomes true, the FSM switches from the current state to a different one.
• State Machine Controller: A MonoBehaviour script attached to your enemy GameObject. This is the engine that runs everything. It holds a reference to the current state, calls its update logic every frame, and manages switching between states.
• ScriptableObject: This is a Unity asset class (API - Application Programming Interface - a way for different software to talk to each other) that stores data independently of any scene. We use these to create reusable state assets that designers can configure in the Inspector.
• FSM (Finite State Machine): The overall pattern that ties everything together. It ensures your AI is only in one state at a time and handles all state transitions cleanly.

Why ScriptableObjects Change Everything

This is where the magic happens. By using ScriptableObjects for our states, we get massive benefits that traditional approaches just can't match.

Modularity and Reusability

I can't stress this enough—you create a generic "PatrolState" once and reuse that same asset for every enemy in your game. Each enemy gets its own copy with unique parameters (patrol speed, detection radius), but the core logic stays the same. No copying and pasting code across multiple enemy scripts.

Empowers Game Designers

This architecture allows designers to create new AI behaviors by creating and configuring ScriptableObject assets in the Unity Inspector. They can mix and match states, adjust transition conditions, and build entirely new enemy types without waiting for a programmer. From my time working on multiple Unity projects, this has cut iteration time by more than half.

Clean and Organized Code

Instead of one 2,000-line AI controller, you have small, focused state files. Each behavior is neatly encapsulated. When I'm debugging and an enemy behaves strangely, I can select it in the editor and immediately see which state asset it's running. That instantly tells me which file to open.

Easier Debugging

The state machine inspector shows you exactly which state your AI is in at any moment. You don't have to add Debug.Log statements everywhere to figure out what's happening. You just look at the component.

Improved Scalability

Adding a new behavior is as simple as creating a new state file and asset. Want to add a "Stunned" state to all your enemies? Create "StunnedState.cs," make the asset, and you're done. You don't touch any existing state logic. The system naturally grows with your game. (Unity Docs - ScriptableObject)

Building the Foundation: Your Base State Class

Let me show you how I approach this. The first thing we need is an abstract base class that all our concrete states will inherit from. This ensures every state has a consistent structure that the State Machine Controller can interact with.

Here's the exact code I use:

// File: State.cs
using UnityEngine;

public abstract class State : ScriptableObject
{
    // This method is called once when the FSM enters this state.
    public virtual void OnEnter(StateMachine machine) { }

    // This method is called every frame while the FSM is in this state.
    public abstract void OnUpdate(StateMachine machine);

    // This method is called every fixed framerate frame while the FSM is in this state.
    public virtual void OnFixedUpdate(StateMachine machine) { }

    // This method is called once when the FSM exits this state.
    public virtual void OnExit(StateMachine machine) { }
}

This is the foundation of our entire system. Every state we create—Patrol, Chase, Attack, Flee—will inherit from this base class. Notice that OnUpdate is abstract, which means every concrete state must implement it. That's intentional. Every state needs to do something each frame. We've also added an OnFixedUpdate method for physics-based logic that needs to be in sync with Unity's physics engine.

The OnEnter and OnExit methods are virtual, not abstract. That means they're optional. Some states need setup logic when they start (like playing an animation), others don't. Some need cleanup when they end (like stopping an animation), others don't. We give states the flexibility to override these only if they need them.

The key detail here is that each method receives a reference to the StateMachine controller. This is how states access components like the NavMeshAgent or Animator without having to call GetComponent repeatedly.

The Engine Room: Creating Your State Machine Controller

Now let's build the engine that drives everything—the state machine itself. This MonoBehaviour lives on your enemy GameObject and is responsible for running the current state's logic and handling transitions.

Here's my implementation:

// File: StateMachine.cs
using UnityEngine;
using UnityEngine.AI;

public class StateMachine : MonoBehaviour
{
    [SerializeField] private State _currentState;

    public NavMeshAgent Agent { get; private set; }
    // We can add more cached components here, like Animator, Rigidbody, etc.

    void Awake()
    {
        Agent = GetComponent<NavMeshAgent>();
    }

    void Start()
    {
        // Initialize the FSM by entering the first state.
        _currentState?.OnEnter(this);
    }

    void Update()
    {
        // Execute the current state's logic.
        _currentState?.OnUpdate(this);
    }

    void FixedUpdate()
    {
        // Execute the current state's physics-based logic.
        _currentState?.OnFixedUpdate(this);
    }

    public void TransitionToState(State nextState)
    {
        _currentState?.OnExit(this); // Call the exit logic of the old state.
        _currentState = nextState;
        _currentState?.OnEnter(this); // Call the enter logic of the new state.
    }
}

This is beautifully simple, and that's exactly the point. The State Machine Controller doesn't need to know anything about what the states actually do. It just needs to call their methods at the right time. We've added an Awake method to cache our NavMeshAgent and a FixedUpdate to handle physics calls.

In Start, we enter the initial state by calling its OnEnter method. In Update, we run the current state's OnUpdate logic every frame. And when a state decides it's time to transition, it calls TransitionToState, which handles exiting the old state and entering the new one cleanly.

The ?. operator is a null-check shorthand. If _currentState is null, nothing happens. This prevents errors during setup.

Your First Concrete State (Patrol Logic That Actually Works)

Now let's create an actual state. This is where your game ai patterns come to life. Here's a patrol state that moves between waypoints and transitions to chase when it spots the player:

// File: PatrolState.cs
using UnityEngine;

[CreateAssetMenu(menuName = "AI/States/Patrol")]
public class PatrolState : State
{
    public override void OnUpdate(StateMachine machine)
    {
        // Implement patrol logic here (e.g., move between waypoints).

        // Check for a transition condition.
        if (IsPlayerInSight(machine))
        {
            // Transition to the Chase state (assuming it's assigned somewhere).
            // machine.TransitionToState(chaseState);
        }
    }

    private bool IsPlayerInSight(StateMachine machine)
    {
        // Logic to detect the player.
        return false;
    }
}

The [CreateAssetMenu] attribute is crucial. This adds a menu item to Unity's right-click menu in the Project window, so designers can create PatrolState assets directly from the editor. That's what makes this system designer-friendly.

Notice how the state receives the StateMachine reference in OnUpdate. This lets it call machine.TransitionToState() when it's time to switch states. The state is checking for transition conditions every frame—in this case, "Is the player in sight?"—and when that condition becomes true, it triggers the transition.

How Games You Love Use These Game AI Patterns

I've seen this technique used brilliantly in some of my favorite games. Let me share three examples that really showcase how powerful these patterns can be when implemented well.

Pac-Man: The Ghost AI That Defined a Generation

What I find fascinating about this approach is how Pac-Man's ghosts use distinct state-driven behaviors that feel alive. Each ghost has several clear states: chasing Pac-Man, fleeing when he eats a power pellet, and returning to their base after being eaten.

Here's how you can adapt this for your own game: Each ghost would have a StateMachine running a ScriptableObject state. They start in a ChaseState. When Pac-Man eats a power pellet, a global event triggers a transition to their FleeState. If caught, they transition to a ReturnToBaseState.

After analyzing dozens of games, this stands out because the ghosts' seemingly complex and distinct behaviors are actually very clean to manage with an FSM structure. The player learns to recognize and predict the ghosts' behavior based on their current state, leading to deep, strategic gameplay. That clear visual and behavioral shift when the ghosts start fleeing? That's a direct result of fsm state transitions.

The Last of Us: Clicker AI That Creates Pure Terror

One of my favorite implementations of this is in The Last of Us with the Clickers. These terrifying enemies wander around idly, making clicking sounds. If they hear a noise, they move to investigate. If they touch the player, they trigger an instant kill.

What makes this brilliant is the simplicity. A Clicker's FSM would have an IdleState (wander and click), an InvestigateState (move towards a sound), and an AttackState (kill the player). The transition from Idle to Investigate is triggered by a sound event, while the transition to Attack is triggered by a physics collision.

From a developer's perspective, this creates immense tension for the player. They must carefully watch and listen to the Clickers to understand their current state and avoid triggering a deadly transition. That's the core of the game's stealth-horror loop, and it's all driven by clean state management.

Hollow Knight: Simple Patterns That Feel Perfect

Let me tell you about how Hollow Knight handles enemy AI. A common enemy like a Vengefly will hover in place. When the Knight gets close, it swoops down to attack, then returns to its hovering position.

Here's the exact method I use when implementing this pattern: The Vengefly's FSM has two simple states—a HoverState and a SwoopAttackState. The transition from Hover to SwoopAttack is triggered by a distance check to the player. After the attack animation is complete, it transitions back to the HoverState.

This is why I always recommend studying this game's approach—the player learns the enemy's simple but predictable pattern, allowing them to time their dodges and attacks. This reliable, state-driven behavior is what makes the combat feel tough but fair. It's a perfect example of how reusable ai system unity principles create engaging gameplay.

The Smart Way to Access Components

Here's a pro tip that took me months to figure out. You don't want your states calling GetComponent every frame. That's wasteful and unnecessary. Instead, pass references to common components through the State Machine Controller.

Here's how I do it:

// In StateMachine.cs
public NavMeshAgent Agent { get; private set; }
void Awake()
{
    Agent = GetComponent<NavMeshAgent>();
}

// In a State's OnUpdate method
public override void OnUpdate(StateMachine machine)
{
    // Access the agent via the machine.
    machine.Agent.SetDestination(...);
}

The State Machine caches the NavMeshAgent reference in Awake. Now every state can access it via machine.Agent without any performance overhead. This same pattern works for Animators, Transforms, Rigidbodies—any component your states need.

I've found this works best when you cache all commonly-used components in the State Machine's Awake method. Then your states just access them through the machine reference. Clean, fast, and no repeated GetComponent calls. (Unity Learn - Creating a Reusable AI System with Scriptable Objects)

Separating Transitions for Maximum Flexibility

For even greater modularity, you can encapsulate transition logic into its own ScriptableObjects. This is an advanced technique, but trust me, you'll thank me later for this tip when you're working on complex AI systems.

Here's the structure:

// A state can hold a list of transitions.
public class State : ScriptableObject
{
    public Transition[] transitions;
    // ...
}

With this approach, a state doesn't hardcode its transitions. Instead, it holds a list of Transition objects. Each Transition asset checks its own condition and knows which state to jump to if that condition is met.

This makes your states even more reusable. Your PatrolState doesn't need to know anything about ChaseState. It just iterates through its list of transitions each frame, checking if any condition is true. If one is, it follows that transition.

From my time at CMU, I learned that this level of abstraction is what separates hobbyist AI systems from professional ones. It takes more setup initially, but the flexibility is worth it when your game grows.

Setup and Cleanup: OnEnter and OnExit

Use the OnEnter method to perform setup actions like starting an animation or setting a NavMeshAgent's speed. Use OnExit to clean up, such as stopping an animation. This is where state transitions feel polished instead of janky.

Let me show you a practical example:

public class AttackState : State
{
    public override void OnEnter(StateMachine machine)
    {
        machine.Animator.SetTrigger("Attack"); // Start attack animation
    }
    public override void OnExit(StateMachine machine)
    {
        machine.Agent.isStopped = false; // Resume movement
    }
}

When the AI enters the Attack state, OnEnter immediately triggers the attack animation. When it exits the Attack state (maybe because the player ran away), OnExit resumes the NavMeshAgent's movement so the AI can chase again.

This one took me a couple of days to crack when I was first learning FSMs. I kept having issues where animations would stick or movement would freeze because I wasn't cleaning up properly. The key insight is that OnEnter and OnExit are your setup and teardown hooks. Use them religiously.

Walking Through a Complete Unity Patrol Chase AI Implementation

Alright, let's tackle this together. I'm going to walk you through building a complete system from scratch. This is a 3D enemy that patrols between waypoints and chases the player when they get close.

What We're Building with Scriptable Finite State Machine Unity

The goal is to create a 3D enemy that patrols between a set of waypoints and transitions to a "Chase" state when the player enters its line of sight. This is the foundation for most enemy ai behavior unity systems.

Unity Editor Setup

Before we write any code, set up your scene:

1. Create a Plane for the ground
2. Bake a NavMesh (Window > AI > Navigation)
3. Create a Player GameObject (e.g., a Capsule) and tag it "Player"
4. Create an Enemy GameObject (e.g., a Cube) with a NavMeshAgent component and the StateMachine.cs script attached
5. Create two empty GameObjects as waypoints in your scene

(Unity Docs - NavMeshAgent)

Step-by-Step Implementation

Let me show you how I approach this. First, we already have our State.cs and StateMachine.cs scripts from earlier. Now we'll create the two concrete states: Patrol and Chase.

Creating the PatrolState

This state handles waypoint navigation and player detection:

// File: PatrolState.cs
using UnityEngine;
using UnityEngine.AI;

[CreateAssetMenu(menuName = "AI/States/Patrol")]
public class PatrolState : State
{
    public Transform[] waypoints;
    public float detectionRadius = 10f;
    public State chaseState; // Assign the ChaseState asset in the Inspector

    private int _waypointIndex = 0;

    public override void OnEnter(StateMachine machine)
    {
        if (waypoints == null || waypoints.Length == 0)
        {
            Debug.LogError("PatrolState: Waypoints are not set up!");
            return;
        }
        // Set the first destination
        machine.Agent.SetDestination(waypoints[_waypointIndex].position);
    }

    public override void OnUpdate(StateMachine machine)
    {
        // 1. Patrol Logic: Cycle through waypoints
        if (machine.Agent.remainingDistance < machine.Agent.stoppingDistance)
        {
            _waypointIndex = (_waypointIndex + 1) % waypoints.Length;
            machine.Agent.SetDestination(waypoints[_waypointIndex].position);
        }
    }

    public override void OnFixedUpdate(StateMachine machine)
    {
        // 2. Transition Logic: Check for player
        Collider[] colliders = Physics.OverlapSphere(machine.transform.position, detectionRadius);
        foreach (var collider in colliders)
        {
            if (collider.CompareTag("Player"))
            {
                machine.TransitionToState(chaseState);
                return;
            }
        }
    }
}

Here's what's happening: In OnEnter, we add a safety check and then set the first waypoint as the destination. In OnUpdate, we now *only* handle the patrol logic—checking if the agent has reached its current waypoint and cycling to the next.

The transition logic has been moved to OnFixedUpdate. This is critical for performance and reliability. Because player detection uses Physics.OverlapSphere, it should run on the fixed physics timestep, not the variable frame rate of Update. This prevents missed detections and ensures our AI's reactions are consistent.

Creating the ChaseState

Now for the chase behavior with our navmesh enemy ai:

// File: ChaseState.cs
using UnityEngine;
using UnityEngine.AI;

[CreateAssetMenu(menuName = "AI/States/Chase")]
public class ChaseState : State
{
    public float loseSightDistance = 15f;
    public State patrolState; // Assign the PatrolState asset in the Inspector

    private Transform _player;

    public override void OnEnter(StateMachine machine)
    {
        _player = GameObject.FindGameObjectWithTag("Player").transform;
    }

    public override void OnUpdate(StateMachine machine)
    {
        if (_player == null) return;

        // 1. Chase Logic: Follow the player
        machine.Agent.SetDestination(_player.position);

        // 2. Transition Logic: Check if player is too far away
        if (Vector3.Distance(machine.transform.position, _player.position) > loseSightDistance)
        {
            machine.TransitionToState(patrolState);
        }
    }
}

In OnEnter, we find and cache the player's transform. In OnUpdate, we continuously set the player's position as the NavMeshAgent destination. The transition logic checks if the player has escaped beyond the "lose sight" distance. If so, we return to patrolling.

Configuring in the Editor

After working on multiple Unity projects, I've learned that the editor setup is just as important as the code:

1. Right-click in the Project window: Create > AI > States > Patrol
2. Right-click again: Create > AI > States > Chase
3. Select the PatrolState asset. Assign your waypoint transforms in the "Waypoints" array and drag the ChaseState asset into the "Chase State" field
4. Select the ChaseState asset. Drag the PatrolState asset into the "Patrol State" field
5. Select your Enemy GameObject. Drag the PatrolState asset into the "Current State" field on the StateMachine component

Press Play. Your enemy now patrols between waypoints, detects the player, chases them, and returns to patrolling when the player escapes. That's a complete, working AI system built with clean, modular code.

When to Use ScriptableObject FSM vs. Simple Switch Statements

Here's a question I get all the time: "When should I actually use this pattern?" Let me break it down with a comparison table:

The honest answer? If your AI only has two or three states and you're certain it won't grow, a simple switch statement is fine. But the moment you find yourself adding a fourth or fifth state, or when you need to reuse behaviors across multiple enemy types, the approach will save you massive amounts of time.

Wrapping Up: Your Path to Clean Enemy AI

So here's what we've covered—a complete scriptable finite state machine unity system that transforms messy, bloated AI code into clean, modular, designer-friendly architecture. Instead of wrestling with thousand-line mega-scripts where every behavior is tangled together, you now have a system where each state is isolated, reusable, and easy to debug.

The beauty of this approach is how it scales. Your first enemy might only need Patrol and Chase states. But when your game designer comes to you asking for enemies that flee when injured, investigate sounds, or call for backup, you're not rewriting your entire AI system. You're just creating new ScriptableObject state assets and configuring transitions. That's the power of this pattern.

From my time working on real game projects, this is the difference between an AI system that becomes your worst maintenance nightmare and one that actually empowers your team to iterate quickly. The upfront investment in setting up the base classes pays itself back the moment you create your third enemy type or add your fifth behavior state.

Ready to Start Building Your First Game?

If this AI system has you excited about what's possible in Unity, you're going to love what comes next. At Outscal, we've designed a course that takes you from "I've never opened Unity" to "I just shipped a working game" in a structured, hands-on way.

The Mr. Blocks: Creating Your First Video Game course walks you through building a complete game from scratch. You'll learn Unity fundamentals, C# scripting, game mechanics, and how to implement systems like the FSM we just covered. By the end, you'll have a finished game you can show off in your portfolio.

This is the course I wish existed when I was making my transition into game development. No fluff, no theory dumps—just practical, project-based learning that gets you building real games.

Key Takeaways

• A Finite State Machine (FSM) ensures your AI can only be in one state at a time, preventing the "mega-script" problem where all behaviors are tangled together in one file.
• Using ScriptableObjects for states creates reusable, designer-friendly AI assets that can be configured in the Unity Inspector without writing new code.
• The State base class defines OnEnter, OnUpdate, and OnExit methods that the State Machine Controller calls at the appropriate times during state lifecycle.
• The StateMachine controller lives on your enemy GameObject and handles running the current state's logic and managing transitions between states.
• Cache component references in the State Machine's Awake method (like NavMeshAgent and Animator) to avoid expensive GetComponent calls in state logic.
• States can hold lists of Transition objects for even greater modularity, allowing you to separate transition logic from state behavior completely.
• Use OnEnter for setup actions (starting animations, setting speeds) and OnExit for cleanup (stopping animations, resuming movement) to make transitions feel polished.
• Real games like Pac-Man, The Last of Us, and Hollow Knight use FSM patterns to create predictable but engaging enemy behaviors that players can learn and master.