Neuroblast Neural Network AIs

This lesson will teach you how implement a generalized learning solution

Neural Networks

    We've used a variety of different machine learning solutions throughout this course. However, for our games so far, most of our game AI has been domain-specific.

    Domain-specific AIs are coded in a way that the programmer assumes something about the inputs and how they relate to the output. When we made our "Learning AI" in the previous lesson, we assumed that the average of the position differences would be a good indicator of when to fire.

    Was it? Maybe. But as there are different inputs and more complicated scenarios, it is helpful to use more general solutions. This is where we can use a Neural Network.

    A Neural network uses a selection of inputs and a number of nodes of hidden layers to arrive at an output.

    The nodes work like synapses of the brain. Given certain conditions, they will trigger and impact whether other nodes will trigger. This triggering process starts at the inputs, and goes all the way to the output.

Our Neural Network AI

    The Neural Network AI that we will implement will use the inputs we have already set up, the difference in position and velocity of the enemy and the player.

    When we make our neural network, we'll decide how many layers the network should have. More layers mean more detailed analysis of the input features, but it also means the model takes longer to train.

    We'll gather inputs and their results, and pass that into our model to train it. Note: Training neural networks takes some time, so we'll change our system to not try to train continually.

    To set up the network, we have to determine how many hidden layers we'll use. We'll experiment with different layer types to see what works best.

Brain Training Code

    To set up our new brain, we're going to do the following:

    1. Add a threshold to

    2. Set up a new class in called NeuralNetworkBrain

    3. Initialize a basic neural network model using Keras.

    4. Set up the model to train on inputs in the training mode

    5. Set up our model to make predictions based on our trained model

    To start, let's update the file with our threshold.
    learning_brain_acceptable_range = 0.25
    # The lower the activation minimum is, the more likely the enemy is to fire
    neural_network_activation_minimum = 0.5

    Next, we'll set up some imports in the file as well as add some new variables to our Brain class.
    from game_utils import display_text
    import constants
    import pygame
    # Importing modules related to keras and other functions necessary for neural network
    from keras.models import Sequential
    from keras.layers import Dense
    import numpy as np
    import keras.backend as K
    from math import fabs
    class Brain:
        def __init__(self):
            self.trained = True
            self.brain_name = "AI: Fire Constantly"
            self.mapShots = {}
            self.mapHits = {}
            # These two variables will be used by our Neural Network Class
            self.train_at_end = False
            self.currentInputs = np.array([list((0,0,0,0))])

    Go to the bottom of the file and set up the NeuralNetworkBrain class.

    Since we're going to have our brain only train when the player finishes the training mode, we'll set the train_at_end variable to True.
    class NeuralNetworkBrain(Brain):
        def __init__(self):
            self.brain_name = "AI: Neural Network"
            self.train_at_end = True

    Next, we're going to initialize our Neural Network using keras. To do that, we're going to:

    1. We'll set up a Sequential keras model, which means we're going to add all of our inputs at the beginning, and expect our output at the end of the model. There are other ways to set up Neural Networks, but this is a simple one that we will use.

    2. Set up the input nodes for the model.

    3. Set up the hidden nodes for the model.

    4. Set up the output node for the model

    5. Set up code that will track the status of the Neural Network for visualization later.
    class NeuralNetworkBrain(Brain):
        def __init__(self):
            self.brain_name = "AI: Neural Network"
            self.train_at_end = True
            # We will use this weights variable when we start to visualize the network
            self.weights = []
            self.keras = Sequential()
            # We add the layers to the neural network here
            # The first layer is the input layer, where we specify the number of nodes, the shape of our input (4 variables), and the activation method
            # The 'relu' activation method stands for Rectified Linear Unit. This determines whether or not the node will count as "active"
            # Using this method means the threshold will only go to a minimum of 0, so there is no negative activation
            self.keras.add(Dense(4, input_shape=(4,), activation='relu'))
            # This adds a hidden layer to our network to make the relationship between variables more complicated.
            self.keras.add(Dense(4, activation='relu'))
            # The final layer is our output layer. The 'Sigmoid' activation will keep the output value between 0 and 1
            self.keras.add(Dense(1, activation='sigmoid'))
            # Once the layers are set up, we compile the neural network so it can be used by training and firing decisions
            # The mean squared error loss method ensures a positive result that minimizes large mistakes.
            # The 'sgd' optimizer stands for Stochastic Gradient Descent. It has a special way of determining how fast the neural network will learn.
            # The 'accuracy' metrics does not alter the way the method is trained, but we'll use that later when we visualize our data
            self.keras.compile(loss='mean_squared_error', optimizer='sgd', metrics=['accuracy'])
            # Now that the layers are set up, we set up initial weights for our visualization later
            for layer in self.keras.layers:

    There are many ways to set up a Neural Network. We won't cover all of them in this course, but this will give you a taste of what the Neural Network looks like.

    Once we finish setting up the network, we will change some of the parameters of the network and see how the network reacts.

    For now, we're going to transition to setting up the train() method for our class
        def train(self):
            # x is the inputs to our model
            # y is the outputs based on those inputs
            x = []
            y = []
            for k,v in self.mapShots.items():
                # v stands for a set of player variables
                # k stands for a bullet
                if k in self.mapHits:
                    a = list(v)
            # Fit the data to the model
            # When we train neural networks, we make multiple passes over the network. 'nb_epoch' stands for the number of times we go over the dataset to train it
            # Batch size stands for the number of sets of data you use for a pass. So the below example will do 150 passes of training, picking 10 sets of data each time
            # The larger the batch size, and the larger the number of epochs, the more accurate the model might be, but it might also take longer to train.
            # We print some output to the interpreter to determine how accurate it is
            scores = self.keras.evaluate(np.array(x), np.array(y))
            print("\n%s: %.2f%%" % (self.keras.metrics_names[1], scores[1]*100))
            # Cache trained weights for visualization
            # Element 0 is weights, 1 is biases
            for layer in self.keras.layers:

    The last method we need to add is how we're making our firing decision inside thte fire_decision method.
        def fire_decision(self, player_variables):
            difference_x = player_variables[0]
            difference_y = player_variables[1]
            difference_velx = player_variables[2]
            difference_vely = player_variables[3]
            # Inputs need to be put into a numpy array
            network_inputs = np.array([list((difference_x, difference_y, difference_velx, difference_vely))])
            self.currentInputs = network_inputs
            #The predict method will return an output based on our inputs
            keras_prediction = self.keras.predict(network_inputs)
            if (keras_prediction >= constants.neural_network_activation_minimum):
                return True
                return False

    The last thing we need to do is update the Play state in to have it only train the model at the end for our neural network (otherwise it will pause the game every time it has to train it.

    First, let's set the state to use our new class inside the init() method
            self.training_mode = training_mode
            if game_utils.trained_brain:
                self.play_brain = game_utils.trained_brain
                # Update to use the Neural Network Brain
                self.play_brain = brain.NeuralNetworkBrain()

    Now, in the update() method, update when we retrain on a timer.
            self.training_timer += delta_time
            # Update condition not to run when our brain specifies to only train at the end
            if (self.training_timer > self.retrain_time and not self.play_brain.train_at_end):
                self.training_timer = 0

    Now, run the code and go to the training mode. After a little bit of time, press "Esc" and watch the training happen on the command line.

    After the training is finished, go to the play mode. The training method here is less specific than our old methods, so you may end up with the result that enemies continually fire, not fire, or fire only when specific conditions are met.

    Important Notes:

    1. The first time you select "Train", it may take up to 2-3 minutes for it to run. This is because it is turning on the GPU processes necessary for performing machine learning. Just be patient, even if the window looks like it is frozen.
    2. After you press "Esc" in the training Mode, the console in Spyder will show a running log of performing the necessary neural network training functions.
    3. When you exit out of play mode, the model will also be retrained.

    In the video above, the AI picked up that I was usually going up and to the right when bullets hit me. So it decided to take that into account when deciding when to fire.

    Feel free to go back and use training mode more if you feel the enemies aren't acting quite logical yet. General intelligence will take longer to train than domain intelligence!

Brain Visualization

    Our visualization for domain intelligence was just a few numbers, but it's harder to visualize the way that Neural Networks think.

    We're going to set up a way to visualize our network by doing the following:

    1. Setting up constants related to visualizing our network.
    2. Create helper methods related to setting up our visualization
    3. Set up our draw method to draw the entire network with all of its layers and activations

    We'll start with the constants that we'll need to keep track of. Add these variables to
    vertical_distance_between_layers = 120
    horizontal_distance_between_neurons = 110
    neuron_radius = 40
    number_of_neurons_in_widest_layer = 4
    node_width = 25
    node_height = 20
    network_left_margin = 10
    network_bottom_margin = 2
    error_bar_x_position = 14
    output_y_position = 15
    network_top_offset = 180
    network_left_offset = 145

    Next, we're going to set up helper methods inside our NeuralNetworkBrain class in to make drawing the network a little easier. We're going to set up the following methods:

    1. The method get_activations will look at our trained model and determine how the inputs we give the model impact its output
    2. The layer_left_margin method will align the network in the window based on our constants
    3. The get_synapse_colour method will color node connections based on weight

    We'll start with the get_activations method, add this method to the NeuralNetworkBrain class.
        def get_activations(self, model, model_inputs, print_shape_only=False, layer_name=None):
            activations = []
            inp = model.input
            model_multi_inputs_cond = True
            if not isinstance(inp, list):
                inp = [inp]
                model_multi_inputs_cond = False
            # Get all the outputs for all the layers
            outputs = [layer.output for layer in model.layers if
              == layer_name or layer_name is None]
            # Get all the trained functions for determining outputs
            funcs = [K.function(inp + [K.learning_phase()], [out]) for out in outputs]
            if model_multi_inputs_cond:
                list_inputs = []
                list_inputs = [model_inputs, 1.]
            # Place the activations in a list, this will be used when visualizing the network
            layer_outputs = [func(list_inputs)[0] for func in funcs]
            for layer_activations in layer_outputs:
            return activations

    Next, we'll set up two methods for simplifying how the network visualization works
        def layer_left_margin(self, number_of_neurons):
            return (constants.network_left_margin + 
                    constants.horizontal_distance_between_neurons * 
                    (constants.number_of_neurons_in_widest_layer - number_of_neurons) / 2)
        def get_synapse_colour(self, weight):
            if weight > 0:
                return 0, 255, 0
                return 255, 0, 0

    Finally, we'll set up our new draw method.
        def draw(self, screen):
            # Set up necessary data to be used
            model = self.keras
            model_inputs = self.currentInputs
            weights = self.weights
            # We create a couple surfaces to draw on, the nsurf will be used to draw the nodes, the surf will be used to draw lines
            surf = pygame.Surface((640,720))
            nsurf = pygame.Surface((640,720))
            # Get the data from the model based on our inputs
            graph = self.get_activations(model, model_inputs)
            y = constants.network_bottom_margin
            # For each layer of nodes
            for layer in range(len(graph)):
                # Align it to the middle of the screen
                x = self.layer_left_margin(len(graph[layer][0]))
                # Draw each node in the layer
                for node in range(len(graph[layer][0])):
                    if (layer+1 != len(graph)):
                        # Draw all connections between the nodes
                        for synapse in range(len(weights[layer+1][node])):
                            lo = constants.network_left_offset
                            to = constants.network_top_offset
                            x2 = synapse * constants.horizontal_distance_between_neurons + self.layer_left_margin(len(graph[layer+1][0]))
                            y2 = y + constants.vertical_distance_between_layers
                                             (int(x+lo), int(y+to)), (int(x2+lo), int(y2+to)),
                    lo = constants.network_left_offset
                    to = constants.network_top_offset
          ,(180,180,200),(int(x+lo), int(y+to)),constants.neuron_radius)
                    display_text(str(round(graph[layer][0][node], 2)), x + 2+lo, y+to, constants.BLACK, nsurf)
                    # After drawing one node, move to the right to draw the next one
                    x += constants.horizontal_distance_between_neurons
                # After drawing one layer, move down to draw the next layer
                y += constants.vertical_distance_between_layers
            #Finally, add the AI name to the screen
            display_text(self.brain_name, 960, 60, constants.WHITE, screen)

    Now, when you play the game, you will be able to see the screen updating with information about the neural network.

    The key node to watch is the bottom node. If that value goes above .5 (our current constant), then the enemy will fire.

    The network will always be drawn for the most recent enemy to appear on the screen


    If you see this error: cannot convert 1.0 of eagertensor of dtype int32 make sure that you have version 2.1.5 of keras installed.

Training and Playing

    Now that we have a basic network set up, we can play with the structure of the network.

    Here are some things you can try

    1. Try adding more hidden layers of different sizes.
    2. Try changing the threshold to activate the enemy firing
    3. Try changing the fire rate of the enemies, that may have an impact on how effective the enemy's strategy is.