Neuroblast Neural Network AIs

This lesson will teach you how implement a generalized learning solution

Neural Networks



    We've used a variety of different machine learning solutions throughout this course. However, for our games so far, most of our game AI has been domain-specific.

    Domain-specific AIs are coded in a way that the programmer assumes something about the inputs and how they relate to the output. When we made our "Learning AI" in the previous lesson, we assumed that the average of the position differences would be a good indicator of when to fire.

    Was it? Maybe. But as there are different inputs and more complicated scenarios, it is helpful to use more general solutions. This is where we can use a Neural Network.

    A Neural network uses a selection of inputs and a number of nodes of hidden layers to arrive at an output.

    The nodes work like synapses of the brain. Given certain conditions, they will trigger and impact whether other nodes will trigger. This triggering process starts at the inputs, and goes all the way to the output.

Our Neural Network AI



    The Neural Network AI that we will implement will use the inputs we have already set up, the difference in position and velocity of the enemy and the player.

    When we make our neural network, we'll decide how many layers the network should have. More layers mean more detailed analysis of the input features, but it also means the model takes longer to train.

    We'll gather inputs and their results, and pass that into our model to train it. Note: Training neural networks takes some time, so we'll change our system to not try to train continually.

    To set up the network, we have to determine how many hidden layers we'll use. We'll experiment with different layer types to see what works best.

Brain Training Code



    To set up our new brain, we're going to do the following:

    1. Add a threshold to constants.py

    2. Set up a new class in brain.py called NeuralNetworkBrain

    3. Initialize a basic neural network model using Keras.

    4. Set up the model to train on inputs in the training mode

    5. Set up our model to make predictions based on our trained model

    To start, let's update the constants.py file with our threshold. 
    learning_brain_acceptable_range = 0.25
# The lower the activation minimum is, the more likely the enemy is to fire
neural_network_activation_minimum = 0.5
#


    Next, we'll set up some imports in the brain.py file as well as add some new variables to our Brain class. 
    from game_utils import display_text
import constants
import pygame
# Importing modules related to keras and other functions necessary for neural network
from keras.models import Sequential
from keras.layers import Dense
import numpy as np
import keras.backend as K
from math import fabs
#


class Brain:
    
    def __init__(self):
        self.trained = True
        self.brain_name = "AI: Fire Constantly"
        self.mapShots = {}
        self.mapHits = {}
        # These two variables will be used by our Neural Network Class
        self.train_at_end = False
        self.currentInputs = np.array([list((0,0,0,0))])
        #


    Go to the bottom of the brain.py file and set up the NeuralNetworkBrain class.

    Since we're going to have our brain only train when the player finishes the training mode, we'll set the train_at_end variable to True. 
    class NeuralNetworkBrain(Brain):
    
    def __init__(self):
        Brain.__init__(self)
        self.brain_name = "AI: Neural Network"
        self.train_at_end = True


    Next, we're going to initialize our Neural Network using keras. To do that, we're going to:

    1. We'll set up a Sequential keras model, which means we're going to add all of our inputs at the beginning, and expect our output at the end of the model. There are other ways to set up Neural Networks, but this is a simple one that we will use.

    2. Set up the input nodes for the model.

    3. Set up the hidden nodes for the model.

    4. Set up the output node for the model

    5. Set up code that will track the status of the Neural Network for visualization later. 
    class NeuralNetworkBrain(Brain):
    
    def __init__(self):
        Brain.__init__(self)
        self.brain_name = "AI: Neural Network"
        self.train_at_end = True
        # We will use this weights variable when we start to visualize the network
        self.weights = []
        self.keras = Sequential()
        
        # We add the layers to the neural network here
        # The first layer is the input layer, where we specify the number of nodes, the shape of our input (4 variables), and the activation method
        # The 'relu' activation method stands for Rectified Linear Unit. This determines whether or not the node will count as "active"
        # Using this method means the threshold will only go to a minimum of 0, so there is no negative activation
        self.keras.add(Dense(4, input_shape=(4,), activation='relu'))
        # This adds a hidden layer to our network to make the relationship between variables more complicated.
        self.keras.add(Dense(4, activation='relu'))
        # The final layer is our output layer. The 'Sigmoid' activation will keep the output value between 0 and 1
        self.keras.add(Dense(1, activation='sigmoid'))
        # Once the layers are set up, we compile the neural network so it can be used by training and firing decisions
        # The mean squared error loss method ensures a positive result that minimizes large mistakes.
        # The 'sgd' optimizer stands for Stochastic Gradient Descent. It has a special way of determining how fast the neural network will learn.
        # The 'accuracy' metrics does not alter the way the method is trained, but we'll use that later when we visualize our data
        self.keras.compile(loss='mean_squared_error', optimizer='sgd', metrics=['accuracy'])
        
        # Now that the layers are set up, we set up initial weights for our visualization later
        for layer in self.keras.layers:
            self.weights.append(layer.get_weights()[0])
        #


    There are many ways to set up a Neural Network. We won't cover all of them in this course, but this will give you a taste of what the Neural Network looks like.

    Once we finish setting up the network, we will change some of the parameters of the network and see how the network reacts.

    For now, we're going to transition to setting up the train() method for our class 
        def train(self):
        # x is the inputs to our model
        # y is the outputs based on those inputs
        x = []
        y = []
        for k,v in self.mapShots.items():
            # v stands for a set of player variables
            # k stands for a bullet
            if k in self.mapHits:
                a = list(v)
                x.append(a)
                y.append(self.mapHits[k])
        
        # Fit the data to the model
        # When we train neural networks, we make multiple passes over the network. 'nb_epoch' stands for the number of times we go over the dataset to train it
        # Batch size stands for the number of sets of data you use for a pass. So the below example will do 150 passes of training, picking 10 sets of data each time
        # The larger the batch size, and the larger the number of epochs, the more accurate the model might be, but it might also take longer to train.
        self.keras.fit(np.array(x),np.array(y),nb_epoch=150,batch_size=10)
        # We print some output to the interpreter to determine how accurate it is
        scores = self.keras.evaluate(np.array(x), np.array(y))
        print("\n%s: %.2f%%" % (self.keras.metrics_names[1], scores[1]*100))

        # Cache trained weights for visualization
        # Element 0 is weights, 1 is biases
        for layer in self.keras.layers:
            self.weights.append(layer.get_weights()[0])


    The last method we need to add is how we're making our firing decision inside thte fire_decision method. 
        def fire_decision(self, player_variables):
        difference_x = player_variables[0]
        difference_y = player_variables[1]
        difference_velx = player_variables[2]
        difference_vely = player_variables[3]
        # Inputs need to be put into a numpy array
        network_inputs = np.array([list((difference_x, difference_y, difference_velx, difference_vely))])
        self.currentInputs = network_inputs
        
        #The predict method will return an output based on our inputs
        keras_prediction = self.keras.predict(network_inputs)
        if (keras_prediction >= constants.neural_network_activation_minimum):
            return True
        else:
            return False


    The last thing we need to do is update the Play state in gamestates.py to have it only train the model at the end for our neural network (otherwise it will pause the game every time it has to train it.

    First, let's set the state to use our new class inside the init() method 
            self.training_mode = training_mode
        if game_utils.trained_brain:
            self.play_brain = game_utils.trained_brain
        else:  
            # Update to use the Neural Network Brain
            self.play_brain = brain.NeuralNetworkBrain()
            #


    Now, in the update() method, update when we retrain on a timer. 
            self.training_timer += delta_time
        # Update condition not to run when our brain specifies to only train at the end
        if (self.training_timer > self.retrain_time and not self.play_brain.train_at_end):
        #
            self.play_brain.train()
            self.training_timer = 0


    Now, run the code and go to the training mode. After a little bit of time, press "Esc" and watch the training happen on the command line.

    After the training is finished, go to the play mode. The training method here is less specific than our old methods, so you may end up with the result that enemies continually fire, not fire, or fire only when specific conditions are met.

    Important Notes:

    1. The first time you select "Train", it may take up to 2-3 minutes for it to run. This is because it is turning on the GPU processes necessary for performing machine learning. Just be patient, even if the window looks like it is frozen.
    2. After you press "Esc" in the training Mode, the console in Spyder will show a running log of performing the necessary neural network training functions.
    3. When you exit out of play mode, the model will also be retrained.


    In the video above, the AI picked up that I was usually going up and to the right when bullets hit me. So it decided to take that into account when deciding when to fire.

    Feel free to go back and use training mode more if you feel the enemies aren't acting quite logical yet. General intelligence will take longer to train than domain intelligence!

Brain Visualization



    Our visualization for domain intelligence was just a few numbers, but it's harder to visualize the way that Neural Networks think.

    We're going to set up a way to visualize our network by doing the following:

    1. Setting up constants related to visualizing our network.
    2. Create helper methods related to setting up our visualization
    3. Set up our draw method to draw the entire network with all of its layers and activations

    We'll start with the constants that we'll need to keep track of. Add these variables to constants.py 
    vertical_distance_between_layers = 120
horizontal_distance_between_neurons = 110
neuron_radius = 40
number_of_neurons_in_widest_layer = 4
node_width = 25
node_height = 20
network_left_margin = 10
network_bottom_margin = 2
error_bar_x_position = 14
output_y_position = 15
network_top_offset = 180
network_left_offset = 145


    Next, we're going to set up helper methods inside our NeuralNetworkBrain class in brain.py to make drawing the network a little easier. We're going to set up the following methods:

    1. The method get_activations will look at our trained model and determine how the inputs we give the model impact its output
    2. The layer_left_margin method will align the network in the window based on our constants
    3. The get_synapse_colour method will color node connections based on weight

    We'll start with the get_activations method, add this method to the NeuralNetworkBrain class. 
        def get_activations(self, model, model_inputs, print_shape_only=False, layer_name=None):
        activations = []
        inp = model.input
    
        model_multi_inputs_cond = True
        if not isinstance(inp, list):
            inp = [inp]
            model_multi_inputs_cond = False
    
        # Get all the outputs for all the layers
        outputs = [layer.output for layer in model.layers if
                   layer.name == layer_name or layer_name is None]
    
        # Get all the trained functions for determining outputs
        funcs = [K.function(inp + [K.learning_phase()], [out]) for out in outputs]
    
        if model_multi_inputs_cond:
            list_inputs = []
            list_inputs.extend(model_inputs)
            list_inputs.append(1.)
        else:
            list_inputs = [model_inputs, 1.]
    
        # Place the activations in a list, this will be used when visualizing the network
        layer_outputs = [func(list_inputs)[0] for func in funcs]
        for layer_activations in layer_outputs:
            activations.append(layer_activations)
        return activations


    Next, we'll set up two methods for simplifying how the network visualization works 
        def layer_left_margin(self, number_of_neurons):
        return (constants.network_left_margin + 
                constants.horizontal_distance_between_neurons * 
                (constants.number_of_neurons_in_widest_layer - number_of_neurons) / 2)
        
    def get_synapse_colour(self, weight):
        if weight > 0:
            return 0, 255, 0
        else:
            return 255, 0, 0


    Finally, we'll set up our new draw method. 
        def draw(self, screen):
        # Set up necessary data to be used
        model = self.keras
        model_inputs = self.currentInputs
        weights = self.weights
        
        # We create a couple surfaces to draw on, the nsurf will be used to draw the nodes, the surf will be used to draw lines
        surf = pygame.Surface((640,720))
        nsurf = pygame.Surface((640,720))
        nsurf.fill((255,0,255))
        nsurf.set_colorkey((255,0,255))
        
        # Get the data from the model based on our inputs
        graph = self.get_activations(model, model_inputs)
        y = constants.network_bottom_margin
        # For each layer of nodes
        for layer in range(len(graph)):
            # Align it to the middle of the screen
            x = self.layer_left_margin(len(graph[layer][0]))
            # Draw each node in the layer
            for node in range(len(graph[layer][0])):
                if (layer+1 != len(graph)):
                    # Draw all connections between the nodes
                    for synapse in range(len(weights[layer+1][node])):
                        lo = constants.network_left_offset
                        to = constants.network_top_offset
                        x2 = synapse * constants.horizontal_distance_between_neurons + self.layer_left_margin(len(graph[layer+1][0]))
                        y2 = y + constants.vertical_distance_between_layers
                        pygame.draw.line(surf,self.get_synapse_colour(weights[layer+1][node][synapse]),
                                         (int(x+lo), int(y+to)), (int(x2+lo), int(y2+to)),
                                         max(1,int(fabs(weights[layer+1][node][synapse]))))
                lo = constants.network_left_offset
                to = constants.network_top_offset
                pygame.draw.circle(nsurf,(180,180,200),(int(x+lo), int(y+to)),constants.neuron_radius)
                display_text(str(round(graph[layer][0][node], 2)), x + 2+lo, y+to, constants.BLACK, nsurf)
                # After drawing one node, move to the right to draw the next one
                x += constants.horizontal_distance_between_neurons
            # After drawing one layer, move down to draw the next layer
            y += constants.vertical_distance_between_layers
        screen.blit(surf,(640,0))
        screen.blit(nsurf,(640,0)) 
        #Finally, add the AI name to the screen
        display_text(self.brain_name, 960, 60, constants.WHITE, screen)


    Now, when you play the game, you will be able to see the screen updating with information about the neural network.

    The key node to watch is the bottom node. If that value goes above .5 (our current constant), then the enemy will fire.

    The network will always be drawn for the most recent enemy to appear on the screen



    Troubleshooting

    If you see this error: cannot convert 1.0 of eagertensor of dtype int32 make sure that you have version 2.1.5 of keras installed.

Training and Playing



    Now that we have a basic network set up, we can play with the structure of the network.

    Here are some things you can try

    1. Try adding more hidden layers of different sizes.
    2. Try changing the threshold to activate the enemy firing
    3. Try changing the fire rate of the enemies, that may have an impact on how effective the enemy's strategy is.