Neural network example using Pylearn2

I was recently looking into using a neural network for a project so I started looking into some of the available Python libraries. The one I ended up using was Pylearn2 which is a fast and powerful library for machine learning that is mainly built upon Theano.

Pylearn2 is under development and is still a bit rough around the edges and the documentation is limited and in some instances not correct. The recommended way of using it is by writing YAML scripts and if you are ok with that you can probably manage with the existing documentation. But if you, like me, want to use it as a standard Python library you have better be prepared to read the code. One thing that would have saved me some time was a complete example of how to use Pylearn2 as a standalone library, so what follows is a simple example of creating a neural network for solving the XOR problem.

The XOR problem is stated as follows, create a neural network that given two binary inputs, 0 or 1, the output should be a 1 if exactly one of the inputs are 1 and 0 otherwise.

Pylearn2 has a dataset implementation that in its simplest form needs a collection of datapoints in a 2D Numpy array named X and a 2D array named y containing the answers. We can create a dataset by creating a new class that inherits from DenseDesignMatrix:

class XOR(DenseDesignMatrix):
    def __init__(self):
        self.class_names = ['0', '1']
        X = [[randint(0, 1), randint(0, 1)] for _ in range(1000)]
        y = []
        for a, b in X:
            if a + b == 1:
                y.append([0, 1])
                y.append([1, 0])
        X = np.array(X)
        y = np.array(y)
        super(XOR, self).__init__(X=X, y=y)

ds = XOR()

Note that we are using two columns in the target variable y, a 1 in the first column signifies a output of 0 and a 1 in the second columns signifies a output of 1.

Next we need to create the layers in the neural net. To be able to solve the XOR problem we need a hidden layer with at least two neurons:

hidden_layer = mlp.Sigmoid(layer_name='hidden', dim=2, irange=.1, init_bias=1.)

The hidden layer uses a standard sigmoid activation function and the weights are initialized in the range -0.1 to 0.1 (using the irange argument). We also add a bias to the two neurons with value 1.0.

We use a softmax layer with two nodes as output layer. The output from the two nodes is between 0 and 1 and the sum of the output from all nodes in the layer is 1.

output_layer = mlp.Softmax(2, 'output', irange=.1)

To train the network we use a Stochastic Gradient Descent (SGD) method which we initialize like this:

trainer = sgd.SGD(learning_rate=.05, batch_size=10, termination_criterion=EpochCounter(400))

We use a simple termination criterion that runs for 400 epochs, more advanced termination criteria are of course available.

To initialize the neural network and setup the training we do like this:

layers = [hidden_layer, output_layer]
ann = mlp.MLP(layers, nvis=2)
trainer.setup(ann, ds)

We put the layers in the Multi-Layer Perceptron class with two inputs and then setup the trainer with the class and the dataset.

We then train the neural network until the termination criteria is reached:

while True:
    if not trainer.continue_learning(ann):

After the training is complete we of course wants to test that it works. We do this by using the fprop-method that takes the inputs as Theano variables:

inputs = np.array([[0, 1]])
print ann.fprop(theano.shared(inputs, name='inputs')).eval()

This should yield a answer like this:

[[ 0.00526688  0.99473312]]

Meaning that the network correctly predicts that the output should be a 1.

See here for the complete source code of the example.

  • Brian

    This is excellent! Pylearn2 looks very capable, but the learning curve is steep. Do you have any examples of how to use regularization in your training above? I see pylearn2 can use L1, L2, dropout but I cant figure out how to train a model (adjusting your code above) to use these three types (and compare them).

    • arngarden

      No sorry, have no experience of using regularization in Pylearn2.

    • Ian Goodfellow

      I think I remember writing about how to use regularization in one of the pylearn2 ipython notebook tutorials. Either logistic regression or MLP, I forget which one.

      • jayjay

        I wish pylearn2 dev could have similar tutorials to this. Thanks a lot!

  • akiniwa

    This is absolutely great!!
    I think official tutorials are so difficult…

  • Steven

    This is amazing. I hate the way they use YAML in the tutorials. I feel like I’m using an executable instead of a library and that I’m simply editing the configuration files for it.

    I just don’t feel like I’m going to learn anything out of editing YAML files let alone extend the library to suit my needs.

  • jayjay

    Love this. I wish pylearn2 dev could have similar tutorials to this. Thanks a lot!

  • peter

    What I’m really missing here is things like:

    from pylearn2.models import mlp
    from pylearn2.training_algorithms import sgd


    I’m sure this would have been a nice example otherwise.

    • arngarden

      I omitted import statements in the blog post, but they are in the complete code that is linked to at the bottom of the post.

  • Pingback: MultiSubject Neural Nets Project – ahyeonhwang