Teaching a Neural Network to See: My First CNN on CIFAR-10

A computer doesn't see a cat. It sees 3,072 numbers.

That's all a CIFAR-10 image is. 32 pixels wide, 32 pixels tall, 3 color channels, which multiplies out to a grid of brightness values between 0 and 255. Somewhere in that grid there's a cat, and your job is to write something that can tell it apart from a dog, a frog, and a truck.

This is where my AI studies at UFPR stopped being abstract. I'd done the statistics and the distributions, which were fine, but you can read about gradient descent for a long time without ever watching a model learn anything. So here's what I built, what the numbers told me, and the thing I had wrong going in.

---

The problem with treating an image like a list

My first instinct was the naive one. Flatten the image into a single row of 3,072 numbers, feed it to a regular dense network, every input wired to every neuron, let the thing figure it out.

That breaks in two places.

The first is that you throw away the geometry. Once you flatten a picture, the pixel above and the pixel below stop being neighbors and become entries sitting 40 slots apart in a list. But "these pixels are next to each other" is the most useful fact about an image. An edge only exists because adjacent pixels differ, and flattening deletes that before the network can use it.

The second is arithmetic. Connect 3,072 inputs to a modest 1,024-neuron layer and you've spent over 3 million weights on the first layer alone. And that network still doesn't know that a cat in the top-left corner is the same cat when it moves to the bottom-right. It would have to learn "cat" over again for every position.

Convolutions deal with both of these, and with the same trick.

---

What a convolution actually does

A convolutional layer doesn't look at the whole image at once. It slides a small window across it, here 3x3 pixels, and at each stop computes a single number from the nine pixels underneath.

The thing that matters is that it's the same nine weights at every stop. That's weight sharing, and it's why any of this works. One filter learns one pattern, say a vertical edge, and then finds it anywhere in the image. Instead of memorizing "cat in the corner" and "cat in the middle" as separate facts, the network learns "cat-ish texture" once and reuses it.

Stack a few of these and a hierarchy shows up on its own. The first layer picks out edges and blobs of color. The second combines those into curves and textures. The third combines those into things you'd actually name, like a wheel or an ear.

Nobody programs that hierarchy. It falls out of the training, which I've now had explained to me several times and still find slightly unreasonable.

---

The data

CIFAR-10 is 60,000 tiny images across 10 classes: airplane, automobile, bird, cat, deer, dog, frog, horse, ship, truck. 50,000 go to training, 10,000 to testing.

cifar10 = tf.keras.datasets.cifar10
(x_train, y_train), (x_test, y_test) = cifar10.load_data()

# pixels come in as 0-255; scale them to 0-1
x_train, x_test = x_train / 255.0, x_test / 255.0

# labels come as [[6], [9], ...]; flatten to [6, 9, ...]
y_train, y_test = y_train.flatten(), y_test.flatten()

That division by 255 looks like housekeeping. It isn't optional. Gradient descent takes steps proportional to the size of its inputs, so feed it raw values in the hundreds and the steps go wild. Training either diverges or crawls. Squeeze everything into 0 to 1 and the optimizer behaves.

The shapes confirm what we're working with:

x_train shape: (50000, 32, 32, 3)
y_train shape: (50000,)

50,000 images, each 32x32 with 3 channels. Small enough to train on a laptop, hard enough to be worth doing.

---

The architecture

i = Input(shape=x_train[0].shape)

x = Conv2D(32,  (3,3), strides=2, activation="relu")(i)
x = Conv2D(64,  (3,3), strides=2, activation="relu")(x)
x = Conv2D(128, (3,3), strides=2, activation="relu")(x)

x = Flatten()(x)

x = Dropout(0.5)(x)
x = Dense(1024, activation="relu")(x)
x = Dropout(0.2)(x)
x = Dense(K, activation="softmax")(x)

model = Model(i, x)

Two phases here, doing genuinely different jobs. The first is feature extraction: three convolutional layers that shrink the image while deepening it.

Spatial size shrinks, depth grows. The network is trading "where exactly" for "what kind of thing", and by the end it's holding 128 learned concepts over a 3x3 grid.

The shrinking comes from strides=2, which makes the filter jump two pixels at a time instead of one. Most tutorials reach for pooling layers to do this. Strided convolutions get the same downsampling as a side effect of the convolution itself, so there's one less layer type in the model. I don't have a strong opinion on which is better yet.

The second phase is the classifier. Flatten those features into a vector, run them through a normal dense network, end with 10 softmax outputs, one probability per class, summing to 1.

---

The parameter count surprised me

Here's the model summary, trimmed to the column worth looking at:

conv2d   (Conv2D)   ->        896 params
conv2d_1 (Conv2D)   ->     18,496 params
conv2d_2 (Conv2D)   ->     73,856 params
flatten             ->          0
dense    (Dense)    ->  1,180,672 params
dense_1  (Dense)    ->     10,250 params

Total: 1,284,170

All three convolutional layers together come to about 93,000 parameters. The single Dense(1024) that follows them uses 1,180,672, roughly 92% of the whole model.

I had this backwards. I'd assumed the convolutions were the expensive machinery and the dense layer was a formality on the end. It's the reverse: the layers doing the visual reasoning are cheap, and one fully connected layer bolted on the back is where the weights actually go. Weight sharing paying for itself, really, since a 3x3 filter is nine weights whether the image is 32 pixels across or 3,000.

It's also the first place I'd look if this thing ever needed to be smaller, since trimming that dense layer to 512 would cut the model roughly in half on its own.

---

Dropout, and why the rates differ

x = Dropout(0.5)(x)   # before the big dense layer
x = Dense(1024, activation="relu")(x)
x = Dropout(0.2)(x)   # after it

Dropout switches off a random fraction of neurons on every training step, which sounds like sabotage the first time you hear it. It works because no single neuron gets to become "the cat detector" if it might vanish at any moment. The network is pushed into spreading the representation around.

The heavier 50% sits where the overfitting risk is worst, right before 1,152 flattened features hit a million-parameter layer. The lighter 20% after it is a gentler nudge, close enough to the output that dropping much more would just wreck the signal.

---

Training, and the moment it started memorizing

15 epochs, Adam, about 6 seconds each:

Epoch  1  accuracy: 0.4209  val_accuracy: 0.5298  val_loss: 1.3040
Epoch  5  accuracy: 0.6462  val_accuracy: 0.6501  val_loss: 0.9933
Epoch 10  accuracy: 0.7317  val_accuracy: 0.7184  val_loss: 0.8088
Epoch 15  accuracy: 0.7751  val_accuracy: 0.7212  val_loss: 0.8103

Random guessing across 10 classes gets you 10%. After a single epoch, six seconds of training, the model was at 53% on images it had never seen. I did not expect the first six seconds to do that much of the work.

Then look at the last five epochs. Training accuracy climbs from 73.2% to 77.5%. Validation accuracy goes from 71.8% to 72.1%, so, nowhere. Validation loss hits its lowest point at epoch 10, at 0.8088, and then drifts back up.

The model kept getting better at the training set and stopped getting better at reality. That's overfitting, and the validation curve is where you watch it happen.

Plot the two loss curves together and it's obvious. Training loss keeps diving, validation loss flattens out and turns. The gap between them is memorization.

plt.plot(r.history['loss'], label='loss')
plt.plot(r.history['val_loss'], label='val_loss')
plt.legend()
plt.show()

I knew all of this in theory before I ran it. Seeing it happen in a chart I'd made from my own model is what made it stick.

---

Looking at the mistakes

An accuracy number tells you how often you're wrong and nothing about what you're wrong about. For that you want a confusion matrix.

y_pred = model.predict(x_test).argmax(axis=1)
cm = confusion_matrix(y_test, y_pred)
plot_confusion_matrix(conf_mat=cm, figsize=(7,7), show_normed=True)

The argmax is doing real work there. Softmax hands back 10 probabilities, and argmax picks the index of the highest one, which is the network's actual answer.

True labels on one axis, predictions on the other. A perfect model lights up only the diagonal, and everything off it is a specific, nameable confusion. On CIFAR-10 the errors aren't random: animals get mixed up with animals, vehicles with vehicles. Cats and dogs are the usual offenders, which is fair enough for two furry quadrupeds rendered in about a thousand pixels.

Then I did the thing I'd tell anyone training a classifier to do. I pulled up the actual failures.

misclassified = np.where(y_pred != y_test)[0]
i = np.random.choice(misclassified)

plt.imshow(x_test[i])
plt.title("True: %s  Predicted: %s" % (labels[y_test[i]], labels[y_pred[i]]))

This is humbling. Several of those images I could not classify myself — I sat there squinting at a deer I'd have called a horse. At 32x32, a blurry brown animal is a blurry brown animal. Some of that 72% ceiling is my model and some of it is just the data, and I don't know the split.

---

What I'd try next

Data augmentation first, randomly flipping and shifting the training images so the model sees more variation, since that goes straight at the overfitting the curves exposed. Batch normalization between the conv layers to steady training. Swapping the strides for pooling, mostly so I can see what changes. More epochs would help too, but only alongside augmentation, or they'd just deepen the memorizing.

---

What actually stuck

Mostly weight sharing. Convolutions aren't just "the layers you use for images", they encode an assumption: a pattern means the same thing wherever it turns up. That assumption is what makes vision models trainable at all, and the parameter split that caught me off guard falls straight out of it.

The other thing is less about CNNs than about habits. Training accuracy will climb happily while the model gets worse at its actual job, and the only reason I caught it was that the validation curve happened to be sitting right there next to it. I'd have missed it plotting one line.

72% isn't a good score and I'm not pretending otherwise. But the whole thing runs in about 90 seconds, and in those 90 seconds something goes from knowing nothing at all to sorting cats from trucks out of pixels and gradients. I've been poking at it for a week now and that hasn't gotten old.