Purple iris flowers in a garden — Photo: Retro Lenses, CC BY 4.0, Wikimedia Commons

Computer Vision, lecture 1

What is computer vision?

Why seeing is hard for computers, how the field started, and what we will learn

What do you see?

Purple iris flowers in front of green leaves, with a soft blurred background — Photo: Retro Lenses, CC BY 4.0, Wikimedia Commons

What the computer gets

Left: the iris photo in grey, with a small red square on the edge of a leaf. Right: the 12 by 12 pixels inside the square, printed as numbers. Bright values near 250 are in the top left, dark values near 90 are in the bottom right — Photo: Retro Lenses, CC BY 4.0, Wikimedia Commons

A picture is a grid of numbers. This photo is 1328 × 2000 pixels × 3 colours: about 8 million numbers. Not one of them says "flower".

This is the most important slide today. Give students time to look at it.

Each number is the brightness of one pixel. 0 means black. 255 means white.
The red square is on the edge of a leaf. Bright background is in the top left (numbers near 250). The dark leaf is in the bottom right (numbers near 90).
Show them the jump from about 250 to about 100. That jump is the edge of the leaf. In lecture 4 we will find edges by looking for jumps like this.
A colour photo has three grids like this: one for red, one for green, one for blue.

Ask: "Look only at the numbers on the right. Can you tell this is a flower?" No. You can only tell that something bright meets something dark. This is where the computer starts.

The key message: computer vision is the work of getting from these numbers to meaning.

If students ask why the numbers stop at 255: each number uses 8 bits, and 8 bits can store 256 different values (0 to 255). Lecture 2 explains how a camera makes these numbers.

A simple definition

Computer vision

Making computers get useful information from pictures and videos.

Three kinds of useful information:

Measure: where things are, how big, how far, how they move
Recognise: what things are
Act: what to do next (a robot picks up a box, a car brakes, a doctor gets a warning)

Write three words on the board: measure, recognise, act. They help students sort every example they will see later.

Measure is about shape and position: cameras, 3D, movement. Lectures 2, 8 and 11 use it a lot.
Recognise is what most people think of: "Is there a person in this photo?" Lectures 6 to 10 and 12.
Act reminds students that vision is usually part of a bigger system. When the system makes a mistake, something real happens. That is why we will care about measuring errors later.

Also explain the neighbours of computer vision:

Image processing: picture in, picture out (make it sharper, remove noise).
Computer vision: picture in, description out ("two cars, 12 metres away").
Computer graphics: description in, picture out (games, animation).
Machine learning: the method that most modern computer vision uses.

Part 1: Why is seeing hard?

We see without effort, so we think it is easy.

Vision is graphics in reverse

Graphics: world → picture

We know the shapes, colours, lights and camera. We calculate the picture.

One correct answer
Physics tells us how

Vision: picture → world

We have only the picture. We must guess the shapes, colours, lights and camera.

Many worlds give the same picture
Some information is lost

A photo loses distance

Diagram: lines from a camera pass through a small image. A small near object, a medium object, a big far object and a tilted object all fit between the same two lines, so they all make the same picture

Every object between the two lines makes exactly the same picture.

Go through the drawing from left to right.

All light lines meet at one point, the camera. Lecture 2 explains why.
Anything between the same two lines lands on the same part of the picture (the red line).
So a small object near the camera and a big object far away look the same. This is how a toy car close to the camera can look like a real car.
Even the tilted object gives the same picture.

Ask: "So how do you know how far away things are?" Collect answers. Good answers: we have two eyes; things move when we move our head; we know how big a car usually is; far things look smaller and more blue; lines meet in the distance.

These are clues. A computer needs clues too. Later lectures teach them: two cameras and motion (lectures 8 and 11), and learning from many examples (lectures 6 to 10).

One object, many pictures

The same cat photo changed eight ways: original, seen from another angle, very dark, very small, partly covered by a block, twisted, placed on a busy Jakarta street, and blurred with noise — Cat: Caleb Woods, CC0. Street: Nugroho Adie, CC BY 2.0. Both via Wikimedia Commons

A program must say "cat" for all of these, and "not cat" for everything else.

Go through the eight pictures. Give each change its name:

Change	What happens to the pixels
Viewpoint (angle)	every pixel moves
Light	every number changes
Size	the cat covers far fewer pixels
Occlusion (something in front)	part of the cat is missing
Shape change	the cat bends (cats bend a lot)
Busy background	the cat must be found among other things
Noise and blur	small details are lost

One more change that is not shown: different cats. Black cats, white cats, fat cats, kittens: all must count as "cat". But a fox, which looks similar, must not.

The main message: the numbers change a lot, but the answer stays the same. A good method must ignore these changes (we say it is invariant to them) but still notice the small differences between a cat and a fox. Remember this idea. It comes back in lectures 4, 5, 7 and 8.

Your eyes guess too

A grey band that goes from dark on the left to light on the right, with two grey squares labelled A and B. A is on the dark side, B is on the light side

Which square is brighter, A or B?

The gap between numbers and meaning

What the computer has

8 million numbers from 0 to 255
no objects, no edges, no distance
no names

What we want

"purple flowers in a garden, on a sunny day"
where each flower is
what kind of flower it is

The distance between pixel numbers and meaning is called the semantic gap. This course is a set of tools to cross it.

Think, pair, share (4 minutes)

With the person next to you, find:

one task that is easy for a 5-year-old child but hard for a computer, and
one task that is easy for a computer but hard for a person.

Say why for each one.

Give 2 minutes to talk in pairs. Then ask 3 or 4 pairs to share.

Good examples to look for:

Easy for a child, hard for a computer: knowing a friend after a new haircut; knowing that a cup can be held by its handle; understanding a cartoon; reading messy handwriting; telling a photo of a person from a real person.
Easy for a computer, hard for a person: counting 4000 grains of rice exactly; measuring a part to 0.01 mm; comparing one face with a million others; checking 10 000 phone screens per hour for a tiny scratch; seeing infrared light.

The "why" is the important part. Computers win when the task is exact, repeated and clearly defined. People win when the task needs common sense, context, or flexibility. This explains why factories use computer vision a lot, but self-driving cars are still hard.

Part 2: How did we get here?

Sixty years in five slides.

A short history

Timeline from 1963 to 2023: blocks world, the Summer Vision Project, Marr's book Vision, the Canny edge detector, eigenfaces, SIFT features, real-time face detection, the ImageNet dataset, AlexNet, ResNet, the Vision Transformer, CLIP, and Segment Anything

Tell it in three parts.

1. Big hopes (1960s). The first programs found 3D blocks in simple line drawings ("the blocks world"). In 1966, researchers planned to build a large part of a seeing machine in one summer, with students. Next slide.

2. Careful tools (1970s to 2000s). David Marr's book Vision (1982) gave the field a way of thinking. Researchers built careful mathematical tools:

edge detectors (Canny, 1986),
camera geometry and 3D,
face recognition with "eigenfaces" (1991),
small image features that can be matched between photos, such as SIFT (1999).

Fast face detection (2001) put computer vision into ordinary cameras. Systems were built from features designed by people + a simple classifier (a program that sorts things into groups).

3. Learning (2012 to now). A very large set of labelled photos (ImageNet) and fast graphics cards (GPUs) made a new method win easily in 2012: deep convolutional neural networks. Since then: bigger models, more data, less hand design. Examples: ResNet (2015), transformers for images (2020), models that learn from pictures and text (CLIP, 2021), and models that can cut out almost any object (Segment Anything, 2023).

Point out that this course follows the same path. Part I of the course is era 2. Part II is era 3.

What computers could do, then and now

1970s results: line drawings of blocks, a face drawn as parts held by springs, a person made of cylinders — 1970s. Szeliski (2022), Fig. 1.7

2010s results: a network diagram, people and surfboards each coloured separately, a 3D body pose, depth from a phone, labelled parts of a forest photo — 2010s. Szeliski (2022), Fig. 1.11

The summer that took sixty years

In 1966, a plan said a group of students could build important parts of a seeing machine in one summer.

Sixty years later, many thousands of people work on it. It is still not finished.

Lesson: seeing feels easy because it happens without thinking, not because it is simple.

2012: the big change

Bar chart of the winning error rate in the ImageNet challenge: 28.2 percent in 2010, 25.8 in 2011, 16.4 in 2012 with AlexNet, 11.7 in 2013, 6.7 in 2014 and 3.6 in 2015 with ResNet. A dashed line shows a trained person at about 5 percent

Winners of the ImageNet challenge (1000 kinds of objects). "Top-5 error": the right answer is not in the computer's 5 best guesses.

Explain the competition first. ImageNet has 1.2 million training photos in 1000 groups (dog breeds, cars, foods, tools…). The computer may give 5 guesses per photo. If the right answer is not among the 5 guesses, it counts as an error.

Now look at the shape of the bars:

With features designed by people, progress was slow: 28% → 26% error.
In 2012, AlexNet, a deep convolutional neural network trained on graphics cards (GPUs), cut the error by about one third in one year.
Three years later, networks made fewer errors than one trained person. (The 5% for a person comes from one researcher who trained himself on the task. It is an estimate.)

Three things came together: a lot of labelled data (ImageNet), fast parallel computers (GPUs), and some new training tricks (lecture 6).

Warn students not to over-read the chart. "Better than people on ImageNet" does not mean computers see better than people. The test is narrow. Computers still fail on small changes that people ignore, like the eight cat pictures earlier.

Old ideas are still inside new methods

Older idea (Part I of this course)	Lives on in… (Part II)
Filtering a picture with a small grid of numbers	the convolution layer, with learned numbers
Looking at a picture at many sizes	detectors that find big and small objects
Edges, corners, small patterns	what the first layers of a network learn
Camera geometry	3D, AR, robots, depth networks
Grouping similar pixels	segmentation

Part 3: What can computer vision do?

The main tasks

The same simple scene of two balls and a box shown four ways: a label for the whole picture, boxes around each object, every pixel coloured by type with both balls the same colour, and every pixel coloured by object with each ball a different colour

Also: body pose · tracking · depth and 3D · motion · describing a picture in words · making new pictures

Explain each task by what it gives as output. This is how engineers think about tasks.

Classification: one label (or a few labels) for the whole picture. Output: a name.
Detection: what and where. Output: a list of boxes, each with a name and a confidence score. Lecture 9.
Semantic segmentation: a label for every pixel. Both balls get the same colour, because both are "ball". Lecture 10.
Instance segmentation: every pixel gets a label and an object number. So the two balls get different colours. Lecture 10.

Ask: "You want to count mangoes on a moving belt. Which output do you need?" Answer: detection or instance segmentation. Classification cannot count. Semantic segmentation joins touching mangoes into one blob. Choosing the task from the output you need is the most useful skill on this slide.

The same tasks on real photos

People on a ferry, each face and body inside a numbered box — Detection. Szeliski (2022), Fig. 1.2a, from Sivic, Zitnick and Szeliski (2006)

People on a beach holding surfboards, each person and each board filled with its own colour — Instance segmentation. Szeliski (2022), Fig. 1.2b, from He et al. (2017)

Vision at work

Six uses of computer vision: reading handwritten numbers, a 3D camera checking metal parts, a robot arm picking items, a 3D medical scan, a self-driving car with sensors on its roof, and a 3D model of a castle made from drone photos — Szeliski (2022), Fig. 1.4

reading handwritten numbers
measuring parts in 3D
robots picking items from boxes
medical scans
self-driving cars
3D models from drone photos

More places that use vision

Phones: face unlock, portrait blur, night mode, document scanning
Factories: finding defects, measuring, sorting, reading serial numbers
Transport: driver help, licence plate reading, counting traffic, parking
Health: checking X-rays and eye scans; counting cells under a microscope
Farming: sorting fruit, finding plant disease, counting crops from drones
Shops and delivery: checking shelves, reading parcels
Satellites: mapping floods, forests and cities
Media: photo editing, search, sports analysis

The parts of a vision system

Light → Camera → Clean-up → Features → Model → Decision

Camera and light: often the cheapest place to fix a problem
Clean-up: remove noise, fix colours, resize
Features: useful numbers, designed by people or learned

Model: rules, a classifier, or a neural network
Decision: a threshold, business rules, or a person who checks
Data and testing are needed for every part

Design task (5 minutes, groups of 3)

A fruit packing house wants to sort mangoes into "ripe" and "not ripe" on a moving belt. 5 mangoes pass every second.

What camera and light would you use?
What exactly is the output? Which task is it?
What makes it hard? (List the changes from the cat slide.)
Which mistake is worse: a ripe mango sorted as not ripe, or the other way round?

Run this in groups. Give 4 minutes. Then take one answer per question, each from a different group.

Good answers:

A closed box with even, soft light (no shadows, no sunlight). A camera with fixed colour settings. A plain belt. Maybe two cameras, or rollers that turn the mango, so the whole fruit is seen.
Classification for each mango. But the camera sees several mangoes at once, so really it is detection + classification, or instance segmentation and then classification.
Different mango types have different ripe colours. Stickers, dirt, spots and shine. Mangoes touching each other. Blur because the belt moves fast. And ripeness is partly inside the fruit, so you cannot see it. Some factories use special near-infrared cameras for this.
It depends on the business. Usually, selling unripe mangoes as ripe causes complaints. A ripe mango sorted as unripe only waits one more day. This difference decides where to put the decision threshold (lecture 9, precision and recall).

The point: students just designed a vision system using only today's ideas. The rest of the course fills in each part.

Part 4: This course

Course map

Part I: Images and classical vision

What is computer vision?
Image formation
Images as signals
Linear filtering and edges
Pyramids and scale

Part II: Learning to see

From classifiers to neural networks
Convolutional networks and transformers
Features and matching
Object recognition and detection
Image segmentation
Video, motion and temporal models
Vision and language

A picture really is a table of numbers

import cv2                         # OpenCV
img = cv2.imread("iris.jpg")       # load the photo

print(type(img))      # <class 'numpy.ndarray'>
print(img.shape)      # (1328, 2000, 3) -> height, width, colours
print(img.dtype)      # uint8           -> whole numbers 0..255
print(img[650, 950])  # [241  88 173]   -> one pixel: B, G, R (!)

gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
print(gray.shape)     # (1328, 2000)

If you can, run this live. It takes 2 minutes and makes "a picture is numbers" real. The iris photo is at lecture.williem.app/figures/photos/iris.jpg. Save it as iris.jpg, and the output will match the slide exactly.

The pixel at row 650, column 950 is on a petal. It has a lot of blue (241) and red (173) and little green (88). Blue + red = purple.

Three warnings for students:

shape is (height, width, colours). Rows come first. Students often mix up x and y. Tell them now: img[y, x]. 1328 × 2000 × 3 = 7 968 000 numbers: the "8 million" from slide 3.
uint8 means whole numbers from 0 to 255. Maths with uint8 wraps around: 200 + 100 gives 44, not 300. Change to float before doing maths. Lecture 3 comes back to this.
OpenCV stores colour as BGR, not RGB. If you show an OpenCV image with matplotlib, red and blue are swapped and skin looks blue. This is the most common beginner mistake in this course.

Summary

Computer vision turns pixels into useful information: measure, recognise, act.
It is hard because a photo loses information, so vision must use clues and guesses.
The numbers change a lot with angle, light, size, covering and shape. Good methods ignore these changes but still tell objects apart.
The field moved from features designed by people to features learned from data (2012). Old ideas still live inside new methods.
Choose the task from the output you need: a label, boxes, outlines, tracks.

Exit ticket (on paper or in chat)

In one sentence: why can't a computer just calculate the 3D world from one photo?
Name three changes a cat detector must handle.
You want to count cars in a parking lot. Which task would you use? Why?

Next lecture: how a camera turns light into numbers.