How do we determine how objects (and/or the camera itself) move in the 3D world? Difficulty comes as motion is geometric whereas optical flow is radiometric (about an origin)

See also: aperture problem

Constraint Equation

Let image intensity be denoted by . Then, applying chain rule, we obtain .

Let and . Then is the 2D velocity space.

If we set , then we get the optical flow constraint equation: .

We assume constant brightness for this, meaning .

We measure each of the following:

  • Spatial Derivative: ,
    • Forward difference
    • Sobel filter
    • Scharr filter
  • Optical Flow: ,
    • We need to solve for this! (the unknown in the optical flow problem)
  • Temporal Derivative:
    • Frame difference

Lucas-Kanade

A dense method to compute motion at every location in an image.

Where can you see movement that can be effectively computed? A corner!

Solve for in where is the 1-by-2 column matrix of and . is the -by-2 column matrix of , partial derivatives evaluated at point ( is actually the same matrix used in Harris corner detection). is the 1-by- matrix consisting of the negative of the temporal partial derivative for each point.

Lucas-Kanade Method

Assumptions

  • Motion is slow enough that partial derivatives , , and are well-defined
  • The optical flow constraint equation holds ()
  • Window size is chosen so that motion is constant in the window
  • Window size is chosen so that the rank of is 2 for the window (required inverse exists)

Stereo

Computing depth from multiple images. Formulated as a correspondence problem: dtermine match between location of a scene point in one image and its location in another.

Disparity: where is baseline, is distance from to epipolar line, and is distance from to epipolar line. is distance from to target .

Simple stereo algorithm

  1. Rectify images (make epipolar lines horizontal)
    1. Rectified images have these properties:
      1. Image planes of cameras are parallel
      2. Focal points are at same height
      3. Focal lengths are the same
      4. Epiolar lines fall along the horizontal scan lines
  2. For each pixel
    1. Find epipolar line
    2. Scan line for best match
    3. Compute depth from disparity

Naive approach, pixel-based often lacks content. What we can try is min SSD-error of a window-based approach.

Another approach is to match the edges (the zero-crossings) at different scales.

Note: Sum squared differences (SSD) is the same as Normalized Cross Correlation (NCC)