Source
Given 4 pairs of points of the form (x,y)→(x′,y′), compute the homography matrix
h=h11h21h31h12h22h32txty1
We write the pre-transformed points as P and the transformed points as P′
P=x1x2x3x4y1y2y3y4,P′=x1′x2′x3′x4′y1′y2′y3′y4′
Defining the equation
x′=h31x+h32y+1h11x+h12y+txy′=h31x+h32y+1h21x+h22y+ty
After cross-multiplying and re-arranging:
h11x+h12y+tx−h31xx′−h32yx′=x′h21x+h22y+ty−h31xy′−h32yy′=y′
x10x20x30x40y10y20y30y40101010100x10x20x30x40y10y20y30y401010101−x1x1′−x1y1′−x2x2′−x2y2′−x3x3′−x3y3′−x4x4′−x4y4′−y1x1′−y1y1′−y2x2′−y2y2′−y3x3′−y3y3′−y4x4′−y4y4′h11h12txh21h22tyh31h32=x1′y1′x2′y2′x3′y3′x4′y4′
We solve Ah=b for h (i. e. h=A−1b).
This gets us the 3x3 h. However, matrix3d takes a 4x4 matrix. Since we don’t care about z, we can make it just map back to itself:
h=h11h210h31h12h220h320010txty01