So start by finding the eigenvalues. If they're different, then A and B are not similar.

If they are the same, then find the eigenvectors and form a matrix with those eigenvectors as columns. Using the eigenvectors of A gives you a matrix that diagonalizes A. That is, let P1 be this matrix. Then

D = P1^(-1) A P1

where D is a diagonal matrix with the eigenvalues on the diagonal. Do the same thing with B. Since it has the same eigenvalues

D = P2^(-1) B P2, or B = P2 D P2^(-1)

So B = P2 P1^(-1) A P1 P2^(-1) and the matrix P you want is equal to P1 * P2^(-1).

View the original article here