Let mathcal S be the set of all 3×3 matrices A with every entry in \0,1\ such that A^2=I, where I is the identity matrix. The number of elements of…

If $A^2=I$ then $A$ is invertible, so $\det A=\pm1$ and $A$ is a $0$-$1$ matrix with $A^{-1}=A$. An invertible $0$-$1$ matrix whose inverse is also a $0$-$1$ matrix must be a permutation matrix, because the product of a row of $A$ with a column of $A^{-1}$ must produce exact $0$ and $1$ entries with non-negative terms only, which forces each row and column to contain a single $1$. So $A$ is the permutation matrix of a permutation $\sigma$ of three objects with $\sigma^{2}=\mathrm{id}$. Such permutations are the identity and the three transpositions, giving $1+3=4$ matrices. An exhaustive computer enumeration of all $2^9$ matrices gives the same count.