I can see no logic in what you have done. Yes, the left side can be written as the matrix multiplication
\(\displaystyle \begin{bmatrix}-D+ 1 & 4 \\ 1 & -D+ 1\end{bmatrix}\begin{bmatrix}y_1 \\ y_2 \end{bmatrix}\) but that is NOT equal to your right side. Where did you get that second column? The right side must be a single column, not a 2 by 2 matrix. Is that next equation supposed to give the determinant? If so the product (-D+ 1)(-D+ 1) is NOT equal to "\(\displaystyle D^2+ 1\)!
Here is different but equivalent way of doing this problem: Differentiate both sides of \(\displaystyle y_1'= y_1+ 4y_2- 2 cos t\) again to get \(\displaystyle y_1''= y_1'+ 4y_2'+ 2sin t\). Use the second equation, \(\displaystyle y_2'= y_1+ y_2- cos t+ sin(t)\), to make the equation \(\displaystyle y_1''= y_1'+ 4(y_1+ y_2- cos t+ sin t)+ 2 sin t= y_1'+ 4y_1+ 4y_2- 4cos t+ 6 sin t\). Now solve the first equation for \(\displaystyle 4y_2\): \(\displaystyle 4y_2=y_1'- y_1+ 2 cos t\). Now substitute that into the previous equation. \(\displaystyle y_1''= y_1'+ 4y_1+ (y_1'- y_1+ 2 cos t)- 4 cos t+ 6 sin t= 2y_1'+ 3y_1- 2 cos t+ 6 sin t\).
So we reduced to the single equation \(\displaystyle y_1''- 2y_1'- 3y_1= 4 cos t+ 6 sin t\). That has characteristic equation \(\displaystyle r^2- 2r- 3= (r- 3)(r+1)= 0\) which has roots r= 3 and r= 1.