That looks to me like the hard way to do it! From \(\displaystyle -y^2+ 3xy= 0\), instead of using the quadratic formula, factor:
y(3x- y)= 0 which gives y= 0 and y= 3x as you get.
If y= 0, then the first equation becomes \(\displaystyle x^2= 2\) so that \(\displaystyle x= \pm\sqrt{2}\), So one solution is \(\displaystyle x= \sqrt{2}\), y= 0 and another is \(\displaystyle x= -\sqrt{2}\), y= 0.
If y= 3, then the first equation becomes \(\displaystyle x^2- 9^2+ 3x^2= -5x^2= 2\). As you say, that has no (real) solution. So there is no solution (in the real numbers) for y= 3. The only solutions to the system are the two given above.
