Stability analysis of a system of 2 second order nonlinear differential equations

This is an answer to Charles' restatement of the question.

Recall that equation F(x,x',x'') = 0 (e.g. x'' + sin x = 0) can be written as a system

X' = f(X), where X = (x,x')^T (e.g. f(x',x) = (-sin x, x')^T) and that system can be linearized about an equilibrium E = (x_,x'_)^T to obtain a linear equation
X' = AX where A is the 2 x 2 matrix given by the derivative of f at E.

So too a larger system F(x,x',x'',y,y',y'') = 0 can be written as a system

X' = f(X) where X = (x,x',y,y')

Given an equilibrium E = (x_,x'_,y_,y'_), the linearization of X' = f(X) about E is
again X' = AX where X = (x',x,y',y) and A is the derivative of f evaluated at E. A is a 4 by 4 matrix. If all eigenvalues of A have negative real part, the system is stable. If one eigenvalue has positive real part, the system is unstable. If there are no eigenvalues with positive real part, and there are eigenvalues which lie on the imaginary axis, then the equilibrium is "spectrally stable," and further analysis is required to determine the nonlinear stability.

With regard to the energy of the system, you are looking for a function of the form
V(x,x',y,y') whose time-derivative along solutions is constant. A good place to start is with the guess 1/2((x')^2 + (y')^2) + F(x,y). You should be able to figure out what F needs to be in this particular example.

For future reference, this forum is (I believe) intended primarily for questions from students and practitioners who are a little further along in their study. I decided to answer the question because I can imagine being very frustrated at not having some information that it would take an expert five minutes to explain and the question didn't strike me as the kind which would encourage others to attempt to turn this forum into a homework help site. If the moderators disagree, I apologize.