We present a fully quantum mechanical treatment of the nondegenerate optical parametric oscillator both below and near threshold. This is a nonequilibrium quantum system with a critical point phase transition, that is also known to exhibit strong yet easily observed squeezing and quantum entanglement. Our treatment makes use of the positive
P representation and goes beyond the usual linearized theory. We compare our analytical results with numerical simulations and find excellent agreement. We also carry out a detailed comparison of our results with those obtained from stochastic electrodynamics, a theory obtained by truncating the equation of motion for the Wigner function, with a view to locating regions of agreement and disagreement between the two. We calculate commonly used measures of quantum behavior including entanglement, squeezing, and Einstein-Podolsky-Rosen (EPR) correlations as well as higher order tripartite correlations, and show how these are modified as the critical point is approached. These results are compared with those obtained using two degenerate parametric oscillators, and we find that in the near-critical region the nondegenerate oscillator has stronger EPR correlations. In general, the critical fluctuations represent an ultimate limit to the possible entanglement that can be achieved in a nondegenerate parametric oscillator.