Introduction When I was a student I had much difficulty with quantum mechanics. It was comforting to find that even Einstein had had such difficulties for a long time. Indeed they had led him to the heretical conclusion that something was missing in the theory: ‘I am, in fact, rather firmly convinced that the essentially statistical character of contemporary quantum theory is solely to be ascribed to the fact that this (theory) operates with an incomplete description of physical systems.’ More explicitly, in ‘a complete physical description, the statistical quantum theory would … take an approximately analogous position to the statistical mechanics within the framework of classical mechanics …’. Einstein did not seem to know that this possibility, of peaceful coexistence between quantum statistical predictions and a more complete theoretical description, had been disposed of with great rigour by J. von Neumann. I myself did not know von Neumann's demonstration at first hand, for at that time it was available only in German, which I could not read. However I knew of it from the beautiful book by Born, Natural Philosophy of Cause and Chance , which was in fact one of the highlights of my physics education. Discussing how physics might develop Born wrote: ‘I expect … that we shall have to sacrifice some current ideas and to use still more abstract methods. However these are only opinions.
Introduction When I was a student I had much difficulty with quantum mechanics. It was comforting to find that even Einstein had had such difficulties for a long time. Indeed they had led him to the heretical conclusion that something was missing in the theory: ‘I am, in fact, rather firmly convinced that the essentially statistical character of contemporary quantum theory is solely to be ascribed to the fact that this (theory) operates with an incomplete description of physical systems.’ More explicitly, in ‘a complete physical description, the statistical quantum theory would … take an approximately analogous position to the statistical mechanics within the framework of classical mechanics …’. Einstein did not seem to know that this possibility, of peaceful coexistence between quantum statistical predictions and a more complete theoretical description, had been disposed of with great rigour by J. von Neumann. I myself did not know von Neumann's demonstration at first hand, for at that time it was available only in German, which I could not read. However I knew of it from the beautiful book by Born, Natural Philosophy of Cause and Chance , which was in fact one of the highlights of my physics education. Discussing how physics might develop Born wrote: ‘I expect … that we shall have to sacrifice some current ideas and to use still more abstract methods. However these are only opinions.
‘Quantum mechanics is very impressive. But an inner voice tells me that it is not yet the real thing. The theory produces a good deal but hardly brings us closer to the secret of the Old One’. These views were contained in a letter written by Albert Einstein to Max Born in December 1926. Though the following year would see the completion of work on the theoretical foundation of the modern physics of atoms and particles, Einstein was never to change his judgement and remained firmly convinced that ‘this business […] contains some unreasonableness’. He was unwilling to sacrifice his own ideas as to the cognitive scope of science even in the face of the important results being achieved by the new theory. To the undeterministic findings of quantum mechanics he opposed his belief in the ‘possibility of giving a model of reality, a theory, that is to say, which shall represent events themselves and not merely the probability of their occurrence’. For this reason he chose to live with the isolation of his scepticism and dissent, ironically accepting the reputation as a obstinate heretic that he had won over the years among his colleagues, and worked to the end on his own research programme to develop a unified field theory of rigorously causal nature. As is generally known, Einstein carried on a strong scientific and philosophical dispute with the defenders of the so-called official interpretation of quantum mechanics.
Now I shall try to give you an idea of the way in which physicists at present endeavour to overcome this failure. One might term it an ‘emergency exit’, though it was not intended as such, but as a new theory. I mean, of course, wave mechanics. (Edding-ton called it ‘not a physical theory but a dodge— and a very good dodge too’.) The situation is about as follows. The observed facts (about particles and light and all sorts of radiation and their mutual interaction) appear to be repugnant to the classical ideal of a continuous description in space and time. (Let me explain myself to the physicist by hinting at one example: Bohr's famous theory of spectral lines in 1913 had to assume that the atom makes a sudden transition from one state into another state, and that in doing so it emits a train of light waves several feet long, containing hundreds of thousands of waves and requiring for its formation a considerable time. No information about the atom during this transition can be offered.) So the facts of observation are irreconcilable with a continuous description in space and time; it just seems impossible, at least in many cases. On the other hand, from an incomplete description—from a picture with gaps in space and time—one cannot draw clear and unambiguous conclusions; it leads to hazy, arbitrary, unclear thinking—and that is the thing we must avoid at all costs! What is to be done?
The dominance of the Copenhagen interpretation is the result of a historical accident (Cushing, 1994). At the Solvay conference in 1927, Louis de Broglie put forward the deterministic pilot wave theory, described later. Never slow to criticize, Pauli jumped up and demolished de Broglie’s proposal, and de Broglie was so discouraged by this that he did not develop his interpretation any further. The Copenhagen interpretation was then developed by Bohr, Heisenberg, Pauli and many others, as described in the previous chapter. It was strongly supported by von Neumann’s proof of the impossibility of hidden variables. Years later, in 1952, Bohm published a deterministic hidden variable theory, showing that there must be something wrong with von Neumann’s proof. The error was found by John Bell in 1966, as described in the last chapter, and he also showed how Pauli’s objections to de Broglie’s original idea can be answered. If this had been done at the conference in 1927, quantum mechanics could have been interpreted deterministically from the beginning. The essential mistake in the Copenhagen interpretation is to treat it as a complete account of the behaviour of each individual system. It is frequently claimed that it enables us to calculate, at least in principle, everything that can be measured. Thus Hooft (1997, p. 11) said that ‘The laws of quantum mechanics have been formulated very accurately. We know exactly how to compute anything we would like to know.’ Also Peierls (1997, p. 25) has said that quantum mechanics ‘had become a complete and consistent scheme capable of giving a unique answer to any questions relating to actual or possible observations’. Even if this were true, it would not imply that it is the final complete theory, for it is always possible that some new phenomenon might be found that cannot be calculated quantum-mechanically. However, it is not true: there are many phenomena such as the time of decay of a radioactive nucleus, or the direction a particle is scattered by a nucleus, that cannot be calculated, even in principle. Quantum mechanics is therefore an incomplete theory. We can only calculate the statistical properties of these systems, such as the half-life of a radioactive decay or the differential scattering cross-section that gives the probabilities that a particle is scattered through various angles. All measurements of quantum systems are of this statistical character. It might be thought that the possibility of making measurements on a single electron provides an exception to this. However, even in such cases the electron is continually bathed in a fluctuating background radiation from nearby atoms. Since this is variable, that electron is a member of an ensemble of electrons subject to different fluctuations. All these examples clearly demonstrate the statistical character of quantum mechanics. ‘I am rather firmly convinced’, Einstein (in Schilpp, 1949, pp. 666, 671) remarked, ‘that the essentially statistical character of contemporary quantum theory is solely to be ascribed to fact that this theory operates with an incomplete description of physical systems’. As a result, ‘the c-function is to be understood as the description not of a single system but of an ensemble of systems’. So, ‘if the statistical quantum theory does not pretend to describe the individual system completely, it appears unavoidable to look elsewhere for a complete description of the individual systems’. Thus ‘the difficulties of theoretical interpretation disappear, if one views the quantum-mechanical description as a description of an ensemble of systems’. If this is achieved, ‘the statistical theory would, within the framework of future physics, take an approximately analogous position to statistical mechanics within the framework of classical mechanics’. ‘Thus, essentially, nothing has changed since Galileo or Newton or Faraday concerning the status of the ‘‘observer’’ or our ‘‘consciousness’’ or of our ‘‘information’’ in physics’ (Popper, 1982, p. 46). Once this is accepted, the quantum paradoxes that plagued the Copenhagen interpretation are easily resolved, as described below. In any textbook of quantum mechanics problems such as the calculation of the energy levels of the hydrogen atom are solved as if that hydrogen atom is alone in the universe. As Feynman (1972) remarked, ‘when we solve a quantum-mechanical problem, what we really do is to divide the universe into two parts – the system in which we are interested and the rest of the universe. We then usually act as if the system in which we are interested comprised the whole universe.’ In fact, however, every hydrogen atom on which measurements are made is surrounded by other atoms and exposed to their radiations. Quantum mechanics somehow takes this into account and gives the average behaviour of an ensemble of hydrogen atoms. A more detailed theory takes these influences into account. One attempt to do this is stochastic electrodynamics, described in a later section.
Quantum mechanics is our most successful physical theory. It underlies our very detailed understanding of atomic physics, chemistry, and nuclear physics, and the many technologies to which physical systems in these regimes give rise. Additionally, relativistic quantum mechanics is the basis for the standard model of elementary particles, which very successfully gives a partial unification of the forces operating at the atomic, nuclear, and subnuclear levels. However, from its inception the probabilistic nature of quantum mechanics, and the fact that “quantum measurements” in the orthodox formulation appear to require the intervention of non-quantum mechanical “classical systems,” have led to speculations by many physicists, mathematicians, and philosophers of science that quantum mechanics may be incomplete. Among the Founding Fathers of quantum theory, Einstein and Schrödinger were both of the opinion that quantum mechanics is in some way unsatisfactory, and this view has been amplified in more recent profound work of John Bell, among others. In an opposing camp, many others in the physics, mathematics, and philosophy communities have attempted to provide an interpretational foundation in which quantum mechanics remains a complete and self-contained system. Among the Founding Fathers, Bohr, Born, and Heisenberg maintained that quantum mechanics is a complete system, and a number of recent proposals have been made to improve upon or to provide alternatives to their “Copenhagen Interpretation.” The debate continues, and has spawned an enormous literature.
The difficulties of the old quantum theory were largely dispelled by the development of quantum mechanics in the 1920s. First Heisenberg (1925) developed the rather abstract matrix mechanics and soon after Schro¨dinger (1926; see also Eckart, 1926) developed the physically more appealing wave mechanics. Subsequently he showed them to be mathematically equivalent. The physicists were greatly relieved; now they had once again a mathematical formalism that enabled the results of experiments to be calculated. It was possible to move forward. It is certainly a very elegant and indeed beautiful formalism, and does enable the results of many physical measurements to be calculated with great accuracy. It is used daily by atomic and nuclear physicists; there is no other way to make calculations. From the earliest days there has been intense debate about the interpretation of quantum mechanics, and it shows no signs of being settled; if anything it is becoming more intense over the years. As Penrose has remarked, no one understands quantum mechanics. Feynman (1967; see also Sykes, 1994) also admitted that ‘nobody understands quantum mechanics’, and Einstein once remarked: ‘I have thought a hundred times as much about the quantum problems as I have about general relativity theory’ (Pais, 1994, p. 57). It should therefore be evident that the most that can be done here is to try to show where the real difficulties lie, and how the failure to recognize that quantum mechanics is an essentially statistical theory has led to a series of confusing quantum paradoxes. It is just these paradoxes that have often been used to support theological speculations. It is useful to ask what ‘understanding quantum mechanics’ actually means. Certainly it means being able to calculate the results of many measurements. The same applies to Newton’s theory of gravitation, which we understand in the sense that we can use it as a basis of calculation, although we do not really understand the gravitational force itself. Very often, especially in modern physics, the mathematical formalism does not give us the sort of intuitive physical feeling for what is going on that we often obtain from classical mechanics. Perhaps we can never achieve more than this for quantum mechanics. In classical mechanics we can assume that a property that is measured objectively exists prior to the interaction of the measuring apparatus with the observed system. Quantum mechanics, however, ‘is incompatible with the proposition that measurement discovers some unknown but pre-existing property’ (Peres, 1997, p. 14). There is an important distinction between mysteries and absurdities. We may be forced to live with mysteries, but there is no reason why we should put up with absurdities such as the wave-particle duality, tunnelling through potential barriers, acausal events and the collapse of the wave function. These are the quantum paradoxes that will be discussed later on. They arise within the Copenhagen interpretation of quantum mechanics and are solved by other interpretations. These alternative interpretations enable us to make conjectures about what is going on, even though we cannot understand it in detail. This is acceptable and a stimulus to further research. It is certainly difficult enough to understand quantum mechanics, but there is no need to try to visualize logical absurdities in the effort to do so. Quantum mechanics is a mathematical formalism that makes it possible to calculate the results of some experiments. Anything more than this, such as questions about the interpretation of what is measured and whether it tells us what is actually going on in the system being studied, belongs to philosophy. Thus we are concerned here with what is essentially a philosophical debate; if it were just a matter of physics the debate would have been settled long ago. One of the difficulties is that it invokes the results of experiments and uses the language of quantum mechanics. Inevitably this language uses philosophical terms and then it is easy to give the impression that the philosophical propositions are in some way entailed by the physical results. These in turn can influence theological thinking. In order to be quite clear which arguments are justified and which are not, it is essential to separate the three levels of discourse: physical, philosophical and theological. Our task is to relate these together, and this immediately raises the traditional problems of the philosophy of science, including the relation of experiment to theory, and of theory to explanation. We begin by discussing experiments, distinguishing between those that are real and those that are imaginary.
A view most physicists adhere to is that scientific research should keep aloof from implicit metaphysical hypotheses. This prompts most of them to make it a principle that the only "existing" physical entities are those that could somehow be observed, directly or indirectly. In quantum mechanics the assumption is made, that, with the exception of nondynamical quantities such as masses and electric charges as well as of time. All these observable quantities are described by Hermitean operators, with the help of which the quantum rules tell us all that can, or could, be observed, in any circumstances whatsoever. One basic feature of the pilot-wave theory is that the quantum mechanical predictions are exactly reproduced for any ensemble E in which the hidden variables are distributed at random. This suffices to show how difficult it is to build up tests to discriminate the two theories on the basis of their observational predictions.