Dark matter and regular matter make up less than a third of the energy budget of the universe. The rest is made up of what we call dark energy.
Anything made of particles becomes less dense as the universe expands – as it should, since the number of particles remain the same while the volume over which they are distributed becomes larger. On the other hand, the density of dark energy remains constant as the universe expands. For this reason, dark energy, which is already the most abundant type of energy in the universe, is certain to become even more dominant as the universe keeps expanding and the density of matter is further reduced. Also, unlike matter, which gravitationally self-attracts, and whose presence therefore slows down the expansion of the universe over time, the presence of dark energy results in an ever faster expanding universe.
So what can explain this mysterious form of energy? Unlike dark matter, for which particle physicists have tons of competing plausible models, there really aren’t any satisfactory ideas for what dark energy might be and why there is as much of it around as we seem to be observing. We observe dark energy experimentally by observing the effects of its gravitational pull (or rather, anti-pull), and while we know it is there, we do not understand what it really is.
The simplest explanation is actually this: General relativity, which is the best theoretical model we have for describing gravity, allows not only matter to be a source of gravitational attraction, but even empty space itself. Moreover, the strength of the gravity created by empty space is not predicted by general relativity. It is a number that can be chosen arbitrarily – a free parameter of the theory that is often called the ‘cosmological constant’ that represents how strongly empty space attracts other empty space. The cosmological constant can even be negative, which would result in empty space repelling itself. This exactly matches what we know about dark energy. It even makes sense that it does not dilute as the universe expands, because the cosmological constant is a property of empty space itself!
So, haven’t we just explained dark energy? Is there any mystery left? The answer is yes, and the problem arises from quantum mechanics. General relativity is a classical theory of gravity, but since we know that nature is fundamentally quantum mechanical, there must be a quantum mechanical description of gravity that supersedes general relativity, a theory we refer to as ‘quantum gravity’. We can describe all other fundamental forces in nature using quantum mechanics, but so far a formulation of quantum gravity has eluded even the best and brightest physicists.
Here is why this matters. Parameters that are arbitrary in the classical theory can have ‘natural’ values in a quantum theory. What do we mean by a natural value? Imagine that you had many pencils, and you tried to balance them on their pointy ends and then timed how long it took them to fall down. Most of them would fall within a second or two, which would be the natural value for the “upright time” of the pencil. Using laws of physics, one can also estimate the upright time theoretically, and it more or less matches the experimental value – no significant discrepancy.
Since we don’t yet know the full theory of quantum gravity we cannot exactly predict the natural value for the cosmological constant. All is not lost however, since we can still make a crude guess for it, based on general principles of quantum mechanics. Unfortunately, the difference between our crude guess for the natural value and the observed value for the cosmological constant is huge — a one followed by over one hundred zeroes. In terms of the above analogy, this is like finding a pencil that balances on its head for 10^100 years before falling. This makes it unlikely that dark energy is just a cosmological constant. But it is just possible that if we figure out the exact theory of quantum gravity one day, we may find that our guess for the cosmological constant was far off the mark, and that the observed value matches the theoretically predicted value after all.