Coincidentally I happen to be reading a book on the history of statistics. So I can give you names and dates.

The method of least squares was invented by Legendre in 1805. (Gauss claimed to have invented it in 1795, but did not popularize it.) He argued for it on mathematical simplicity. It was easy to apply, obviously fair, and in several special cases it simplified to the obviously right thing.

However at that point least squares was one of a great many averaging techniques that existed. Why did it beat out Boscovitch's method which was already in widespread use and minimized the sum of the absolute values of the errors?

The next step was provided by Gauss in a poorly reasoned section of Theoria Motos Corporum Coelestium in Sectionibus Conicis Solum Ambientium in 1809. What Gauss proved was that the only error distribution that made least squares best was the normal distribution, and vice versa the normal distribution made least squares best. (The poor reasoning was a circularity where he assumed that least squares was best, which lead him to the normal distribution, which lead him back to least squares.)

What do I mean by that? Well suppose that the measurements can be expected to be normally distributed with some mean and variance. Then for any given variance, the mean that makes the observed data most likely is the result of a least squares fit. Furthermore it is then possible to figure out the variance that best fits the data, allowing fairly detailed analysis to proceed.

Of course it wasn't stated that way because the concept of variance wasn't yet developed, nor was the normal curve well enough known to be called "normal".

I should note that the relationship between the normal curve and being able to carry out this analysis in an obvious way is special. For instance Laplace had tried previously to figure out a good error curve from a priori considerations and then analyze the error. He did come up with a reasonable family of curves, but the analysis got hopelessly difficult, and the best answer depended on which curve out of his family you chose. (Contrast the normal where the unknown variance doesn't affect what mean is best.)

In 1810 Laplace improved on Gauss. In 1785 he already had a version of the Central Limit theorem. After reading Gauss, this motivated him to combine the two to point out that the normal distribution was the natural result if the error term was itself the sum of several other error curves. Therefore there were good reasons to believe that the normal distribution was likely to come up, in which case least squares was the best possible estimate.

In 1811 Laplace further proved that of all estimation techniques of the form of one linear combination divided by another, the one that leads to least squares has (for a large number of observations) the smallest expected error in estimating the mean of the actual distribution, no matter what the error distribution is.

In 1812 Laplace presented both of these results in the Théorie analytique des probabilités. By 1815 least squares had entered standard use in French astronomy and among geometers there. It was standard in England as well by 1825.

In 1823 Gauss updated Laplace's 1811 analysis to point out that if one is interested in minimizing the expected squares of the error in estimating the mean to the error, then you can remove the condition "for a large number of observations" from Laplace's result. (The expected square of the error is, of course, the variance.)

Let me summarize this. To the extent that the normal curve is likely to be encountered in practice, the least squares method is the best estimate. Furthermore of all simple estimation techniques to calculate (ie one linear combination divided by another), least squares is best (either by a particular measure, or by any reasonable measure that you want if there are many measurements). These facts earned the technique rapid acceptance and caused it to replace previous methods of estimation.

Furthermore there are other possible estimation techniques that you can try that will be better with specific hypotheses and criteria. But the calculations involved with any of these are far more complex than least squares, and if you have enough data, all reasonable ones converge on the same answers.

I should also note that in many applications there are good physical interpretations behind least squares. For instance in a spectrum, the sum (OK, integral) of the squares of the signal is the energy of that signal. So by minimizing the sum of the squared error you are minimizing the unaccounted for energy in the signal.

Is this enough for you?

Ben

The number of events observed is the energy delivered in a Poisson distribution of radiation.

I don't see that naturally leading to the physical interpretation, although the underlying phenomena measured (and described) in another way does.

Cheers,
Ben

You can strengthen your "it may not be" though. The 1809 result from Gauss proved that if you have a family of possible error distributions which are the same up to translation (ie move the center) and scale (multiply by a constant factor), then least squares can only be best if that family is the normal distribution.

That is, if the real error curve differs from the normal at all, then least squares cannot be the best possible technique. The best one is very likely, however, to be far more complicated than "use a different exponent". There would be little point in going through this effort unless you had good reason to believe that the distribution was not normal, and you had a pretty good idea what the real distribution was.

Cheers,
Ben