History of mathematical statistics and sampling theory

Introductory note

In the 1980s, while at the University of Birmingham's Centre for Russian and East European Studies, I wrote a Ph.D. thesis on the mathematical-statistical methodology of the Soviet Family Budget Survey. Macmillan offered to publish a book based on the thesis, but I would have had to revise the manuscript and considerably reduce its length. It was hard to decide what to omit. I got distracted by other things and never got the book written. The thesis was read only by a handful of specialists who borrowed it from the university library. 

The other day I discovered to my delight that the library has digitized the theses submitted to the university. They are now all freely accessible. To download my thesis go here. While most of the contents are indeed of interest only to a few specialists, I believe that a few chapters may be of interest to a somewhat wider readership. One such chapter, reproduced below, should appeal to people interested in the history of science and mathematics. It shows that from the late nineteenth century to the end of the 1920s Russian statisticians were about a decade ahead of their Western colleagues in the fields of mathematical statistics and sampling theory -- achievements that were almost completely destroyed by Stalinism.

The historical development of mathematical statistics and sampling theory in the West, tsarist Russia and the USSR

1 Introduction

The term "statistics" is used in two distinct senses. It can refer to quantitative information about socio-economic phenomena, or to the study of the general problems of collecting, presenting, analyzing, interpreting and using such information.[1] Alternatively, it can refer to the discipline of mathematical statistics, which deals with a particular set of mathematical methods, based on probability theory, for collecting and analyzing data of many kinds. Mathematical statistics can be applied both to the collection and analysis of socio-economic statistics and to the collection and analysis of other kinds of data - for example, to the design and analysis of scientific experiments. Conversely, socio-economic statistics can be collected and analyzed either with or without the help of mathematical statistics. 

This situation also applies to the practice and theory of sampling. Modern probabilistic sampling theory constitutes a branch of mathematical statistics. As such, it can be applied not only to the collection and analysis of socio-economic statistics but also in other fields, such as production quality control. Likewise, socio-economic statistics can be collected with or without the application of modern sampling theory. This theory is not applied both when data are collected by complete enumeration of the population and when non-probabilistic sampling methods of pre-modern origin are used. 

The central focus of this thesis is on the Explication of mathematical statistics, and of modern sampling theory in particular, to the exercise in socio-economic statistics represented by the Soviet Family Budget Survey. We do not therefore cover the full range of issues either of socio-economic or of mathematical statistics, but consider the intersection between the two. 

Contemporary Soviet statistical practice can be properly understood only in a historical perspective. This is much truer of Soviet than of Western statistical practice, because the full application of mathematical statistics to socio-economic statistics continues to be obstructed by factors which have their origin in the Stalinist period. In particular, pre-modern forms of sampling remain in much wider use in the USSR than they do in the West. 

This chapter outlines the pattern of historical development of socio- economic statistics, of mathematical statistics, and especially of the interaction of the two. Sections 2 and 3 are devoted to the development of mathematical statistics in general and of its application in socio-economic statistics, while Sections 4 and 5 cover the same ground with respect to sampling in particular. The pattern of development in the West - Western Europe and North America - is dealt with in Sections 2 and 4, the more or less independent pattern of development in Tsarist Russia and then in the USSR in Sections 3 and 5. 

2 The historical development of mathematical statistics in the West 

Socio-economic statistics, in the form of State statistics, can be traced back to the censuses of the ancient riverine civilizations. Mathematical statistics, by contrast, has emerged relatively recently. Its precursor was the school of "Political Arithmetic" founded in seventeenth-century England by Graunt and Petty, who used elementary probabilistics to study such public issues as the causes of disease (Pearson 19?8). Further advances in the analysis of statistical variation were made by such nineteenth-century mathematicians as Poisson, Gauss and Quetelet. Mathematical statistics finally emerged in its contemporary form during the period 1890-1940. The most crucial contributions to this process were Galton's early theory of correlation and regression, the "classical" theory of statistical inference of Neyman and Pearson, and the work of Fisher on the analysis of variance. 

The socio-economic orientation of Political Arithmetic notwithstanding, the development of mathematical statistics in the nineteenth and early twentieth centuries was primarily motivated by the needs of the physical and especially the biological sciences. The Gaussian study of statistical variation was concerned mainly with the problem of experimental error in applied science. Galton was interested above all in heredity; Gossett ("Student") was led to the t-test of statistical significance by the needs of beer production in the Guinness brewery at Dublin; Fisher was occupied in the design of field experiments between the wars at the Rothamsted Experimental Station in Hertfordshire, England. The new discipline was for some time known by the name of "biometrics". 

The application of mathematical statistics to socio-economic statistics was delayed by the isolation of State statistics from the new theoretical developments. The officials responsible for State statistics lacked the mathematical training necessary to appreciate the potential value for their work of probabilistic methods, while most of the mathematicians who pioneered these methods appear to have taken no interest in State statistics. Interaction between socio-economic and mathematical statistics was at first promoted only by a few reformers who understood both subjects, such as Professor Bowley, the occupant of the first Chair devoted to statistics in the social sciences (at the London School of Economics). In Britain the gap was substantially overcome when a more professional Government Statistical Service was built up after World War Two, although the institutional relationship between State statistics and mathematical statistics remains a source of some difficulty even today. 

3 The historical development of mathematical statistics in Tsarist Russia and the USSR 

There developed in Russia in the late nineteenth century an autonomous tradition of socio-economic statistics, mainly based in the statistical services of the local-government zemstva,[3] which was more sensitive to the potential applications of mathematical statistics than the Western State statistics of the time. Such classical Russian statisticians as Chuprov were simultaneously socio-economic and mathematical statisticians. 

Soviet statistics of the 1920s in many ways represented a continuation of the zemstvo tradition. The statisticians of the Central Statistical Administration (TsSU) enjoyed both generous State support and a considerable degree of professional autonomy
(Wheatcroft 1980). They took a great interest in the work of Western mathematical statisticians, whose methods they developed further and applied to economic analysis. At the same time they criticized Western mathematical statistics for its "empty empiricism" and lack of interest in substantive issues (Yastremskii 1927).

At the end of the 1920s a group of mathematical statisticians associated with the New Economic Policy came under Stalinist attack. Some Stalinist statisticians used this campaign to attempt to discredit mathematical statistics as such, on the grounds that probabilistic methods, while eminently suited to analysis of the anarchy of the capitalist market, were alien to a planned socialist economy. Other- argued that, while the "wreckers" had misused mathematical statistics for anti-State purposes, planning could not eliminate all probabilistic phenomena (for example, the weather) and therefore the correct application of mathematical statistics remained necessary (Smit 1930). Conflict between these two points of view continued throughout the Stalinist period, and the defenders of mathematical statistics were vindicated in the years following Stalin's death. 

The dogmatic opponents of mathematical statistics at no time achieved total dominance. Thus, even in the years after World War Two, when they took advantage of the "anti-cosmopolitan" campaign to denounce "enemies of the people ... who propagate bourgeois theories under the slogan of defense of mathematics" (Methodology 1952), they did not succeed in suppressing statistical methods of quality control in the aviation industry, the practical need for which was realized by the leadership. However, the prolonged influence of the dogmatists has had a powerful impact on Soviet statistics. 

First, the interaction of mathematical with socio-economic statistics was terminated. "The possibility of applying the methods of mathematical statistics to the statistical study of social phenomena", and often even "the expediency of mathematical methods of any complexity in statistics", were denied (Nemchinov 1955). Many mathematical statisticians left socio- economic statistics to work in other fields.[4] A "general theory of statistics" emerged which expounded a methodology uninformed by mathematical statistics.[5]

A great deal of work on the socio-economic application of mathematical statistics has been carried out since the 1950s. Nevertheless, most of this work has been done in institutes outside TsSU, and has not greatly affected the methods in use within TsSU itself. It is published in such journals as Uchenye zapiski po statistike (''Scholarly Notes on Statistics") and Ekonomika i matematicheskie metody ("Economics and Mathematical Methods") rather than in the TsSU journal Vestnik statistiki ("Statistical Courier"). TsSU has not been among the organizations participating in the series of conferences on the application of mathematical statistics in economics held from 1972 onwards.[6] 

Moreover, the teaching of mathematical statistics and the teaching of socio-economic statistics remain to a considerable extent isolated from one another in educational institutions. Mathematical statistics rarely occupies a prominent position in the curricula of the institutions which train staff for TsSU - the statistical tekhnikumy (vocational schools), the Moscow Economic-Statistical Institute, etc. - while the mathematics faculties of higher educational institutions teach a very abstract and non-applied kind of mathematical statistics. 

Second, the dogmatic positions of the Stalin period have not yet been completely overcome. One still occasionally comes across expositions of the view that probabilistic schemas are inapplicable to socio-economic phenomena (Maslov 1971 pp. 35-6),[7] or of the view that probabilistic methods contradict the nature of a planned economy (Lipkin 1977). It may well be that positions of this kind will disappear from circulation when the older generation, educated in the Stalin period, leave the scene. However, less explicit attitudes at least partly originating in the earlier dogma may be more persistent.

Soviet approaches to forecasting provide an example of such attitudes (Shenfield 1983a). A. Ya. Boyarskii, the Head of TsSU’s Scientific Research Institute, observes that State statisticians are accustomed to dealing with figures that are (supposedly) uniquely accurate and must therefore overcome a "psychological barrier" before accepting the non-unique results of probabilistic forecasting (Metodologicheskie 1977 pp. 8-9). 

A discussion of significance testing by Boyarskii (1980) shows that he himself remains influenced by another tenet of the Stalinist doctrine of statistics -- the idea that the function of statistics is to illustrate theories already known to be true rather than to assess tentative hypotheses. He argues that, even if a test of statistical significance rejects an apparent correlation between the scale of production and productivity of 0.1 as a chance deviation from zero, it is natural for an economist with a theoretical knowledge of economies of scale to consider this a high correlation. From this point of view there is no way that any statistical analysis could ever discredit prior assumptions. 

4 The historical development of sampling theory in the West 

Most nineteenth-century socio-economic statisticians took the view that only the data of complete censuses could be regarded as "statistics properly speaking". This was an understandable attitude at a time when the only sampling methods practised were widely known to be unreliable. 

The most important of these very early sampling methods was "the monographic method", invented by the social reformer LePlay, who from 1829 onwards collected hundreds of detailed "monographs" about the budget and way of life of workers' families (Lazarsfeld 1961). In a monographic survey of a population, an extremely detailed quantitative and qualitative description of a fairly small number of units is obtained. The surveyed units are supposed to be carefully selected by experts in such a way that each "type" of unit in the population is represented in the sample by one unit, or a few units, "typical" of that type (thus the alternative term "typological sampling"). As the critics of monography pointed out, there was no way of verifying the judgement of the sampler regarding typicality. Furthermore, most populations are not comprised of a few known homogeneous "types", and a monographic sample by its very nature cannot reflect heterogeneity within types. 

In the 1890s Kiaer, the Director o: the new Bureau of Statistics in Norway, developed and used a new form of sapling which he called "the representative method". Kiaer's method was imitated by Wright, the Director of the US National Bureau of Labor. The method was, in Kiaer's words: 

a partial enquiry in which the observed units are distributed so that their totality forms a miniature of the whole country, a photograph which reproduces the details of the original in its true relative proportions. 

To achieve this aim Kiaer used complex multi-stage sample designs incorporating intensive stratification and elements of systematic selection (for example: select males aged 17, 22, 27 ... with names beginning with A, B, C, L, M and N). Strata proportions were determined on the basis of the results of previous censuses, which also served as a means of assessing the representativeness of the sample. The main difference between such "purposive" or "balanced" samples and modern multi-stage sample designs is the absence of random selection within strata. 

Probability theory was first systematically applied to sampling in the West by Bowley, who introduced the basic theory of simple random sampling in 1906.[9] The great advantage of random sampling is that, by controlling the probabilities of inclusion of population units in the sample,[10] it makes it possible to estimate by means of probability theory the precision of sample estimates in the form of standard errors or confidence limits. The first social survey using probability sampling was conducted by Bowley in Reading in 1912 (Maunder 1977). The theory of probability sampling was extended to stratified random sampling by Neyman and Pearson in 1934.

In the 1930s large-scale practical experimentation with probability sampling was undertaken by US agencies such as the Bureau of the Census and by the newly formed Indian Statistical Institute. In some countries, such as Sweden, survey sampling before the war continued to rely on "the representative method" (Medin 1983). Probability sampling replaced earlier forms of sampling in State statistics after the war. In market research and opinion polling, however, balanced sampling still remains in use under the name of "quota sampling". Moreover, sampling theorists in the 1970s have taken renewed interest in the possibility of putting balanced sampling on a sound basis.[11]

The relatively late emergence of sampling theory is perhaps the most striking manifestation of the former isolation of socio-economic from mathematical statistics. Both the practical need for sound sampling and the mathematical apparatus for its development already existed in the nineteenth century, but the necessary interaction between the potential suppliers and the potential consumers of sampling theory was lacking. 

5 The historical development of sampling theory in Tsarist Russia and the USSR 

As in the West, State statistics in nineteenth-century Russia relied mainly on complete censuses. However, various forms of non-probability sampling came into use towards the end of the century. 

Most of the studies of peasant household budgets which the statisticians of several zemstva undertook from the 1870s onwards were monographic surveys, based on the selection by one method or another of households supposedly "typical" of different regions (Wheatcroft 1980). 

Apparently peculiar to Russia was the form of incomplete enumeration known as "the census method" (tsenzovoi metod). The "census" (tsenz) here was a register of all those population units considered important enough to be included in the statistics; data were collected on all these units and only on them. Thus, the Tsarist Ministry of Finance maintained in the late nineteenth century a list of "census railway stations" for each type of freight; these lists were used to compile statistics of railway transport (Poplavskii 1927). “Census industry" consisted of enterprises with a minimum size of workforce, depending on the level of mechanization (Wheatcroft 1981). The rationale of the census method was to use limited resources to cover the main bulk of the phenomenon of interest. However, there was no way in which the results could be extrapolated to the population as a whole, as the relatively few large units covered were very far from representative of the many small units neglected. 

It seems that, unknown to the West at the time, the theory of probability sampling was developed independently in Russia several years in advance of corresponding Western work. The application of probability theory to sampling was first proposed in a paper which Chuprov presented to a congress of scientific research workers as early as 1894. The theory of optimal allocation in stratified random sampling, generally attributed to a 1934 paper by Neyman, had already been set out in a book on sampling theory by Kovalskii, published in Saratov in 1924 (Zarkovic 1956, 1962).[13]

In the 1920s TsSU felt a great need to develop sampling methods, with practical experimentation often proceeding in advance of theory.[14] The State required statistical information to regulate the economy, but the coverage of all economic units was not necessary to the economic means of regulation used during the New Economic Policy. Nor was complete enumeration practicable given the scattered nature of the NEP economy. Such conditions were very favorable to the development of sampling. 

However, although the theory of probability sampling was worked out by some statisticians in the 1920s, probability sampling did not completely replace earlier forms of sampling. The census method, in particular, remained in quite wide use - for example, in the statistics of rail and water freight transport (Poplavskii 1927) and in the study of labor productivity in industry (Akinshina 1966).

The attack on the application of mathematical statistics in socio-economic statistics at the end of the 1920s had an especially deleterious effect on sampling practice and theory, the development of which seems to have been "frozen". The application of sampling theory was in general neglected during the Stalin period (Nemchinov 1955). The census method - now renamed "the method of the basic mass" (metod osnovnogo massiva) - continued to be used in such fields as rail transport statistics (Kochetov 1966), and was also applied in the new survey of collective farm markets, which covered only the largest urban centers (Belyaevskii 1962). As we shall see in Chapter A4, the method remains in use even today. 

The administration of the command economy set up in the 1930s required the collection of much statistical information on the basis of complete enumeration of economic units. The concept of ‘statistics’ was replaced by that of ‘national-economic accounting’,[15] in which sampling naturally had no place. But, as we shall argue in the next chapter when we consider the position of sampling in the post-Stalin period, even within a command economy sampling could very often substitute for complete statistical reporting, and it was neglected at great cost. 

6 Conclusion

The interaction of socio-economic and mathematical statistics has proceeded along very different paths in the West on the one hand and in Russia and the USSR on the other. 

In the West the virtual isolation of the two fields from one another which prevailed in the nineteenth century was broken down in the first half of the twentieth. Probability sampling in particular has become a central tool of State statistics, and has on the whole displaced earlier non-probabilistic forms of sampling. 

In Russia an independent statistical tradition developed in the few decades before 1917 which proved capable of integrating socio-economic with mathematical statistics and which reached its apogee in the Soviet 1920s. Up to that time Russian and Soviet statistics were somewhat in advance of the West in the field of sampling theory and practice. 

However, progress was frozen at the onset of the Stalin period, when the application of mathematical statistics in socio-economic statistics came under sharp attack. An isolation of mathematical from socio-economic statistics was imposed, similar to that which was now disappearing in the West. Sampling was neglected, and the early non-probabilistic forms of sampling remained in wide use. Since Stalin this legacy has been overcome only to a limited extent.

Notes

[1] Methods of collecting, presenting, analyzing, interpreting and using socio-economic statistics can be usefully classified into three categories: 

(a)  the methods based on relatively simple mathematics which mathematical statisticians call "descriptive statistics"; 

(b)  the methods of mathematical statistics, based on probability theory; and 

(c)  relatively complex mathematical methods not based on probability theory (for example, analysis of indices or non-stochastic programming). 

[2] In 1906 Professor Bowley addressed the British Association for the Advancement of Science as follows:

"Edgeworth's illustrations in 1885 of the importance of mathematical methods in testing the truth of practical deductions have as yet borne singularly little fruit... It is time that mathematical statistics was brought to bear on the criticism and analysis of existing industrial statistics... Most of our statistics remain untested and their significance not analyzed" (Maunder 1977). 

[3] The zemstva, institutions of local government set up in 1861 by Tsar Alexander II, enjoyed a degree of independence from the central government and were open to external intellectual and political influences. 

[4] For example, V. S. Nemchinov turned to the design and analysis of agricultural experiments at Bezenchukskaya Experimental Station, developing a computational system based on Chebychev's polynomials (Nemchinov 1946).

[5] Reformers in the post-Stalin period have pressed for the methods of the "general theory" to be combined with methods from mathematical statistics. For example, Yuzbashev (1967) criticizes the method of "analytical grouping" for ignoring the confounding effect of uncontrolled variables in comparing groups of units, and suggests that it be combined with the analysis of variance. 

[6] For an account of one such conference, the All-Union Scientific-Technical Conference on the Application of Multivariate Statistical Analysis in Economics and on Production Quality Control, held at Tartu (Estonia) in 1977, see Aivazyan et al. (1978). 

[7] Professor P. P. Maslov was a veteran Soviet statistician, prolific in a number of branches of State statistics. According to his obituary, he was "one of the greatest of contemporary statisticians" (Ryabushkin and Sinokov 1975).  

[8] For discussions of the early history of sampling, see Stephan (1948), You (1951), and O'Muircheartaigh and Wong (1981), of which the last is the most perceptive. 

[9] In fact, it was Bortkiewicz, who was in 1901 the first in the West to suggest applying probability theory to sampling problems. He recommended the use of Poisson's formula to determine whether differences between census control proportions and sample proportions could have arisen by chance. 

[10] Simple random sampling provides for equal probabilities of inclusion, which was at first regarded as an essential principle. Random sampling in general provides for known, but not necessarily equal, probabilities of inclusion. 

[11] It was Royall (1970) who resurrected balanced sampling in the context of a non-Bayesian superpopulation approach. For a discussion of the issue see O'Muircheartaigh and Wong (1981 pp. 12-14), who are skeptical as to whether the possible gains from balanced sampling in raising representativeness are likely to outweigh the risks from giving up the ability to estimate precision. Moreover, balanced sampling has been justified only under certain conditions. 

[12] The census method is sometimes referred to as "concentrated sampling". 

[13] It is symptomatic of the subsequent fate of sampling in the USSR that Soviet authors do not make reference to Kovalskii's work when discussing the origins of sampling. It was the Yugoslav statistician Zarkovic who rediscovered Kovalskii's book in the Lenin Library in Moscow. 

[14] Conferences of statisticians instructed the Methodological Section of TsSU to develop the theory of methods found necessary in practice, such as cluster sampling (Zarkovic 1956). For an account of the sampling methods used in Soviet sociology in the 1920s, see Sheregi (1978). 

[15] This is reflected in the fate of TsSU itself. In 1930 the Central Statistical Administration ceased to exist under that name; its staff were incorporated into the State Planning Agency (Gosplan) as its Economic-Statistical Sector, renamed in 1931 the Sector of National Economic Accounting. In 1941 TsSU got back its original name but remained subordinate to Gosplan. Only in 1948 did TsSU regain the status it had in the 1920s.