Ministry of Agriculture, Fisheries and Food: Statistics (Census and Prices) Division: Agricultural and Horticultural Census Datasets
The Agricultural and Horticultural Census is one of the main data gathering exercises for agriculture and horticulture in England and Wales. It is a survey of main agricultural holdings in both countries, covering about 99% of the total agricultural area, and is conducted annually on the first working day of June. It provides the Ministry of Agriculture, Fisheries and Food (MAFF) and the National Assembly for Wales with 'the basic physical statistics of farming: areas of land use and crops; numbers of main livestock types; and the numbers and kinds of person working on the farm summarised by parish. To 1994, the base data for the summaries was collected for all agricultural holdings. From 1995, only the larger holdings submitted annual returns, and smaller holdings were sampled on a three-yearly basis.
Broadly speaking, the 1989-1999 censuses covered the following areas:
- The extent and type of crops and fallow, land use and land tenure, and the size of holdings.
- The agricultural labour force: numbers and types of farmers, managers and workers (full-time and part-time, male and female).
- Livestock: numbers of cattle, pigs, sheep, goats, poultry and other types of farm animals (e.g. horses, farmed deer).
- Horticulture: extent of vegetables, bulbs, flowers and fruit grown in the open, orchards, nursery stock, glasshouses and covered structures, and glasshouse crops.
- Changes in the area of a holding (land given up or taken over), and changes in a farmer's name and address.
Most of the data collected by the 1989-1999 Agricultural and Horticultural Censuses related to the situation of farmers on the census date, the first weekday in June each year. The census was a snapshot of holdings at a time of prime activity rather than a picture of holdings over an entire year. Consequently, it was complemented by surveys taken at other times, although these were not as comprehensive in their coverage.
These datasets fall into three main sub-series:
(1) Parish Summaries, in which the data was aggregated by MAFF, before transfer, to the level of the agricultural parish (on agricultural parishes, see Counties and Parishes lookup tables, below). In each dataset, the data drawn from the census is held in a single table in which there are two types of field: those which record the total of the responses in the parish to a question asked in the census, and those which record the number of holdings which answered the question. While a correlation can be drawn between most of the census questions and the fields in the datasets, this is not always the case:
- Some questions which appear in the census forms do not have corresponding fields in the datasets, because the data was not held in MAFF's system, or because a statistical decision was taken not to use the data.
- Where the census form required respondents to provide an answer other than a numeric figure (e.g.'Yes/No'), MAFF has not been able to aggregate the data in a meaningful way. For example, question 135 in the 1989 census asked respondents to answer 'Yes' or 'No' to the question 'Do you intend to keep any turkeys on your holding in the next 12 months'. Rather than there being separate fields for the numbers in each parish answering 'Yes', 'No', or not responding, the values representing 'Yes' and 'No' have been summed within a single field to produce a single numeric total (i.e. fields like this have been aggregated in the same way as fields relating to questions where farmers are asked to provide a numeric figure). Some of the fields which are affected by this problem have been excluded by MAFF from the datasets transferred.
(2) Counties and Parishes lookup tables. The Parish Summaries datasets include codes for agricultural parishes, in which the first one or two digits identify the county, and the remaining digits identify the MAFF/Welsh Office 'parish'. In England, MAFF's 'parishes' generally correspond to the lowest level of local authority, which typically is a parish council, although it may also be a larger administrative area where there are no parishes. The 'parishes' used by the Statistical Directorate of the National Assembly for Wales (and its predecessors in the Welsh Office) relate to units which predate community councils (the equivalent in Wales of English parish councils). In some cases, the names of MAFF 'parishes' are not identical to the names of modern local authority areas; where the names are the same, MAFF has advised that the MAFF 'parish' and the civil parish may not cover the same areas.
Lookup tables were transferred, by MAFF and by the Welsh Office, which translate the codes used for parishes in the Parish Summaries datasets. MAFF also supplied a table which translates the codes used for counties. As these tables do not relate to any particular sweep of the census, they have been treated as distinct datasets rather than as part of the Parish Summaries datasets.
(3) A series of datasets of County Summaries held in MAF 410.
The datasets in this series are available to download. Links to individual datasets can be found at piece level.
Agricultural Census and Survey System (ACS) (1968-1993):
- Hardware: ICL mainframes.
- Operating System: GEORGE (II & III).
- Application Software: Inhouse system written in COBOL. Enhanced and refined until its replacement by the Farm Survey System (FSS) in 1993. Tabulation Utility (TAU) was used in connection with the EC Structure Survey. In addition, there were 470 job control macros.
Farm Survey System (FSS) (1993-?):
- Hardware: Client/Server architecture operated via a Local Area Network (LAN). The FSS LAN was connected to the MAFF data communications network (MAFFnet). The FSS ran on two servers: FSS1, the main server. This managed the Ingres relational database in which data from censuses and surveys was stored. FSS2, was used by statisticians in Stats(C&S) to carry out complex statistical analyses of the data held on FSS1.
- Operating System: UNIX.
- Application Software: Ingres database management.
User Interface: Not known for ACS. FSS allowed up to 80 users to connect to the main server (FSS1) concurrently. Users moved through the system via a hierarchy of screens and menus.
Logical Structure and Schema: Each of the Parish Summaries datasets consists of a single table of aggregated data for all English and Welsh parishes for a particular census. The Counties and Parishes lookup tables likewise consist of a single table each.
In MAFF's view, the system can be divided into three main sub-systems:
- A fixed application, the same for all censuses and surveys run on the system.
- A survey application specific to each survey.
- Software tools to manipulate and analyse the data.
How data was originally captured and validated: During the 1989-1996 censuses, the basic method of data capture was paper forms. Farmers were asked to return the forms within about a week of the census date in mid June. Non-respondents were sent reminders. MAFF had the power to prosecute non-respondents.
Before 1997, farmers were sent either a 'short' or a 'long' form. The short omitted the sections on horticulture and glasshouses. Farmers were sent a long form unless previous census returns indicated that no horticulture or glasshouse data was likely to be present.
Before 1995 forms were sent to all holdings which were classed as 'main' holdings. In 1995 an element of sampling was introduced. Thereafter, sampling was applied to main holdings with a Standard Gross Margin (SGM) of less than 9600 (SGM was 'an economic measure of a farm's profitability . . . derived from the activity being recorded on the Agricultural Census return for the farm'). 'Small main holdings', were sent a census form every three years on a rolling basis; 'very small main holdings' (those with an SGM of less than 3600) were sampled randomly each year at a rate of 10%.
MAFF also attempted to integrate data collection for the census with data collection for its Integrated Administration and Control System (IACS). By 1996 Stats(C&S) were investigating the possibility of establishing closer links between IACS and the census. By 1999 this has progressed to the point that census data was derived from IACS for about 6,000 farms whose production was covered by an IACS claim.
Data inputting and validation: In the 1989-1996 censuses, the forms were returned by farmers to the Parish List Section of Stats(C&S). The rate of return of forms could be as high as 10,000 in a single day. Under the ACS, receipting was done by batching the forms and sending them to ITD's PCK section for the recording of the holding numbers ('CPHing'). With the introduction of the FSS in 1993, bar-coded forms were introduced. After forms were receipted, they were scrutinised by the Parish List clerks. Data inputting was done by batching the forms and sending them to PCK section for speed keying. This continued to be the normal method of data entry after the introduction of the FSS. However, the Parish List clerks acquired the ability to input the data directly to the FSS via input screens.
Once the forms were loaded into the system, a parameter sheet was generated which listed batches loaded, and the number of forms in each. A data load report (recording forms that failed to load) was also produced, and a validation error report (listing validation queries by county in the order of CPH numbers). The data was validated against previous returns, against other values on the form, and for credibility via validation rules written by statisticians. Validation queries were cleared by Parish List staff by reference to the original survey forms, by following standard instructions, and by contacting the farmer directly.
Provisional results for the census were published after about 60% of the forms had been received. Final results were calculated when about 80% of the despatched forms were deemed sufficiently 'clean' to be used. In order to account for non-responses statistical techniques were employed to generate a dataset covering 100% of main holdings. Two techniques were used:
- Ratio estimation: 'using previous years data as a base . . . the returned sample is poststratified by farm size (SGM) and farm type to minimise non-response bias. Each item being measured is raised independently of others and so different raising factors apply. Coverage and standard errors for the estimates vary between items but relative standard errors are typically under 2 per cent for the main crops and under 5 per cent for the livestock'.
- Imputation: 'this involves applying an appropriate trend (obtained from responding units) to the previous years data for a missing farm to give a best estimate of the current year's information'.
Production of final results involved the use of estimation to compensate for holdings which had not returned a form, or for returns which were being queried.
Constraints on the reliability of the data: MAFF stated that the Census was 'subject to some margin of statistical error on account of the quality of the basic material and the methods which are used in adjusting and statistically treating data for non-respondents'.
Validation performed after transfer: Details of the content and transformation validation checks performed by NDAD staff on each Agricultural and Horticultural Census dataset are contained in the catalogues of individual datasets.
MAFF unpublished documents relating to the Agricultural and Horticultural Census have been transferred to NDAD, and are listed in the Dataset Documentation Catalogue.
For earlier summaries see MAF 68
Datasets summarising the results of the census at parish level are also held by the UK Data Archive (www.data-archive.ac.uk/), covering the years 1975-1988 (study numbers 2064-2066, 2075, 2082-2084, 2089-2091, 2178, 2381, 2562, 2681). The Edinburgh University Data Library http://datalib.ed.ac.uk/) holds Parish Summary datasets for some years between 1969 and 1989, as well as a separate series of datasets in which parish level data is related to grid squares at various levels of aggregation.
The National Archives, Kew
|Former reference in The National Archives:
Ministry of Agriculture, Fisheries and Food, 1955-2001
21 datasets and documentation
|Restrictions on use:
This dataset was formerly closed until 2020 under the Public Records Act. Following implementation of the Freedom of Information Act this closure was reviewed and the Department agreed that no FOI exemptions apply to the data. The dataset was therefore opened in January 2006.
Subject to 30 year closure
Further accruals are not expected.
Aggregated data from the Agricultural and Horticultural Census is routinely made available by MAFF. Provisional and final results for England and the UK were published by MAFF, via Statistical Notices or Statistical News Releases. Data from the census was also incorporated in the annual Digest of Annual Census Statistics United Kingdom. G. Clark, The Agricultural Census - United Kingdom and United States, Concepts and Techniques in Modern Geography no. 35 (Norwich: Geo Books, 1982)
|Administrative / biographical background:
The Agricultural and Horticultural census dates back to the nineteenth century. The first agricultural statistics to have complete coverage of Great Britain were collected in 1866 by the Board of Inland Revenue on behalf of the Board of Trade, which was responsible for the processing and publication of the returns. In 1883 publication was undertaken for the first time by the Agricultural Department of the Privy Council, to which, in 1884, the grant made to the Board of Trade for their collections was transferred. The Agricultural Department of the Privy Council began to process the forms for itself although the actual collection remained in the hands of the Inland Revenue. These arrangements continued unchanged in the succeeding years, during which (in 1889) the Agricultural Department of the Privy Council became a part of the newly formed Board of Agriculture. Later, in 1909, the excise duties of Inland Revenue were transferred to the Board of Customs. This Board became responsible for the collection of the returns until 1912, when the duty was transferred to the Board of Agriculture and Fisheries. The Board became the Ministry of Agriculture and Fisheries (MAFF) under the Ministry of Agriculture and Fisheries Act 1919. Responsibility for the census in Scotland was devolved in 1912 to the Board of Agriculture for Scotland (part of the Scottish Office).
The completion of the census returns by farmers was done on a voluntary basis from 1866 until the passing of the Corn Production Act 1917. The Act provided for the payment of a subsidy to farmers based on the acreage of their wheats and oats, and in turn made the completion of the census returns obligatory. Voluntary completion was re-introduced in 1921, when the subsidy paid under the Act was withdrawn. The completion of census returns became compulsory again with the Agricultural Returns Act 1925, which established the statutory authority for the census. Successive enactments since then have laid an obligation on farmers to complete the forms for the Agricultural and Horticultural Census. Since 1956 farmers have returned the forms directly to MAFF's central statistical departments. Before then, forms were collected at a local level by MAFF's County Advisory Officers (1949-1956); by a network of MAFF-employed crop reporters (1919-1949); and by excise officers (1866-1918).
The current statutory basis for the census is the Agricultural Statistics Act 1979, as amended by the Agriculture (Amendment) Act 1984, and Section 12 of the European Communities Act 1972. The legislation requires farmers in England and Wales to provide information for statistical purposes, as required by the Minister of Agriculture, Fisheries and Food and the Secretary of State for Wales. It also requires that strict confidentiality be observed with respect to the data, to protect against the disclosure of information on individual holdings. One effect of this is that data gathered in the Agricultural and Horticultural Census has only been published and made available in aggregated form.
In addition to data for publication, the Agricultural and Horticultural Census provides MAFF with the core data from which sample selections for other surveys are taken. Data from the census is often correlated to data collected in other surveys: e.g. the area of wheat is correlated to the yield. Where there is a correlation, information from the June census may be used to raise the aggregates from the sample in a survey to estimate the aggregates for the population as a whole. Thus the June census is not only important in its own right but also for the processing of many other surveys.
Information derived from the census is provided to MAFF economists and policy sections, and is used in the formulation of agricultural policy, assessment of the impact of European Community policy proposals, identification of market trends, assessment of research and development needs, and in agricultural advisory work. MAFF economists carry out survey work based on data from the June census, a prime example being the Farm Business Survey. Information from the census has also been supplied to agricultural industry customers (such as the Meat and Livestock Commission and the Milk Marketing Board, now succeeded by Milk Marque), and to other government departments: e.g. the Department of the Environment (for research into changes in land use) and the Office for National Statistics (for Social Trends and Regional Trends publications, and for national employment data). Trade associations and non-departmental public bodies produce economic analyses using census data, whilst academics use the material for both research and teaching. Local authorities rely on Agricultural and Horticultural Census data for administration and the provision of statutory services: e.g. structure plans and strategic planning. Results are also used by international bodies. The census and the parallel June censuses carried out in Scotland and Northern Ireland are the main UK sources for the European Community's Farm Structure Survey, a survey conducted by EC member states every two or three years.
Within the Ministry of Agriculture, Fisheries and Food, the responsibility for the Agricultural and Horticultural Census rested (in 2000) with Statistics (Census and Surveys) Division (Stats(C&S)) in the Economics and Statistics Group. Stats(C&S) design the census, gather and analyse the data, produce statistical publications, and maintain the English part of the Parish List (a register of agricultural and horticultural holdings which is used to contact farmers for the census.
Prior to the creation of the National Assembly for Wales in 1999, responsibility for maintaining the Welsh part of the Parish List, and for controlling and correcting Welsh data for the June census and larger surveys, rested with the Statistical Directorate of the Welsh Office and the Directorate's predecessors. Smaller surveys were processed by MAFF on behalf of the Welsh Office. Welsh Office staff therefore had access to MAFF's Farm Survey System, and performed their own data processing, cleaning and validation for the census using the same suite of programs. Although the Welsh Office Agriculture Department appears on some of the Welsh census forms, and as the joint author of some publications relating to the census, NDAD was told that the Department was not directly involved in the processing of census data. The census forms used in Wales from 1989 to 1993 have 'Welsh Office Agriculture Department' at the top of the form, but the contact address is given as Economic and Statistical Services Division, the predecessor of the Welsh Office's Statistical Directorate. In 1999 the Welsh Office Statistical Directorate became the Statistical Directorate of the National Assembly for Wales Executive, to which responsibility for the census in Wales was also transferred.