Metropolitan Police: Crime Statistics Database

Details of MEPO 36
Reference:MEPO 36
Title:
Metropolitan Police: Crime Statistics Database
Description:

The datasets derived from the Metropolitan Police's Crime Statistics System (known within the Metropolitan Police by the code 'ME') contain data relating to crimes reported within the Metropolitan Police District which were input to the Crime Statistics System between 1990 and 1997.

The datasets derived from the ME Crime Statistics System fall into the following categories:

  • Datasets for the years 1990-1992, where the division between one year and the next is based on a calendar year. Each of these datasets comprises two flat files: a year end extract file (YTDEXTRACT), consisting of data relating to offences, clear-ups, arrests and victims input during that year; and a year end 'No Crime' extract file (YTDNOCREXT) holding data on allegations classed as 'No Crime', also input during the year in question.
  • Datasets for 1992-1993, 1993-1994 and 1994 where the division between one year and the next is based on a financial year. The 1994 dataset does not include a 'No Crime' file.
  • Datasets for 1994-1995 and 1995-1997, again based on a financial year, with the second including two years' worth of data. Each dataset comprises a single YTDEXTRACT file which reflects changes to the ME System to allow for compatibility with the Crime Report Information System (CRIS).
  • A final dataset is believed to correspond to the data in the ME database at the end of the System's lifespan. It includes data which was input to the system between February 1992 and March 1997.

It is thought that the ME datasets originated in 'year-to-date' files which were produced as a cumulative record of data input to the database each year. They were updated on a monthly basis with data extracted from the database, and were 'archived' at the end of the calendar or financial year.

Calendar year datasets for 1990-1992; financial year datasets for 1992-1994: These datasets predate the changes to the structure of the ME System which were made to accommodate CRIS. Each consists of a year end extract (YTDEXTRACT) file and a 'No Crime' extract file (YTDNOCREXT).

The YTDEXTRACT files contain the following types of information relating to offences, clear-ups, arrests and crime victims:

  • The record type: i.e. whether the record contains details of an offence which has been reported but not cleared up, an offence which has been cleared up, a clear-up not connected with a previously recorded offence, an arrest, or a victim of crime.
  • Classifications of the type of crime according to the classification schemes used by the Home Office (known as 'HO classes') and the Metropolitan Police (known as 'CO classes').
  • The type of Metropolitan Police Crime Report Book in which the paper crime report was filed; the serial number of the crime report within the Crime Book.
  • When the crime took place: the date that the offence was reported or that the arrest was reported; the hour of the day and day of the week when the offence occurred; and the year and month when the details were input to the database.
  • Where the crime took place.
  • The manner in which the offence came to the attention of the police.
  • Details of the commission of the offence: i.e. any means of attack, weapon and means of getaway used; the number of attackers; any relationship between the victim and the attacker; the type of any drug involved; the nature and value of stolen property; the value of recovered property; whether the vehicle was recovered (in the case of crimes involving vehicles); and the number of times that stolen cheques or credit cards were used (in the case of offences of obtaining by deception).
  • The number of victims of the offence and of prisoners held.
  • In the case of arrests: the manner in which the arrest occurred; the type of police officer who made the arrest; the nature of any proceedings against the prisoner; the age, sex, and ethnicity of the prisoner; whether the prisoner was bailed; and whether the prisoner had previously been convicted of an offence.
  • In the case of victims of crime: the age, sex and ethnicity of the victim and the nature of any injuries sustained.

More limited details are entered in the year end 'No Crime' extract files (YTDNOCREXT), which only hold data on allegations classed as 'No Crime':

  • The record type.
  • The type of Metropolitan Police Crime Report Book in which the paper crime report of the 'No Crime' was filed (this acts as a broad classification of the allegation: e.g. burglary, cheque fraud, shoplifting); the serial number of the crime report within the Crime Book.
  • Classifications of the type of 'No Crime' allegation according to the classification schemes used by the Home Office (known as 'HO classes') and the Metropolitan Police (known as 'CO classes').
  • The date that the 'No Crime' allegation was reported, and the year and month that the data was input to the database.
  • Where the 'No Crime' allegation was reported.

Financial year datasets for 1994-1997; final database: These datasets reflect changes to the structure of the ME System which were apparently made in 1994-1995 to facilitate concurrent running with CRIS. There are a number of fundamental differences to previous versions of the database:

  • 'No crime' data is recorded along with data relating to offences, arrests, clear-ups and crime victims in a single extract file (YTDEXTRACT in the case of 1994-1995 and 1995-1997, DBEXTRACT in the case of the final database extract). These files are slightly smaller than the YTDEXTRACT files in previous datasets, in terms of the number of fields (60 fields versus 65). Different field names are employed than in the preceding datasets, though in the majority of cases there was no change in fields' functions.
  • Crimes and 'no crimes' are no longer classified according to the Metropolitan Police's crime classification system ('CO classes'); only the Home Office classification scheme is employed.
  • There are fields recording detailed classifications of the 'venue' (location) where an offence occurred (VENUE-CODE, represented as VENUE-CODE-MAJOR and VENUE-CODE-MINOR), and any vehicle or plant involved in an offence (VEHICLE-CODE), which do not appear in the earlier datasets.
  • Types of stolen property are classified in greater detail than in the earlier datasets, with four fields (PROPERTY-1, PROPERTY-2 etc) set aside for stolen property details.

The datasets in this series are available to download. Links to individual datasets can be found at piece level.

Date: 1990-1997
Arrangement:

Hardware: ICL 3960 mainframe. An ICL 2966 and an ICL 2988 were used for development work.

Operating System: VME, with an SCL-based interface.

Application Software: IDMSX (Integrated Database Management System), manufactured by ICL, was used to create the ME database. ICL's ReportMaster and the Office for Population Censuses and Surveys' Tabulation Utility (TAU) were used for the production of tables. TAU was phased out in 1991 and replaced with an in-house program. SPSS-X 3.0 was used for ad hoc enquiries. This package and ICL's QueryMaster replaced two tabulation packages called FIND and SDTAB which had been used with the MC System. QueryMaster was used for ad hoc enquiries of the database, monthly, year-to-date and historical files. ICL's Transaction Processing Management System (TPMS(X)) was used to manage G10's access to the database via the transaction processing service. By 1992 Microsoft Excel 3.0 was used by PIB to manipulate data downloaded from the ME database onto local PCs.

Logical structure and schema: The calendar year datasets for 1990-92 and the financial year datasets for 1992-93 and 1993-94 consist of two files for each dataset (a YTDEXTRACT file and a YTDNOCREXT file). The 'truncated' financial year dataset for 1994 consists solely of a YTDEXTRACT file. The financial year datasets for 1994-95 and 1995-97, by contrast, each consist of a single YTDFEXTRACT file holding all data on offences, arrests, victims and 'No Crime' allegations for that year. A further dataset consists of a file which holds the data in the ME database at the end of its operational life. The format of this file (DBEXTRACT) is essentially identical to the YTDEXTRACT files in 1994-95 and 1995-97.

A distinction should be made betwen the structure of the datasets - most of which originated in cumulative extract files archived each year - and the structure of the original IDMSX database. This is believed to have been a hierarchical database centered around tables holding data for different types of records, consisting of:

  • 'A', recording details of arrests.
  • 'B', recording details of offences not cleared up before being entered into the database.
  • 'C', recording details of offences reported and cleared up before being entered into the database.
  • 'D', recording clear-ups not matched to a previous offence record.
  • 'V', recording details of crime victims.
  • 'N', recording allegations classed as 'No Crime'.

In addition to tables for these types of records, the database also included files concerned with the validation of data loaded onto the database and files designed to assist the searching of the database.

How data was originally captured and validated: The ME Crime Statistics datasets are based on data extracted by G10 and PIB from paper crime reports completed by police officers and their supervisors. These reports (known as 'Form 478') recorded details of individual incidents reported to the police. Photocopies of crime report forms, authorised by supervising officers, were required to be sent to G10. Copies of Form 478 were also sent to other branches of the Metropolitan Police. Original forms were retained in 'crime books' at police stations according to whether they related to major crime, burglary, 'beat' crime, vehicle crime, or child protection teams.

Information from crime reports was encoded and input in G10's input section. At the time that the ME System was implemented in 1989, this local system was a Microdata 8000 series. The system replaced an earlier Datapad input system, installed in 1977. The Datapad system had produced paper tape which was sent to Department of Computing Services (DCS) for inputting to the MC Crime Statistics database, supplemented by manually produced punch cards at times of high workload or during processor failures. By contrast, under the Microdata system data was loaded onto magnetic tapes which were sent by G10 to DCS on a daily basis.

One major change that accompanied the development of ME was the introduction of a transaction processing service which allowed users in G10 to input, update, view and amend records; correct errors arising from the daily loading of data onto the database; and conduct ad hoc enquiries of the database and the monthly, year-to-date and historical files. An ICL DRS300 system consisting of a processor stack, printer and four terminals was installed. This was connected to the ME System running on an ICL 3960 at Jubilee House by a MICROLAN communications network. The Microdata system was also able to store up to eight months' worth of reports in its own local database.

Validation checks on the data were performed using the Microdata software at the time of inputting by G10. Further checks were carried out at the time that data was loaded onto the mainframe at DCS.

G10's successor, PIB, acquired a new inputting system based on networked PCs and a server. This system was connected to the mainframe running the database at Jubilee House.

Constraints on the reliability of the data: The datasets derived from the ME Crime Statistics System do not cover unreported crimes, or crimes and allegations reported to the police but not recorded by them, and will be vulnerable to changes in police recording practices and in the level of reporting to the police. Victimisation surveys such as the British Crime Survey (conducted by the Home Office) have been developed to measure the 'true' extent of crime.

All queries intended to produce meaningful statistical results from the YTDEXTRACT files should include selection by record type: a count of all the records in the file without specifying record type will not give the total of all the offences input to the ME System in the year in question.

The significance of certain values relating to Metropolitan Police divisions and subdivisions, Home Office and Metropolitan Police crime classifications, and several other indicator codes in the data remains unexplained by either the metadata or other documentation provided by the Metropolitan Police.

The ME coding system is extensive and exhaustive, and complex relationships exist between the Home Office (HO) and Metropolitan Police (CO) classifications of offences and other indicator codes in the records which are largely undocumented

Care should also be taken when interpreting fields with zero or blank values. In conception, the records in the YTDEXTRACT files and the DBEXTRACT file are effectively divided into four sections:

  • 1. Data about the record (record type, dates reported and input).
  • 2. Data about the offence reported.
  • 3. Data about an arrest.
  • 4. Data about a victim.

In the event that an offence is a clear-up, with one prisoner and one victim, all this information can be contained in a single record. Additional arrest or victim records will contain significant data in the section for offence details, and in one other section, according to the record type; information in the remaining sections will be redundant or blank. 'No crimes' in the YTDEXTRACT and the DBEXTRACT file appear as single records, with the 'no crime' type being indicated by the EXT-REC-TYPE and the NO-CRIME fields.

Validation performed after transfer: Details of the content and transformation validation checks performed by NDAD staff on each ME System dataset are contained in the catalogues of individual datasets.

Held by: The National Archives, Kew
Former reference in The National Archives CRDA/1
Legal status: Public Record
Language: English
Creator: Metropolitan Police Force, 1829-
Physical description: 10 datasets and documentation
Restriction on use: Copyright of the datasets derived from the ME Crime Statistics System and related documentation rests with the Commissioner of Police for the Metropolis; copies may be made for private study and research purposes only. Subject to registration under the Data Protection Act. Authorisation for the provision of subject access to data rests with the Metropolitan Police.
Access conditions: Open unless otherwise stated
Immediate source of acquisition: In 2010 the United Kingdom National Digital Archive of Datasets
Custodial history: Originally transferred by the Metropolitan Police Force in 1998. The United Kingdom National Digital Archive of Datasets (NDAD) then held the datasets until 2010 when they were transferred to The National Archives.
Accruals: No further accruals are expected.
Publication note: Crime statistics derived from the ME Crime Statistics System were incorporated in the annual Report of the Commissioner of Police of the Metropolis for the years covered by the datasets.
Unpublished finding aids: Extent of documentation: 306 documents, Dates of creation of documentation: 1975-2000
Administrative / biographical background:

The Crime Statistics System (ME) was designed to replace an earlier version of the Crime Statistics System known as MC. This system also consisted of a database and associated files maintained by the Department of Computing Services (DCS) and, as with the ME System, inputting and analyses of data from paper crime reports were carried out by G10. The replacement of MC by ME was prompted by DCS's desire in the mid-1980s to move its applications from a DME to a VME operating system, and to decommission an ICL 2966 mainframe computer at Jubilee House, Putney which was used for running the MC System. DCS and G10 took the opportunity of the move from DME to VME to revise the structure of the Crime Statistics database. The new database was intended to use ICL's Quickbuild products (ICL ReportMaster, ApplicationMaster, etc), and to reduce maintenance overheads by providing a means for quicker enhancements and amendments to software and inbuilt security through standardised utilities. At the same time, steps were taken to improve G10's ability to access and manipulate crime statistics data through the development of a transaction processing service.

Work on the conversion of DCS's DME applications to VME started in 1985 with two pilot projects involving data on traffic accidents and juvenile bureaux systems. In the following year DCS began work on designing a new Crime Statistics database. At that time the new database was scheduled to be operational by mid-1986. In the event, a series of delays and postponements meant that final implementation of ME does not appear to have occurred until late 1989. A service level agreement relating to the System was signed between G10 and DCS in March 1990. Delays were caused by the following factors: the need to develop new statistical tables as a result of the reorganisation of the Metropolitan Police into eight areas in 1986; changes in G10's requirements regarding the amount of data to be stored in the database; the need to test the new report writing and tabulation packages; and apparent communication difficulties between G10 and DCS. A working party of G10 and DCS officers met for the first time in May 1986 to co-ordinate the conversion of the Crime Statistics System.

It should be noted that the ME System was originally designed as an interim system which would meet G10's needs prior to the implementation of the Metropolitan Police's Crime Report Information System (CRIS). The aim of CRIS was to replace paper crime reports and the central inputting of crime data with a network of dispersed terminals, which would allow police officers to input details of crimes directly to a database and to conduct on-line searches of the data. Development of CRIS started in 1981 and continued throughout the 1980s (£22.3 million were spent on the development of CRIS between 1987/88 and 1992/93). A number of slippages occurred in the system's implementation date. By mid-1988 it was believed within DCS that CRIS would replace the ME System for the production of all new crime statistics in 1990 or 1991, after which ME would be used for about five years solely for access to historical data. In the event, field trials of CRIS did not start until 1992. The system began to be rolled out on an Area by Area basis within the Metropolitan Police beginning in 1994, and finally went live throughout the Metropolitan Police District on 1 October 1996. This meant that the ME System continued to be used for the recording of crime data and the production of crime statistics well beyond its intended lifespan.

As the development of CRIS progressed, G10 and later PIB became increasingly concerned to ensure that CRIS data and data generated on the ME System should be compatible during the transitional period when both systems would be operating simultaneously. During the initial stages of the introduction of CRIS, paper printouts of crime reports produced by divisions using CRIS were input to the ME System by PIB, though this proved to be labour intensive. By 1992 DCS had started work on developing software to allow for the electronic data interchange (EDI) of ME and CRIS data. This required certain modifications to the ME System: initially, a shift from a calendar year to a financial year basis for producing datasets; later, more substantial modifications to the ME database. New software for tabulating data and running ad hoc queries of the redesigned database appears to have been developed by the contractors working on the CRIS system. Some retrospective conversion of data generated on the older version of the database may also have occurred.

Context of this record

Browse by Reference

Add a tag

Users have not yet tagged this record
  • You need to sign in to tag records. If you don't have an account please register.

 

Found an error?

If you find an error with this record description let us know

Fields marked with * are mandatory.

Your details

If you provide your details, we will contact you within 10 working days if we cannot act on your suggestion