Prior to the outbreak of the Second World War, colonial defence matters had been dealt with by the General Division. The dramatic increase in defence issues being handled by that division from September 1939 led almost at once to the creation of a separate branch within the Division just to handle colonial defence questions (Department B, or the Defence Department). It had a military staff for liaison with the armed services attached to it.
The new Department dealt with questions relating to the defence of particular territories, and also with general defence matters such as cable and wireless communications, censorship, aviation, broadcasting, aliens, passports, emigration and immigration, extradition and deportation. In 1940 the War Office assumed control of colonial forces, but it continued to exercise that control via the Colonial Office's military staff.
By the close of the war, the Department's brief had expanded to include war crimes, displaced persons, demobilisation and rehabilitation, and the acquisition of land for defence purposes. In 1948, there was a general reorganisation of the Office, and the division was reconstituted as the Defence and General Department.
The defence and the general aspects of the department's work were separated in 1955 when a Defence Department independent of the General Department was for the first time created. By this time its responsibilities had expanded to include general forces liaison, (formerly exercised by the military staff, and including matters relating to visiting forces), colonial forces, military duties in aid of civil power, war and military pensions, and, later, responsibility was added for the compilation of the official war history.
In 1961 the department was merged with the former Intelligence, Police and Security Department to form a Defence, Intelligence and Security Department, though there remained a distinct defence branch within the new department. In 1963 the department assumed responsibility for Hong Kong, becoming the Defence, Intelligence and Hong Kong Department, though in 1965 Hong Kong was passed on to the West Indies Department, leaving a Defence and Intelligence Department. This was the last organisational change before the creation of the Commonwealth Office, when responsibility for colonial defence passed to a new Commonwealth Office Defence Department.