It makes sense that any data collection using epidemiological methods should be preceded by a careful consideration of the purposes for which they will be used. Yet convention often seems to overtake such considerations.
A key question should be whether the purpose is research, audit, or surveillance. Research should aim to answer a key question and related subsidiary questions and should be designed and planned as such. The purpose of audit is to improve a process or service by examining its functioning against defined objectives and standards; audit should not be carried out without inclusion of a process for correcting flaws in the process or service being audited. Surveillance is similar to audit but is designed to monitor trends in populations over time and to alert decision makers to any changes that require early or urgent action.
A further difficulty may lie in a misunderstanding of the differences between clinical and epidemiological assessment. The measurement of psychiatric disorders in national surveys, which aim to provide population estimates, differs substantially from methods used in judicial and clinical decisions, for example, for benefits, compensation, and need for services. Methods used for assessing individuals, say, for a treatment program, are rarely the same as those required in epidemiology for classifying populations. In the case of the former, detailed and time-consuming assessments by a range of professionals of contexts such as family situations are justified because such methods are sensitive to individual circumstances and produce results that are equitable between individuals. In surveys, provided that the methods used deal adequately with the majority of people and can be done within a relatively short time, it matters little if a small minority have been classified differently than they would had more detailed procedures been used.
Quality Of Data
Conclusions regarding epidemiological estimates, the modeling of possible determinants, service planning, and needs assessment all require the use of comparisons. Comparisons have little value or legitimacy if the data employed are not collected reliably. The types of measures available are discussed later in this research paper. Because it is essential that epidemiological methods are reproducible (reliable), comparison may be either too costly or impractical when purely clinical measures are used. However, it is also important in developing either a new measure or in using an existing measure in a population with different language or culture, that it is generally accepted as representing descriptions used in clinical settings. Small-scale comparisons of epidemiological questionnaires and systematic clinical evaluations have been carried out occasionally in general population samples. At first sight the level of agreement between these appears to be poor and likely to be limited by the uncertain consistency with which clinical assessments can be used outside in the general population where most conditions are less severe and more fluctuant over time. It is reassuring therefore that clinical and epidemiological measures, when compared in clinical settings, can show high levels of agreement.
Design Issues
Study designs can be divided into the experimental and nonexperimental; the former are infrequently used in the collection of information on psychiatric and psychological disorders in the general population. New mental health policies are seldom subjected to the rigors of experimental evaluation in the general population, although where feasible this can be particularly informative. One experimental design variant that may be particularly feasible in practice is the cluster randomized evaluation. Examples are beginning to appear in the mental health field in which the cluster is an organizational unit, for example, a primary care practice, a specialist team: existing and new policies can be compared between randomly allocated cluster units (MacArthur et al., 2002). Another example makes use of the classroom or work team unit, which could be used in prevention policy trials. As in any experiment involving human subjects, ethical considerations arise: interventions can have wanted and unwanted effects, and those participating should be informed. Careful consideration should be given to anticipating possible harm and minimizing its likelihood; experiments are only justifiable when there is genuine uncertainty about the outcome.
A detailed discussion of the use of different designs in psychiatric epidemiology can by found in a chapter by Zahner and colleagues (1995). Here we focus selectively on the factors that are sometimes neglected in the design of studies in the general population. Most first-generation mental health surveys were typically small, based in a local community, and carried out by a small academic team. The past two decades have seen the emergence of adequately powered, large sample size surveys using complex, often clustered sampling methods, systematically developed and tested structured instruments, and specialized ‘survey’ methods of analysis, carried out by collaborations between professional survey organizations and academic researchers, which are referred to in examples that follow.
Cross-Sectional Surveys
This research paper refers mainly to the use of cross-sectional and longitudinal designs. Great attention is paid in such surveys to ensuring the representativeness of subjects so that inferences are generalizable and can be used, for example, at a national level and sometimes in small area statistics. Because we regard this as a fundamental issue, which is often inadequately handled in surveys other than those carried out by large (generic) survey organizations, we give this particular emphasis here.
The approach taken by one of us to the challenge of sampling children nationally illustrates some of the factors that can and need to be considered. Several methods of obtaining a representative sample of children in Great Britain were considered: carrying out a postal sift of the general population to identify households with children, sampling through schools, using administrative databases, and piggybacking on other surveys. It was decided to draw the sample from administrative records, specifically, the Child Benefit Records held by the Child Benefit Centre (CBC). Parents of each child under 16 living in the United Kingdom are entitled to receive child benefits unless the child is under the care of social services. Using these centralized records as a sampling frame was preferred to carrying out a postal sift of over 100 000 addresses or sampling through schools. The postal sift would have been time consuming and expensive. The designers did not want to sample through schools because they wanted the initial contact to be with parents who then would give signed consent to approach the child’s teacher. A two-hour survey was regarded as too long for piggybacking on other surveys.
Use of centralized records (see later also) does have some disadvantages; access to the records can be problematic and the frame may not be accurate or comprehensive. It was realized that some child benefit records did not have postal codes attributed to addresses: 10% in a first survey and 5% in the repeat. The Child Benefit Centre had no evidence that records with postal codes were different from those without. The addresses with missing postal codes probably represent a mixture of people who did not know their postal code at the time of applying for child benefits and those who simply forgot to enter the details on the form. If there are other factors which differentiate between households with and without postal-coded addresses, the key question is to what extent these factors are related to the mental health of children. Because these factors are unknown, one does not know what biases may have been introduced into the survey by omitting the addresses without postal codes.
The survey design dealt with the problem of omission of children in the care of social services by carrying out a separate survey of this vulnerable group even though such children represent only 0.5% of the population in England. Previous research had indicated high rates of mental disorders among this group. Also excluded from the original sample were cases in which ‘action’ was being invoked, such as occur with the death of the child or a change of address. These are administrative actions as distinct from some legal process concerning the child and hence should not bias the sample in any way.
Small-area statistical estimates are population approximations developed by specialized statistical procedures such as spatial interpolation. They allow policymakers, local planners, and individual users of services to extrapolate census and survey data to specific local areas. Very large survey sample sizes may be used in conjunction with census data allowing linkages between different information sources on economics, health, crime, quality of the environment, and so forth. The estimates obtained may be subject to census (or survey) or modeling error. Trend estimates may not be directly comparable from year to year because of changes in boundaries, data, and methodology. Estimates can be improved by the use of spatially more precise, geographically coded, building identifiers. Alternatively postal code data or an equivalent coding system are essential but will not be available in all parts of the world.
Screening And Full Assessment
Although in the children survey example all respondents were administered the same set of questions, many surveys employed two-stage or multiphase survey designs. The first stage consists of a self-report questionnaire administered to the full sample by lay interviewers who do not need to have any clinical training. In a second stage, a randomly selected subsample will undergo a more complex and detailed assessment, possibly involving some degree of clinical expertise. The second phase might involve a full clinical assessment of one or more forms of mental disorder, analogous to a second stage of a population dietary survey in which a random subsample of first stage respondents undergoes a more detailed inventory of all food kept in a household, and of food purchases within a past number of days, which may be carried out by a smaller team of research dieticians.
There are considerable advantages and also disadvantages to both the single and two-phase designs; these are well discussed in the literature (Newman et al., 1990; Shrout and Newman, 1989). The advantages of the single stage approach are as follows:
- Detailed information is collected on all respondents. A sample distribution can be produced on all subscales even though only those with an above-threshold score will have psychopathology.
- With the possibility of a longitudinal element in the survey, there is a large pool of respondents from which to select controls who could be matched on several characteristics to the children who exhibit significant psychiatric symptoms during the first-stage interview.
- A one-phase design is likely to increase the overall response rate compared with a two-phase (screening plus clinical assessment) design.
- A one-stage design reduces the burden put on respondents. Ideally, a two-phase design would require a screening questionnaire to be followed up with an assessment interview administered to the selected respondent. A one-stage design only requires one interview.
One of the advantages of a one-phase over a two-phase design is that it can be carried out in a far shorter time scale. The main disadvantage of a one-stage design is cost. The administration is far cheaper in two-stage designs, although they are likely to have more biases and less precision.
Cohort And Longitudinal Surveys
Cross sectional surveys explore associations between two variables (x and y) whereas prospective surveys allow the researcher to investigate the problem of unknown direction of causality (x → y; x←y; x ↔ y). In relation to mental disorder they can also provide invaluable information on the persistence and duration of mental disorder over time in the population (in general the commoner and less severe forms of disorder in adulthood tend to be of short duration rather than long lasting). However, they are not appropriate for examining treatment and service effectiveness prospectively because, if anything, at a given time, those having treatment are likely to be more severely ill and to have a poorer subsequent health outcome.
In our experience the more common obstacle to successfully carrying out a longitudinal study is a failure to include this element in the design from the beginning of the study, a ‘keeping in touch component.’ This is essential if the number followed up successfully is to be maximized. In an adequately funded ‘keeping in touch’ program, respondents are regularly sent reminders and updates, often including personal birthday greetings, on the progress and successful achievement of the study, thus sustaining what can be a considerable commitment on the part of respondents. This of course requires sustained funding over much longer periods some of which could be used to provide incentives.
Informed Consent
According to Martin (2000), the validity of survey research depends crucially on obtaining as high response rates as possible from the sample selected for a survey. However, individuals selected have a right to make an informed decision about whether or not to participate. Martin argues that without personal contact with an interviewer, the conditions needed to obtain informed consent cannot be achieved. Use of opt-out procedures based on a letter (and opt-in procedures to an even great extent) guarantees neither that the information needed to ensure that consent is informed reaches the right person nor that the person has read and fully understood what participation entails. Martin has concluded that truly informed consent can be obtained only if it is sought by an interviewer trained to explain the survey fully and answer all questions. This is analogous to a personal approach to take part in a clinical trial in which informed consent must also be obtained.
The survey research literature and standard survey practice both emphasize the importance of motivating people to agree to take part in surveys by stressing the importance of the survey and of their personal contribution. There is no evidence that such motivation is seen as exerting undue pressure. The voluntary nature of surveys is always explained, and respondents are free to refuse to answer particular questions or to terminate their participation at any stage. Additional safeguards in the case of the longitudinal and panel surveys are that explicit permission is sought to continue at each stage of the survey, so initial consent does not imply consent to every part of the survey process (Martin, 2000).
Case Control Studies
Measures of exposure status can be compared in which disease status is determined at the time of subject selection. This design may be useful in studying rare conditions, events, or exposures. However, unless these are obtained in the same population setting from which controls are drawn, comparisons and resulting estimates may be biased. This design is rarely used in this field.
Information Sources
Types of information sources can be divided into administrative and survey or census. Some very useful studies can be performed by means of existing well-collected administrative data, thus avoiding the high cost of new data collection and using those resources instead for the important tasks of analysis and dissemination.
Administrative Sources
For cultural and legal reasons, administrative authorities will probably have the best data on causes of death such as suicide and homicide. Data can exist in registers of independent organizations, provider services, or of the authority organization itself.
Censuses
Censuses are a similar source of data; the main advantage is that they collect whole-population data. But census sources are limited by the fact that questions on health are rare and virtually unheard of in relation to psychological and mental health. Other disadvantages are that they are carried out relatively infrequently, rely on proxy information (provided by one household member), and rely on another service such as the postal service to convey data.
Surveys
Surveys can be an efficient and relatively low-cost method for obtaining large numbers of observations in a systematic, reliable way and provide powerful methods for addressing questions on health. Such survey data sets are increasingly archived after a period of time and therefore remain available to the wider research community and to policy advisers (see the Relevant Websites section at the end of this research paper). This is good in that better use is made of such data at little extra cost, but it can have bad effects also, particularly in a complex area such as mental health, unless the original survey designers can contribute the analysis and interpretation of the data in order to obviate serious misunderstandings.
Mental Health Surveys
Early mental health surveys tended to be carried out on small samples obtained in single communities and were typically conducted by an individual or small academic team. Estimates from these studies carried little precision, and only substantial effects were demonstrable. Large-scale, statistically more powerful, survey methodologies began to impinge on the field of psychiatric surveys with the Epidemiologic Catchment Area Survey of Mental Disorders (ECA) (Robins et al., 1984), British National Psychiatric Morbidity Surveys ( Jenkins et al., 1997) (see Relevant Websites), the National Comorbidity Survey (NCS) (Kessler et al., 1994), and now the WHO World Mental Health Survey (Demyttenaere et al., 2004), all of which have served to transform the field. Not only did this development herald the introduction of new instruments for assessing mental disorders (see below) but it also served to refine sampling and statistical methods used to generate scientific and policy information. Such surveys have probably been more successful in answering questions about effects within subgroups of the population such as those who are economically disadvantaged. They have also been effective in collecting representative data from difficult-to-reach populations, including low-income populations in less economically developed parts of the world. General health surveys and social and economic surveys have also made increased use of measures of mental and physical health and functioning, thus bringing evidence to a much wider community of policymakers and decision makers.
Registers
Disease and service registers have also played a key part, although few countries have been willing to or capable of operating these over sustained periods. Scandinavian countries, most notably Denmark, have made a contribution in this regard to studying, for example, rarer conditions such as bipolar disorder and other specific forms of psychosis within a whole population context. External researchers have also been able to work collaboratively with those responsible for maintaining such registers. The use of such sources must include a careful consideration of the quality of the register’s coding of data.