DESIGNING QUESTIONNAIRES FOR USE IN MULTICULTURAL CONTEXTS

INTRODUCTION Measurement is an integral part of the assessment process in social work. Whether or not this is done formally, social workers inevitably quantify the functioning of client systems during assessment: “This community lacks access to social resources; this family experiences high levels of conflict; this person has a severe drug addiction.” All of these statements, typical of social work assessments, involve measurement.


INTRODUCTION
Measurement is an integral part of the assessment process in social work.Whether or not this is done formally, social workers inevitably quantify the functioning of client systems during assessment: "This community lacks access to social resources; this family experiences high levels of conflict; this person has a severe drug addiction."All of these statements, typical of social work assessments, involve measurement.
Formal measuring tools, such as summated rating scales, provide a more reliable and valid method for quantifying social functioning.In social work these tools are referred to as ecometric instruments.The term "ecometrics" was coined by Van Zyl (1995:31) to mean "die tegnologie in maatskaplike werk wat te make het met die kwantifisering van mens-inomgewingpassing" [the technology in social work concerned with the quantification of the person-in-environment fit].The Ecometrics Committee of the SA Council for Social Service Professions (SACSSP) has recently reworked this definition.In the ninth draft of the SACSSP's Policy Document on Ecometric Technology, ecometrics is defined as "the technology in social work concerned with accurate description and/or measurement of the adaptation of the person within the context of the environment" (SACSSP, 2006:18).
There has been an increase over recent years in the development and use of measurement tools in social work.This has been prompted by, among other factors, the press for measuring treatment or programme outcomes (Bloom, Fischer & Orme, 1995;Hudson, 1982), the need to demonstrate efficacy in order to secure scarce welfare funding (Faul, 1995), and the need to establish the scientific basis of our profession on a par with other professions (Dangel, 1994;Hudson & Faul, 1997).The SACSSP's recent approval of a policy on ecometrics may contribute to a further increase in the development and use of ecometric tools in South Africa (SACSSP, 2006).
The few social work scales available for use, however, often suffer from the limitation of having been developed within a monocultural context, typically the United States of America.South Africa, like most countries, is strongly multicultural.The legacy of apartheid has led to sensitivity in this country towards applying different measurement rules to different racial or cultural groups.Consequently, it is essential that measuring instruments are not only culturally appropriate, but also "multicultural".This implies that the instruments must be able to be used by a professional of any culture with a client of any culture.This paper, therefore, purposes to present a methodology for the design of measuring instruments, such as questionnaires, that can be used within the multicultural South African context.The central tenet of the paper is that instruments must be purposely developed for use within multicultural contexts; they should not be designed in the researcher's own culture and then subsequently applied onto other cultures.Thus, the focus of this paper will be on the initial three phases of instrument design according to Thomas and Rothman's Developmental Research (Rothman & Thomas, 1994;Thomas, 1984;Van Breda, 2004), viz. the analysis, design and development phases.The fourth phase of scale development, evaluation or validation, which requires sophisticated statistical competence and resources, is not the focus of this paper.
The paper will, therefore, begin by detailing the procedural steps that can be followed across the analysis, design and development phases.In each of these three phases, the steps will be described and then illustrated with the design of a new instrument called the Military Social Health Index (MSHI).Thereafter, selected results of the multicultural validation of the MSHI will be briefly presented to provide empirical evidence of the efficacy of the design process in producing multiculturally valid instruments.It is the author's hope that the procedural steps will be sufficiently straightforward, and the validation results sufficiently convincing, that social workers who undertake to design new questionnaires, scales or research tools, will feel empowered to utilise the procedures.

THE INSTRUMENT DEVELOPMENT PROCESS
Although many books describe the procedures involved in instrument development, few provide a comprehensive, step-by-step guide to the entire process and even fewer address the issue of multiculturalism.Nunnally and Bernstein's (1994:95) seminal text on psychometric theory, for example, contains only a single reference to the question of culture.Similarly, Faul's (1995) model of scale development, which is probably the most developed model in social work and which is based on the work of Walter Hudson (the grandparent of scale development in social work), makes no reference to the word "culture".Furthermore, her model has not been published and thus is not accessible to most social workers.

ANALYSIS
A IDENTIFY PROBLEM 1 ANALYSE THE PROBLEM 2 Determine the study end results

B FORMULATE THEORY
3 Identify and describe the theoretical framework within which the scale is to be developed 4 Identify the operational assessment area(s) that will be measured by the scale  Hudson (1982) and Faul (1995), and the developmental research of Thomas (1984) and Rothman (1994).This model comprises five phases, viz.analysis, design, development, evaluation, and diffusion and adoption (Figure 1).

ANALYSIS PHASE Introduction
In the first phase, analysis, the researcher identifies the need for an ecometric instrument measuring a particular phenomenon (or set of phenomena), determines the theoretical framework that should undergird the instrument and identifies the constructs and research that elucidate the phenomenon.Of great importance in this phase is the exploration of the crosscultural comparability of the phenomenon.This comparability must be established before the researcher begins to formulate items to be included in the instrument.

The procedure
This phase opens with clear and unambiguous formulations of the problem and the end result of the instrument design process.A new instrument must meet a specific need or problem faced by an agency or community, and this need or problem must be made explicit very early in the process.
It is helpful to conduct a state-of-the-art review (Thomas, 1984), which involves identifying existing instruments and evaluating whether they are suitable for use.Such a review will assist in ensuring that one does not "re-invent the wheel".In addition, a feasibility study (Thomas, 1984) could also be conducted, where the researcher ensures that the researcher and/or agency has the necessary time, expertise, resources and political will to develop a new instrument.
It is important at this early stage to formulate the innovation objective and requirements (Thomas, 1984).The innovation objective is a single sentence stating what the instrument should measure, while the innovation requirement is a listing of the desired characteristics of the instrument, such as the languages it should include, the approximate length of the instrument, the level of literacy required to complete the instrument, etc.
With the first steps completed, the researcher needs to formulate the theory that underpins the instrument.This may entail conducting a literature review on the subject and determining a theoretical and empirical framework for the instrument.It leads to the identification of the operational assessment area or areas that should be measured, as well as formal operational definitions of these areas.
A key step in this phase is exploring the cross-cultural comparability of the assessment areas.In simple terms, the researcher must determine whether the constructs "work" or "make sense" in each of the target cultures before attempting to design items to measure the construct.In formal terms, this is an assessment of the conceptual equivalence of the constructs (Berry, 1980;Pareek & Rao, 1980;Reddy, Knowles & Reddy, 1995).Conceptual equivalence can be assessed directly by conducting qualitative research with people from various cultures to "explore the similarities and differences in the understanding of the concepts underlying the instrument" (Kuyken, Orley, Hudelson & Sartorius, 1994:20).Alternatively, conceptual equivalence can be assessed indirectly by reviewing existing research and literature that addresses the manifestation of the constructs in the various cultures (Kuyken et al., 1994).

The Military Social Health Index
The need for the MSHI was determined by the Director Social Work of the South African National Defence Force (SANDF).The Directorate needed to screen the social health of thousands of soldiers on a routine basis, for which a standardised assessment instrument was required.A state-of-the-art review identified a number of existing instruments, notably the Heimler Scale of Social Functioning (Heimler, 1990).A trial implementation of the Heimler Scale, however, demonstrated that it was not suitable for the Directorate's purposes (Van Breda, 2002a, 2002b, 2006).Thus it was agreed that a new scale needed to be developed.The feasibility study was positive, based on the availability of full-time social work researchers in the SANDF, the availability of funding and infrastructure for a large national study and strong personal commitment to the project by the Director Social Work.
The innovation objective, which was extensively negotiated with the managers in the Directorate Social Work, was a scale "that measures the social health of soldiers earmarked for cross-border military operations" (Van Breda, 2004:162).Fourteen innovation requirements were identified and contracted, including the inclusion of four languages (English, isiZulu, Afrikaans and Setswana), the location of the instrument within resilience theory, and the sensitivity of the instrument to diverse family structures (Van Breda, 2004).
A detailed review of the various theories of resilience was conducted (Van Breda, 2001).This led to the identification of four operational assessment areas, viz.Social Support, Problem Solving, Stressor Appraisal and Generalised Resistance Resources.The theories and research concerning each of these constructs were content analysed to extract key themes (Van Breda, 2004).These themes, in combination, provided a composite description of what it meant to have, for example, social support.
These themes were further decomposed to form "facet maps", one for each construct.The facet maps, like a mind map, depicted the construct in the centre of map (e.g.Social Support) and the various facets of what it means to have social support radiating out from the centre (e.g.Feel important to others; Regular contact with others; Others will help in an emergency).The facet map for Social Support, which was defined as "the ability of the family system to access quality and sufficient support systems in times of need" (Van Breda, 2004:230), is appended to this paper by way of illustration.
The cross-cultural comparability of the seven assessment areas was explored by a team of eight military social workers -two isiZulu, two Setswana, one White English, one Asian English, one White Afrikaans and one Coloured Afrikaans.All members of the team had experience of the resilience model that formed the theoretical foundation of the MSHI.
A work session was held with the research team, during which the construct definitions were discussed and the initial facet maps reviewed.Through this process a number of adjustments were made to the formulation of the construct definitions and facet maps.For instance, the original definition of Generalised Resistance Resources was "The presence of a variety of creative and dynamic resources that enable families to resist the stress of deployments".This was subsequently reformulated as "The presence of a variety of creative and dynamic resources in family members and the family system, that enable families to resist life stress, which contributes to increased resilience and social health."The adjustments served to incorporate the cultural perspectives of the research team and ensured clear and unambiguous statements.The team reached consensus that all four constructs were meaningful and measurable in the respective cultures.
By the end of the analysis phase there was a clear rationale for the MSHI, a specific agreement about the requirements for the instrument, an extensive theoretical foundation for the instrument, four operationally-defined constructs, agreement that the constructs were culturally comparable and a set of facet maps for each instrument.

DESIGN PHASE Introduction
The second phase, design, involves designing the actual instrument, including formulating the items, translating them (if applicable), scaling the items and typesetting the instrument.Of great importance, the individual items should be designed by a multicultural team of researchers to ensure their acceptability in a multicultural context.

The procedure
The steps involved in the Design Phase are, on the whole, straightforward, and are well described in most research textbooks.Thus attention will be given here only to those aspects of the design phase that are specifically multicultural.
The researcher must determine what kind of response scale the instrument will have, for example, a relative frequency scale (always, often, half the time, sometimes, never) or a traditional Likert scale (strongly agree, agree, uncertain, disagree, strongly disagree).A key issue to consider here is the number of partitions or response options.Although many scales have seven or more partitions, this is often too many for people who have limited literacy or education.It is also better not to have fewer than five partitions, as this will reduce the variance of the instrument, which would then require more items to ensure reliability and validity (Frary, 1996).A five-point scale is a good rule of thumb in multicultural work (Faul, 1997;Skevington & Tucker, 1999;Szabo, Orley & Saxena, 1997).
In the formulation or design of items, the multi-focus approach is strongly recommended (Van Breda, 2004).This approach, derived from the dual-focus approach (Erkut, Alarcon, Coll, Tropp & Garcia, 1999), entails convening a team of researchers who represent the various culture groups that will later complete the instrument.A member of the team proposes an item in a language that all members hold in common, most typically English.The team members immediately ask, "How would we say that in Pedi?" or "How could I phrase that in Afrikaans?"Members would translate the proposed item into their own language and then back translate it into English.If the item does not translate directly and easily, it is reworded or discarded.Only when an item can be phrased in all the languages in the same way, using simple and direct terms, is it accepted as a multiculturally equivalent item.
The multi-focus approach seems to imply that the instrument must be translated into multiple languages, and while this may be desirable, it is not inevitable.The core notion here is that if a sentence can be phrased equivalently in various languages then the concept underlying the sentence is equivalent.In this way, "it is a concept-driven rather than a translation-driven approach to attaining conceptual and linguistic equivalence" (Erkut et al., 1999:207).Thus, the multi-focus approach to designing instruments should be used even for instruments that will be produced in only one language.Indeed, any instrument that will be used in a multicultural context should be designed using this approach.

The Military Social Health Index
The multicultural team of social workers from the Analysis Phase generated 99 items over a period of two days.The "list method" (Faul, 1995) was used, in which one item was generated for each facet.For example, the facet "Feel important to others" resulted in the item "We know that our family is important to others".This process facilitated thorough sampling of the construct domains, and thus ensured content validity (Nunnally & Bernstein, 1994).
Social Work/Maatskaplike Werk 2008:44(1) Items were designed using the multi-focus approach.English was used as the main language of communication, since it was the only language shared by all eight researchers.In the process of proposing, translating, back translating, adjusting and discarding items, a number of insights were gained.
• Metaphors had to be avoided.For example, the English item "I work long hours" was proposed.It appeared to be a perfectly acceptable, short, easy English statement.One of the isiZulu researchers, however, stated that an hour is only 60 minutes -it cannot be longer or shorter than that.The item was prepared in isiZulu ("Ngisebenza amahora amaningi kakhulu") and back translated as "I work too many hours".This new item has an 'African feel' (the use of the phrase "too many"), but was accepted as clear and understandable.• There are no African words for depression, anxiety, worry, etc.These emotions cannot stand alone, as in English and Afrikaans -they must be linked to a specific situation.The focus is thus on the situation causing the worry, rather than on the worry itself.The facets "depression" and "anxiety" were thus rendered in English as "Someone in my family is depressed (feels down)".In the African languages the sentence was rendered simply as "Someone in my family feels down" ("Emndenini wami kukhona ilunga elikhathazekile emoyeni" in isiZulu and "Mongwe mo lapeng mowa wa gagwe o kwa tlase" in Setswana).• There was no way to say "Someone in my family has housing problems" in the African languages.The concept 'housing problems' had to be very specific (e.g.house leaks, no housing, house too small, etc.).The item was eventually phrased as "Someone in my family has no place to stay." • The facet "helplessness" could not be translated into the African languages -there was no word for "helpless".It was eventually agreed to drop the word 'helpless' and rather state, "Someone in my family feels unable to cope with his/her life situation" ("Emndenini wami kukhona ilunga elingaphathekile kahle ngesimo sempilo yalo elingazi lenzenjani" in isiZulu and "Mongwe mo lapeng o bona fa a ka se kgonane le maemo a botshelo jwa gagwe" in Setswana").Although the meaning of this item, or its content validity to the concept "helpless", has been somewhat reduced, the research team agreed that the item was at least translatable and thus equivalent.This led to a general rule being formulated by the research team: rather compromise slightly on meaning in one language to achieve conceptual equivalence between languages than retain the depth of meaning in one language at the expense of equivalence.• On several occasions, the process of back translation produced better items in the original version.For instance, the item "Someone in my family cannot pay their accounts" was translated into Afrikaans as "Iemand in my familie kan nie hul skuld betaal nie" and then back translated as "Someone in my family cannot pay his/her debts".The team believed this revised English item to be better than the original.• In the African cultures the word "health" refers to the total health of a person (body, mind and spirit combined).In Western cultures the word "health" refers primarily to the physical health of a person, indicating the tripartite view of humanity.This fundamental paradigm difference between cultures resulted in the items being lengthier, so as to ensure conceptual equivalence: "Someone in my family has health problems (physical, emotional or spiritual)".This inclusion of the three components of health in brackets was incorporated across language versions to ensure linguistic equivalence, even though it is clearly implied by the word "health" in isiZulu and Setswana ("Emndenini wami kukhona ogulayo (emoyeni, ngenqondo, emzimbeni") in isiZulu).
• The expression "bounce back after a difficult time", which is an English metaphor that taps into resilience, could only be translated directly and meaningfully into isiZulu ("Umndeni wami uyakwazi ukuvuka uzithathe emva kokuhlangabezana nobunzima").In Setswana the concept was translated as "carry on after a difficult time" ("Lapa lame le kgona go tswelapele ka botshelo morago ga nako e thata") and in Afrikaans as "recover quickly after a difficult time" ("My familie het die vermoë om vinnig te herstel na moeilike tye").The team agreed that although a different phrasing was used across language versions, the items tapped into the same construct.• Several facets were abandoned when it was found to be extremely difficult to formulate items in all four languages.For instance, the facet "psychological problems" meant different things in the different cultures and was eventually discarded.Similarly, the facet "abuse of prescription medication" introduced such complex issues around traditional medicines that it was abandoned.The final Social Support subscale of the MSHI is appended to this paper by way of illustration.

Introduction
The third phase, development, involves testing the instrument in the field so as to ensure that the instrument measures what it is supposed to measure, to ensure that the items are well formulated and to ensure that the translations of the instrument are equivalent.

The procedure
A researcher should never design an instrument in isolation.It is always helpful to request experts in the field to review the instrument.There are two groups of experts that the researcher may wish to approach.Firstly, it is wise to approach subject experts -people with expertise around the constructs that the instrument measures (De Vellis, 1991;Haynes, Richard & Kubany, 1995;Messick, 1989).These experts will be able to advise on the content validity of the instrument.This can entail their reviewing the extent to which each item is relevant to the construct being measured and the extent to which the items as a group cover all aspects of the construct.Secondly, it is wise to approach linguistic experts -people who have expertise in the languages that are included in the instrument.These experts will be able to advise on the grammar and formulation of the items.
Field-testing the instrument involves having it reviewed by a group of people who are representative of the people for whom the instrument was developed.Small (5-10 people), culturally homogenous groups should be convened by a researcher from that culture group.The items in the instrument should be presented one at a time to the group and discussed.The researcher should ask, "What do you understand by this item?"If the group understands the item differently from what was intended, the researcher can involve the group in reformulating the item (Butcher, 1996).

The Military Social Health Index
The research team that designed the MSHI was utilised as the pool of expert reviewers.The team spent two days reviewing the items to assess their content validity.This process was closely tied with the design phase.Items that the team considered to lack content validity were immediately adjusted.The revised items were then formally reviewed.Consequently, the team's review of items was extremely positive.In retrospect, it would have been good to obtain subsequently established that the majority of Afrikaans-speaking participants (77,3% of White and 57,8% of Coloured participants) read primarily the Afrikaans version of the MSHI, while most African participants (57,1% of Setswana-speakers and 82,8% of isiXhosa-speakers) read primarily the English version.

Reliability
Reliability refers to the degree to which an instrument produces similar results when administered on separate occasions (Anastasi, 1982).Reliability, therefore, concerns the consistency of an instrument, both within itself (internal consistency) and over time (temporal stability) (De Vellis, 1991).Coefficient Alpha is a measure of the internal consistency of a scale, that is the degree to which all the items in a scale hold or work together and therefore measure a construct consistently.Coefficient Alpha should be above .90 in the case of narrow constructs (Cronbach, 1990;De Vellis, 1991;Nunnally & Bernstein, 1994), but above .80 in the case of broader constructs (Kline, 1986).The reliability coefficients for the four MSHI scales across the four culture groups are presented in Table 1.It can here be seen that all alpha coefficients exceed the .90standard, with a mean of .945.They range from a low of .917(Social Support for African isiXhosa-speaking participants) to a high of .970(Problem Solving for White Afrikaans-speaking participants).The equivalence of the alpha coefficients was investigated using a procedure formulated by Van De Vijver and Leung (1997).Two thirds of the coefficient pairs were equivalent, with the main nonequivalence found in the Problem Solving scale.
Social Work/Maatskaplike Werk 2008:44(1) These results, which are impressive in comparison with many other measuring instruments used in South Africa, show that the MSHI has very strong reliability or internal consistency across four culture groups.This suggests that the scale development procedures used in the previous three phases are effective in producing instruments that have the potential to be multiculturally reliable.

Construct and factorial validity
Construct validity at item level refers to the degree to which each item in a scale measures the construct it is supposed to measure, and not some other construct.Multiple Groups Confirmatory Factor Analysis (Hudson, 1982;Nunnally & Bernstein, 1994) was used to examine the construct validity of the MSHI.For each culture group separately, each item was correlated with its own corrected scale total (having removed the item from the total to prevent the confounding effects of item-self correlations) and with the total scores for the three other scales.
Two criteria were used to determine whether an item evidenced construct validity (Faul, 1995;Hudson, 1982).Firstly (Criterion 1), each item was required to correlate with its own corrected scale total more highly than with any of the other scales.This would indicate that the item was measuring the construct it was supposed to measure and not one of the other constructs.Secondly (Criterion 2), each item was required to correlate with its own corrected scale total at .40 or higher.This would indicate that the item was measuring its own construct "strongly" or "well".
The analyses indicated that, with the exception of the African isiXhosa responses to one item in the Social Support scale, all other items in the MSHI met the first criterion, and that all items met the second criterion.The instrument thus demonstrates strong factorial validity within and across cultures.
The item-total correlations were then averaged for each of the scales for each culture group.It was expected that the average corrected item-total correlations for item-own-scale correlations should be markedly higher than the average item-total correlations for item-other-scale correlations.This would indicate that, on average, the items in a scale were measuring their own construct more strongly than other constructs.Table 2 presents the results of these analyses.Here it can be seen that the own item-total correlations are higher than the other item-total correlations in all cases.This confirms the factorial validity of the MSHI across culture groups.
The corrected item-total correlations (shown in bold italics) serve as coefficients of construct validity (Faul, 1995).The minimum standard for validity coefficients is set at .60 (Faul, 1995;Hudson, 1982;Nurius & Hudson, 1993).Table 2 indicates that all sixteen validity coefficients exceed this standard -indeed, the mean construct validity coefficient is .73,well above the standard of .60.This provides evidence for strong construct validity of the MSHI at item level for all culture groups.
Once again, these results suggest that the multicultural instrument development procedures used to design the MSHI have been effective in producing an instrument that has factorial and construct validity across four cultural groups.This is particularly noteworthy given that one of the groups (isiXhosa-speakers) was completing the instrument in a second language.

Practice recommendations and conclusions
This paper has endeavoured to present a series of steps that can be followed when developing a measuring instrument that will be used in a multicultural context.These steps have been described and then illustrated with the experience of developing the Military Social Health Index.Finally, selected validation data have been mobilised to provide evidence that these Social Work/Maatskaplike Werk 2008:44(1) steps can indeed be effective in producing an instrument that is both reliable and valid across culture groups.
It is likely that there are a handful of crucial steps in the process of developing a new instrument that contribute most significantly to the multicultural validity of the instrument.These steps should be followed in the design of all instruments that will be used in multicultural contexts.
• A clear rationale for (steps 1 and 2) and theory underpinning (step 3) the new instrument are required.While these steps are not unique to multicultural instruments, they provide the firm foundation that is essential in good quality instruments.This foundation is especially critical for multicultural instruments because of the vulnerability of such instruments to poor measurement properties.• The constructs that will be measured should be critically evaluated for cultural comparability (step 5).This will ensure that, at a broad conceptual level, the instrument is measuring constructs that "make sense" across the target cultures.• The items should be designed using the multi-focus approach (step 8), regardless of whether the instrument will be translated into different languages.The author has followed this approach in the development of two subsequent instruments, one of which was not translated, and has found the process to be effective at identifying English sentences that do not make sense in different languages.The principle here is that if an English sentence cannot be translated into isiXhosa, for instance, the English sentence will probably not make sense to a isiXhosa-speaker, which would reduce the reliability and validity of the instrument for isiXhosa speakers.Ensuring that all the items can easily be translated into several other languages contributes significantly to the linguistic equivalence of the instrument.
• The items should be field tested with culturally homogenous groups of respondents (step 13), as this will elicit any potential misunderstandings that the research team may have missed.It is the author's contention that building these steps into the development of all new instruments in South Africa will produce instruments that are much more likely to be reliable and valid across the various culture groups where they are used.This will make a significant contribution to the eradication of measuring instruments that are biased against or in favour of particular culture groups.This in turn will contribute to fairness and respect for cultural diversity within social work practice in South Africa.