Aggregation, Bias and Confusion - Distortions in Policy

All analyses involve simplification, frequently through aggregation. As has been seen in debate on "Closing the Gaps", aggregation that groups by ethnicity can be challenged. This paper briefly considers some of the theoretical issues surrounding aggregation, asking under what conditions aggregations might be acceptable. It then looks at possible distortions that can arise from inappropriate aggregation, including both analytical and political perspectives. Some safeguards will also be suggested.

While the criteria are of general relevance, points will be illustrated with examples arising from gendered approaches.

In this section I consider the underlying factors that might result in distortions in policymaking and policy analysis. In the next section, I shall use criteria identified here to illustrate how the current policy environment is distorted.

Policymaking involves trying to do as well as possible in relation to chosen objectives subject to limitations on what can be achieved. In other words, it is a matter of constrained optimisation. Such optimisations need not always involve overall optimisation. It is sometimes possible to break down a problem into several components. The extent to which this can be done without distorting the results depends on the extent to which the components permit independent analyses. If there are interrelationships across components, they cannot be correctly assessed in isolation.

1) Complete independence – in which decisions can be made with no regard for other factors.

2) General linkages – in which optimal decisions are a function of disaggregated measures of circumstances elsewhere.

There is a range of other, intermediate possibilities. For example, there could be aggregate linkages only, in which decisions can be made area by area, while only having regard for some aggregate measure of circumstances elsewhere. This would permit a multi-stage optimisation, optimising over broad areas (as in budget allocations to several sectors, such as health and education), and then optimising within each of those areas.

For (1), optimal solutions can be identified, area by area, as a function of internal variables only. This is a special case. Distortions could arise if it is falsely assumed that the required conditions are met.

With (2), optimal solutions would generally be a function of disaggregated variables across areas.

Intermediate options can permit some independent analysis, but still require consideration of outside linkages at some stage in the policymaking process.

It is common, both on institutional and political grounds, to undertake policy analysis in a segmented way even if this may be problematic. For example, the State Services Commission, looking at the quality of policy advice, identified problems with “insufficient incentives for active co-operation by departmental chief executives on cross-cutting policy issues” and “the counter-productive and debilitating consequences of departmental patch-protection” (State Services Commission, 1999, p.5).

Similarly, microeconomic theory relies on ceteris paribus assumptions with respect to numerous omitted variables.

Policy analysis generally involves simplification through aggregation. Rather than consider each individual unit (individual, household, business), they are commonly grouped. This reduces the number of variables, which makes analysis easier, but can also be misleading unless the aggregation involves grouping units with common characteristics within each group [1], and distinct characteristics between groups. Macroeconomics is an extreme example of the use of aggregation.

One illustration of misleading aggregation is NZDep, a small area index of deprivation. The index is calculated from the proportions of people in each area having specified characteristics. As described in Salmond and Crampton (2002), NZDep96 is very poor as a measure of individual deprivation due to the wide variety of individual deprivation within each area. Hence, “if the three most deprived deciles of NZDep96 were used for targeting, comprising one quarter of the study population, these areas would miss 13.9% - over half of the study population who are individually most multiply deprived” (pp.669-70).

As another example, Wallis (2003) described a blind person in the US who was denied insurance because blind people had a shorter life expectancy. While diseases such as diabetes can result in blindness and shorter life, this person was blind from birth and could expect a normal length of life.

Looked at another way, it could be possible to incorrectly specify a group as deprived simply by including it in an aggregation with a genuinely deprived group, or to conceal a group’s deprivation through aggregation with a better endowed group.

The nature of the analysis shapes the perspectives taken, and hence can determine the policy issues that are identified, the variables that are monitored, and perceptions of outcomes.

For example, we could consider people on an individual or a household basis. Each would give a different picture. Alternatively, we could attempt to see people on a family basis, where family is defined to include inter-household links, rather than the current definition, where the family is restricted to one household only.

Similarly, we could consider all working women as one group, or distinguish between career women and those whose primary interest is family [2], or working women under and over 30, for example.

An aggregated view can suggest homogeneity within a group that is actually composed of distinct sub-groups. In other words, it can fail to acknowledge discrete differences. Differences can also be continuous, and so breaking down an aggregate may still not result in homogeneity with the resulting sub-groups. As a general point, therefore, in addition to consideration of group averages, variation within groups should also be considered.

Political factors can influence the choice of approach. To quote Tapp, Geddis and Taylor (in relation to child maltreatment and domestic violence):

The present policies in this area in New Zealand are not the end result of logical, coherent evolution, but rather reflect an ad hoc compromise in the face of pressures applied from vested interests.

Lobby groups want their specific focus to be highlighted. This will inevitably impact on the choice and definition of variables, the relationships to be emphasized between variables, and the solutions to be promoted. It is to be hoped, but by no means expected, that public sector analyses will compensate for distortions arising from lobbying, and that lobby groups will be sufficiently diverse, so as to cover a range of opinions and prevent an overly-unbalanced understanding.

Incorrect breakdowns or groupings can distort analyses and can be reflected in or arise from political discourse. The next section considers these points in relation to groupings by gender.

How suitable is gender as a category? Can we divide the population on gender lines to consider each group separately? A fundamental requirement is that functional relationships can be separated into those involving men and those involving women, with little or no interaction between them. Ministry of Women’s Affairs (2002a) specified the areas of: income, work and family; education; safety and justice; housing; disability; positive aging; and health. Just to take income, work and family as an example, given the extent to which men and women co-habit, share income, and share responsibilities for their children, is it appropriate to consider these issues for women in isolation? Are the underlying relationships for women between policy variables and outcomes independent of the choices made by men, and, given the relevance of these issues to men also, can policies for men be formulated without regard to the aspirations and behaviour of women? If we cannot separate them, what are we trying to optimize when we consider one in isolation?

“…feminist scholars have increasingly focused on the need to acknowledge differences [between women] associated with ethnicity, class, religion, colonial conquest, sexualities, and disabilities” (du Plessis, 1997, p.221)

Homogeneity within an aggregate is only required in relation to relevant functional relationships. These will vary according to the issues being considered. Similarly, for some issues there may be few differences between common aggregates (such as men and women), in which case there would be little analytical gain from distinguishing between them. As du Plessis says, feminists’ internal political debates have stressed certain dimensions of heterogeneity. Many of these could be of similar significance to men.

In addition to the above list, an analyst might also want to consider such variables as age, education level, marital/partnership status, parental status, motivation for work (to supplement household income, for job satisfaction, for social contact, career ambitions, etc.).

Despite current requirements to undertake gender analysis in the New Zealand public sector, a publication on child maltreatment (Ministry of Social Policy, 2000) has expressly warned of the dangers of crude groupings and simple comparisons:

"There have been some attempts to establish links between child maltreatment and the ascribed characteristics of perpetrators, in particular their age, sex, and ethnicity. In general, however, it has been recognised that any patterns related to these are most likely to be driven by the prevailing conditions that perpetrators face in relation to family, neighbourhood, and culture." (p.13)

"A large body of evidence suggests that the incidence of reported child maltreatment is over-represented among single parent families and blended families. Much of that research, however, is based on relatively simple comparisons of the proportion of such families in the general population compared to their representation among families reported in maltreatment statistics. That type of analysis must be treated with considerable caution." (pp.20-21)

“Relatively simple comparisons” are common, so this advice might be more generally applicable. Hopefully it is purely coincidental that, in the above two cases, a gendered analysis would present women in an unfavourable light.

The choice of gender as a variable for analysis is not neutral in its effect. Just as using NZDep can result in policies targeting geographic areas rather than deprived individuals, the use of gender as the central defining characteristic may mislead. There are numerous other possible variables that could be chosen as a basis for grouping observations, and some factors may actually be hidden due to the use of gender.

One clear distortion for many issues arises from gender being defined for individuals (naturally), although many aspects of people’s circumstances are determined by their participation in groups. One of the most fundamental of these is the family (however defined), and most families include both sexes. Hence an individual’s personal income or wealth may be only loosely related to the person’s economic circumstances, preferences may exist and decisions may be made with others in mind, and well-being may also depend on relationships with other people. Nevertheless, the current government takes a specifically individualistic approach. The Ministry of Women’s Affairs (2002a) specifies economic autonomy for women as a requirement for “New Zealand’s success, prosperity and wellbeing” (p.5), and this is included as part of the 2002 Labour-headed government’s agenda.

Just as macroeconomic issues such as inflation and growth can only be observed through taking a macroeconomic approach, family-based issues can only be observed through a family, rather than an individual, approach. While individual issues may also be worthy of analysis, it may not be possible to undertake a realistic analysis if family connections are ignored or misrepresented.

Emphasis on gender is likely to result in identification of and relative emphasis on gendered, individual perspectives on issues. Hence we see, to list just a few: “gender wage gap”; “glass ceiling”; battered women’s syndrome; the offence of “male assaults female”; possible downplaying of men’s perspectives in such issues as parental leave, “work-life balance”, “caregiving”, “sole parent families”; dominance of the patriarchal power and control approach to domestic violence; downplaying of men’s lower life expectancy, higher suicide rates, lower representation among current tertiary students, greater workplace injury and fatality rates; and redistributions from men to women through the tax and benefit systems and state provided goods and services. Similarly, it could be wondered whether feminist writers would have been as accepting of the current household-base definition of family, were fathers to have had first claim on custody of children.

Gendered perspectives can also affect the specific questions that are asked such that they influence the results. For example, the first two sub-questions in Question 36 of Gendall (2002) ask how much respondents agreed or disagreed that, (a) both parents are equally important to children, and (b) mothers are always more important to children than fathers. Although these questions were next to each other in the written questionnaire, from (a) we find that about 96% of men and women agree that both parents are equally important, but from (b) nearly 14% agree that mothers are more important than fathers. Full details are in the Appendix. It should not be assumed, therefore, that gendered studies necessarily give an accurate picture of people’s opinions.

Evaluations of impacts, costs and benefits require a specification of the perspective to be taken. Are effects to be measured in relation to government, society as a whole, affected individuals, specific groups? Hence the impact of student debt could be evaluated with respect to society, the taxpayer, industry, the workforce, the skilled workforce, men, women, ethnic groups, students as a whole, students by course, students by age, students by future work plans, etc.. The choice of perspective affects our view of the issues. Current gendered approaches may not be the most useful, and we should be aware that many alternative, legitimate perspectives might be overlooked.

The above discussion indicates that distorted analyses and policies are possible. Political factors make them inevitable. Rosemary du Plessis writes:

"Claims about women's common interests have been used as a strategy to ensure that international organisations such as the United Nations attend to the political status of women. It has also been vital in developing women-focused structures in particular nation states (such as New Zealand's Ministry of Women's Affairs) and at the level of community organisations, trade unions, churches, workplaces, professional associations, and educational institutions." (du Plessis 1997, p.221)

This is particularly worrying as not only is it an incorrect grouping, but also it results in a focus on the interests of half of the population to the exclusion of the other half. This is clearly seen if we consider the issue of “disadvantage”. To show disadvantage, it is necessary first to identify indicators which give different numbers for the specified groups, and second to give an interpretation of those differences whereby the chosen group is worse off. Naturally, therefore, such conclusions can be challenged through questioning the suitability of the indicators and/or presenting alternative interpretations of the results.

For example, in New Zealand the Coalition for Equal Value for Equal Pay has a poster with the caption, “Prepare your daughter for working life. Give her less pocket money than your son.”[3] It is based on the statistic that women’s average total weekly earnings are lower than men’s. The indicator could be challenged due to being based on data across generations, and differing education, career choices, hours worked and reasons for working [4], for example. An alternative interpretation could be to suggest, “Prepare your son for working life. Make him earn his pocket money, then give some of it to his sister.”

We see one-sided research methods built in to the structure of the Ministry of Women’s Affairs’ version of gender analysis, as discussed in Birks (1999). Other parts of the public sector have undertaken gender-biased research, as in the Law Commission’s Women’s Access to Justice study, critiqued in Birks (1998), and the then Department of Justice’s “Hitting Home” study (Leibrich et al, 1995).

Research elsewhere may also be focused more on achieving political objectives than the pursuit of academic enquiry. According to the web site for the Women’s Studies Association 2003 conference (http://www.womenz.org.nz/wsa/conf.htm), the conference “addresses feminist goals”. The theme is “Celebrating All Women”, but it is not clear how they intend to celebrate non-feminist women. The call for papers includes a suggested area, “economic and social systems which disadvantage women”. They are apparently less welcoming of papers which consider advantages experienced by women.

Political self-interest (and the heterogeneity of women) might also be indicated in the following quote:

"...although equal employment opportunities initiatives have benefited some women, those most advantaged have tended to be the women who most resemble those who fill the equal employment opportunities (EEO) positions in organisations." (du Plessis, 1997, p.222)

and gendered groupings may result in pressure for quite different treatment according to gender:

"Increasingly, feminist theorists have considered how citizenship implies a male subject and inhibits attention to forms of difference between women and men that might legitimate the two sexes making different claims on the state." (du Plessis, 1997, p.224)

There is a clear danger that treatment would depend on the average characteristics of a chosen group, rather than the underlying individual circumstances. Moreover, separate policy development could result in inconsistent application of principles, such as human rights. The Human Rights Commission has a special section on women based on the International Convention on the Elimination of All Forms of Discrimination Against Women (CEDAW). Might this be the result of lobby groups, as described in 1.4 above, and the initiatives described by du Plessis at the start of this section?

Article 4 refers to: “…temporary special measures aimed at accelerating de facto equality between men and women shall not be considered discrimination as defined in the present Convention, but shall in no way entail as a consequence the maintenance of unequal or separate standards; these measures shall be discontinued when the objectives of equality of opportunity and treatment have been achieved.” Note that equality is specified in terms of opportunity and treatment, not, for example, equal average pay for men and women in the public sector. This latter is the objective of the recently established taskforce as part of the current government’s policy on pay and employment equity [5]. [6]

Article 5 calls on states: “to modify the social and cultural patterns of conduct of men and women, with a view to achieving the elimination of prejudices and customary and all other practices which are based on the idea of the inferiority or the superiority of either of the sexes or on stereotyped roles for men and women.” Presumably those who call for preferential treatment for women as “the carers” in society are contravening the spirit of this Article. A Google search for “carers” and “caregivers” commonly finds these terms linked to women. For example, Resolution 9 of 27^th Conference of the International Federation of University Women, calls for action on “access to legal services, especially for women caregivers”[7]. It should be noted that gendered approaches can accentuate and even create differences between genders.

Article 16 “shall ensure, on the basis of equality of men and women: …the same rights and responsibilities as parents, irrespective of their marital status, in matters relating to their children… the same rights to decide freely and responsibly on the number and spacing of their children and to have access to the information, education and means to enable them to exercise these rights… the same rights and responsibilities with regard to guardianship, wardship, trusteeship and adoption of children.” These aspects are seldom, if ever, mentioned.

We see the effects of gendered stereotyping in policymaking. The 1996 coalition agreement statement on child support talked only of “parents who desert their families”[8]. Margaret Wilson, on 29 February 2000 in the parliamentary debate on the Matrimonial Property Amendment Bill [9], focussed on women who “have been discarded for a newer and younger model”. Page 159 of Fleming and Atkinson (1999) talks of policy analysts with “an implicitly punitive attitude towards non-custodial fathers”.

As Callister (2000) discusses, there is now an accepted principle that women must play a key role in researching women, and Maori in researching Maori. By the same principle, men should play a key part in research on men. On page 43, Callister refers to a research project run by women in the Office of the Commissioner for Children. While finding that almost half the men and a third of women respondents in a random telephone survey thought that the Family Court discriminated against men, this was discounted as a "misperception". This is not the only all-woman research project involving male subjects. Given gender politics, can such researchers be expected to provide balanced assessments?

There are politically defined groupings which are used to define policy issues and approaches, while disregarding the differences within and linkages across groups. They distort our perceptions and result in inappropriate policies. Another way of putting it is that policy is determined by political pressure with little regard for analysis or consideration of possible outcomes.

Gender is not the only significant dimension in current discourse on policy, with ethnicity also being prominent. There is a difference, however. With ethnicity, it is possible, and possibly desirable, to have a degree of diversity and separate development. In comparison to ethnicity, divisions by gender impinge on the living and working arrangements of all or most of society. Gendered analysis does not convey a strong vision of men and women working together. In fact, it could be questioned whether feminist writing displays much empathy for men at all.

The message in this paper is not that the issues are necessarily unimportant, they may merit attention, but that we are looking at them in a way that is harmful. Separation by gender, rather than considering common interests and interdependencies, has the potential to split society down the middle. There is evidence of social engineering, gender politics, and lack of analysis, and signs that a social agenda is being pursued which has not been debated, even less approved.

Appendix

			Both parents equally important to kids					Total
			Strongly agree	Agree	Neither agree nor disagree	Disagree	Cant choose	Total
sex	Male	Count	302	144	10	5	2	463
	Male	% within sex	65.2%	31.1%	2.2%	1.1%	.4%	100.0%
	Female	Count	350	119	9	13	2	493
	Female	% within sex	71.0%	24.1%	1.8%	2.6%	.4%	100.0%
Total		Count	652	263	19	18	4	956
Total		%	68.2%	27.5%	2.0%	1.9%	.4%	100.0%

			Mothers always more important to kids than fathers						Total
			Strongly agree	Agree	Neither agree nor disagree	Disagree	Strongly disagree	Cant choose	Total
sex	Male	Count	7	60	68	236	92		463
	Male	% within sex	1.5%	13.0%	14.7%	51.0%	19.9%		100.0%
	Female	Count	9	53	86	245	87	8	488
	Female	% within sex	1.8%	10.9%	17.6%	50.2%	17.8%	1.6%	100.0%
Total		Count	16	113	154	481	179	8	951
Total		%	1.7%	11.9%	16.2%	50.6%	18.8%	.8%	100.0%

Birks S (1998) Gender Analysis and the Women's Access to Justice Project, Issues Paper No.2, Centre for Public Policy Evaluation, Massey University, Palmerston North

Birks S (1999) “Gender, policy and social engineering”, Nuance, 1, pp.22-42

Callister P (2000) “Researching fathers in New Zealand: Whose voices are we hearing”, Chapter 4 of Birks S (ed.) Inclusion or Exclusion: Family Strategy and Policy, Issues Paper No.9, Centre for Public Policy Evaluation, Massey University, Palmerston North

du Plessis R (1997) "Women, Feminism and the State", chapter 12 in Rudd C and Roper B, The Political Economy of New Zealand Auckland: OUP.

Fleming R and Atkinson T (1999), Families of a different kind: life in the households of couples who have children from previous marriages or marriage-like relationships: a social research report, Waikanai: Families of Remarriage Project

Gendall P (2002) The Roles of Men and Women in Society, International Social Survey Programme, Department of Marketing, Massey University, Palmerston North, New Zealand

Hakim C (2003) Competing family models, competing social policies, Paper presented to the annual conference of the Australian Institute for Family Studies, 12 March

Leibrich J, Paulin J and Ransom R (1995) Hitting home: men speak about abuse of women partners, Wellington: Department of Justice

Salmond C and Crampton P (2002) “Heterogeneity of deprivation within very small areas”, Journal of Epidemiology and Community Health, Vol.56, pp. 669-670

State Services Commission (1999), Essential Ingredients: Improving the Quality of Policy Advice, Occasional Paper 9, State Services Commission: Wellington

Tapp P, Geddis D and Taylor N (1992), “Protecting the Family”, in Henaghan M and Aitken B (eds.) (1992) Family Law Policy in New Zealand, Oxford: OUP

Wallis A (2003) “Blind led sighted on Sept 11”, Manawatu Standard, 12 May, p.1

[1] For functional relationships, it is enough that the composition of the aggregate stays the same as its magnitude changes. In this case, we could consider a “representative unit”, being a weighted average of the various components.

[2] Hakim (2003) suggests three types of women, “work-centred”, “home-centred” and “adaptive”, where the latter group prefer to combine employment and family work without giving a fixed priority to either (p.6).

[4] Results of a study (Rowntree 1999) for families with teenage children published by the Joseph Rowntree Foundation found men’s provider role to be central to fathering, with about 85% of men and 45% of women reporting that they worked out of financial necessity, whereas about 15% of men and 43% of women worked for ‘a sense of independence”. Also, “Women were also much more likely than men to see paid work as a choice and to link it to questions of personal freedom or self-development” (p.2). Hakim (2003) also discusses diverse motivations among women. Some of these motivations require the co-operation of a partner, and so suggest that separate analyses would be misleading.

[6] Hakim (2003) discusses UK policies in this area in the context of women’s diverse priorities: “Did the New Labour government of Tony Blair take any notice of the results of the Listening to Women research programme? Of course not. The studies revealed more diversity of values and complexity of opinion than was politically useful. So the findings were used selectively to support the government's predetermined policy positions - in particular policies promoting paid work as women's central life activity.” (pp.3-4)