Downloads & Free Reading Options - Results

A new method for the identification of differential item functioning (DIF) by using recursive partitioning techniques is proposed. We assume an extension of the Rasch model that allows for DIF being induced by an arbitrary number of covariates for each item. Recursive partitioning on the item level results in one tree for each item and leads to simultaneous selection of items and variables that induce DIF. For each item it is possible to detect groups of subjects with different item difficulties, defined by combinations of characteristics that are not pre-specified. An algorithm is proposed that is based on permutation tests. Various simulation studies, including the comparison with traditional approaches to identify items with DIF, show the applicability and the competitive performance of the method. Two applications illustrate the usefulness and the advantages of the new method.

Three statistical testing procedures well-known in the maximum likelihood approach are the Wald, likelihood ratio (LR), and score tests. Although well-known, the application of these three testing procedures in the logistic regression method to investigate differential item function (DIF) has not been rigorously made yet. Employing a variety of simulation conditions, this research (a) assessed the three tests' performance for DIF detection and (b) compared DIF detection in different DIF testing modes (targeted vs. general DIF testing). Simulation results showed small differences between the three tests and different testing modes. However, targeted DIF testing consistently performed better than general DIF testing; the three tests differed more in performance in general DIF testing and nonuniform DIF conditions than in targeted DIF testing and uniform DIF conditions; and the LR and score tests consistently performed better than the Wald test.

Whether judges on bias review committees can identify test items that function differently for black and white examinees was studied. Judges (n=42) on three bias review committees were asked to examine a set of items and predict differential item functioning (DIF) without empirical data. Test items from teacher certification tests in the content fields of early childhood (n=11), administration and supervision (n=15), and middle childhood (n=16) were examined. Each committee examined 40 items. Agreement between judgmental and empirical indices of DIF were determined. The results suggest that the agreement between the bias review judges and the empirical indices are generally not beyond what would be expected by chance, although each field had one to two judges who exhibited statistically significant agreement with the empirical indices of DIF. The data also indicate that the judges were unlikely to classify items as "favoring blacks." Suggestions for future research on the identification of biased items and the practical implications of this study are discussed. Five tables present the data. (Author/SLD)

Two techniques for detecting differential item functioning (DIF) of test items are compared: (1) item response theory (IRT), using the three-parameter model; and (2) Mantel-Haenszel chi square techniques (MHCS). The steps necessary for identifying differentially functioning items are identified for both approaches. Using data from the 1988 Maryland Test of Citizenship Skills (MTCS), the following parameters were investigated: (1) the stability of the MHCS statistic across sample sizes; (2) the stability of the DIF index (MH Alpha) in the MHCS approach across several score groups; (3) the correlation between IRT DIF indices and MH Alpha; and (4) agreement between the IRT and MHCS techniques in identifying biased items. The MTCS was administered to about 50,000 ninth-graders in January and February of 1988. For IRT purposes, random comparison groups of 1,000 students each were created, while the MHCS procedure used samples of 1,000, 750, 500, and 200 examinees. The IRT DIF procedure identified four items in male/female comparisons and three items in white/black comparisons in the MTCS that have significant unequal probabilities of a correct response. While correlation results indicate that the two techniques disagree regarding item bias, agreement in terms of total hits indicates that the MHCS technique is an adequate substitute for the three-parameter IRT approach if the sample size is at least 750. In terms of the percentage of items identified as biased, the techniques appear to be equally good in detecting DIF. Five graphs and 11 tables present study data. A 35-item list of references is provided. (SLD)

In this project, we will assess Differential Item Functioning (DIF) with respect to four demographically different age (children, teenagers and adults) and cultural groups (Brazilians, Dutch, Nigerians and Latin Americans) on Raven’s standard Progressive Matrices (SPM) to determine whether these groups can be merged for further analysis that seeks to estimate latent ability given unweighted sum scores. In the preregistration, we refer to certain scripts, equations or appendences. These can be found under files of this project in the folder 'DIF analysis preregistration'. For the most up to date documents (e.g., scripts, results), we refer you to our Github page incl. the README files: https://gitlab.uvt.nl/tsb-rst/tests_computed_ability.git. The review, equations and appendences can be found under 'Preregistration_DIF_analysis_SPM.pdf'.

Within the educational context, students' assessment tests are routinely validated through Item Response Theory (IRT) models which assume unidimensionality and absence of Differential Item Functioning (DIF). In this paper, we investigate if such assumptions hold for two national tests administered in Italy to middle school students in June 2009: the Italian Test and the Mathematics Test. To this aim, we rely on an extended class of multidimensional latent class IRT models characterised by: (i) a two-parameter logistic parameterisation for the conditional probability of a correct response, (ii) latent traits represented through a random vector with a discrete distribution, and (iii) the inclusion of (uniform) DIF to account for students' gender and geographical area. A classification of the items into unidimensional groups is also proposed and represented by a dendrogram, which is obtained from a hierarchical clustering algorithm. The results provide evidence for DIF effects for both Tests. Besides, the assumption of unidimensionality is strongly rejected for the Italian Test, whereas it is reasonable for the Mathematics Test.

This article is from BMC Medical Research Methodology , volume 11 . Abstract Background: For the Low Vision Quality Of Life questionnaire (LVQOL) it is unknown whether the psychometric properties are satisfactory when an item response theory (IRT) perspective is considered. This study evaluates some essential psychometric properties of the LVQOL questionnaire in an IRT model, and investigates differential item functioning (DIF). Methods: Cross-sectional data were used from an observational study among visually-impaired patients (n = 296). Calibration was performed for every dimension of the LVQOL in the graded response model. Item goodness-of-fit was assessed with the S-X2-test. DIF was assessed on relevant background variables (i.e. age, gender, visual acuity, eye condition, rehabilitation type and administration type) with likelihood-ratio tests for DIF. The magnitude of DIF was interpreted by assessing the largest difference in expected scores between subgroups. Measurement precision was assessed by presenting test information curves; reliability with the index of subject separation. Results: All items of the LVQOL dimensions fitted the model. There was significant DIF on several items. For two items the maximum difference between expected scores exceeded one point, and DIF was found on multiple relevant background variables. Item 1 'Vision in general' from the "Adjustment" dimension and item 24 'Using tools' from the "Reading and fine work" dimension were removed. Test information was highest for the "Reading and fine work" dimension. Indices for subject separation ranged from 0.83 to 0.94. Conclusions: The items of the LVQOL showed satisfactory item fit to the graded response model; however, two items were removed because of DIF. The adapted LVQOL with 21 items is DIF-free and therefore seems highly appropriate for use in heterogeneous populations of visually impaired patients.

Academic test scores are used to determine the effectiveness of education interventions across numerous randomized controlled trials (RCTs). Test scores are compared across treatment and control groups to determine whether an intervention has had a positive impact on academic outcomes. These aggregate scores are typically representative of broad academic abilities in math or reading but can also be focused on more narrowly defined constructs. However, there is limited analysis of the item-level test score data. We propose to examine such data to ask whether observed treatment effects are due to generalized gains or gains that largely manifest with respect to specific items. To do this, we will use techniques for identification of Differential Item Functioning (DIF). DIF analysis is typically used to examine whether certain items used on educational or psychological measures show bias as a function of group membership; such analyses do so by asking whether item response behavior varies across groups once we have controlled for underlying ability. Here, we anticipate findings of DIF if the effects of a RCT intervention on academic outcomes are specific to certain items. We would not interpret such findings as bias but as potential evidence for a localized treatment effect. For example, if an intervention leads to increased math test scores, DIF analysis will demonstrate if the positive effect is uniform across all of the items or if the gains are in items for specific skill areas (e.g., the intervention improved performance on items focusing on fractions). This will inform policymakers and researchers what skills are improved from the intervention, will allow insight into the appropriateness of certain measures in certain RCT settings, and will suggest skills that may need to be targeted in future interventions.

Academic test scores are used to determine the effectiveness of education interventions across numerous randomized controlled trials (RCTs). Test scores are compared across treatment and control groups to determine whether an intervention has had a positive impact on academic outcomes. These aggregate scores are typically representative of broad academic abilities in math or reading but can also be focused on more narrowly defined constructs. However, there is limited analysis of the item-level test score data. We propose to examine such data to ask whether observed treatment effects are due to generalized gains or gains that largely manifest with respect to specific items. To do this, we will use techniques for identification of Differential Item Functioning (DIF). DIF analysis is typically used to examine whether certain items used on educational or psychological measures show bias as a function of group membership; such analyses do so by asking whether item response behavior varies across groups once we have controlled for underlying ability. Here, we anticipate findings of DIF if the effects of a RCT intervention on academic outcomes are specific to certain items. We would not interpret such findings as bias but as potential evidence for a localized treatment effect. For example, if an intervention leads to increased math test scores, DIF analysis will demonstrate if the positive effect is uniform across all of the items or if the gains are in items for specific skill areas (e.g., the intervention improved performance on items focusing on fractions). This will inform policymakers and researchers what skills are improved from the intervention, will allow insight into the appropriateness of certain measures in certain RCT settings, and will suggest skills that may need to be targeted in future interventions.

Differential item functioning for minority examinees on the SAT Educational Testing Service, May 1988 Screening for "differential item functioning" insures that no test question is more biased than the test as a whole.

As part of a project to update the Military Equal Opportunity Climate Survey (MEOCS), items from eight of its scales (Sexual Harassment and Discrimination, Differential Command Behavior toward Minorities and Women, Positive Equal Opportunity (EO Behavior, Racist/Sexist Behavior, Reverse Discrimination (Behavior), Discrimination against Minorities and Women, Reverse Discrimination (Attitude), and Attitudes toward Racial/Gender Separatism) have been rewritten to make them more neutral (e.g., replacing terms such as 'men' and 'women' with 'gender'). A comparison was made of the psychometric characteristics of the original and revised versions of these items using differential item functioning (DIF) from item response theory (IRT). DIF was found for the majority of the 40 items examined, although in many cases the DIF indicated improvements in the revised items. Implications for these scales and for the use of IRT with the MEOCS are discussed.

This study assessed unidimensionality and occurrence of Differential Item Functioning (DIF) in Mathematics and English Language items of Osun State Qualifying Examination. The study made use of secondary data. The results showed that OSQ Mathematics (-0.094 = r = 0.236) and English Language items (-0.095 = r = 0.228) were unidimensional. Also, there was occurrence of DIF items in both Mathematics and English Language multiple-choice items of the OSQE for 2008. Fourteen items representing 28% of the 50 items in the Mathematics examination exhibited DIF and 10 items, representing 20% of the 50 items in the English Language examination exhibited DIF. The study concluded that the Examination contained considerable number of items that exhibited DIF and therefore requires adequate item quality improvement to justify its use as the inclusion or exclusion criterion of state candidate in West African Examination Council.

The purpose of this study was to identify differentially functioning items on operational administrations of the Graduate Management Admission Test (GMAT) through the use of the Mantel-Haenszel statistic. Retrospective analyses of data collected over 3 years are reported for black/white and female/male comparisons for the Verbal and Quantitative Tests. In general, one to six percent of the items were identified as being differentially difficult per comparison with a greater number of items flagged in the female/male analyses than in the black/white analyses. Although the analyses suggested some content characteristics that may be related to differential item functioning, these findings about GMAT items should be considered tentative since only a small number of items was studied, and all investigations were post hoc analyses. Correlations between item difficulty and differential item functioning were generally low, with the exception of quantitative items in the black/white analyses. For these items, a moderately positive relationship existed between item difficulty and the differential item functioning statistic, showing that black examinees performed differentially better than matched whites as item difficulty increased. Eleven tables in the text and four appendixes (one with six tables) provide information on item classification, means and standard deviations, and problem solving and sentence correction items. (Contains 43 references.) (Author/SLD)

This study examined gender-related differential item functioning (DIF) using a mathematics performance assessment, the QUASAR Cognitive Assessment Instrument (QCAI), administered to middle school students. The QCAI was developed for the Quantitative Understanding: Amplifying Student Achievement and Reading (QUASAR) project, which focuses on reasoning, problem solving, and communication. By including two QCAI administration occasions, it was possible to examine the stability of gender-related DIF over time. On one occasion, 1,947 students took 4 forms of the QCAI, and on the other 1,999 students took the 4 forms. Student responses were rated by middle school mathematics teachers trained in the scoring procedure. Results indicated that 4 of the 36 tasks favored females and 2 favored males with respect to uniform DIF. The tasks that favored females were set in a real-world context, suggesting that some of the features that have been associated with gender-related DIF in the past may not hold true for performance assessments. For example, male students may have been disadvantaged in providing explanations for their answers in this performance assessment. (Contains 4 figures, 13 tables, and 43 references.) (SLD)

The "TOEFL Junior"® Standard Test is an assessment that measures the degree to which middle school-aged students learning English as a second language have attained proficiency in the academic and social English skills representative of English-medium instructional environments. The assessment measures skills in three areas: listening comprehension, language form and meaning, and reading comprehension. This study focused on two specific psychometric characteristics of the assessment: (a) For which segments of its score scales does the assessment provide sufficient information to support the use of the scores in placement decisions? (b) Do items exhibit significant (or C-level) differential item functioning (DIF) when comparisons are made between test-takers from different countries? For the first question, both of the forms we analyzed appear to provide sufficient information to support placement decisions across the majority of the score scale for all 3 sections of the assessment. For the second question, we found that a moderate number of items exhibited significant DIF, while the linguistic analyses conducted on the DIF results showed plausible construct-relevant explanations for most of the findings.

Differential Item Functioning (DIF) analysis is a key element in evaluating educational test fairness and validity. One of the frequently cited sources of construct-irrelevant variance is gender which has an important role in the university entrance exam; therefore, it causes bias and consequently undermines test validity. The present study aims at investigating the presence of DIF in terms of gender in a high stakes language proficiency test in Iran, the National University Entrance Exam for Foreign Languages (NUEEFL). The participants' responses (N = 5000) were selected randomly from a pool of examinees who had taken the NUEEFL in 2015. The results displayed DIF between male and female test takers. Hence, on the basis of the findings, it is concluded that the NUEEFL test scores are not free of construct-irrelevant variance and the overall fairness of the test is not confirmed. Also, both Rasch assumptions (i.e., unidimensionality and local independence) are hold in the present research.

Reliability and differential item functioning (DIF) analyses were conducted on testlets displaying local item dependence in this study. The data set employed in the research was obtained from the answers given by 1,500 students to the 20 items included in six testlets given in English Proficiency Exam by the School of Foreign Languages of a state University in Turkey. One of the purposes of this study was to determine the influences of the tests composed of testlets on reliability, so the reliability coefficients obtained for cases where the influences of testlets were considered and those for cases where the testlet influences were not considered were compared. In consequence of the G theory analyses conducted in this context, it was found that the G and Phi coefficients estimated by not considering the testlet effects were higher than those estimated by considering the testlet effects. It was concluded that the reliability was estimated to be relatively higher when the influences of the testlet were not considered. Two methods were used in this study so as to determine the effects of testlets on differential item functioning and the results were compared. In the DIF-determining method considering the testlet effect, both the number of items displaying DIF at the significant and estimated levels of DIF were found to be higher than in the method not considering the testlet effect.

The aim of this research is to determine differential item functioning (DIF) by hierarchical linear modeling (HGLM) on test items and comparing these results by the DIF results determined by logistic regression (LR) and IRT-likelihood ratio (IRT-LR) techniques. Investigating the concordance between the techniques in determining the items with DIF, we have found significant relations between items with DIF determined in Turkish and Science sub-tests using LR and IRT-LR. In Social Studies test, we have found significant relation between the results of the HGLM and LR, HGLM and IRT-LR, LR and IRT-LR techniques. The number of items with DIF per gender determined by all three different techniques has been found to be almost half of the number of test items. Though all the level of the items determined by LR technique have been negligible, with IRT-LR technique only a very few of the items have been found with negligible DIF. Comparing the Social Studies and Science sub-tests with Turkish sub-test using HGLM technique, more than half of the items have been found to have DIF. Turkish sub-test has the maximum number of item with DIF.

The effect of calculator type on student performance on a mathematics examination was studied. Differential item functioning (DIF) methodology was applied to examine group differences (calculator use) on item performance while conditioning on the relevant ability. Other survey questions were developed to ask students the extent to which they used a calculator, the perceived usefulness of the calculator, and how often it was used in the classroom. In addition, content experts were asked to identify whether an item was sensitive to calculator usage, the potential for being useful or distracting, and how the type of calculator would affect student performance. Student test data was obtained from the Tennessee Gateway assessment end-of-course test in Algebra 1. Six forms of the test were spiraled with approximately 7,000 students taking each form. No evidence of pervasive uniform difference in DIF resulting from calculator use was detected in this data. The type of questions in this assessment tended not to be sensitive to differential calculator use and did not impact test performance significantly. DIF was also not evident for students who used calculators versus those who did not. Calculator type, use, and familiarity were associated with differences in the univariate comparison of test scores. For example, students who responded that they used a graphing calculator performed higher than the other groups. Classroom practice and experience with calculators appeared to vary widely. Results from the expert judgment indicate that items with more egregious sensitivity to calculator use could be identified, although experts tended to over-identify items as being sensitive to calculator use. One appendix contains survey items related to calculator use, and the other describes the Linn-Harnisch DIF procedure. (Contains 3 figures, 4 tables, and 15 references.) (SLD)

Based on initial Scholastic Aptitude Test (SAT) Verbal pretest data and hypotheses advanced in the research literature, 7 sentence completion and 16 analogy items with extreme levels of differential item functioning (DIF) were selected and then systematically revised and re-administered in an attempt to reduce or eliminate DIF. The apparent success of the effort makes similar attempts worth continuing. The particular terminology used in stems and keys, rather than the underlying skill being measured, seemed to be a recurring source of DIF in the SAT-Verbal items. Larger sample sizes, especially for minority focal groups, would help to stabilize the DIF categories used by Educational Testing Service (ETS) test developers. In addition, because the ETS delta metric is unbounded at the extremes, the use of both the Standardization (p-metric) and Mantel-Haenszel (delta-metric) methodologies is recommended for classifying the level of DIF for very easy and very difficult items. Further research is suggested to study the possible relationship between DIF and predictive validity. Nine tables and four figures present analysis results. An appendix summarizes DIF hypotheses. (Contains 26 references.) (Author/SLD)

Differential item functioning (DIF), or item bias, occurs when individuals in a focal group respond differently to a test item than do individuals in a reference group even when comparisons are restricted to individuals with similar overall skill levels on the trait in question. It is common in constructing a questionnaire or survey to recommend that an item analysis be conducted in a manner similar to that used in cognitive measurement, but it is not common to be concerned with items as they perform differently. DIF has apparently not yet been widely recognized as a tool for developing a survey or for understanding survey responses. The Mantel Haenszel procedure is one of the empirical methods most commonly used to identify DIF, and its use in survey development is explored. Two examples, one involving the evaluation of student achievement for 777 male and 773 female elementary school students, and the other involving the evaluation of human service workshops for 798 participants aged over 40 years and 884 who were younger, illustrate the way in which information about DIF could have aided in the development of the instrument and interpretation of the data. DIF detection would seem a useful adjunct to the traditional item analysis that could be of substantial value at the pilot or revision stage of instrument development. (Contains 4 figures and 20 references.) (SLD)

The purposes of this study were to introduce the iterative purification procedure and to compare this with the two-step purification procedure, to compare false positive error rates and the power of five observed score approaches and to identify factors affecting power and false positive rates in each method. This study used 2,400 data sets that were divided into uniform, symmetric nonuniform, and nonsymmetric nonuniform differential item functioning (DIF) data sets. The sample size pairs were either 500,500 or 1,000,1,000 for the reference group and the focal group when the means of ability distributions for the 2 groups were the same, and either 1,000,500 or 1,000,250 for the reference and focal groups when the means of ability distributions for the 2 groups were different. Each dataset included four items with uniform, symmetric nonuniform, or nonsymmetric nonuniform DIF, with each DIF item having either a 0.4 or 0.8 amount of DIF (that is, the area between two item characteristic curves). The purification procedures reduced false positive error rates and/or increased power. The Mantel Haenszel method was superior to other methods with uniform DIF data sets, and the Absolute Mean Deviation method using the iterative purification procedure was superior to the others in nonuniform data sets when the means of ability distributions for the two groups were different. The ability estimation and the sample size affected detection rates and false positive error rates for all methods. The DIF effect size was also a strong influence on detection rates. (Contains 21 tables and 25 references.) (Author/SLD)

When tests contain few items, observed score may not be an accurate reflection of true score, and the Mantel Haenszel (MH) statistic may perform poorly in detecting differential item functioning. Applications of the MH procedure in such situations require an alternate strategy; one such strategy is to include background variables in the matching criterion. Techniques for incorporating external information are presented here that match on a weighted score that combines the observed score and background data, using either ordinary least squares regression or a multilevel model. The regression and multilevel models were constructed using data obtained with the Grade 3 North Carolina End of Grade Mathematics Test. A simulation study was performed in which the prediction models were used to generate data, and three MH statistics were computed matching on observed scores, regression weighted scores, and multilevel weighted scores. The results showed similar performance for the regression and multilevel weighted score methods. The observed score and weighted score methods demonstrated advantages over the observed score method for test lengths of 5 and 10 items, but the improvement was small and inconsistent. Techniques for improving the performance of the weighted score methods are discussed. (Contains 21 references and 12 tables.) (SLD)

This study introduced and demonstrated a new methodology for item and test bias studies: moderated differential item functioning (DIF). This technique expands the DIF methodology to incorporate contextual and sociological variables as moderating effects of the DIF. The study explored differential domain functioning (DDF), so that the focus of interpretation for this test is on the "domain" rather than the item. This moderated DDF effect was shown in a multiple choice and constructed response provincial assessment that was designed to match a specific mathematics curriculum. Participants were 45,728 fourth graders, 45,022 seventh graders, and 43,525 tenth graders in British Columbia, Canada. Data were narrowed to create four contrast groups of communities that reflect differences in contextual variables: rural low-income, rural affluent, urban low-income, and urban affluent. Gender DDF was explored using a general liner statistical model. After statistically matching males and females on their mathematical ability, gender DDF was moderated by the contextual variables. Thus, this "moderation" approach allowed the investigation of sociological, community-based contextual variables that may help explain the complex functioning of DIF in large scale testing. The paper advocates a more "sociological" and "ecological" approach to help explain differences in item and test performance. (Contains 22 references.) (Author/SLD)

Simulated data were used to investigate the performance of modified versions of the Mantel-Haenszel and standardization methods of differential item functioning (DIF) analysis in computer-adaptive tests (CATs). Each "examinee" received 25 items out of a 75-item pool. A three-parameter logistic item response model was assumed, and examinees were matched on expected true scores based on their CAT responses and on estimated item parameters. Both DIF methods performed well. The CAT-based DIF statistics were highly correlated with DIF statistics based on nonadaptive administration of all 75 pool items and with the true magnitudes of DIF in the simulation. DIF methods were also investigated for "pretest items," for which item parameter estimates were assumed to be unavailable. The pretest DIF statistics were generally well-behaved and also had high correlations with the true DIF. The pretest DIF measures, however, tended to be slightly smaller in magnitude than their CAT-based counterparts. Also, in the case of the Mantel-Haenszel approach, the pretest DIF statistics tended to have somewhat larger standard errors than the CAT-DIF statistics. Appendix A contains 10 supplementary tables; and Appendixes B, C, and D present additional information about the expected table estimator. Twenty-two tables in Appendix D present analysis results. (Contains 24 references.) (Author/SLD)

Differential item functioning (DIF) occurs when the responses of students with approximately equal ability differ systematically based on their group membership. Idaho contracted with the Northwest Evaluation Association (NWEA) to provide the Idaho Standards Achievement Tests of reading, language arts and mathematics at grades 4, 8 and 10. The tests included items in the NWEA Rasch-calibrated item bank and items written by Idaho teachers that were equated to the item bank metric. NWEA provided a random sample of data representing about half of the students taking part in the spring 2003 assessment. This study screened for gender-based (female-male) and ethnic-based (Hispanic-White) DIF. The Simultaneous Item Bias Test (SIBTEST) and the Mantel-Haenszel Chi Square Test were used to detect statistically significant DIF. The Bonferonni correction was employed to control for Type I error. Effect size procedures were used to determine whether statistical differences were large enough to have practical meaning. DIF findings from a NWEA analysis of spring 2003 test results were also noted. The proportion of items exhibiting gender-based DIF ranged from seven percent for the Grade 4 reading test to 37 percent for the Grade 10 mathematics test. The proportion of items exhibiting ethnic-based DIF ranged from seven percent for the Grade 8 mathematics test to 19 percent for the Grade 4 tests of reading and language arts. The study recommended that items exhibiting DIF be reviewed by curriculum specialists before further use, that the impact of the high occurrence of DIF on the ability to accurately interpret the spring 2003 student scores be examined, and that language requiring DIF analyses on all items used in future test administrations be added to the contract with the test developer.

The purpose of this study was to screen for gender-based (female-male) and ethnic-based (Hispanic-White) differential item functioning (DIF) in the 2003 Idaho Standards Achievement Tests of reading, language arts and mathematics at grades 4, 8 and 10. The vendor, the Northwest Evaluation Association (NWEA), assembled the computer-administered tests using items from its Rasch-calibrated item bank and items from Idaho teachers that were equated to the item bank metric. The absence of DIF statistics for any of these test items led to this study. NWEA provided random samples of data representing about half of the students completing each grade-level test that ranged from 8,659 for the tenth grade mathematics test to 9,676 for the fourth grade language usage test. This study used the Simultaneous Item Bias Test (SIBTEST), which assesses item dimensionality, and the Mantel-Haenszel Chi Square Test to detect statistically significant DIF in the individual items. A Bonferonni correction controlled for Type I error. Effect size procedures described whether the observed statistical differences were large enough to have practical meaning. This study also noted DIF findings from a concurrent NWEA analysis of item parameter invariance in the 2003 Idaho tests. The proportion of items exhibiting moderate to high gender-based DIF ranged from seven percent for the fourth grade reading test to 37 percent for the tenth grade mathematics test. The proportion of items exhibiting moderate to high ethnic-based DIF ranged from seven percent for the eighth grade mathematics test to 19 percent for the fourth grade tests of reading and language arts. The study recommended that NWEA and independent curriculum specialists review all items exhibiting moderate to high DIF before further use, that the state examine the impact of the high occurrence of DIF on the ability to accurately interpret the spring 2003 student scores, and that the state add language to the vendor contract requiring DIF analyses of all items used in future administrations of the statewide tests. (Contains 6 tables, 1 figure, and 3 appendices.)

The differential item functioning of mathematics performance of Hispanic, Asian, and White students on the National Assessment of Educational Progress (NAEP) was studied, using a modification of the Mantel-Haenszel Procedure. The sample for grades 3, 7, and 11, respectively, included: (1) 1,367, 1,570, and 1,580 Whites; (2) 265, 613, and 760 Asians; (3) 1,238, 1,602, and 1,022 Mexican Americans; (4) 566, 624, and 458 Puerto Ricans; and (5) 292, 347, and 566 Cubans. The data are from the 1985-86 NAEP regular assessment and the NAEP special supplemental study of language minority students. With the exception of third graders, no simple explanation could be offered as to why some items were more difficult or easier for ethnic group members. For the third grade, items with little or no text, involving simple arithmetical operations, were differentially easier for many ethnic group members. Overall, as many items favored the language groups as favored the majority comparison group at each grade level. Four tables summarize the study data. (SLD)

Research objective is comparing the objective methods often used in literature for determination of differential item functioning (DIF) and the subjective method based on the opinions of the experts which are not used so often in literature. Mantel-Haenszel (MH), Logistic Regression (LR) and SIBTEST are chosen as objective methods. While the data of an extensive examination in Turkey applied for objective methods, the data that are obtained from Expert Opinions Form used to evaluate the items of the same examination. The data obtained from 5077 female and 5271 male students are used for the objective methods, and 23 experts' opinions are used for subjective method. The concordance between the objective and subjective methods is calculated by using the compatibility rate and Cohen's kappa coefficient in the research. While the highest concordance related to the existence of DIF is obtained between MH and SIBTEST methods (0.90; ? = 0,79) and the lowest concordance is between LR and SIBTEST methods (0.75; ? = 0,50) in objective methods, When the concordance of the objective method with the subjective methods is examined, at least moderate concordance (0.75; ? = 0,47) is obtained in the decision. When items which have DIF is examined according to DIF level, three items indicate low level of DIF and one item indicates moderate or high level of DIF for both methods. In addition, in subjective method, a decision study is made on the number of the experts presenting opinion within the generalizability theory and the acceptable reliability value is reached with 13 experts' opinions.

The "Assessment of Culturally and Contextually Relevant Supports" (ACCReS) was developed in response to the need for well-constructed instruments to measure teachers' cultural responsiveness and guide decision-making related to professional development needs. The current study sought to evaluate the presence of differential item functioning (DIF) in ACCReS items and the magnitude of DIF, if detected. With a national sample of 999 grade K-12 teachers in the U.S., we examined measurement invariance of ACCReS items in relation to responses from (a) racially and ethnically minoritized (REM) and White teachers (teacher race), (b) teachers in schools with 0-50% and 51-100% REM youth (student race), and (c) teachers with <1-5 years of teaching experience and teachers with >5 years of experience. Findings suggested that ACCReS items exhibited negligible levels of DIF. The lack of DIF found provides additional evidence for the validity of scores from the ACCReS to assess teachers' cultural responsiveness. Furthermore, descriptive analyses revealed that teachers were more likely to agree with items pertaining to their own classroom practice than items related to access to adequate training and support. Results inform implications for future educational and measurement research. [This paper was published in "Journal of Psychological Assessment" 2021 (EJ1312232).]

Two studies investigated factors affecting the scores of Hawaii students taking the verbal subtest of the Scholastic Aptitude Test (SAT). For the past several years, the mean verbal scores of Hawaii students have consistently been among the lowest 10% of all states. The first study addressed the identification of items and types of items that have been answered differentially by Hawaii students in comparison to mainland United States students. The items were identified through differential item functioning (DIF) procedures which assess performance differences between groups of individuals with the same overall scholastic aptitude. Results indicated that Hawaii students performed less well than the mainland reference group (students of equal overall scholastic aptitude) on the early items in each of the antonym sections and better than the reference group on the more difficult or later items in each section. Carelessness and unfamiliarity with the item type were identified as possible causes. The second study utilized a sample of Hawaii public school students who received brief instruction addressing the low performance on the types of items identified in the first study. Independent groups t-tests conducted between the treatment students and a sample from the original pool of 1988 examinees showed equivalence of performance on the pretest. Treatment students performed statistically significantly better than the comparison groups did after receiving the treatment. (Five tables of data are included. Contains 17 references.) (Author/RS)

The Mantel-Haenszel (MH) procedure has become one of the most popular procedures for detecting differential item functioning (DIF). One of the most troublesome criticisms of this procedure is that while detection rates for uniform DIF are very good, the procedure is not sensitive to non-uniform DIF. In this study, examinee responses were generated to simulate both uniform and non-uniform DIF. Responses for 3 groups of 1,000 examinees (1 reference group and 2 focal groups) were simulated using a 3-parameter logistic item response theory model. A standard MH procedure was used first. Then examinees were split into two samples, by breaking the full sample at approximately the middle of the test score distribution. The tests (25 simulated tests) were then re-analyzed, first with the low performing sample, and then with the high performing sample. This variation improved detection rates of non-uniform DIF considerably over the total sample procedure without increasing the Type I error rate. Items with the largest differences in discrimination and difficulty parameters were most likely to be identified. Two tables present study data. (Author/SLD)

Studies evaluating hypotheses about sources of differential item functioning (DIF) are classified into two categories: observational studies evaluating operational items and randomized DIF studies evaluating specially constructed items. For observational studies, advice is given for item classification, sample selection, the matching criterion, and the choice of DIF techniques, as well as how to summarize, synthesize, and translate DIF data into DIF hypotheses. In randomized DIF studies of specially constructed items, specific hypotheses, often generated from observational studies, are evaluated under rigorous conditions. Advice for these studies focuses on the importance of carefully constructed items to assess DIF hypotheses. In addition, randomized DIF studies are cast within a causal inference framework, which provides a justification for the use of standardization analyses or logistic regression analysis to estimate effect sizes. Two studies that have components spanning the observational and controlled domains are summarized for illustrative purposes. Standardization analyses are used for both studies. Special logistic regression analyses of an item from one of these studies are provided to illustrate a new approach in the assessment of DIF hypotheses using specially constructed items. (Contains 5 figures and 39 references.) (Author)

This article is from BMC Musculoskeletal Disorders , volume 14 . Abstract Background: The SF-36 is a very commonly used generic measure of health outcome in osteoarthritis (OA). An important, but frequently overlooked, aspect of validating health outcome measures is to establish if items work in the same way across subgroup of a population. That is, if respondents have the same ‘true’ level of outcome, does the item give the same score in different subgroups or is it biased towards one subgroup or another. Differential item functioning (DIF) can identify items that may be biased for one group or another and has been applied to measuring patient reported outcomes. Items may show DIF for different conditions and between cultures, however the SF-36 has not been specifically examined in an osteoarthritis population nor in a UK population. Hence, the aim of the study was to apply the DIF method to the SF-36 for a UK OA population. Methods: The sample comprised a community sample of 763 people with OA who participated in the Somerset and Avon Survey of Health. The SF-36 was explored for DIF with respect to demographic, social, clinical and psychological factors. Well developed ordinal regression models were used to identify DIF items. Results: DIF items were found by age (6 items), employment status (6 items), social class (2 items), mood (2 items), hip v knee (2 items), social deprivation (1 item) and body mass index (1 item). Although the impact of the DIF items rarely had a significant effect on the conclusions of group comparisons, in most cases there was a significant change in effect size. Conclusions: Overall, the SF-36 performed well with only a small number of DIF items identified, a reassuring finding in view of the frequent use of the SF-36 in OA. Nevertheless, where DIF items were identified it would be advisable to analyse data taking account of DIF items, especially when age effects are the focus of interest.

The purpose of this study was to compare two non-parametric procedures, the Mantel-Haenszel (MH) procedure and the simultaneous item bias (SIB) procedure, with respect to their Type I error rates and power, and to investigate the conditions under which asymptotic distributional properties of the SIB and MH were obtained. Data were simulated to reflect a variety of conditions; the factors manipulated were sample sizes, ability distributions of the focal and the reference groups, percent of DIF items in the test, types of item and DIF effect sizes. Investigations of the distribution of the SIB and the MH statistics revealed that the SIB statistic had the theoretical asymptotic distributions when the sample sizes of the focal and reference groups exceeded 200, whereas the MH statistic did not have the theoretical asymptotic distributions under any condition. The MH and the SIB procedures were equally powerful in detecting DIF for equal ability distributions, and the SIB procedure was more powerful for unequal ability distributions than the MH procedure. The Type I error rates for the MH statistic were within limits, whereas they were higher for the SIB statistic than those for the MH statistic. Comparisons between the detection rates of the two procedures were made with respect to the various factors manipulated in the study. Suggestions for future research are made. (Contains 16 references and 11 tables.) (Author)

This study investigated the possible impacts of language and curriculum differences on the performance of test items by subpopulations of students. Focusing on Measurement and Geometry items completed by students in French- and English-language schools in Ontario made it possible to explore the differences and to compare the item response theory (IRT) and Mantel-Haenszel (MH) approaches to finding items with differential item functioning (DIF). The tests in this study might have had DIF as a result of translation or curriculum. Data came from the 2001 administration of the School Achievement Indicators Program (SAIP) mathematics assessment. The dataset contained information on 793 13-year-old and 677 16-year-old students from English-language schools and 487 13-year-old and 546 16-year-old students from French-language schools. Thirteen of the 27 items studies were flagged as exhibiting DIF by the MH approach, and 6 of these 13 were also flagged by IRT. Differences in curricula and practice between the French- and English-language schools are very difficult to separate from language differences, and, in fact, results point to the possibility of another source of DIF: different test-taking approaches in the two populations. Results demonstrate the complexity of the factors that contribute to how items are understood and approached by different groups of students. (Contains 5 tables, 7 figures, and 40 references.) (SLD)

This study was designed to answer two key questions about the fixed portions of the fourth, eighth, and tenth grade spring 2003 Idaho Standards Achievement Tests (ISAT) in Reading, Language Usage and Mathematics: (1) To what extent do test items exhibit substantive differences in functioning across student groups formed based on ethnicity and based on gender? and (2) What is the collective impact on test characteristics of any substantive differentially functioning items? It is worth mentioning that Differential Item Functioning (DIF) is not quite the same as what has commonly been called "bias." Differential item functioning is an item characteristic that occurs whenever groups differ noticeably in their correct answer frequency, when achievement level is matched. It may be used as part of the process to identify biased items, but is almost always viewed as secondary to a sensitivity review. Because of its statistical nature, many items that are identified as having DIF are not identified as having any difficulty in the sensitivity review. (Contains 14 tables and 18 figures.)

Data from a 50-item translated test used for certification were used to assess the percentage and type of agreement between the Mantel-Haenszel (MH) and Differential Functioning of Items and Tests (DFIT) techniques for the detection of differential item functioning (DIF). The DFIT procedure flagged 10 of 30 items as exhibiting significant DIF while the MH technique flagged 2 of 30 items for significant DIF. In both methods items were flagged for significant DIF when translation differences appeared in the item stems. The DFIT method was more sensitive in detecting DIF, resulting exclusively from differences in the item answer options. The overall percent agreement between the two techniques for the detection of DIF in this investigation was 20 percent. The MH technique detected 1 of 10 items as exhibiting nonuniform DIF and 1 of 10 items as displaying uniform DIF. The DFIT procedure detected 4 of 10 items as exhibiting nonuniform DIF and 6 of 10 as displaying uniform DIF. Four appendixes contain tables of descriptive statistics and the English and back-translated item versions. (Contains 34 references.) (Author/SLD)

The purpose of this study is to illustrate the use of explanatory models based on Rasch measurement theory to detect systematic relationships between student and item characteristics and achievement differences using differential item functioning (DIF), differential group functioning (DGF), and differential person functioning (DPF) techniques. The major focus of the analyses in this study was to demonstrate a set of methodological techniques that can be used to better understand subgroup performance on a large-scale writing assessment, rather than to conduct bias or sensitivity reviews. Tables and figures are appended.

The purpose of this study is to help ensure that strategies for differential item functioning (DIF) detection for students with disabilities are appropriate and lead to meaningful results. We surveyed existing DIF studies for students with disabilities and describe them in terms of study design, statistical approach, sample characteristics, and DIF results. Based on descriptive and graphical summaries of previous DIF studies, we make recommendations for future studies of DIF for students with disabilities. Differential Item Functioning (DIF) Studies Surveyed are appended. (Contains 3 tables, 6 figures and 7 notes.)

The proposed study will investigate potential shifts in participants' response patterns in the My Resiliance in Adolescence (MYRIAD) trial

Differential Item Functioning (DIF) exists when examinees of equal ability from different groups have different probabilities of successful performance in a certain item. This study examined gender differential item functioning across the PhD Entrance Exam of TEFL (PEET) in Iran, using both logistic regression (LR) and one-parameter item response theory (1-p IRT) models. The PEET is a national test consisting of a centralized written examination designed to provide information on the eligibility of PhD applicants of TEFL to enter PhD programs. The 2013 administration of this test provided score data for a sample of 999 Iranian PhD applicants consisting of 397 males and 602 females. First, the data were subjected to DIF analysis through logistic regression (LR) model. Then, to triangulate the findings, a 1-p IRT procedure was applied. The results indicated (1) more items flagged for DIF by LR than by 1-p IRT (2) DIF cancellation (the number of DIF items were equal for both males and females), as revealed through LR, (3) equal number of uniform and non-uniform DIF, as tracked via LR, and (4) female superiority in the test performance, as revealed via IRT analysis. Overall, the findings of the study indicated that PEET suffers from DIF. As such, test developers and policymakers (like NOET & MSRT) are recommended to take these findings into serious consideration and exercise care in fair test practice by dedicating effort to more unbiased test development and decision making.

Differential Item Functioning (DIF) exists when examinees of equal ability from different groups have different probabilities of successful performance in a certain item. This study examined gender differential item functioning across the PhD Entrance Exam of TEFL (PEET) in Iran, using both logistic regression (LR) and one-parameter item response theory (1-p IRT) models. The PEET is a national test consisting of a centralized written examination designed to provide information on the eligibility of PhD applicants of TEFL to enter PhD programs. The 2013 administration of this test provided score data for a sample of 999 Iranian PhD applicants consisting of 397 males and 602 females. First, the data were subjected to DIF analysis through logistic regression (LR) model. Then, to triangulate the findings, a 1-p IRT procedure was applied. The results indicated (1) more items flagged for DIF by LR than by 1-p IRT (2) DIF cancellation (the number of DIF items were equal for both males and females), as revealed through LR, (3) equal number of uniform and non-uniform DIF, as tracked via LR, and (4) female superiority in the test performance, as revealed via IRT analysis. Overall, the findings of the study indicated that PEET suffers from DIF. As such, test developers and policymakers (like NOET & MSRT) are recommended to take these findings into serious consideration and exercise care in fair test practice by dedicating effort to more unbiased test development and decision making.

The Mantel-Haenszel (MH) differential item functioning (DIF) parameter for uniform DIF is well defined when item responses follow the two-parameter-logistic (2PPL) item response function (IRF), but not when they follow the three-parameter-logistic (3PL) IRF, the model typically used with multiple choice items. This research report presents a general formulation of the MH DIF population parameter for any IRF and presents results for numerous 3PL uniform DIF conditions. The results indicate that for items of medium or high difficulty, the 2PL DIF parameter formulation can overestimate the 3PL DIF parameter and the MH DIF estimator may exhibit less than expected power to identify even substantial DIF in certain circumstances. Implications of this study on the routine operational task of identifying DIF at the Law School Admission Council are still not known, and may in fact be minimal. However, because some items on the Law School Admission Test (LSAT) are known to exhibit guessing behavior, the results suggest that additional research is warranted. (Contains 2 tables, 5 figures, and 13 references.) (Author/SLD)

This study examines performance differences between students with disabilities and students without disabilities students using differential item functioning (DIF) analyses in a high-stakes reading assessment. Results indicated that for Grade 9, many items exhibited DIF. Items that exhibited DIF were more likely to be located in the second half of the assessment subscales. After accounting for reading ability using a proxy score from items on the first half of the subscales, students with disabilities consistently underperformed on items located in the second half relative to the items located in the first half, as compared with students without disabilities. These results were seen in Grade 9 for data from two different states. These results were not seen for Grade 3. This study has several limitations. There was no access to information regarding the testing accommodations that students with disabilities might have received, and no access to the type of disabilities. Results of this study can shed light on potential factors affecting the accessibility of reading assessments for students with disabilities, in an ultimate effort to provide assessment tools that are conceptually and psychometrically sound for all students. A companion report is available examining differential distractor functioning for students with disabilities. (Detailed DIF Results are appended. Contains 30 figures and 45 tables.)

The Mantel Haenszel (MH; 1959) approach of Holland and Thayer (1988) is a well-established method for assessing differential item functioning (DIF). The formula for the variance of the MH DIF statistic is based on work by Phillips and Holland (1987) and Robins, Breslow, and Greenland (1986). Recent simulation studies showed that the MH variances tended to be larger when items were administered to "examinees" who were randomly selected from a population than when items were administered adaptively. An analytic perspective shed some light on this puzzling result. Although the general form of the MH variance is complex and does not provide an intuitive understanding of the phenomenon, application of certain Rasch model assumptions yields a simple expression that appears to explain the difference in variances for adaptive versus nonadaptive administration. One table, two figures. (Contains 13 references.) (Author)

The purpose of this study was to improve both statistical and judgmental methods for detecting potentially biased test items in an attempt to examine the agreement between the results obtained with these methods. If greater agreement between methods can be achieved, test items can be more effectively screened using judgmental methods prior to field testing or actual test administrations. Several methodological shortcomings of current empirical and judgmental methods were addressed. The test data came from samples of 2,000 Native Americans and 2,000 Anglo-Americans who took a 4-choice 150-item Statewide Proficiency Test. Fifteen Native American educators provided item bias reviews. To reduce computer time and facilitate the analyses, 75 items on the test were analyzed. The Mantel-Haenszel procedure was applied to the item responses for the two subject groups. The results suggest that s somewhat higher level of agreement between methods was obtained than has been observed in other studies. The use of cross-validation in empirically identifying potentially biased items was one reason for the higher level of agreement. However, the judgmental method implemented in this study appeared to have several shortcomings. Practical implications of the findings are presented. Two tables and one figure are included. (Author/RLC)

The term differential item functioning (DIF) refers to whether or not the same psychological constructs are measured across different groups. If an item does not measure the same skills or subskills in different populations, it is said to function differentially or to display item bias. A multilevel approach to DIF is proposed. In such a model, the dependency between observations due to cluster effects is explicitly taken into account. Results of a multilevel logit model and of a multilevel logistic regression model are compared with results of analogous unilevel models. The procedure is illustrated with data from a national assessment of geography performed with respect to gender bias. Each of the 294 items was answered by an average of 2,161 respondents. Analysis supports the use of multilevel models, which have the advantage of accounting for cluster effects in data from a hierarchical population. DIF does seem to be more stable according to multilevel models than to unilevel models. Five tables and four figures present analysis results. Appendixes provide parameter estimates. (Contains 19 references.) (SLD)

In multiple-choice items the response probability on an item may be viewed as the result of two distinct latent processes--a cognitive process to solve the problem, and another random process that leads to the choice of a certain alternative (the process of giving the actual response). An incomplete latent class model is formulated that describes the first process by a Rasch model and the second process by a guessing model. Alternative models are specified that contain additional parameters describing differential item functioning (DIF) in the two processes. DIF with respect to either known or unknown subgroups can be tested by a likelihood ratio test that is asymptotically distributed as chi-square. As an example of the model, four five-choice items from the Second International Mathematics Study (1987) with a sample of 3,002 secondary students were considered. A 42-item list of references and 3 data tables are included. (Author/SLD)

An approach to detecting differential item functioning using the Rasch model with equivalent-group cross-validation was investigated. College students taking the Dental Admission Test, were divided by gender (936 females and 1,537 males) into 2 different samples. Rasch analyses were performed on both samples. Data were recalibrated after misfitting persons and items were removed. Resulting difficulties from the two samples were compared, and some potentially biased items were identified. Each sample was then randomly divided into two equivalent samples, resulting in four groups, two of males and two of females. Rasch calibrations were again performed. Items were rank-ordered, and items with the same rankings were selected for equating. Link constants were calculated by selecting one of the four groups as the ground scale. Using the t-test, item-by-item comparisons were conducted within each group pair after the equating. Different difficulty calibrations were compared across the different groups. The significantly different difficulties may have been caused by sampling fluctuation if the t-value was significant when comparing groups with the same gender membership. Finally, identified items were examined for bias. The final set of biased items was then distributed to the test construction committee for possible modifications. One table contains descriptive statistics from the analysis. (Author/SLD)