Downloads & Free Reading Options - Results

This study investigated the number-correct scoring method based on different theories (classical true-score theory and multidimensional item response theory) when a standardized test requires more than one ability for an examinee to get a correct response. The number-correct scoring procedure that is widely used is the one that is defined in classical true-score theory (CTT). In CTT, a test score is equal to the number of items an examinee answered, so that all items are weighted "one." It is also possible to use a form of number-correct scoring in which the weights of items are different. In this study, the accuracy of estimated number-correct scores relative to true number-correct scores under CTT, multidimensional item response theory (MIRT) and both MIRT and CTT were studied using simulated data for a standardized test in which true scores and estimated scores were known. A method in which item weights were based on MIRT and test scores based on CTT (MIX method) was found to be the most accurate method used to estimate the true score on an examinee. This MIX method was also significantly different from the other three scoring methods using the bootstrap analysis. An appendix contains definitions of the notations representing the various parameters and approaches. (Contains 20 references.) (SLD)

A confirmatory approach to assessing test structure using multidimensional item response theory (MIRT) was developed and evaluated. The approach involved adding to the exponent of the MIRT model an item structure matrix that allows the user to specify the ability dimensions measured by an item. Various combinations of item structures were fit to two sets of simulation data with known true structures, and the results were evaluated using a likelihood ratio chi-square statistic and two information-based model selection criteria. The results of these analyses support the use of the confirmatory MIRT approach, since it was found that the procedures could recover true item structures. It was also found that adding an additional ability dimension that forces together items that ought not to be together noticeably deteriorates the quality of the solution. On the other hand, imposing structures different from, but not inconsistent with, the true structures does not necessarily yield worse fit. Finally, in terms of model fit statistics, the consistent Akaike information criterion performed better than the simple Akaike information criterion, while the likelihood ratio chi-square was clearly inadequate. (Contains 12 tables and 30 references.) (Author/SLD)

Several methods for estimating item response theory scores for multiple subtests were compared. These methods included two multidimensional item response theory models: a bi-factor model where each subtest was a composite score based on the primary trait measured by the set of tests and a secondary trait measured by the individual subtest, and a model where the traits measured by the subtests were separate but correlated. Composite scores based on unidimensional item response theory, with each subtest borrowing information from the other subtest, as well as independent unidimensional scores for each subtest were also considered. Correlations among scores from all methods were high, though somewhat lower for the independent unidimensional scores. Correlations between course grades and test scores, a measure of validity, were similar for all methods, though again slightly lower for the unidimensional scores. To assess bias and RMSE, data were simulated using the parameters estimated for the correlated factors model. The independent unidimensional scores showed the greatest bias and RMSE; the relative performance of the other three methods varied with the subscale. (Contains 6 tables, 4 figures, and 2 footnotes.)

This report summarizes the work in the development and evaluation of multidimensional item response theory models (MIRT) and in the evaluation of the use of computerized testing in the instructional environment. The work on MIRT included the definition of descriptive statistics for items that require more than an ability for successful completion, the evaluation of calibration programs, the comparison of competing MIRT models, and the application of MIRT procedures to practical testing problems. The work on computerized testing investigated medium and order effects for classroom achievement tests in a military training setting and the feasibility of computerized adaptive testing in that setting. Multidimensional item response theory (MIRT) is a theoretical framework for describing the interaction between a person and a test item when it is believed that performance on the test item is sensitive to person differences on more than one dimension. (kr)

Test data generated according to two different multidimensional item response theory models were compared at both the item response level and the test score level to determine if measureable differences between the models could be detected when the data sets were constrained to be equivalent in terms of item p-values. Although differences could be detected at the item level, these differences decreased as the correlation between examinee abilities increased. Furthermore, these item differences were small in magnitude and could be considered unimportant or insignificant from a practical standpoint. No differences were found at the total test score level, and it was concluded that, at least for the data used in this study, the models were indistinguishable.

Confirmatory multidimensional item response theory (CMIRT) was used to assess the structure of the Graduate Record Examination General Test, about which much information about factorial structure exists, using a sample of 1,001 psychology majors taking the test in 1984 or 1985. Results supported previous findings that, for this population, there exists a weak analytical factor defined by the logical reasoning items and not by the analytical reasoning items. This finding was more straightforward than the same finding based on the full-information factor analysis (FIFA) approach used in the previous study; however, an advantage of FIFA was that it allowed the researcher to assess the proportion of variance explained by each factor in the orthogonal solution--the CMIRT approach is unlikely to yield such a statistic. Examination of the results shows the CMIRT approach to hold much promise. (SLD)

A two-dimensional, compensatory item response model and a unidimensional model were fitted to the reading and mathematics items in the Woodcock-Johnson Psycho-Educational Battery-Revised for a sample of 1,000 adults aged 20-39 years. Multidimensional information theory predicts that if the unidimensional abilities can be represented as vectors in the two-dimensional solution, then the multidimensional model can be used to obtain ability scores with smaller standard errors. In reading, the multidimensional model yielded scores with smaller standard errors, but multidimensional scores from subtests within reading were identical to the overall reading score. In mathematics, unidimensional scores were non-linearly related to multidimensional ability estimates, and for some subjects, multidimensional ability scores had larger standard errors. There is a 16-item list of references. One table and 12 figures provide analysis information. (Author/SLD)

Multidimensional Item Response Theory (MIRT) is widely used in assessment and evaluation of educational and psychological tests. It models the individual response patterns by specifying functional relationship between individuals' multiple latent traits and their responses to test items. One major challenge in parameter estimation in MIRT is that the likelihood involves intractable multidimensional integrals due to latent variable structure. Various methods have been proposed that either involve direct numerical approximations to the integrals or Monte Carlo simulations. However, these methods are known to be computationally demanding in high dimensions and rely on sampling data points from a posterior distribution. We propose a new Gaussian Variational EM (GVEM) algorithm which adopts a variational inference to approximate the intractable marginal likelihood by a computationally feasible lower bound. In addition, the proposed algorithm can be applied to assess the dimensionality of the latent traits in an exploratory analysis. Simulation studies are conducted to demonstrate the computational efficiency and estimation precision of the new GVEM algorithm in comparison to the popular alternative Metropolis-Hastings Robbins-Monro (MHRM) algorithm. In addition, theoretical results are also presented to establish the consistency of the estimator from the new GVEM algorithm. [This paper will be published in "British Journal of Mathematical and Statistical Psychology."]

A coordinate system free definition of complex structure multidimensional item response theory (MIRT) for dichotomously scored items is presented. The point of view taken emphasizes the possibilities and subtleties of understanding MIRT as a multidimensional extension of the ``classical'' unidimensional item response theory models. The main theorem of the paper is that every monotonic MIRT model looks the same; they are all trivial extensions of univariate item response theory.

Hooker and colleagues addressed a paradoxical situation that can arise in the application of multidimensional item response theory (MIRT) models to educational test data. We demonstrate that this MIRT paradox is an instance of the explaining-away phenomenon in Bayesian networks, and we attempt to enhance the understanding of MIRT models by placing the paradox in a broader statistical modeling perspective.

It is not unusual for several tests measuring different abilities to be given in one test administration. A common practice is to estimate these abilities independently of each other, but this paper proposes a more efficient method of estimating these abilities that takes into account the correlational structure of the abilities. The method uses a hierarchical Bayesian approach to simultaneous estimation of abilities based on a simple structure multidimensional item response model. Whether the simultaneous estimation of abilities from different dimensions yields more accurate estimates was studied, as well as how the number of dimensions, the number of items in each dimension, and the degree of correlation between abilities affect the accuracy of these estimates. For each combination of number of abilities and number of items, item parameters were randomly drawn from a pool of 550 items from nationally standardized mathematics tests, with 1,000 examinees simulated for each experimental condition. Results show that the proposed multidimensional approach gives more general outcomes, yielding results similar to those of the unidimensional method when abilities are uncorrelated. When abilities are correlated, taking the correlation into account can result in noticeable improvements in ability estimates. (Contains 24 tables and 15 references.) (SLD)

It is not unusual for several tests measuring different abilities to be given in one test administration. A common practice is to estimate these abilities independently of each other, but this paper proposes a more efficient method of estimating these abilities that takes into account the correlational structure of the abilities. The method uses a hierarchical Bayesian approach to simultaneous estimation of abilities based on a simple structure multidimensional item response model. Whether the simultaneous estimation of abilities from different dimensions yields more accurate estimates was studied, as well as how the number of dimensions, the number of items in each dimension, and the degree of correlation between abilities affect the accuracy of these estimates. For each combination of number of abilities and number of items, item parameters were randomly drawn from a pool of 550 items from nationally standardized mathematics tests, with 1,000 examinees simulated for each experimental condition. Results show that the proposed multidimensional approach gives more general outcomes, yielding results similar to those of the unidimensional method when abilities are uncorrelated. When abilities are correlated, taking the correlation into account can result in noticeable improvements in ability estimates. (Contains 24 tables and 15 references.) (SLD)

The assessment of dimensionality of data is important to item response theory (IRT) modelling and other multidimensional data analysis techniques. The fact that the parameters from the factor analysis formulation for dichotomous data can be expressed in terms of the parameters in the multidimensional IRT model suggests that the assessment of the dimensionality of the latent trait space can also be approached from the factor analytical viewpoint. Some problems connected with the assessment of the dimensionality of the latent space are discussed, and the conclusions are supported by simulated results for sample sizes of 250 and 500 on a 15-item test. Five tables contain data from the simulation; and 48 graphs illustrate eigenvalues and plotted mean residuals. A 46-item list of references is included. (Author/SLD)

A study was undertaken to develop guidelines for the interpretation of the parameters of three multidimensional item response theory models and to determine the relationship between the parameters and traditional concepts of item difficulty and discrimination. The three models considered were multidimensional extensions of the one-, two-, and three-parameter logistic models. The three models were found to have quite distinct definitions of difficulty and discrimination, making it imperative that the user take care to ensure that the characteristics of the model used match the characteristics of the data to which it is applied. The guidelines developed as a result of this study provide information that should be helpful in ensuring that the appropriate model is used for a particular application. (Author)

The purpose of this paper is to demonstrate how graphical analyses can enhance the interpretation and understanding of multidimensional item-response theory (IRT) analyses. Conceptually many of the unidimensional IRT concepts such as item characteristic curves, information, etc., can be extended to multiple dimensions. However, as the dimensionality increases, new problems and issues arise, most notably how to represent these conceptual features within a multidimensional framework. This paper provides examples of different graphical representations including item-response surfaces, information vectors, and centroid plots of conditional two-dimensional ability distributions given the number-correct score. All of these are intended to supplement quantitative and substantive analyses and thereby help the testing practitioner determine more precisely what a test is measuring, the degree of the measurement precision, and the consistency of the assessment process. Fifteen figures illustrate the analyses. (Contains 13 references.) (Author/SLD)

The concept of a user-specified validity sector is discussed. The idea of the validity sector combines the work of M. D. Reckase (1986) and R. Shealy and W. Stout (1991). Reckase developed a methodology to represent an item in a multidimensional latent space as a vector. Item vectors are computed using multidimensional item response theory item parameter estimates obtained from calibration programs such as NOHARM II (C. Fraser, 1983). The direction indicated by an item's vector indicates the composite of skills that the item is sensitive to measuring. Shealy and Stout developed a procedure (Simultaneous Item Bias) to detect item bias that encourages the user to identify (and condition on the scores from) only the most valid test items. T. A. Ackerman (1991) suggested that the most valid items for a given test could easily be identified from a plot of the item vectors. That is, the construct valid items should be measuring similar composites and thus lie within a definable sector. Items that lie outside this sector are assessing unintended-to-be-measured skills, and this could be considered construct invalid. This paper describes several uses of the validity sector, including how it can be used to construct tests, detect biased items, and define the raw score scale for a test. One table and 14 vector plots support the discussion. There is a 15-item list of references. (Author/SLD)

Test data generated according to two different multidimensional item response theory (IRT) models were compared at both the item response level and the test score level to determine whether measurable differences between the models could be detected when the data sets were constrained to be equivalent in terms of item "p"-values. The models chosen were the multiplicative multidimensional IRT model (non-compensatory) and a linear multidimensional IRT model (compensatory). Data on a 20-item test for 2,000 examinees were used. Although differences could be detected at the item level, these differences decreased as the correlation between examinee abilities increased. Furthermore, these item differences were small and could be considered unimportant or insignificant from a practical standpoint. No differences were found at the total test score level, and it was concluded that, at least for the data used in this study, the models were indistinguishable. Nine tables and eight figures present study data. (Author/SLD)

The Mirte computer program, which provides estimates of item parameters and individuals' proficiencies (abilities) based on the multidimensional two-parameter logistic (M2PL) item response theory model, is described. The program uses a modified Newton-Raphson algorithm to iteratively estimate the parameters and proficiencies. The algorithm, use of the program, and some results based on simulated datasets are described.

The multidimensional item response theory (MIRT) models with covariates proposed by Haberman and implemented in the "mirt" program provide a flexible way to analyze data based on item response theory. In this report, we discuss applications of the MIRT models with covariates to longitudinal test data to measure skill differences at the individual and group levels. In particular, we describe the differential item functioning procedure to identify common items with item drift across test occasions, and model selection and evaluation based on model comparison, fit statistics, and skill estimates. A real dataset on algebra tests is used to demonstrate the applications.

A coordinate-free definition of complex-structure multidimensional item response theory (MIRT) for dichotomously scored items is presented. The point of view taken emphasizes the possibilities and subtleties of understanding MIRT as a multidimensional extension of the classical unidimensional item response theory models. The main theorem of the paper is that every monotonic MIRT model looks the same; they are all trivial extensions of univariate item response theory.