Downloads & Free Reading Options - Results

The specification of a computerized adaptive test, like the specification of computer-assisted instruction, is easier and can be done by personnel who are not proficient in computer programming if an authoring language is provided. The Minnesota Computerized Adaptive Testing Language (MCATL) is an authoring language specifically designed for specifying adaptive tests. Its fourteen statements can be grouped into five functions: test division, administration control, scoring, reporting, and customizing. The first four categories provide statements for specifying most types of adaptive tests with minimal programming effort; the fifth category provides an interface with standard programming languages for tests that cannot be directly specified in MCATL. A formal specification of MCATL in Backus Naur Form and a practical example are provided. (Author)

This report is the Proceedings of the 1979 Computerized Adaptive Testing Conference held June 27 to 30, 1979, at the Spring Hill Conference Center in Wayzata, Minnesota. These Proceedings include 23 of the 25 papers presented at the conference, discussion of these papers by invited discussants, and symposium papers by a group of leaders in adaptive testing and latent trait test theory research and applications. The papers are organized into the following topical sessions: (1) Adaptive Testing Strategies for Measuring Ability; (2) Adaptive Testing in Germany; (3) Adaptive Mastery Testing; (4) Estimating Response Functions without Assuming a Parametric Model; (5) Item Linking and Equating; (6) Latent Trait Models which Incorporate Response Time as well as Response Appropriateness; (7) Person-Item Interaction; (8) Use of Latent Trait Models with Estimated Item Parameters; (9) Longitudinal Measurement with Latent Trait Models. Symposium and Discussion: State of the Art of Adaptive Testing and Latent Trait Test Theory.

Bias-reduced and non-bias-reduced conventional paper-and-pencil and computerized adaptive tests of word knowledge were administered to Black and White high school students to study differential effects on ability estimates and psychological reactions. Independent variables examined were bias-reduction, the presence or absence of knowledge of results after each item, mode of administration (paper-and-pencil or computerized adaptive), order of administration, and race. Dependent variables were three test performance variables (the ability estimates derived from both conventional paper-and-pencil and computerized adaptive tests, the variance of those estimates, and the number of omitted responses) and four psychological reaction variables (reaction to knowledge of results, nervousness, motivation, and guessing) the computer-administered adaptive tests appeared to reduce these differences in comparison to the conventional per-and-pencil tests. These data imply the need for further study of the effects of test administration on members of minority groups to determine those administration conditions which maximize ability estimates directly or through their effects on the psychological environment of testing.

Because scores on high-stakes tests influence many decisions, tests need to be secure. Decisions based on scores affected by preknowledge of items are unacceptable. New methods are needed to detect the new cheating strategies used for computer-administered tests because item pools are typically used over time, providing the potential opportunity for test takers to share items with future test takers. Because of the serious ramifications of accusing someone as being a user of item preknowledge (or "cheater"), it may be more useful for operational computer-administered test developers to focus on item security rather than the behavior of individual test takers. This research explores the development and use of a fit index to detect items that have been memorized so that these items may be removed from the item pool, while leaving secure items in the pool. The results from this initial simulation for the developed Bayesian posterior log odds ratio index are promising. It is hoped that this work and future work will enable testing programs to determine more effectively how long to leave an item pool (or specific items) in the field. (Contains four tables, five figures, and five references.) (Author/SLD)

Test security has often been a problem in computerized adaptive testing (CAT) because the traditional wisdom of item selection overly exposes high discrimination items. The a-stratified (STR) design advocated by H. Chang and his collaborators, which uses items of less discrimination in earlier stages of testing, has been shown to be very successful in balancing and hence maximizing the usage of all items in the pool. However, under specific conditions such as the early stages of utilizing completely new item pools, it is possible that the STR strategy is slightly less efficient than the most widely used maximum information (Max-I) approach. In this series of simulation studies with variable length CAT in which testing terminates at a targeted test information, researchers examined whether the use of more items in STR to attain accuracy similar to the Max-I approach in ability estimation would result in a greater exposure of all items. Simulations with self-generated items as well as an operational pool support the usefulness of the STR method in general. However, the results suggest that it is desirable to have fewer in number but less discriminating items at earlier stages of testing and have a larger number of highly discriminating items at later stages. The limitations of this study and implications for future studies are discussed. (Contains 6 tables, 8 figures, and 12 references.) (Author/SLD)

Computerized adaptive tests (CATs) pose major obstacles to the traditional assessment of differential item functioning (DIF). This paper proposes a modification of the SIBTEST DIF procedure for CATs, called CATSIB. CATSIB matches test takers on estimated ability based on unidimensional item response theory. To control for impact-induced Type I error inflation, the SIBTEST regression correction is shown to have an easily implemented and theoretically justified counterpart in the CAT setting. A simulation study was conducted to evaluate the performance of CATSIB. Simulated test takers were adaptively administered 25 simulated operational items from a pool of 1,000 and were linearly administered 16 simulated pretest items that were evaluated for DIF. The pretest items were designed to represent varying levels of discrimination, difficulty, and amounts of DIF. Sample size varied from 250 to 500 in each group. Simulated levels of impact ranged from 0 to 1 standard deviations difference in mean ability levels. Results show that CATSIB with the regression correction displays impact-induced Type I error inflation. In terms of power, even with as few as 250 test takers in each group, CATSIB had detection rates of 64% or greater for large values of DIF. When sample size was increased to 500 in each group, these power rates increased to more than 90%. CATSIB displayed nearly unbiased estimation under nearly all the simulated conditions. (Contains 7 tables, 1 figure, and 11 references.) (Author/SLD)

The relationship between several individual difference variables and test performance using computerized adaptive testing (CAT) was studied with 208 South Korean college students. The general individual difference variable selected was test anxiety, and the math-related individual difference variables were mathematics aptitude, math test anxiety, and math self-concept. Computer-related individual difference variables were computer literacy, computer anxiety, and computer experience. The CAT was an algebra test. Test anxiety and other individual difference variables were measured with instruments translated into Korean for the study. The seven difference variables and their squares were submitted to an analysis using the RSQUARE Procedure from Statistical Analysis System software. Math aptitude, math self-concept, math test anxiety, and computer literacy, in that order, were significantly related to test performance using CAT, but test anxiety, computer experience, and computer anxiety were not. Results suggest that the relationship between individual difference variables and test performance using CAT may be curvilinear. Two tables present analysis results. (Contains 27 references.) (SLD)

Includes bibliographical references (p. 273-286) and indexes

This thesis is a case study of mental testing in the military as it applies to mental qualifications for service. The thesis begins with a review of the literature concerning the history of mental testing, particularly in the military services, through the current Armed Services Vocational Aptitude Battery (ASVAB) forms -8, -9, and -10. Then, a discussion of issues facing mental testing in general is presented, followed by a report of research into Computerized Adaptive Testing (CAT) currently conducted at the Navy Personnel Research and Development Center, San Diego. Finally, a concluding chapter discusses some considerations involved in the implementation of CAT.

In 1979, the Computerized Adaptive Testing version of the Armed Services Vocational Aptitude Battery (CAT-ASVAB) program was initiated by the Department of Defense. One objective of the (Joint-Service) CAT-ASVAB program was to develop a distributed processing computer system capable of deploying CAT at aptitude testing sites of the United States Military Entrance Processing Command (USMEPCOM). In 1985, the CAT-ASVAB program was redirected with the initiation of the Accelerated CAT-ASVAB Project (ACAP). The purpose of ACAP was to test the feasibility of using CAT at USMEPCOM testing sites prior to initiation of another full scale development effort. To accomplish this purpose, ACAP bad to develop a computer system with all of the essential Joint-Service specifications for deploying CAT. The UNIX-based computer system developed under ACAP meets this goal.

The Accelerated CAT-ASVAB Project (ACAP) may serve as the pilot version for national implementation of computerized adaptive testing (CAT) for the Armed Services Vocational Aptitude Battery (ASVAB). Two major decisions in ACAP involve the introduction of new items into the tests, and setting time limits. This Research Memorandum takes the position that the long-term benefits which CAT may provide are more important than purely technical concerns and makes recommendations based on this position. Keywords: Naval personnel, Recruits.

This study explored the utility, from a psychometric and cost effectiveness standpoint, of a computerized adaptive measurement system in an Air Force technical training environment. The phase 1 effort was designed to take the study to the point of producing an operational system ready to actually test technical training students adaptively. A literature review indicated that two testing techniques showed considerable promise: flexilevel testing and hierarchical testing; two courses were selected to implement these procedures. One preliminary conclusion was that adaptive testing appears to offer the potential for time savings of up to 50%. Furthermore, it was found that a very flexible computer system to drive the adaptive testing strategies could be relatively easily developed. (WH)

Overexposure and underexposure of items in the bank are serious problems in operational computerized adaptive testing (CAT) systems. These exposure problems might result in item compromise, or point at a waste of investments. The exposure control problem can be viewed as a test assembly problem with multiple objectives. Information in the test has to be maximized, item compromise has to be minimized, and pool usage has to be optimized. In this paper, a multiple objectives method is developed to deal with both types of exposure problems. In this method, exposure control parameters based on observed exposure rates are implemented as weights for the information in the item selection procedure. The method does not need time consuming simulation studies, and it can be implemented conditional on ability level. The method is compared with Sympson Hetter method for exposure control, with the Progressive method and with alpha-stratified testing. The results show that the method is successful in dealing with both kinds of exposure problems. (Contains 3 figures and 3 tables.)

The degree to which item review on a computerized adaptive test (CAT) could be used by examinees to inflate their scores artificially was studied. G. G. Kingsbury (1996) described a strategy in which examinees could use the changes in item difficulty during a CAT to determine which items' answers are incorrect and should be changed during item review. In CAT, a correctly answered item will typically be followed by a more difficult item, while an incorrectly answered item will typically be followed by an easier item. The results of two studies involving groups of 77 and 62 undergraduates suggest that examinees are not highly proficient at discriminating item difficulty, a skill needed for successful application of the Kingsbury strategy. In the third study, which used 243 introductory statistics students, the Kingsbury strategy, which examinees would use only for guessed items, was compared to a generalized strategy used for all sequential item pairs. The Kingsbury strategy yielded a small average score gain, while the generalized strategy yielded an average score loss. These results suggest that only the Kingsbury strategy would enable examinees to inflate their scores successfully. (Contains 2 tables, 3 figures, and 14 references.) (Author/SLD)

Problem Statement: Computerized adaptive testing (CAT) is a sophisticated and efficient way of delivering examinations. In CAT, items for each examinee are selected from an item bank based on the examinee's responses to the items. In this way, the difficulty level of the test is adjusted based on the examinee's ability level. Instead of administering very long tests, CAT can estimate examinees' ability levels with a small number of items. A number of operational testing programs have implemented CAT during the last decade. However, CAT hasn't been applied to any operational test in Turkey, where there are several standardized assessments taken by millions of people every year. Therefore, this study investigates the applicability of CAT to a high-stakes test in Turkey. Purpose of Study: The purpose of this study is to examine the applicability of CAT procedure to the Entrance Examination for Graduate Studies (EEGS), which is used in selecting students for graduate programs in Turkish universities. Methods: In this study, post-hoc simulations were conducted using real responses from examinees. First, all items in EEGS were calibrated using the three-parameter item response theory (IRT) model. Then, ability estimates were obtained for all examinees. Using the item parameters and responses to EEGS, post-hoc simulations were run to estimate abilities in CAT. Expected A Posteriori (EAP) method was used for ability estimation. Test termination rule was standard error of measurement for estimated abilities. Findings and Results: The results indicated that CAT provided accurateability estimates with fewer items compared to the paper-pencil format of EEGS. Correlations between ability estimates from CAT and the real administration of EEGS were found to be 0.93 or higher under all conditions. Average number of items given in CAT ranged from 9 to 22. The number of items given to the examinees could be reduced by up to 70%. Even with a high SEM termination criterion, CAT provided very reliable ability estimates. EAP was the best method among several ability estimates methods (e.g., MAP, MLE, etc.). Conclusions and Recommendations: CAT can be useful in administering EEGS. With a large item bank, EEGS can be administered to examinees in a reliable and efficient way. The use of CAT can help to minimize the cost of the test since test booklets, examinee response sheets, etc. won't be needed anymore. It can also help to prevent cheating during the test.

Problem Statement: Computerized adaptive testing (CAT) is a sophisticated and efficient way of delivering examinations. In CAT, items for each examinee are selected from an item bank based on the examinee's responses to the items. In this way, the difficulty level of the test is adjusted based on the examinee's ability level. Instead of administering very long tests, CAT can estimate examinees' ability levels with a small number of items. A number of operational testing programs have implemented CAT during the last decade. However, CAT hasn't been applied to any operational test in Turkey, where there are several standardized assessments taken by millions of people every year. Therefore, this study investigates the applicability of CAT to a high-stakes test in Turkey. Purpose of Study: The purpose of this study is to examine the applicability of CAT procedure to the Entrance Examination for Graduate Studies (EEGS), which is used in selecting students for graduate programs in Turkish universities. Methods: In this study, post-hoc simulations were conducted using real responses from examinees. First, all items in EEGS were calibrated using the three-parameter item response theory (IRT) model. Then, ability estimates were obtained for all examinees. Using the item parameters and responses to EEGS, post-hoc simulations were run to estimate abilities in CAT. Expected A Posteriori (EAP) method was used for ability estimation. Test termination rule was standard error of measurement for estimated abilities. Findings and Results: The results indicated that CAT provided accurateability estimates with fewer items compared to the paper-pencil format of EEGS. Correlations between ability estimates from CAT and the real administration of EEGS were found to be 0.93 or higher under all conditions. Average number of items given in CAT ranged from 9 to 22. The number of items given to the examinees could be reduced by up to 70%. Even with a high SEM termination criterion, CAT provided very reliable ability estimates. EAP was the best method among several ability estimates methods (e.g., MAP, MLE, etc.). Conclusions and Recommendations: CAT can be useful in administering EEGS. With a large item bank, EEGS can be administered to examinees in a reliable and efficient way. The use of CAT can help to minimize the cost of the test since test booklets, examinee response sheets, etc. won't be needed anymore. It can also help to prevent cheating during the test.

Problem Statement: Computerized adaptive testing (CAT) is a sophisticated and efficient way of delivering examinations. In CAT, items for each examinee are selected from an item bank based on the examinee's responses to the items. In this way, the difficulty level of the test is adjusted based on the examinee's ability level. Instead of administering very long tests, CAT can estimate examinees' ability levels with a small number of items. A number of operational testing programs have implemented CAT during the last decade. However, CAT hasn't been applied to any operational test in Turkey, where there are several standardized assessments taken by millions of people every year. Therefore, this study investigates the applicability of CAT to a high-stakes test in Turkey. Purpose of Study: The purpose of this study is to examine the applicability of CAT procedure to the Entrance Examination for Graduate Studies (EEGS), which is used in selecting students for graduate programs in Turkish universities. Methods: In this study, post-hoc simulations were conducted using real responses from examinees. First, all items in EEGS were calibrated using the three-parameter item response theory (IRT) model. Then, ability estimates were obtained for all examinees. Using the item parameters and responses to EEGS, post-hoc simulations were run to estimate abilities in CAT. Expected A Posteriori (EAP) method was used for ability estimation. Test termination rule was standard error of measurement for estimated abilities. Findings and Results: The results indicated that CAT provided accurateability estimates with fewer items compared to the paper-pencil format of EEGS. Correlations between ability estimates from CAT and the real administration of EEGS were found to be 0.93 or higher under all conditions. Average number of items given in CAT ranged from 9 to 22. The number of items given to the examinees could be reduced by up to 70%. Even with a high SEM termination criterion, CAT provided very reliable ability estimates. EAP was the best method among several ability estimates methods (e.g., MAP, MLE, etc.). Conclusions and Recommendations: CAT can be useful in administering EEGS. With a large item bank, EEGS can be administered to examinees in a reliable and efficient way. The use of CAT can help to minimize the cost of the test since test booklets, examinee response sheets, etc. won't be needed anymore. It can also help to prevent cheating during the test.

Problem Statement: Computerized adaptive testing (CAT) is a sophisticated and efficient way of delivering examinations. In CAT, items for each examinee are selected from an item bank based on the examinee's responses to the items. In this way, the difficulty level of the test is adjusted based on the examinee's ability level. Instead of administering very long tests, CAT can estimate examinees' ability levels with a small number of items. A number of operational testing programs have implemented CAT during the last decade. However, CAT hasn't been applied to any operational test in Turkey, where there are several standardized assessments taken by millions of people every year. Therefore, this study investigates the applicability of CAT to a high-stakes test in Turkey. Purpose of Study: The purpose of this study is to examine the applicability of CAT procedure to the Entrance Examination for Graduate Studies (EEGS), which is used in selecting students for graduate programs in Turkish universities. Methods: In this study, post-hoc simulations were conducted using real responses from examinees. First, all items in EEGS were calibrated using the three-parameter item response theory (IRT) model. Then, ability estimates were obtained for all examinees. Using the item parameters and responses to EEGS, post-hoc simulations were run to estimate abilities in CAT. Expected A Posteriori (EAP) method was used for ability estimation. Test termination rule was standard error of measurement for estimated abilities. Findings and Results: The results indicated that CAT provided accurateability estimates with fewer items compared to the paper-pencil format of EEGS. Correlations between ability estimates from CAT and the real administration of EEGS were found to be 0.93 or higher under all conditions. Average number of items given in CAT ranged from 9 to 22. The number of items given to the examinees could be reduced by up to 70%. Even with a high SEM termination criterion, CAT provided very reliable ability estimates. EAP was the best method among several ability estimates methods (e.g., MAP, MLE, etc.). Conclusions and Recommendations: CAT can be useful in administering EEGS. With a large item bank, EEGS can be administered to examinees in a reliable and efficient way. The use of CAT can help to minimize the cost of the test since test booklets, examinee response sheets, etc. won't be needed anymore. It can also help to prevent cheating during the test.

A nominal response model-based computerized adaptive testing procedure (nominal CAT) was implemented using simulated data. Ability estimates from the nominal CAT were compared to those from a CAT based upon the three-parameter logistic model (3PL CAT). Furthermore, estimates from both CAT procedures were compared with the known true abilities used to generate the simulated data. Results showed that the nominal CAT's ability estimates were highly correlated with those of the 3PL CAT as well as with the true abilities. Furthermore, the nominal CAT had a significantly higher association with negative true thetas than did the 3PL CAT, and it also had significantly lower standard errors of estimate than did the 3PL CAT. However, the nominal model-based CAT had difficulty estimating positive thetas and had a poor convergence rate. In contrast, the 3PL CAT had a high convergence rate and its performance was not affected by whether the true abilities were positive or negative. Potential reasons for the nominal CAT's high nonconvergence rate as well as implications for computerized adaptive testing were discussed. (Author)

Thesis (M.S. in Management)--Naval Postgraduate School, 1980

The goal of this paper is to present and analyse the potential of E-learning and computerized adaptive testing, as a part of it, within a corporate context. Over the years, technological growth led to creation of learning organizations and knowledge economy so the solution for constant and cost-effective training of staff had to be found. That is where E-learning proved to be one of the most efficient solutions in a business organization. In the paper, features and segments of corporate E-learning will be presented. Meanwhile, fundamentals of computerized adaptive testing, item response theory and algorithm procedure are going to be described. Finally, potential of computerized adaptive testing used in determining employees knowledge and skills development is going to be discussed.

Two distinct approaches, one based on item response theory and the other based on observed item responses and standard summary statistics, have been proposed to identify unusual response patterns. A link between these two approaches is provided by showing certain correspondences between Sato's S-P Curve Theory and item response theory. This link makes possible several extensions of Sato's caution index that take advantage of the results of item response theory. Several such indices are introduced and their use illustrated by application to a set of achievement test data. Two of the newly introduced extended indices were found to be very effective for purposes of identifying persons who consistently use an erroneous rule in attempting to solve signed number arithmetic problems. The potential importance of this result is briefly discussed, and 15 references are listed. (Author/LLS)

Multilog (Thissen, 1991) was used to estimate parameters of 225 items from the Quebec Adaptive Behavior Scale (QABS). A database containing actual data from 2,439 subjects was used for the parameterization procedures. The two-parameter-logistic model was used in estimating item parameters and in the testing strategy. MicroCAT (Assessment Systems Corporation, 1989) was then used to manage the item banks and Computerized Adaptive Testing (CAT) environment during a simulation run using data from a randomly selected sample of 200 subjects taken from the larger data base. The simulation of the QABS-CAT testing indicates that levels of ability can be estimated for each of the seven skill domains by using only 30% of the items of the conventional version. The numerous advantages of item response theory and CAT as applied to the assessment of adaptive behavior with regard to the changing definition of mental retardation (Luckasson et al., 1992) are discussed. (Author/SLD)

Among the changes in education called for under the No Child Left Behind act is the need for states to test students in a number of grades and subject areas. Scores from these tests are to be used for a variety of purposes, from identifying whether individual students are proficient, to helping determine whether schools are causing adequate growth for their students. This study investigates several testing approaches, and their potential impact on the use of test scores for these varied purposes. (Contains 1 table and 2 figures.)

The purpose of this paper is to derive optimal rules for variable-length mastery tests in case three mastery classification decisions (nonmastery, partial mastery, and mastery) are distinguished. In a variable-length or adaptive mastery test, the decision is to classify a subject as a master, a partial master, a nonmaster, or continuing sampling and administering another test item. The framework of minimax sequential decision theory is used; that is, optimal sequential rules minimizing the maximum expected losses associated with all possible decision rules. The binomial model is assumed for the conditional probability of a correct response given the true level of functioning, whereas the threshold loss is adopted for the loss function involved. Monotonicity conditions are derived, that is, conditions sufficient for optimal sequential rules to be in the form of cutting scores. The paper concludes with an empirical example of a computerized adaptive mastery test for concept-learning in medicine. (Contains 4 tables and 63 references.) (Author)

As lead laboratory in a joint-service project, NAVPERSRANDCEN is contracting for design, development, testing, and evaluation of a system for automated adaptive administration of military personnel selection tests. This report describes the planned contracting approach and system requirements for developing a computerized adaptive testing system (CAT). (Author)

This paper examines some characteristics of the statistics associated with the sampling distribution of the proficiency level estimate when the Rasch model is used. These characteristics allow the judgment of the meaning to be given to the proficiency level estimate obtained in adaptive testing, and as a consequence, they can illustrate the meaningfulness of the proficiency level estimate. A Monte Carlo method computer-based simulation was used to investigate these characteristics. Findings show that with the Rasch model, and the expected a posteriori estimation method, if the theoretical distribution of the proficiency level is postulated as normal, then the stopping rule according to the number of items administered should be applied only if at least 13 items are administered. In general, it is preferable not to use the stopping rule according to the standard error with a standard error retained above 0.40. The results also suggest the application of the correction developed by R. Bock and R. Mislevy no matter which stopping rule is used, to reduce the bias of the proficiency level estimate over the entire proficiency range. (Contains 2 tables and 24 references.) (SLD)

This paper examines some characteristics of the statistics associated with the sampling distribution of the proficiency level estimate when the Rasch model is used. These characteristics allow the judgment of the meaning to be given to the proficiency level estimate obtained in adaptive testing, and as a consequence, they can illustrate the meaningfulness of the proficiency level estimate. A Monte Carlo method computer-based simulation was used to investigate these characteristics. Findings show that with the Rasch model, and the expected a posteriori estimation method, if the theoretical distribution of the proficiency level is postulated as normal, then the stopping rule according to the number of items administered should be applied only if at least 13 items are administered. In general, it is preferable not to use the stopping rule according to the standard error with a standard error retained above 0.40. The results also suggest the application of the correction developed by R. Bock and R. Mislevy no matter which stopping rule is used, to reduce the bias of the proficiency level estimate over the entire proficiency range. (Contains 2 tables and 24 references.) (SLD)

This paper examines some characteristics of the statistics associated with the sampling distribution of the proficiency level estimate when the Rasch model is used. These characteristics allow the judgment of the meaning to be given to the proficiency level estimate obtained in adaptive testing, and as a consequence, they can illustrate the meaningfulness of the proficiency level estimate. A Monte Carlo method computer-based simulation was used to investigate these characteristics. Findings show that with the Rasch model, and the expected a posteriori estimation method, if the theoretical distribution of the proficiency level is postulated as normal, then the stopping rule according to the number of items administered should be applied only if at least 13 items are administered. In general, it is preferable not to use the stopping rule according to the standard error with a standard error retained above 0.40. The results also suggest the application of the correction developed by R. Bock and R. Mislevy no matter which stopping rule is used, to reduce the bias of the proficiency level estimate over the entire proficiency range. (Contains 2 tables and 24 references.) (SLD)

After a brief review of recent advances in sequential analysis involving sequential generalized likelihood ratio tests, we discuss their use in psychometric testing and extend the asymptotic optimality theory of these sequential tests to the case of sequentially generated experiments, of particular interest in computerized adaptive testing. We then show how these methods can be used to design adaptive mastery tests, which are asymptotically optimal and are also shown to provide substantial improvements over currently used sequential and fixed length tests.

Two-stage and multistage test designs provide a way of roughly adapting item difficulty to test-taker ability. All test takers take a parallel stage-one test, and, based on their scores, they are routed to tests of different difficulty levels in subsequent stages. These designs provide some of the benefits of standard computerized adaptive testing (CAT), such as increased precision of ability estimates over a paper-and-pencil test design. The item selection and scoring algorithms in two-stage and multistage designs may also be easier for test takers and test score users to understand--an important feature for gaining public acceptance of new test designs. This study incorporates testlets (bundles of items) into two-stage and multistage designs, and compares the precision of the ability estimates derived from these designs with those derived from a standard CAT design and from paper-and-pencil test designs. For the group that was used to establish the cutoffs for the two-stage and multistage testlet designs, 50,000 simulated test takers were created randomly. The group of simulated test takers used to simulate all test designs totaled 25,000. Results indicate that all testlet-based designs resulted in improved precision over the same-length paper-and-pencil test, and almost as much precision as the paper-and-pencil test of double length. Given the many other (nonpsychometric) advantages of these designs, they may be viable options for computer-administered tests. (Contains six figures and nine references.) (Author/SLD)

Recently, several person-fit statistics have been proposed to detect nonfitting response patterns. This study is designed to generalize an approach followed by Klauer (1995) to an adaptive testing system using the two-parameter logistic model (2PL) as a null model. The approach developed by Klauer is described, and some difficulties in generalizing it to a computerized adaptive testing model are explored. Alternative approaches are presented, and the results of a power study and the consequences for person-fit measurement in adaptive testing situations are discussed. The first part of the simulation concerns the ability to ignore the process that causes missing data and the impact of the fact that data in adaptive testing are not observed at random. Then the power of the proposed three tests is studied. Results suggest that tests against local independence in the 2PL have little power, but that the power for testing against nonvariant abilities is relatively larger but still low. (Contains 4 tables and 18 references.) (SLD)

Recently, several person-fit statistics have been proposed to detect nonfitting response patterns. This study is designed to generalize an approach followed by Klauer (1995) to an adaptive testing system using the two-parameter logistic model (2PL) as a null model. The approach developed by Klauer is described, and some difficulties in generalizing it to a computerized adaptive testing model are explored. Alternative approaches are presented, and the results of a power study and the consequences for person-fit measurement in adaptive testing situations are discussed. The first part of the simulation concerns the ability to ignore the process that causes missing data and the impact of the fact that data in adaptive testing are not observed at random. Then the power of the proposed three tests is studied. Results suggest that tests against local independence in the 2PL have little power, but that the power for testing against nonvariant abilities is relatively larger but still low. (Contains 4 tables and 18 references.) (SLD)

Item-exposure control in computerized adaptive testing is implemented by imposing item-ineligibility constraints on the assembly process of the shadow tests. The method resembles J. Sympson and R. Hetter's (1985) method of item-exposure control in that the decisions to impose the constraints are probabilistic. However, the method does not require time consuming simulation studies to set values for control parameters prior to the operational use of the test. Instead, the probabilities of item ineligibility can be set "on the fly" using an adaptive procedure based on the actual item-exposure rates. An empirical study using an item pool from the Law School Admission Test showed that application of the method yielded perfect control of the item exposure rates and had negligible impact on the bias and mean squared error functions of the ability estimates. (Contains 5 figures and 26 references.) (Author/SLD)

This paper discusses the minimum system requirements needed to develop a computerized adaptive test (CAT). It lists some of the benefits of adaptive testing, establishes a set of operational constraints, and reviews both software and hardware requirements based on those operational constraints. An experimental CAT system that is currently in use is reviewed in detail.

Most large-scale testing programs facing computerized adaptive testing (CAT) must face the challenge of maintaining extensive content requirements, but content constraints in computerized adaptive testing (CAT) can compromise the precision and efficiency that could be achieved by a pure maximum information adaptive testing algorithm. This simulation study first evaluated whether realistic content constraints could be met by carefully assembling testlets and appropriately selecting testlets for each test taker that, when combined, would meet the content requirements of the test and would be adapted to the test takers ability level. The second focus of the study was to compare the precision of the content-balanced testlet design with that achieved by the current paper-and-pencil version of the test through data simulation. The results reveal that constraints to control for item exposure, testlet overlap, and efficient pool utilization need to be incorporated into the testlet assembly algorithm. More refinement of the statistical constraints for testlet assembly is also necessary. However, even for this preliminary attempt at assembling content-balanced testlets, the two-stage computerized test simulated with these testlets performed quite well. (Contains 5 figures, 5 tables, and 12 references.) (Author/SLD)

A model for constrained computerized adaptive testing is proposed in which the information on the test at the ability estimate is maximized subject to a large variety of possible constraints on the contents of the test. At each item-selection step, a full test is first assembled to have maximum information at the current ability estimate fixing the items previously administered. Then the item with maximum information is selected from the test. All test assembly is optimal due to the use of a linear programming model that is automatically updated to allow for the attributes of the items already administered as well as the new value of the ability estimator. A simulation study using a pool of 753 items from the Law School Admission Test (LSAT) shows that for adaptive tests of realistic lengths the ability estimator did not suffer any loss of efficiency from the presence of 433 constraints on the item selection process. (Contains 2 figures, 3 tables, and 35 references.) (Author/SLD)

This brief overview notes that an adaptive test differs from standardized achievement tests in that it does not consist of a certain set of items that are administered to a group of examinees. Instead, the test is individualized for each examinee. The items administered to the examinee are selected from a large pool of items on the basis of the correctness of the examinee's previous responses. Several steps are involved in constructing a computerized adaptive test (CAT): (1) the item pool must be developed; (2) some item characteristics must be calculated for each item; and (3) a strategy for deleting items from the item pool must be chosen, along with a scoring procedure. Latent trait theory assumes that an underlying trait is the sole determinant of an examinee's responses to test items, and that performance on an item is governed by two factors: the examinee's ability and the item's difficulty. Several item selection strategies have been developed, including the Robbins Monro procedure, fixed step size, the flexilevel method, the Bayesian approach, and the stratified-adaptive approach. Advantages of CAT are its increased efficiency, optimization of the difficulty level of the test for each examinee, and administration convenience. The most obvious constraint on the use of CAT is the problem of availability of appropriate computer hardware and software. (LMO)

Research on the major computerized adaptive testing (CAT) strategies is reviewed, and some findings are reported that examine effects of examinee demographic and psychological characteristics on CAT strategies. In fixed branching strategies, all examinees respond to a common routing test, the score of which is used to assign examinees to a second-stage test. The currently popular statistically branched adaptive strategies are based on item-response theory, and include maximum likelihood strategy and Bayesian strategy. Two alternative strategies are the use of self-adapted testing and testlet strategies. Examinee characteristic variables are divided into: (1) demographic variables; (2) computer-use variables; (3) test-taking strategy variables; (4) cognitive characteristics; and (5) affective characteristics. Although research on the relationship between examinee psychological characteristics and CAT has been inconclusive, the basic findings are that examinees of different ethnic, gender, age, grade, ability, academic self-concept, test anxiety, computer anxiety, math anxiety, and computer experience groups are differentially affected by the adaptive testing strategies. Implications for research and practice are discussed. (Contains 67 references.) (SLD)

The purpose of the study was to investigate the effects of test motivation on estimated ability, test anxiety, and attitudes toward computerized adaptive testing (CAT). Korean college students (n=208) were given the Math Aptitude Test, Math Self-Concept Scale, Math Test Anxiety Scale, Computer Competence Instrument, Computer Anxiety Scale, and Test Anxiety Inventory in the regular classroom. The two groups (motivated and non-motivated) were randomly assigned by each course section. The motivated group was given special test instructions. The paper-and-pencil test (PPT) and the CAT algebra tests were given to each group in random order (PPT-CAT or CAT-PPT) under the counterbalanced design at the computer laboratory. They were also given a 10-item paper test anxiety scale, a 10-item computer test anxiety scale, and a paper-and-pencil version of the Questionnaire on Computerized Adaptive Testing. A multivariate analysis of covariance, with the math aptitude and the test anxiety as covariates, demonstrated that test motivation influenced improvement in estimated ability and reduction in test anxiety, but did not affect CAT attitudes. (Contains 4 tables and 45 references.) (Author)

The principal objectives of this conference were to exchange information, discuss theoretical and empirical developments, and to coordinate research efforts. The papers and their authors are: "The Graded Response Model of Latent Trait Theory and Tailored Testing" by Fumiko Samejima; (Incomplete Orders and Computerized Testing" by Norman Cliff; "Adaptive Testing Research at Minnesota: Overview, Recent Results, and Future Directions" by David J. Weiss; "Adaptive Testing Research at Minnesota: Some Properties of a Bayesian Sequential Adaptive Mental Testing Strategy" by James R. McBride; "An Empirical Investigation of Weiss' Stradaptive Testing Model" by Brian K. Waters; "Using Computerized Tests to Add New Dimensions to the Measurement of Abilities Which Are Important for On-Job Performance: An Exploratory Study" by Charles H. Cory; "A Broad-Range Tailored Test of Verbal Ability" and "Some Likelihood Functions found in Tailored Testing" by Frederic M. Lord; "Bayesian Tailored Testing and the Influence of Item Bank Characteristics" by Carl J. Jensema; "Reflections on Adaptive Testing" by Duncan N. Hansen; "Computer-Assisted Testing: An Orderly Transition from Theory to Practice" by Richard H. McKillip and Vern W. Urry; "Five Years of Research: Is Computer-Assisted Testing Feasibility?" by Vern W. Urry; "Effectiveness of the Ancillary Estimation Procedure" by John F. Gugel, Frank L. Schmidt, and Vern W. Urry; and "Item Parameterization Procedures for the Future" by Frank L. Schmidt and Urry. (Author/DEP)

The interest in the application of large-scale adaptive testing for secure tests has served to focus attention on issues that arise when theoretical advances are made operational. Many such issues in the application of large-scale adaptive testing for secure tests have more to do with changes in testing conditions than with testing paradigms. One such issue is that of insuring item and pool security in the continuous testing environment made possible by the computerized administration of a test, as opposed to the more periodic testing environment typically used for linear paper-and-pencil tests. In the continuous testing environment of adaptive testing, methods have been developed in the past to use the computer to control the rate at which particular items are exposed to test-takers. These methods have typically employed randomization schemes, sometimes in reference to a particular target distribution of test-taker ability. This paper presents a new multinomial method of controlling the exposure rate of items conditional on the ability level of an individual test-taker. The properties of such conditional control on the exposure rates of items, when used in conjunction with a particular adaptive testing algorithm, are explored through five studies with simulated data. (Contains 1 table, 7 figures, and 17 references.) (Author)

Computerized adaptive testing (CAT) has unsurpassable advantages over traditional testing. It has become the mainstream in large scale examinations in modern society. This paper gives a brief introduction to CAT including differences between traditional testing and CAT, the principles of CAT, psychometric theory and computer algorithms of CAT, the advantages and cautions of CAT. In the end, the development of CAT in China is reviewed. (Contains 1 figure and 1 table.)

Many large-scale testing programs routinely pretest new items alongside operational (or scored) items to determine their empirical characteristics. If these pretest items pass certain statistical criteria, they are placed into an operational item pool; otherwise they are edited and re-pretested or simply discarded. In these situations, reliable ability estimates are usually available for each examinee based on operational items, and they may be treated as fixed. If so, polynomial (in ability, theta) logistic regression analyses can be conducted using a variety of statistical software packages. In this study, a cubic logistic model (theta, theta-2, theta-3) was found to fit standard three-parameter (i.e. discrimination, difficulty, and lower asymptote) logistic item response theory (IRT) model items very well. When employing a polynomial logistic model, well-known selection routines (such as stepwise elimination) can be utilized to reduce the number of required parameters for certain items, thus reducing the sample sizes needed for reliable estimation. With this model, simultaneous confidence bands are easily calculated. As an added benefit, given that a polynomial logistic function is not necessarily monotonically increasing with ability, poor quality items and incorrect alternative responses can also be fit using the same estimation procedures. (Contains 19 figures, 4 tables, and 22 references.) (Author/SLD)

This paper presents a derivation of an average between-test overlap index as a function of the item exposure index, for fixed-length computerized adaptive tests (CAT). This relationship is used to investigate the simultaneous control of item exposure at both the item and test levels. Implications for practice as well as future research are also discussed. An appendix demonstrates that the expected value of the between-test overlap for a fixed-length CAT under completely randomized item selection, is equal to the fixed test length divided by the item pool size. (Contains 5 tables, 2 figures, and 12 references.) (Author/SLD)

In this paper we follow our previous research in the area of Computerized Adaptive Testing (CAT). We present three different methods for CAT. One of them, the item response theory, is a well established method, while the other two, Bayesian and neural networks, are new in the area of educational testing. In the first part of this paper, we present the concept of CAT and its advantages and disadvantages. We collected data from paper tests performed with grammar school students. We provide the summary of data used for our experiments in the second part. Next, we present three different model types for CAT. They are based on the item response theory, Bayesian networks, and neural networks. The general theory associated with each type is briefly explained and the utilization of these models for CAT is analyzed. Future research is outlined in the concluding part of the paper. It shows many interesting research paths that are important not only for CAT but also for other areas of artificial intelligence.

A two-stage testing strategy is one method of adapting the difficulty of a test to an individual's ability level in an effort to achieve more precise measurement. A routing test provides an initial estimate of ability level, and a second-stage measurement test then evaluates the examinee further. The measurement accuracy and efficiency of item response theory (IRT) based two-stage testing was investigated in comparison with an individualized computerized adaptive test (CAT). Eighteen simulated two-stage tests and three fixed-length CATs differing in the number of test items administered were compared. Abilities were generated for a sample of 1,600 simulees. Results indicate that the statistical characteristics of the routing test have a major influence on measurement precision in ability estimation. Overall, it was apparent that a fixed-length CAT is superior to the two-stage tests of equivalent length in terms of measurement accuracy and efficiency. IRT-based two-stage tests using rectangular distribution of item difficulties in the routing test and an odd number of second-stage tests produced more accurate theta estimates than did the other two-stage test configurations studied. IRT-based two-stage tests may sometimes be practical alternatives to CAT, considering its limitations. Two tables and 18 graphs present analysis results. (SLD)

Computerized adaptive testing for language teaching and learning takes advantage of two properties of the computer: its number-crunching and multiple-branching capabilities. Adaptive testing has also been called tailored testing because it aims at presenting items that suit the student's competence and that are informative, using an item bank and an algorithm for selecting items. The most widely used theoretical framework for these components is item response theory, which allows for the measurement of the trait that corresponds to the subject's ability. Creation of the item bank includes planning, field testing, item analysis, calibration, and item inclusion. Test bank management requires additional features for updating, importing items, listing items, and obtaining item information. The item selection procedure estimates the examinee's ability after an answer and finds the next item that is most appropriate. Test administration involves gathering information about the examinee, a self-rating of general proficiency, and sub-tests using the item selection procedure. Results are available immediately, an advantage appreciated by examinees and administrators. The adaptive procedure allows for shorter tests, and the items are never too difficult or too easy. Limitations include the artificiality of the computer environment, restriction of answer type and test content, non-applicability for small-scale testing, the theoretical assumption of unidimensionality, and cost. (MSE)

The Sympson and Hetter (SH; J. Sympson and R. Hetter; 1985; 1997) method is a method of probabilistic item exposure control in computerized adaptive testing. Setting its control parameters to admissible values requires an iterative process of computer simulations that has been found to be time consuming, particularly if the parameters have to be set conditional on a realistic set of values for the examinees ability parameter. Formal properties of the method are identified that help us explain why this iterative process can be slow and does not guarantee admissibility. In addition, some alternatives to the SH method are introduced. The behavior of these alternatives was estimated for an adaptive test from an item pool from the Law School Admission Test. Two of the alternatives showed attractive behavior and converged smoothly to admissibility for all items in a relatively small number of iteration steps. (Contains 4 figures and 14 references.) (Author/SLD)

Expert systems can be used to aid decisionmaking. A computerized adaptive test is one kind of expert system, although not commonly recognized as such. A new approach, termed EXSPRT, was devised that combines expert systems reasoning and sequential probability ratio test stopping rules. Two versions of EXSPRT were developed, one with random selection of items (EXSPRT-R) and one with intelligent selection (EXSPRT-I). Two empirical studies were conducted in which these two new methods were compared to the traditional SPRT and to an adaptive mastery testing (AMT) approach based on item response theory (IRT). The EXSPRT-I tended to be more efficient than the AMT, EXSPRT-R, and SPRT models in terms of average test lengths. Although further research is needed, the EXSPRT-I initially appears to be a strong alternative to both IRT- and SPRT-based adaptive testing when categorical decisions about examinees are desired. The EXSPRT-I is clearly less complex than IRT, both conceptually and mathematically. It also appears to require many fewer examinees to establish empirically a rule base when compared to the large numbers needed to eliminate parameters for item response functions in the IRT model. (20 references) (Author/BBM)