Parametric complexity of tasks in intelligence tests

Publication type Article
Status Published
Occupation: Professor, Chief Researcher at the Institute of Psychology, RAS
Affiliation: Institute of Psychology, RAS
Address: Moscow, Yaroslavskaya St., 13
Occupation: Senior Researcher at the Institute of Psychology, RAS
Affiliation: Institute of Psychology, RAS
Address: Yaroslavskaya St., 13
Journal namePsikhologicheskii zhurnal
EditionVolume 41 Issue 6

The article is devoted to the poorly studied problem of the component composition of the complexity of individual test items. The ambiguity and lack of certainty of the concept of “complexity” of the task leads to a confusion of problems in the development and evaluation of psychodiagnostic tools. The article consistently reveals the content of the concept of “complexity” in the framework of various testing theories: The Classical Theory of Mental Tests and The Item Response Theory. The meaning of the concept “complexity” of a task is determined for different types of testing: linear, multistage and adaptive. The article discusses various connotations of the concept of “complexity”. The diversity of complexity characteristics in the framework of complexity science, in various linguistic and cognitive studies is shown. “Complexity” and “difficulty” of the task are close, but not coinciding concepts, and the article attempts to dilute and reformulate them. In this connection, the “complexity” of an individual test item is considered as a construct, the constituent parts of which represent individual parameters. The main parameters of the complexity of the tasks of intelligence tests are described: the number of homogeneous and / or heterogeneous elements of the task, the number of irrelevant elements, the number of rules performed with the elements of the task, the combination of different types of stimulus material, the change in several signs of the elements of the task, the use of different types of answers, time restrictions on task performance, dynamics of variability of conditions, violation of the principles of good shape, “convexity” of elements, number of distractors, non-transitivity of objects, game components in the structure of tasks, use of contexts of varying degrees of concreteness in tasks. Varying the described complexity parameters makes it possible to construct test items with a given probability of a correct answer.

KeywordsCognitive ability, intelligence, psychodiagnostics, the complexity of test items, the difficulty of the task
AcknowledgmentThe research is supported by a grant of the Russian Foundation for Basic Research (project № 20-013-00495 “Empirical verification of the structural-functional model of the cognitive resource”).
Publication date27.11.2020
Number of characters23070
100 rub.
When subscribing to an article or issue, the user can download PDF, evaluate the publication or contact the author. Need to register.

Number of purchasers: 0, views: 958

Readers community rating: votes 0

1. Avanesov V.S. Forma testovyh zadanij. Uchebnoe posobie. 2 izd. Moscow: Centr testirovaniya, 2005. 156 p. (in Russian)

2. Voĭtov V.K. Raschet znacheniĭ slozhnostej zadanij dlya adaptivnogo testa intellekta. Eksperimental'naya psihologiya. 2013. V. 6. № 2. P. 120−128. (in Russian)

3. Kornilova T.V., Kornilov S.A., Zirenko M.S., CHumakova M.A. Psihometricheskie svoĭstva modificirovannoĭ batarei Internacional'nogo Resursa Kognitivnyh Sposobnosteĭ (ICAR). Nacional'nyĭ psihologicheskiĭ zhurn. 2019. № 3(35). P. 32–45. (in Russian)

4. Kuravskij L.S., Artemenkov S.L., YUr'ev G.A., Grigorenko E.L. Novyj podhod k komp'yuterizirovannomu adaptivnomu testirovaniyu. Eksperimental'naya psihologiya. 2017. V. 10. № 3. P. 33−45. (in Russian)

5. Kuravskij L.S., YUr'ev G.A., Ushakov D.V., YUr'eva N.E., Valueva E.A., Lapteva E.M. Diagnostika po testovym traektoriyam: metod patternov. Eksperimental'naya psihologiya. 2018. V. 11. № 2. P. 77−94. (in Russian)

6. Nizbett R. CHto takoe intellekt i kak ego razvivat'. Moscow: Al'pina nonfikshn, 2013. (in Russian)

7. Podd'yakov A.N. Opyt razrabotki ob"ektov, nahodyashchihsya v netranzitivnyh otnosheniyah prevoskhodstva. Kognitivnaya nauka v Moskve: novye issledovaniya. Materialy konferencii 15 iyunya 2017 g.. Ed. E.V. Pechenkova, M.V. Falikman. Moscow: OOO “Buki Vedi”, IPPiP, 2017. 596 p. (in Russian)

8. SHeronov E.A. Razrabotka psihologicheskogo instrumentariya dlya opredeleniya prigodnosti k obucheniyu na medicinskih special'nostyah. Vestnik KGU im N.A. Nekrasova, 2013. V. 19. P. 51−54. (in Russian)

9. YUsupov F.M. O sootnoshenii verbal'noj i neverbal'noj sostavlyayushchih v strukture intellekta. F.M. YUsupov. Psihologicheskij zhurnal. 1995. № 1. P. 102−106. (in Russian)

10. Carreiras M., Quiñones I., Hernández-Cabrera J.A., Duñabeitia J.A. Orthographic Coding: Brain Activation for Letters, Symbols, and Digits. Cerebral Cortex, 2015. 25. P. 4748−4760.

11. Crocker L. Introduction to Classical and Modern Test Theory. L. Crocker, J. Algina. N.Y.: Holt, Rinehart and Wilson, 1986.

12. Cucina, J., Byle, K. The bifactor model fits better than the higher-order model in more than 90% of comparisons for mental abilities test batteries. Journ. of Intelligence. 2017. 5(27). 1–21.

13. Dubois, J., Galdi, P., Paul, L.K., Adolphs, R. A distributed brain network predicts general intelligence from resting-state human neuroimaging data. Philosophical Transaction of the Royal Society B: Biological Sciences, 2018. 373 (1756).

14. Fahrenberg J. “Ökologische Validität [ecological validity],” in Dorsch-Lexikon der Psychologie, ed. H. Wirz (Bern: Huber), 2017.

15. Gell-Mann M. The quark and the jaguar: adventures in the simple and the complex. N.Y., 1994.

16. Gottfredson L.S. Why g Matters: The Complexity of Everyday Life. Intelligence, 1997. 24 (1). Р. 79−132.

17. Green K.E., Kluever R.C. Components of Item Difficulty of Raven's Matrices. The Journal of General Psychology. 1992. 119:2. P. 189−199.

18. Greiff S., Wüstenberg S., Funke J. Dynamic problem solving – A new assessment perspective. Applied Psychological Measurement. 2012. 36. P. 189−213.

19. Gupta S., Karahanna E. Technology adoption in complex systems. 7th Annual Conference of the Southern Association for Information Systems. Savannah GA: University of Georgia, 2004.

20. Guttman L. A basis for analyzing test-retest reliability. Psychometrika. 1945. 10. P. 255−282.

21. Hammarström H. Complexity in numeral systems with an investigation into pidgins and creoles. Language complexity: typology, contact, change. Amsterdam, 2008.

22. Juola P. Assessing linguistic complexity. Language complexity: typology, contact, change. Amsterdam, 2008.

23. Kessler Y., Rac-Lubashevsky R., Lichtstein C., Markus H., Simchon A., Moscovitch M. Updating visual working memory in the change detection paradigm. Journal of Vision. 2015. 15(9). 18. P. 1−12.

24. Kirschenbaum S.S. What Makes Decision Tasks Difficult?. Newport. RI: Naval Undersea Warfare Center, 1999.

25. Kovacs, K., & Conway, A.R.A. (2016). Process overlap theory: A unified account of the general factor of intelligence. Psychological Inquiry. 27(3). 151–177.

26. Krieger F., Zimmer H.D., Greiff S., Spinath F.M., Becker N. Why are difficult figural matrices hard to solve? The role of selective encoding and working memory capacity. Intelligence. 2019. 72. P. 35−48.

27. Malone S., Altmeyer K., Vogel M., Brünken R. Homogeneous and heterogeneous multiple representations in equation-solving problems: An eye-tracking study. J Comput Assist Learn. 2020. P. 1−18.

28. McWhorter J. Linguistic simplicity and complexity: why do languages undress? Berlin, 2011.

29. Meo M., Roberts M. J. Marucci F. S. Element salience as a predictor of item difficulty for Raven’s Progressive Matrices. Intelligence. 2007. 35. P. 359−368.

30. Miestamo M. Implicational hierarchies and grammatical complexity. Language complexity as an evolving variable. Oxford, 2009.

31. Pérez-Salas C.P., Streiner D.L., Roberts M.J. A comparison between element salience versus context as item difficulty factors in Raven's Matrices. Intelligence. 2012. 40. P. 325−332.

32. Primi R. Complexity of geometric inductive reasoning tasks: Contribution to the understanding of fluid intelligence. Intelligence. 2002. 30(1). P. 41−70.

33. Ren X., Wang T., Sun S., Deng M., Schweizer K. Speeded testing in the assessment of intelligence gives rise to a speed factor. Intelligence. 2018. 66. P. 64−71.

34. Rouder J.N., Morey R.D., Morey C.C., Cowan N. How to measure working memory capacity in the change detection paradigm. Psychon Bull Rev. 2011. 18(2). P. 324−330.

35. Rüsseler J., Scholz J., Jordan K., and Quaiser-Pohl C.M. Mental rotation of letters, pictures, and three-dimensional objects in German dyslexic children. Child Neuropsychology. 2005. 11. P. 497−512.

36. Sonnleitner P., Brunner M., Greiff S., Funke J., Keller U., Martin R., ... Latour T. The Genetics Lab: Acceptance and psychometric characteristics of a computer-based microworld assessing complex problem solving. Psychological Test and Assessment Modeling. 2012. 54. P. 54−72.

37. The Evolution of Cognitive Ability Tests and Score Interpretation //Applied Measurement in Education. V. 32. № 3. 2019.

38. Waldrop M.M. Complexity: The emerging science at the edge of order and chaos. Touchstone, New York, 1993.

39. White A. P., Zammarelli J. E. Convergence principles: Information in the answer sets of somemultiple choice tests. Applied Psychological Measurement. 1981. 5. P. 21−27.

40. Wood R.E. Task complexity: Definition of the construct. Organizational Behavior and Human Decision Processes. 1986. 37. Р. 60−82.

Система Orphus