Test-based measurement of skill mismatch: a validation of five different measurement approaches using the NEPS

Abstract

Skill mismatch is a key indicator of labour market research that has received significant attention. To date, various approaches of test-based measurement of skill mismatch have been used in research, generating differing results. However, it remains unclear which method is the most valid for measuring skill mismatch. This study provides a comparative validation of five commonly used approaches to test-based measurement of skill mismatches in reading and mathematics to detect the most valid method. Drawing on the 2016 wave of the German National Educational Panel Study (NEPS) Adult Cohort, I find significantly varying distributions for the different measurement approaches, and highly valid skill mismatch measures for the statistical and the mixed approach. Overall, the mixed approach emerges as the most valid method. The findings highlight the critical importance of measurement approaches in skill mismatch research.

Key points

• Empirical validation of five different approaches of test-based measurement of skill mismatch.

• Job analysis approach possesses high construct validity but classifies few workers as matched.

• Worker assessment approach and task approach lack sufficient coverage of mismatch categories.

• Statistical approach and mixed approach possess high construct and criterion-related validity.

• Mixed approach is most valid overall.

1 Introduction

Skill mismatch is a significant determinant of labour market returns which contributes, for example, to differences in job satisfaction and wages (Mateos‑Romero and Salinas-Jiménez 2018; McGuinness et al. 2018; Rohrbach-Schmidt and Tiemann 2016). The major relevance of this issue has recently been underlined by the European Commission’s declaration of the European Year of Skills 2023, with the objective of improving workforce skill matching (European Commission 2022). As a central indicator of labour market research, skill mismatch measures are crucial for identifying which individuals are affected by mismatches, and for analysing how mismatches affect these individuals. Thus, both policymaking and research require accurate and valid information on the incidence and type of individual-level skill mismatches (Flisi et al. 2017).

Previous studies on the validation of mismatch indicators have mainly focused on qualification mismatch measures (e.g. Capsada-Munsech 2019 for overeducation). In contrast to the concept of qualification mismatch which defines mismatches based on educational qualifications (educational level or field of education), skill mismatches refer to the matching of workers’ skill levels to their jobs (International Labour Office 2018). Typically, skill mismatch measures are either based on workers’ subjective self-assessments of their skill mismatch situation, or on objective (test-based) approaches that compare individuals’ skill levels measured by skill tests to the level of skill requirements in their occupational group. This study focuses on test-based measures of skill mismatch which are considered to be less biased. Such test-based measures can draw on valid information about workers’ skills in specific skill domains. However, they lack perfectly corresponding information on the level of skill requirements within these domains in individuals’ occupations (Pérez Rodríguez et al. 2023).

To date, various approaches have been used to proxy the skill requirements in occupations and consequently, approaches to test-based measurement of skill mismatch also vary. Five commonly used approaches in the literature operationalise skill requirements in occupations. They are based on the average skills of all workers (statistical approach) resp. of subjectively well-matched workers only (mixed approach), or on expert-based assessments (job analysis approach), worker-based assessments (worker assessment approach), or the complexity of job tasks (task approach). However, these approaches in part indicate highly different patterns of skill mismatches (cf. Flisi et al. 2017; Pérez Rodríguez et al. 2023; Perry et al. 2014), and so far, no common consensus exists on which of them is most appropriate. This raises questions on which of the approaches are appropriate for test-based measurement of skill mismatch.

This study provides a comparative empirical evaluation of these five approaches to test-based measurement of skill mismatch. It adds to the previous literature in several ways. First, this study pioneers in empirically analysing the plausibility of different methods to measuring skill requirements in occupations. Previous studies have only validated the resulting skill mismatch measures. However, this overlooks the decisive relevance of valid occupational skill requirements for the quality of the resulting test-based measures of skill mismatch (for example, determining the skill level required for workers in a given occupation and establishing the level at which workers are categorized as either underskilled or overskilled). In contrast, I empirically validate five different occupational skill requirement measures, underlining their relevance as an essential component for valid test-based measures of skill mismatch.

Second, this study is the first to provide a comprehensive comparative validation of five different test-based measurement approaches for skill mismatch in order to test the quality of the measures based on empirical analyses. Some previous studies have validated single approaches of test-based measurement of skill mismatch (cf. Allen et al. 2013; Desjardins and Rubenson 2011; Pellizzari and Fichen 2017). However, these studies lack comparison to other measures or restrict to single facets of the mismatch (e.g. Flisi et al. 2017 for overskilling). Furthermore, three studies to date provide comparative analyses of different approaches to test-based measurement of skill mismatch. They validate the different measures based on their link to key labour market outcomes such as wages or job satisfaction (cf. Pérez Rodríguez et al. 2023 for the statistical, mixed, and job analysis approach; Perry et al. 2014 for the statistical, mixed, and skill use approach; van der Velden and Bijlsma 2019 for the statistical, mixed, and effective skill approach). These studies demonstrate that the incidence of skill mismatches and their link to labour market outcomes are highly dependent on the underlying measurement approach. However, they do not suggest how these skill mismatches are explained by relevant characteristics of workers. This study, by contrast, applies a range of validation methods to assess and compare the validity of skill mismatch measures based on the statistical, mixed, job analysis, worker assessment, and task approach.

Third, this study validates skill mismatch measures in two conceptually distinct skill domains: reading and mathematics. Previous studies using test-based skill mismatch measures usually base on the Programme for the International Assessment of Adult Competencies (PIAAC) (e.g. Allen et al. 2013; Pérez Rodríguez et al. 2023). The PIAAC provides valid information on skill levels for literacy and numeracy,¹ but these skills exhibit extremely high intra-individual correlations exceeding 0.90 (Levels et al. 2014). In contrast, this study draws on the German National Educational Panel Study (NEPS) Adult Cohort which offers test-based information on workers’ skills in reading and mathematics with a correlation of 0.49. This suggests that the NEPS Adult Cohort provides information on two conceptually different skill domains, unlike the PIAAC, which exhibit overwhelmingly high correlations.

The remainder of this paper is organised as follows. Chapter 2 discusses the five approaches to test-based measurement of skill mismatch. Chapter 3 presents the data and describes the operationalisation of the central measures and the analytical plan. This is followed by the empirical validation of the occupational skill requirement measures and the resulting skill mismatch measures of the five approaches in Chapter 4. Finally, Chapter 5 discusses the findings and limitations, and provides an outlook and implications for future research.

2 Test-based measurement of skill mismatch

Test-based measures of skill mismatch compare an individual’s skill level assessed by skill tests to the level of skill requirements in their occupation. An individual is categorized as being underskilled if their skill level in the specific skill is lower than the level of skill requirements in their occupational group, and as being overskilled if their skill level is higher. Individuals possessing the required level of skills are categorized as being matched. Large-scale surveys such as PIAAC or NEPS provide information on the supply side of skills, i.e. the skill levels of individuals, but not on the demand side of skills, i.e. the level of skills required in their respective jobs.

This study analyses five commonly used approaches to operationalising occupational skill requirements and their resulting skill mismatch measures. The statistical approach (STA) defines occupational skill requirements based on the averaged test scores of workers in occupations which belong to the same occupational group. Similarly, the mixed approach (MA) defines occupational skill requirements based on the averaged test scores of workers within the same occupational group, but only considers those individuals who subjectively assess themselves as having an adequate skill level for their job (cf. Pellizzari and Fichen 2017). These two approaches possess a high level of objectivity, as the occupational skill requirements are based on objective test scores. Moreover, the MA prevents biases through test scores of mismatched workers, as it only considers subjectively matched workers. Both approaches, however, have been criticised for defining the average worker as being matched, which might not necessarily represent the actual skill requirements in a job (van der Velden and Bijlsma 2019). In the job analysis approach (JA), the level of occupational skill requirements is assessed by professional experts. This approach benefits from the high field expertise of occupational experts and from a transparent and objective evaluation process (McGuinness et al. 2018). However, expert-based ratings may also involve subjective bias or fallible human judgement (Morgeson and Campion 1997). The worker assessment approach (WA) uses subjective evaluations of workers on the level of skills required to perform their job, defining the occupational skill requirements by the averaged self-assessed scores of workers belonging to the same occupational group. The direct questioning of workers offers the opportunity to promptly identify current trends and changes in occupational requirements (Hartog 2000). On the other hand, workers’ subjective assessments lack objectivity. This might lead to biases since workers may tend to overstate their job requirements in order to upgrade their status (Handel 2016; Sparreboom and Tarvid 2016). The task approach (TA) refers to workers’ subjective assessments on the complexity of tasks they regularly employ in their job. Occupational skill requirements are defined by the average complexity-of-tasks scores of employees working in the same occupational group. This provides reliable information on the complexity of job tasks in occupations (Kracke and Rodrigues 2020). Yet, there are some concerns regarding the appropriateness of this measure to represent occupational skill requirements, as the complexity of job tasks is conceptually different from the level of skill requirements. Nevertheless, both concepts are closely related because the complexity of job tasks that have to be performed varies with the skill requirement levels of the occupation (Kracke and Rodrigues 2020). Table 1 provides an overview of the operationalisation method as well as the strengths and limitations of the five different approaches.

Table 1 Five approaches to measuring occupational skill requirements

3 Data and methods

3.1 Data and samples

The empirical analyses of this study are based on the NEPS Adult Cohort which comprises extensive information on educational and employment biographies of adults in Germany from several waves (Blossfeld and Roßbach 2019; NEPS Network 2018). This data provides test-based information on skill levels of employed adults in different skill domains, which is essential to operationalise test-based measures of skill mismatch. The main empirical analyses draw on cross-sections of the 2016 wave of the NEPS Adult Cohort. This wave comprises adults born between 1944 and 1986 within three subsamples with a gross sample of 10,078 individuals (4427 of the Working and Learning in a Changing World (ALWA) sample; 2641 of the NEPS wave 2 enhancement and refreshment sample; 3010 of the NEPS wave 4 refreshment sample). I restrict the analytical sample to dependent-employed core workers defined as having a maximum age of 65 years, being employed for at least 15 h per week, and excluding the self-employed, persons in pre-professional employment (e.g. internship, student assistant, etc.), freelancers, family workers, and individuals employed in active labour market programmes or seasonal work. Additionally, I exclude individuals who did not participate in the 2016 wave and those with missing skill mismatch information. This results in a gross sample size of 4889 individuals.

3.2 Measurements

I employ five different measures of occupational skill requirements in both reading and mathematics. The operationalisation of each approach’s measures is based on different data sources, as the relevant information is either provided in different waves of the NEPS Adult Cohort or had to be gathered from external data. Given the use of different sources which vary in the richness and accuracy of available data, information on skill requirements refers to the ISCO-08 two-, three-, or four-digit occupational groups, depending on the approach. In this context, ISCO-08 occupational unit groups (four-digits) provide the most specific information because of their high within-occupational group homogeneity of occupations, which is also high for ISCO-08 occupational minor groups (three-digits). Conversely, the skill requirement information for ISCO-08 occupational sub-major groups (two-digits) is considerably less specific, which may result in aggregation biases (Pérez Rodríguez et al. 2023).

The STA occupational skill requirement measures are based on the 2016 wave of the NEPS Adult Cohort and refer to the average skill level in reading or mathematics of all dependent-employed core workers of the same ISCO-08 three-digit occupational minor group plus and minus one standard deviation.² Based on the 2018 wave of the NEPS Adult Cohort, the MA occupational skill requirement measures are defined by the average skill level in reading or mathematics plus and minus one standard deviation of dependent-employed core workers belonging to the same ISCO-08 three-digit occupational minor group, who subjectively assess that their skills match the requirements of their job.³

The JA occupational skill requirement measures build on expert-based information on the critical level of skill requirements in reading or mathematics in ISCO-08 four-digit occupational unit groups in the OECD countries including Germany (Pérez Rodríguez et al. 2023). Given that the information on skill requirements refers to OECD countries in general, it does not provide country-specific information for the German labour market context. I apply a linear transformation to convert the original eleven-point scales (ranging from level 0, level 0.5, level 1, etc., to level 5) into scales ranging from 0 to 100. This transformation involves multiplying the original scale values by 20. I define the occupational skill requirements as the transformed expert-based scores per ISCO-08 four-digit occupational unit group plus and minus one standard deviation.

The WA occupational skill requirement measures are based on the 2014 wave of the European Skills and Jobs Survey (ESJS) restricted to the sample of Germany. I build on worker’s self-assessed levels of reading or mathematical skill requirements in their job, each distinguishing three levels (0: not required, 1: basic level, 2: advanced level).⁴ I transform the three-point scales into a 0 to 100 range using linear transformation, achieved by multiplying the original scale values by 50. The occupational skill requirement levels in reading or mathematics are defined by the average self-assessed score of dependent-employed core workers belonging to the same ISCO-08 two-digit occupational sub-major group plus and minus one standard deviation.

The TA occupational skill requirement measures build on the 2019 wave of the NEPS Adult Cohort. In a first step, I calculate the complexity-of-tasks scores in reading and mathematics per individual by aggregating information from several items on reading or mathematical job tasks each into a five-point scale (0: not required, 1: low complexity, 2: moderate complexity, 3: advanced complexity, 4: high complexity). This follows the procedure of Matthes et al. (2014).⁵ Subsequently, I multiply the scale values by 25 to result in scales ranging from 0 to 100 by means of linear transformation. I define the occupational skill requirements as the average complexity-of-tasks scores in reading or mathematics of dependent-employed core workers working in the same ISCO-08 three-digit occupational minor group plus and minus one standard deviation.⁶ Table 5 in the Appendix gives an overview on the operationalisation of the different occupational skill requirement measures.

The five measures of skill mismatch in both reading or mathematics are operationalised by comparing an individual’s skill level in reading or mathematics with the level of reading or mathematical skill requirements in their occupational group. Individuals’ skill levels in reading and mathematics are based on skill tests assessed in wave 2016 (cf. National Educational Panel Study 2019 for detailed information on the measurement of skill levels in the NEPS Adult Cohort).⁷ Workers are categorized as being underskilled, matched, or overskilled depending on whether they possess lower, higher, or the required level of skills in the respective domain (see Fig. 1).

Fig. 1

Source: Own illustration

Operationalisation of skill mismatch measures.

3.3 Analytical strategy

The study provides empirical validations of five different measures of occupational skill requirements and test-based measures of skill mismatch, each for reading and mathematics. This involves four validation methods. First, I evaluate the plausibility of threshold values of occupational skill requirement measures (1). This entails an assessment of whether the distributions of the occupational skill requirements’ lower and upper limit values vary across occupational groups in line with theoretical expectations, and whether the threshold values allow for differentiation at both ends of the scale. Next, a validation of the empirical distributions of skill mismatch measures (2) involves analyses of whether the skill mismatch measures possess significant coverage of all categories and a meaningful proportion between matched and mismatched individuals. Subsequently, I assess the construct validity of skill mismatch measures (3) by testing whether they link to key predictors of skill mismatch in a theoretically expected manner (Döring and Bortz 2016). For this purpose, I run separate logistic regression models for both underskilling (US_i) and overskilling (OS_i) to test how they relate to five core sociodemographic and occupational predictors of skill mismatch (cf. Eqs. 1a and 1b). These predictors include individuals’ age cohort (agec_i) (up to 35 years, 36–45 years, 46–55 years, 56–65 years) and educational mismatch (educmis_i) (undereducation, education match, overeducation) applying the statistical realised matches approach (cf. Hartog 2000), as well as dummies for female (gender_i), tertiary education (terteduc_i) (ISCED-1997: 5A, 6), and high complex occupational level (highocc_i) (working in an occupation with highly complex tasks defined by the fifth-digit classification of the KldB 2010).

Equation 1a Logistic regression models for predicting underskilling vs. not

$${US}_{i}= \alpha +{\beta }_{1}{gender}_{i}+{\beta }_{2}{agec}_{i}+ {\beta }_{3}{terteduc}_{i}+{\beta }_{4}{educmis}_{i}+{\beta }_{5}{highocc}_{i}+\varepsilon$$

(1a)

Equation 1b Logistic regression models for predicting overskilling vs. not

$${OS}_{i}= \alpha +{\beta }_{1}{gender}_{i}+{\beta }_{2}{agec}_{i}+ {\beta }_{3}{terteduc}_{i}+{\beta }_{4}{educmis}_{i}+{\beta }_{5}{highocc}_{i}+\varepsilon$$

(1b)

Finally, I test the criterion-related validity of skill mismatch measures (4) by analysing how they are associated with individuals’ wages, which represent a central labour market outcome (Döring and Bortz 2016). Therefore, I run OLS regressions based on the so-called Overeducation-Required education-Undereducation (ORU) model of Duncan and Hoffman (1981), modified for skill mismatches. I test how both underskilling and overskilling relate to individuals’ ln gross hourly wages (W_i), measured by respondents’ self-reported gross income divided by their actual monthly working hours and trimmed by dropping the 1st and 99th percentiles. The ORU models include information on the skill requirement level in an individual’s occupational group (RS_i) and a set of relevant control variables (C_i) (cf. Eq. 2).

Equation 2 Specification of the ORU model modified for skill mismatches

$${W}_{i}= \alpha +{\beta }_{1}{US}_{i}+ {\beta }_{2}{OS}_{i}+ {\beta }_{3}{RS}_{i}+{\beta }_{4}{C}_{i}+\varepsilon$$

(2)

These control variables cover female (as opposed to male), age (in years), immigration background (individuals of the first and second generation), education level (lower secondary, upper secondary, post-secondary non-tertiary, tertiary), field of education (general, STEM, humanities/social sciences, business/law/services, education, health/welfare, unknown), part-time work (as opposed to full-time), public sector work (as opposed to private sector), economic sector (agriculture/industry, services, information sector, public administration, unknown), workplace (East Germany, West Germany incl. Berlin, unknown), and job experience (number of years in the current job). Table 6 in the Appendix provides an overview of the descriptive sample statistics.

4 Validation

4.1 Occupational skill requirements

The occupational skill requirement measures are validated with regard to their plausibility of threshold values (1). Threshold values are defined as the lower and upper limit of occupational skill requirements in an occupational group. These threshold values specify the range of skill requirements for the various occupational groups and thus determine the skill level from which an individual is considered underskilled (below the lower limit), matched (between the lower and upper limit), or overskilled (above the upper limit). In the individual approaches, these thresholds are determined by the average scores (skill tests, experts’ or workers’ assessments, job tasks) in the respective occupational group minus (lower limit) or plus (upper limit) one standard deviation.

Plausible threshold values indicate relatively higher skill requirement levels in occupational groups that are perceived to possess higher skill levels. The ISCO-08 indicates different skill requirement levels (basic, moderate, high, extended) in reading and mathematics between the occupational major groups (International Labour Office 2018). Specifically, this implies extended skill requirements in reading and mathematics in occupations belonging to the ISCO-08 one-digit major groups 1 (managers) and 2 (professionals), high requirements in 3 (technicians and associate professionals), moderate requirements in 4 (clerical support workers), 5 (services and sales workers), 6 (skilled agricultural, forestry and fishery workers), 7 (craft and related trades workers), and 8 (plant and machine operators and assemblers), as well as basic requirements in 9 (elementary occupations).

Moreover, plausible threshold values need to allow for differentiation in both directions. Therefore, lower limit thresholds that, by definition, cannot fall below, and upper limit thresholds that cannot be exceeded, are problematic, because even individuals with extraordinarily low or high skill levels cannot be classified as underskilled or overskilled. Figures 2 and 3 depict for each approach the threshold values of occupational skill requirements in reading and mathematics (vertical axis) for the different ISCO-08 minor groups (horizontal axis).⁸

Fig. 2

Sources: NEPS Adult Cohort (SC6: 9.0.1; SC6: 11.1.0; SC6: 12.0.1), Pérez Rodríguez et al. (2023), ESJS (2014), own illustration

Occupational skill requirement threshold values in reading per ISCO-08 minor group (three-digits).

Fig. 3

Sources: NEPS Adult Cohort (SC6: 9.0.1; SC6: 11.1.0; SC6: 12.0.1), Pérez Rodríguez et al. (2023), ESJS (2014), own illustration

Occupational skill requirement threshold values in mathematics per ISCO-08 minor group (three-digits).

For both skill domains, the STA and MA possess middle-ranging thresholds across each occupational minor group. The occupational skill requirements range from minimum lower limit values of approximately 20 (reading) and 30 (mathematics) to maximum upper limit values of about 70 (reading) and 80 (mathematics). This indicates that both STA and MA allow for differentiation at both ends of the scale because individuals with exceptionally low or high skill levels are classified as underskilled or overskilled. Conversely, the JA, WA, and TA mark job requirements across the full spectrum of the scale, with JA and TA applying to both reading and mathematics, and WA applying exclusively to mathematics. The JA exhibits comparatively narrow bandwidths for skill-matching, defined by the corridor between the lower limit and upper limit values, whereas the WA and TA demonstrate significantly broader bandwidths for skill-matching. These broader bandwidths may reduce differentiability as almost all workers might be classified as being matched by definition. Moreover, the JA, WA, and TA each possess several absolute thresholds (lower limit values of 0 resp. upper limit values of 100), which cannot be fallen below or exceeded by definition. This is problematic because even workers with extraordinarily low or high skill levels cannot be categorized as being underskilled or overskilled in the occupational groups concerned.

The distributions of skill requirements in reading and mathematics are basically in line with the ISCO-08 expectations for the occupational major groups (International Labour Office 2018), tendentially indicating higher requirement levels for managers and professionals compared to workers and lowest for elementary occupations. This is particularly reflected in the distributions of the JA. Overall, STA and MA have a comparatively higher plausibility of threshold values as they do not possess any problematic absolute threshold values.

4.2 Distribution of skill mismatches

The skill mismatch measures are validated with regard to their empirical distributions, construct validity, and criterion-related validity. Valid empirical distributions (2) of skill mismatch possess significant coverage for all categories of the measure. Moreover, valid empirical distributions correspond to major expectations of skill mismatch; that is, the majority of workers are matched. Figure 4 presents the empirical distributions of the five skill mismatch measures for both reading and mathematics.

Fig. 4

Sources: NEPS Adult Cohort, https://doi.org/10.5157/NEPS:SC6:9.0.1, own calculations

Distributions of skill mismatches (reading, mathematics). N = 4889.

The distributions significantly differ between the five approaches, except for the STA and MA. In both reading and mathematics, the STA and MA classify about three-fourths of workers as being matched and show fairly balanced ratios between underskilling and overskilling. Conversely, the JA identifies approximately three-fourths of workers in reading and two-thirds in mathematics as mismatched, with more than three times as many underskilled individuals in reading and more than two times as many in mathematics compared to overskilled individuals. These three approaches do not show substantial differences in distributions between skill mismatches in reading and mathematics.

In contrast, the skill mismatch distributions of WA and TA significantly differ between reading and mathematics. For example, the WA classifies 84.52 percent as underskilled and 15.48 percent as matched in reading compared to 12.80 percent underskilled and 87.07 percent matched in mathematics. The TA indicates 39.87 percent as being underskilled in reading compared to 1.28 percent in mathematics as well as 60.03 percent as being matched in reading compared to 93.41 percent in mathematics. In both WA and TA, either no individuals (WA in reading) or hardly any individuals are classified as overskilled in reading or mathematics.

The STA, MA, and JA provide a significant coverage of all skill (mis)match categories, which is not the case for the WA and TA. However, the JA classifies the vast majority of individuals as mismatched. Overall, the STA and MA demonstrate the most valid empirical distributions in both reading and mathematics, aligning closely with previous findings on test-based measures of skill mismatch (e.g. Allen et al. 2013; Pérez Rodríguez et al. 2023; Perry et al. 2014).

4.3 Construct validity

The construct validity (3) of skill mismatch measures is analysed based on their link to relevant sociodemographic and occupational predictors. For each approach, I run separate logistic regression models for underskilling and overskilling in both reading and mathematics to examine whether individuals’ gender, age cohort, education, educational mismatch, and occupational level predict both types of mismatches in a theoretically expected manner. Following theoretical expectations (cf. Becker 1964; Sicherman and Galor 1990) and previous empirical findings on determinants of skill mismatches (e.g. Livanos and Núñez 2017, McGowan and Andrews 2015, Pellizzari and Fichen 2017 for tertiary education; Desjardins and Rubenson 2011, Pellizzari and Fichen 2017 for age; Allen et al. 2013 for educational mismatches; Allen et al. 2013, Cedefop 2022 for occupational level), I expect older workers, undereducated workers, and individuals working in a job with a highly complex occupational level more likely to be underskilled but less likely to be overskilled in both reading and mathematics. In contrast, workers with tertiary education and overeducated workers are assumedly less likely to be underskilled, but more likely to be overskilled. Furthermore, women and men demonstrate varying skill mismatch risks in reading and mathematics (Desjardins and Rubenson 2011; Pellizzari and Fichen 2017). Hence, I assume that female workers are less likely to be underskilled in reading but more likely to be underskilled in mathematics and more likely to be overskilled in reading but less likely to be overskilled in mathematics. Tables 2 and 3 present the associations between the predictors and underskilling resp. overskilling separately for each approach.

Table 2 The association between predictors and underskilling (reading, mathematics), logistic regressions

Table 3 The association between predictors and overskilling (reading, mathematics), logistic regressions

The STA and MA show similar results for underskilling in reading, indicating tertiary-educated workers and overeducated workers to be statistically significantly less likely underskilled. In contrast, tertiary-educated workers (JA and TA) and underskilled workers (JA, WA, and TA) are statistically significantly more likely to be underskilled in the other approaches. Across all approaches, as expected, older age cohorts and workers in highly complex occupations are statistically significantly more likely to be underskilled. Interestingly, the relationships between gender and underskilling in reading vary considerably between the approaches. While there are no statistically significant findings for STA, MA, and JA, females are statistically significantly more likely to be underskilled in the WA, but statistically significantly less likely to be underskilled in the TA.

For underskilling in mathematics, similar results show across the different approaches, excepting the TA. In the case of the STA, MA, JA, and WA, females, workers of older age cohorts, and workers in highly complex occupations are statistically significantly more likely to be underskilled, whereas tertiary-educated workers are statistically significantly less likely to be underskilled (excepting the JA). Moreover, undereducated workers are somewhat (WA 5.4 percentage points) resp. strongly (JA 17.3 percentage points) more likely to be underskilled in comparison to workers with a matching education. Interestingly, overeducated workers are statistically significantly less likely to be underskilled in the STA, MA, and WA, but more likely to be underskilled in the JA. Basically, the findings for underskilling in reading and mathematics exhibit similarities across the approaches. This is particularly evident in the STA and MA, which also align most closely with theoretical expectations.

The results for overskilling in reading between STA and MA are also quite similar. Neither approach reveals statistically significant associations between gender and overskilling. However, workers in older age cohorts are statistically significantly less likely to be overskilled, whereas tertiary-educated workers and overeducated workers are statistically significantly more likely to be overskilled. The JA differs in that tertiary-educated workers, undereducated workers, and overeducated workers are statistically significantly less likely to be overskilled. Moreover, both the MA and JA show workers in highly complex occupations to be statistically significantly less likely overskilled. Given that no individual is classified as overskilled in reading, predictions for the WA cannot be made. Additionally, the TA fails to provide estimations for some predictors due to the very small number of workers classified as overskilled.

With respect to overskilling in mathematics, again similar findings exist for STA and MA, which indicate statistically significantly lower risks of being overskilled for female workers, older age cohorts, and workers in highly complex occupations, but higher risks for workers with a tertiary education as well as for overeducated workers. The results for overskilling in the JA and TA differ from the other approaches insofar as they do not show a statistically significantly higher likelihood of being overskilled for overeducated workers. Additionally, undereducated workers face a 14.4 percentage points lower likelihood of being overskilled in the JA than properly educated workers, and female workers are not statistically significantly less likely to be overskilled than male workers in the TA. The WA fails to provide estimations for some predictors, given the very small number of workers categorized as being overskilled.

Considering the predictions for both types of skill mismatches (underskilling and overskilling) in both reading and mathematics, the findings of the STA and MA are most consistent with theoretical expectations, followed by the JA. This implies that the MA, STA, and JA possess the highest construct validity. In summary, the MA results for overskilling in reading and mathematics align most closely with expectations which is why the MA is slightly superior to the STA.

4.4 Criterion-related validity

Finally, I test the criterion-related validity (4) of the skill mismatch measures based on their association with individuals’ wages which represents a key labour market outcome (cf. Table 4). Given the context of ORU models, I analyse wage differences among individuals with different skill levels but working in occupations with the same level of skill requirements. Accordingly, while underskilled workers exhibit skill deficits and may be less productive, overskilled workers possess surplus skills that may potentially enhance productivity beyond that of workers with matched skills. Drawing on the human capital theory (Becker 1964), which postulates that individuals’ wages are commensurate with their productivity, I expect underskilled workers to earn lower wages and overskilled workers to earn higher wages than matched individuals working in occupations with similar skill requirements.

Table 4 The association between skill mismatches (reading, mathematics) and wages, OLS regressions

Across the approaches, workers underskilled in reading earn statistically significantly lower wages than matched workers with similar occupational requirements. These underskilling wage penalties range from approx. 3.6 percent (JA) to approx. 7.5 percent (STA) lower wages. Conversely, workers overskilled in reading earn statistically significantly higher wages than workers with matching reading skills, but only in the STA (approx. 5.4 percent) and MA (approx. 5.2 percent). With regard to skill mismatches in mathematics, there are statistically significant wage penalties for underskilling, ranging from approx. 4.3 percent (STA) to approx. 18.7 percent (TA) lower wages, with the exception of the JA. In contrast, overskilled workers earn statistically significantly higher wages than workers with matching mathematical skills in the STA (approx. 3.7 percent), MA (approx. 3.5 percent), and JA (approx. 8.5 percent).

This means similar findings for skill mismatches in reading and mathematics between the approaches. However, the expected statistically significant underskilling wage penalties and overskilling wage benefits in both reading and mathematics are only evident for the STA and MA. Moreover, these two approaches indicate stronger wage differences for underskilling than for overskilling in both reading and mathematics. This aligns with assumptions that skill deficits substantially decrease worker productivity, while there are also ceiling effects on worker productivity resulting from skill surpluses, as individuals cannot perform their jobs beyond excellence (Humburg et al. 2013). I conclude that the wage differences resulting from skill mismatches are most convincing in the STA and MA, and therefore possess the highest criterion-related validity.

4.5 Discussion of results and robustness checks

Overall, the empirical validations illustrate that different approaches may produce quite different results. In terms of occupational skill requirements, the STA and MA demonstrate the most plausible threshold values. This is because both approaches allow for both underskilling and overskilling within each occupational group, unlike the other approaches. The STA and MA also exhibit the most valid empirical distributions of skill mismatch, providing significant coverage of all categories and classifying the majority of workers as matched. In contrast, the JA classifies the vast majority of individuals as mismatched, while the WA and TA do not adequately cover each skill mismatch category in both reading and mathematics. The MA demonstrates the highest construct validity overall, slightly surpassing the STA. The JA also exhibits high construct validity, whereas both the WA and TA show considerably lower construct validity. Finally, the STA and MA also have the highest criterion-related validity, both in reading and mathematics.

I replicated the validation of the distributions, construct validity, and criterion-validity of skill mismatches in both reading and mathematics, restricting on individuals working at least 35 h per week (cf. Figure 5 and Tables 10 to 12 in the Supplementary Material). This is to assess the stability of findings for full-time employment. The empirical distributions of skill mismatch as well as construct validity remain essentially the same across all approaches. This holds also for criterion-related validity of skill mismatches in reading. However, for skill mismatches in mathematics, the MA exhibited slightly lower criterion-related validity compared to the sample not restricted to full-time employment.

Additionally, I replicated the validation by consistently using the ISCO-08 two-digit level for all five approaches (cf. Figure 6 and Tables 13, 14, 15 in the Supplementary Material). This is to ensure that the validity of the different approaches is not driven by using differently detailed ISCO-08 classifications for the operationalisation of skill mismatches. Again, the empirical distributions and the construct validity remain essentially the same for all approaches in both reading and mathematics. Conversely, the criterion-related validity remains essentially the same for STA, WA, and TA, whereas criterion-related validity is slightly lower for the MA in mathematics and considerably lower for the JA in reading. This underlines the relevance of assessing occupational skill requirements at the most detailed classification level possible to avoid loss of validity.

Finally, I replicated the validation using skill mismatch measures operationalised based on the KldB 2010, which specifically pertains to the German labour market (cf. Figure 7 and Tables 16, 17, 18 in the Supplementary Material). However, the validations are only conducted for the STA, MA, and TA because the JA and WA occupational group classifications are not available for the KldB 2010.⁹ The empirical distributions and the construct validity remain essentially the same for the three approaches, with minor differences for the TA. The criterion-related validity remains essentially the same for the MA, but is slightly lower for the STA in mathematics and considerably lower for the TA in reading. Using the KldB 2010 instead of the ISCO-08, however, produces no significant differences overall.

In summary, the robustness checks confirm the main findings. The STA and MA consistently demonstrate the most valid skill mismatch measures, followed by the JA, while the WA and TA exhibit significantly lower validity. Overall, the MA is slightly superior to the STA due to its somewhat higher construct validity. Furthermore, the MA defines occupational skill requirements based on the average skill levels of only subjectively matched workers. This covers subjective and objective characteristics in one measure and may represent a more accurate proxy for skill requirements compared to the average skill levels of all workers used by the STA.

The robustness checks also indicate that the validity differences are not attributable to the use of specific occupational classifications or a more detailed classification level. The fact that the scale metric between skill levels and skill requirements is originally the same for the STA and MA may be beneficial, as it reduces potential bias compared to the transformed skill requirement scales of the JA, WA, and TA. However, these three approaches exhibit several shortcomings, such as absolute threshold values, which may more seriously impair their overall validity. Thus, the different scale metrics are not expected to substantially drive the findings.

5 Conclusions

This study provides a comparative empirical validation of five different approaches to measuring occupational skill requirements as well as test-based measures of skill mismatch to analyse which approach is most adequate to measuring skill mismatches. Drawing on the 2016 wave of the NEPS Adult Cohort, it validates the five approaches with regard to the empirical distributions of their occupational skill requirement and skill mismatch measures, as well as their skill mismatch measure’s link to relevant predictors and outcomes. The findings illustrate significant variances between the different approaches in both reading and mathematics, which underlines the need for careful reflection when choosing skill mismatch measures for analyses. Given their plausible distributions of occupational skill requirements and skill mismatches, and their high construct and criterion-related validity, the skill mismatch measures of the STA and MA are considered more valid. In contrast, the skill mismatch measures of the other approaches are less valid, for example, due to classifying the vast majority of workers as being mismatched (JA), or because they possess considerably lower construct and criterion-related validity (WA and TA). Overall, the MA is considered the most valid approach for test-based measurement of skill mismatch, as it exhibits slightly higher construct validity than the STA and integrates both subjective and objective characteristics into one measure.

Interestingly, the overall validity of the various approaches’ skill mismatch measures does not substantially differ between reading and mathematics. This may suggest that the validity of skill mismatch measurement approaches is not specific to particular skill domains. Therefore, the findings and conclusions may also be applicable to test-based skill mismatches in other relevant skill domains, such as ICT, science, or problem-solving.

Nevertheless, this study has some limitations. The quality of the different approaches to test-based measurement of skill mismatch is evaluated using the NEPS Adult Cohort, but the implications cannot be transferred one-to-one to other contexts or other data. This is particularly evident for JA, WA, and TA, whose scaling of the occupational skill requirements does not perfectly match the metric of the NEPS skill level scaling. Thus, these three approaches should not be considered unsuitable for test-based measurement of skill mismatch in general, but rather in the specific context of the NEPS study. In the case of the JA, for example, this might be mainly due to lacking country-specific data, as information on skill requirements pertains to OECD countries in general but not specifically to the German labour market context. Moreover, this study illustrates that test-based measurement of skill mismatch is always associated with some degree of arbitrariness regarding the definition of threshold values, the skill domains used, or the selection of measurement approach. Considering that the incidence of skill mismatches and the resulting conclusions are highly driven by these arbitrary choices, results and conclusions should be interpreted with caution and in reflection of the measure which is used.

Future research might build on these findings, for example, to test the validity of the different approaches using other data or in another country-specific context. Subsequent skill mismatch research could benefit if large-scale surveys were to capture fitting information on occupational skill requirements in addition to skill tests. Moreover, future research may develop more holistic approaches to better understanding the skill mismatch phenomenon. Previous test-based measures, which focus solely on mismatches in single skill domains, might be expanded by adopting a multidimensional perspective, such as capturing the overall skill mismatch situation of individuals across multiple skill domains, for example.

Appendix

See Tables 5, 6

Table 5 Operationalisation of the occupational skill requirement measures

Table 6 Descriptive sample statistics