You’ll never seek alone: The impact of active labour market policies on finding a job

Abstract

Using the administrative records of the Spanish Public Employment Service for the period 2018–2019, we analysed the impact of participation in job search assistance and training programmes on the chances of unemployed jobseekers finding employment, as well as the quality of the job obtained. Propensity score matching techniques were used to evaluate the effectiveness of these programmes. We found that participation exerted a positive and significant differential effect on the probability of transition from unemployment to employment and on the probability of finding a job of intermediate quality compared to non-participation. This positive impact was largely due to the effect of training programmes, the magnitude of which persisted even as the period elapsed between participation and exit towards employment increased, whereas job search assistance was less effective over time. The positive influence of participation appeared to be more intense for specific socioeconomic groups.

1 Introduction

Research on the impact of active labour market policies (ALMPs) is an area of labour economics that has grown considerably in the last three decades. ALMPs have gained prominence as a result of the development of so-called ‘activation strategies’ (Malo 2018). It is common to find economic policy reports and documents that describe these programmes as a fundamental tool in the fight against unemployment, and the role of these policies has in fact been and remains crucial in Europe as a strategy to improve employability (Martin 2015). Since the European Employment Strategy was first approved in 1997, the budgets for programmes to promote the employment of specific groups (young people, long-term unemployed, women, people at risk of social exclusion) have increased considerably.

Along the same lines, according to the European Commission, Public Employment Services (PES) – and the ALMPs they carry out – should play a leading role in mitigating the adverse effects of economic shocks and reducing long-term unemployment. Concerns about how to achieve more effective targeting of ALMPs and resource allocation have gained special attention from policymakers in EU Member States. As a result, many research papers have focussed on the causal impact of specific active measures. More evaluations of the impact of these programmes have been published at the international level, and some studies have systematized the results of this large empirical literature using meta-analysis (Card et al. 2010, 2018; Kluve 2010).

This article focusses on the effects of ALMPs on job search outcomes in Spain and provides an evaluation of the impact of participation in such programmes (grouped into ‘job search assistance’ and ‘training’) on the probability of an unemployed worker’s finding a job. Job search assistance programmes are aimed at helping unemployed workers to find a job through orientation, search advice, support and intermediation services, as well as providing access to available information on the situation and trends of the labour market.¹ Training programmes, meanwhile, are measures that seek to improve human capital by updating, renewing and increasing the skills and qualifications of workers.

Expenditure on ALMPs in Spain expanded until 2010, suffered budgetary cuts in the period 2011–2013 and increased again starting in 2014. The share of job search assistance services was below 15% of the total expenditure on ALMPs until 2014 and rose to 20%–25% in the period 2016–2018. Training programmes have traditionally accounted for around 20% of the budget, although their share decreased to about 15% in the period 2017–2019. Although successive governments have been promoting policy evaluation, so far this kind of exercise is quite scarce in Spain. One reason for this paucity of studies is related to the difficulty of accessing adequate databases. Many of existing works have focussed on training programmes and used information for certain regions, because control over the implementation of ALMPs in Spain are transferred to the regional governments.² In general, the evaluation studies have found positive effects for participation on workers’ outcomes.

The contribution of this article to the existing economic literature on the evaluation of ALMPs is threefold. First, we use representative, adequate data that come from a database that is difficult to access: the administrative records of the PES of one of the Spanish regions (the Community of Madrid). These data are of high quality. They allow us to build a rigorous definition of the comparison group, as well as the delimitation and use of alternative treatment groups, depending on when the last service received by an individual begins; this allows a refinement that improves on previous studies on ALMP effects. The comparison group is made up of those jobseekers who did not participate in any programme in 2018, whereas the treatment groups comprise those who received their ‘last’ service in the months of September to December 2018. By narrowing this period, we can examine whether the effect of participation in ALMPs changes.

Second, we focus on two types of activation measures – job search assistance and training – due to the availability of information on the dataset. This choice separates us from other studies on ALMPs, because evaluation of the job search assistance is less common than that for training, as the former has been less studied in the international empirical literature. For instance, in the meta-analysis carried out by Card et al. (2018), job search services account for 15% of the estimates, this share being lower among studies focussed on non-Nordic European countries; in Kluve et al. (2019), the proportion is 10%. Our analyses are also carried out in an aggregate way but also in a disaggregated manner for more specific programmes. This applies particularly to job search assistance programmes.

Third, the article expands the types of indicators used to measure the employment outcomes of unemployed workers when analysing the potential impact that ALMPs can produce. In this sense, we analyse the effect not only on the probability of finding a job but also on the quality of the job found by considering an indicator based on features related to the type and duration of the labour contract and working hours.

To carry out the empirical analysis, we use microdata from administrative records of the PES of the Autonomous Community of Madrid for the period January 2018–December 2019. These records contain information about (a) jobseekers registered in public employment offices; (b) the services received by the participants in ALMPs related to job search assistance and training; and (c) the employment contracts of those who have found a job. To evaluate the effectiveness of these programmes, we use the matching method based on the probability of assignment to treatment (propensity score matching, PSM), because the delimitation of the groups of people who receive services (treatment group) and those who do not receive them (control group) was not carried out through a randomized experiment (or lottery). The objective of this technique is to restore the conditions of an experiment by constructing a comparison group appropriate to the treatment group, both groups being as similar as possible in terms of their observable characteristics.³

Our findings suggest that there is a positive and significant differential effect on the probability of transition from unemployment to employment and on the probability of finding a job of ‘intermediate quality’ among jobseekers who receive employment services compared to jobseekers who do not receive them. This positive impact is largely due to effects of training and qualification programmes.

The remainder of this article can be outlined follows. In section two, we briefly review the empirical literature focussed on the assessment of the effects of ALMPs on labour outcomes. In section three, the PSM technique is presented. Section four describes the database and the delimitation of the treatment and comparison groups. Section five contains the results of the estimation of the models. Finally, section six presents the conclusions.

2 Review of the literature

ALMPs have different aims⁴ and can be grouped into different categories.⁵ Concerns about the effectiveness and efficiency of such measures have gained special attention from policymakers in the EU Member States. Numerous microeconomic studies on the impact of these types of measures in developed countries can be found in the literature. The review carried out in this section focusses on analyses of the impact of ALMPs on the participants and offers an overview of its main results in national and international literature. In particular, we focus on job search services, including measures such as counselling and monitoring, search assistance and placement management, and training, both in the classroom and on the job.

First, although the results of the existing impact assessment studies are diverse, they suggest that ALMPs produce positive treatment effects, albeit small in size, on the participating people. This occurs with training measures for the unemployed (Martin and Grubb 2001; Raaum and Torp 2002; Bergemann et al. 2009; Fitzenberger et al. 2012), although there are studies that find insignificant or even negative treatment effects (Sianesi 2004), while it also happens in the case of job search assistance programmes, with many studies finding positive treatment effects (Dolton and O'Neill, 2002; Blundell et al. 2004; Graversen and van Ours 2008; Malmberg-Heimonen and Tge 2016), although others have found them null or sometimes negative (Van den Berg and van der Klaauw 2006; Centeno et al. 2009). The meta-analysis results tend to confirm the previous findings (see Heckman et al. 1999; Kluve and Schmidt 2002; Kluve 2010; Card et al. 2010, 2018; Vooren et al. 2019).

Second, the impacts of ALMPs differ according to the type of action. In the sample in Kluve (2010), the estimates with significantly positive treatment effects represent 73.9% of those referring to hiring incentive programmes, 71.4% of those for job search assistance services, 54.3% among those of training actions and 26.9% among those for direct job creation. These differences according to the type of programme are maintained when estimates are made that control for programme characteristics, the techniques used in the studies or participant attributes.

Third, the effects of ALMPs also differ according to the time horizon considered. Training programmes (and incentives for hiring in the private sector) tend to have null (or even negative) average effects in the short term, together with more positive average impacts in the medium and long term (Forslund et al. 2011; Crépon et al. 2012; Card et al. 2018). In contrast, job search assistance programmes are more likely to be effective primarily in the short term or show more stable positive impacts over time (Sorensen 2016). In Card et al. (2018), the treatment effects of job search services are around 1–2 percentage points (pp) in all time horizons, while those of training actions are 2 pp in the short term and almost 7 pp in the medium and long term. These time profiles for effects are consistent with the nature of the programme groups. This would imply that participants obtain higher labour gains, on average, through actions that emphasize the accumulation of human capital than through those that focus on getting workers back to work as quickly as possible.

Fourth, the impact of these programmes varies for different groups of participants. ALMPs produce very positive treatment impacts for the long-term unemployed, especially in the case of training programmes and, to a lesser extent, job search assistance services (Card et al. 2018). Although ALMPs are less likely to produce positive effects (and these effects are more likely to be small) for participants who are young or older workers, specific meta-analyses referring to these groups suggest that training seems to be beneficial for young people, especially if it involves on-the-job training or is combined with internships or work experiences (Kluve et al. 2019; Ghisletta et al. 2021), while a combination of training and job search services (and wage subsidies) are better for older workers (Orfao and Malo 2023).

In the Spanish case, the evaluation of training and job search programmes has been scarcer due to the lack of available data. Most such work has analysed training actions, used regional data, and relied on PSM techniques. Thus, there is a growing body of impact evaluation studies of training programmes for the unemployed carried out by PES (Mato and Cueto 2008; Ramos et al. 2009; Cueto and Mato 2009; Arellano 2010; Cueto et al. 2010; Cansino and Sánchez Braza 2011; Clemente et al. 2014; Alegre et al. 2015; Blázquez et al. 2012, 2019). The results of most of these works are positive. Normally, the estimated treatment impacts imply a positive differential in the employment rate of participants with respect to that of non-participants (or an increase in the probability of being employed) of between 5 and 10 pp. The size of this effect is similar to that found by studies for training measures developed in other countries that have obtained positive results.

In any case, the effects differ according to the characteristics of the unemployed people and the training actions. In general, higher effects tend to be found for the most disadvantaged groups in terms of employment, such as women, people with lower levels of education and the long-term unemployed. The few works that have examined the duration of actions (e.g. Cueto et al. 2010) find that longer courses produce greater effects, but these are only slightly higher than those of medium duration, which, given the cost, would imply that the latter are more efficient. In addition, programmes that provide relatively more specific training or that are offered in combination with other types of active policies, such as individualized guidance, improve employment access results.

The evidence regarding evaluations of job search assistance services in Spain is much more limited (Herrarte and Sáez 2008; Ramos et al. 2009; Blázquez et al. 2019). The treatment effects found are usually positive, with a differential impact for participants (compared to non-participants) that ranges between a 1 and 5 pp greater probability of moving towards employment or being employed at a given time after participation. The size of this effect differs depending on the workers’ attributes and the type of action. It is higher when job search services are combined with other types of programmes, mainly training courses.

Finally, Malo and Cueto (2016) carried out a meta-analysis of the impact evaluations of Spanish ALMPs that used a comparison group. Their meta-analysis included 12 studies (published between 2008 and 2014) that make up a total of 144 impact estimates. Grouping the evaluations into training courses, job orientation/intermediation and hiring incentives, the authors found positive impacts on average. Training actions and job search assistance services increased the probability of accessing a job by up to 5–6 pp, while hiring incentives did so by less than 4 pp.

3 Methodology

Our objective here is to estimate the impact of jobseeker participation in ALMPs on the probability of finding a job using impact evaluation techniques. To do so, we follow the Roy–Rubin econometric model (see Roy 1951; Rubin 1974), which involves determining what the impact of treatment of individuals would have been on an outcome of interest if they had not been treated. The problem in this case is that, to calculate the average treatment effect (ATE) of the programme, we must compare the results of a ‘control group’ or ‘counterfactual’ with those of the ‘treated group’:

$$ATE=E\left({Y}_{1}|T=1\right)-E\left({Y}_{0}|T=0\right)$$

(1)

where T is a binary treatment variable that takes the value 1 if the individual receives treatment and 0 otherwise, and Y(T) is the potential outcome for each individual receiving the treatment (Y₁) or not (Y₀). The ATE corresponds to a situation in which a randomly chosen individual from the population is assigned to participate in the programme, so participating and non-participating individuals have an equal probability of receiving the treatment.

The problem is that the delimitation of the group of people who receive services provided by the PES (treatment group) and that made up by those who do not receive them (control group) has not been carried out through a random experiment. In the absence of an experiment, we can estimate the average treatment on the treated (ATT) of the programme given by the following expression:

$$ATT=E\left({Y}_{1}|T=1\right)-E\left({Y}_{0}|T=1\right)$$

(2)

where the expected value of the ATT is the difference between the expected outcome values with and without treatment (first and second term, respectively) for those who participate in treatment.

In this context (see Caliendo and Kopeinig 2008; Khandker et al. 2010), the impact of the intervention is ATE = ATT + B, where the term B (E [Y₀|T = 1] – E[Y₀|T = 0]) is the extent of the selection bias that appears when we do not have a random experiment (with an experiment, B = 0). The basic objective of a good impact assessment is then to find ways either to get rid of the selection bias or to account for it. For that reason, because the second term in Eq. (2) is not observable and we do not have a random specific assignment rule to the programme, we use a quasi-experimental method – that is, propensity score matching (PSM) – to identify the ATT.

The PSM method constructs a statistical comparison group by modelling the probability of participating in the programme on the basis of observed characteristics unaffected by the programme. Participants are then matched to non-participants based on this probability using various methods (algorithms). The ATT of the programme is calculated as the mean difference in outcomes between these two groups.

Carrying out the matching method can be difficult if it is conditioned on many variables, because this would imply finding a pair for all the participants among the non-participants with the same observable characteristics. This is known as the ‘curse of dimensionality’. To avoid this problem, Rosenbaum and Rubin (1983) proposed synthesizing all the information from multiple variables into a single variable and then calculating, for each unit of the treatment and potential comparison groups, the probability of participating in the programme conditioned on the observed values of its characteristics. This is the so-called ‘propensity score’, P(X), which is calculated as follows:

$$P(X)=Pr(T=1|X)$$

(3)

This score is a real number between 0 and 1. As the first step, this probability is normally estimated using a discrete choice model (logit or probit).

There are two strong assumptions for identifying the estimates of the ATT of a programme using PSM: (1) the conditional independence assumption (CIA); and (2) the presence of common support. The CIA states (see Rosenbaum and Rubin 1983) that, given a set of explanatory variables (X) that are unaffected by the treatment (T), the potential outcomes (Y) are independent of the treatment assignment:

$$\left( {{\text{Y}}_{{1}} ,{\text{Y}}_{0} } \right) \bot {\text{ T}}|{\text{X}}$$

(4)

This implies that participation in the programme is based on observed characteristics and there is no difference between the treated and comparison groups conditioned on X. As Imbens (2004) points out, if this condition is met, conditioning on the propensity score removes all biases due to observable attributes. CIA is a strong assumption. We support this by making it possible to control for many observed characteristics affecting the programme (assuming that unobserved selection is limited). Nevertheless, if any unobserved factors affect participation status and the outcome of interest, a hidden bias (unobserved heterogeneity) is involved (see Rosenbaum 2002). This bias is related to the potential impact of confounding variables not available to researchers, such employment histories or soft skills and personality traits (Lechner and Wunsch 2013; Caliendo et al. 2017). Later, we test to ensure there is no hidden bias and, therefore, that the CIA holds.

In relation to the assumption of common support, the following must be met:

$$0 \, < {\text{ P}}\left( {{\text{D}} = {1}|{\text{X}}_{{\text{i}}} } \right) \, < {1}$$

(5)

The basic intuition is that there must be at least one similar individual in the potential comparison group for each treated individual. This assumption ensures that the observations of the treatment group have ‘close’ comparison observations in the propensity score distribution (Rosenbaum and Rubin 1983; Heckman et al. 1999).

In the second step, treated and untreated individuals are then matched based on this closest probability or score with matching algorithms. By pairing treated individuals with counterfactuals with a similar probability of participating in the programme, the problem of sample selection bias is avoided.⁶ If CIA holds, as well as common support in P(X) across treated and non-treated individuals, the PSM estimator for the ATT can be written as the mean difference in Y over the common support, weighting the comparison group by the propensity score distribution of the treated group⁷:

$${\text{ATT}}_{{{\text{PSM}}}} = {\text{ E}}_{{{\text{P}}\left( {\text{X}} \right) \, |{\text{ T }} = { 1}}} \left\{ {{\text{E}}\left[ {{\text{Y}}_{{1}} |{\text{T}} = {1},{\text{P}}\left( {\text{X }} \right)} \right] \, {-}{\text{ E}}\left[ {{\text{Y}}_{0} |{\text{T}} = \, 0,{\text{ P}}\left( {\text{X }} \right)} \right]} \right\}$$

(6)

Different matching methods can be used to do this and obtain estimates of the outcomes of those assigned to the treatment and comparison groups.⁸ All these matching methods use algorithms to calculate the ATT but differ in the definition of the neighbours for the treated individuals and the weight assigned to them. The question is how one should select a specific matching algorithm. Caliendo and Kopeinig (2008) compare the trade-offs in terms of bias (distance of estimated treatment effect from true effect) and variance or efficiency (precision of estimated treatment effect) for different matching algorithms (the nearest neighbour with and without calliper, with single or multiple neighbours, with and without replacement, with radius or kernel matching, etc.). They concluded that there is not a winner for all situations: the choice depends on the data structure, so that if the results are similar, the choice may be unimportant. Moreover, all PSM estimators should yield the same results asymptotically, because they all become closer to comparing exact matches with large sample sizes (Smith 2000).

In our case, the data source provides us with a large sample. We have tried several approaches and tested the sensitivity of the results with respect to the algorithm chosen. In the end, we have chosen ‘the nearest neighbour (one to one) matching with replacement and without calliper’ because, being just one matching variant, it is the most straightforward estimator. This algorithm also increases the average quality of matching and decreases the bias compared to other estimators without replacement (see Caliendo and Kopeinig 2008). Nevertheless, we have tried other PSM estimators and obtained similar results (see Table A of the Appendix).

4 Data

4.1 Dataset

To carry out the analysis of the impact of the services provided by the PES on the job placement probabilities of the participants, the administrative records of the PES for the region of Madrid for the period January 2018–December 2019 were used. These provide information contained in three types of microdata files: (a) jobseekers registered with the employment offices; (b) services related to job search assistance and training received by participants; and (c) employment contracts of those who have accessed a job. The links between these microdata can be made thanks to the availability of a unique anonymized personal identification number (PIN) for each person in the three files. This PIN makes it possible to identify the monthly information of the same person in all files, merge it and save it in a new database.

First, the jobseekers’ records include the universe of (employed and unemployed) jobseekers registered at public employment offices.⁹ Registration is voluntary, except for those receiving benefits and those who want access to ALMPs and support for active job search. These records contain information on personal and labour characteristics (sex, age, education, nationality, specific occupational work experience, unemployment benefit recipiency, time enrolled at PES, etc.) and features related to the job search process (geographical area of job search, type of workday selected in their job applications, desired occupation, etc.). On average, 430,000–450,000 unemployed jobseekers were registered as such each month in 2018.

Second, records of the services provided by the PES have been merged with the jobseekers for each month and person to identify which individuals in the sample have participated in an active action and which have not each month in 2018. All the jobseekers are entitled to participate in programmes if they request it, so those who ask for a service receive it. The caseworkers at the employment offices decide on the type of service they offer to the jobseekers after a personal interview according to the individual’s request¹⁰ Jobseekers do not compete for services. We focussed on training programmes and services intended to facilitate the job search process of the worker¹¹ In 2018, there were around 866,000 actions provided by the PES, and 274,190 different people participated in them.

The duration of the different actions is diverse. Because information on the start and the end date of each service is available in the dataset, we can calculate their duration. On average, the actions that took place in 2018 had a duration of 29 days. This mean that duration differs according to the type of service: 197 days for employment pathway, 1.2 days for professional information, 4.4 days for career guidance and 96 days for training, on average. Moreover, the mean number of services jobseekers received from employment offices in 2018 was 3.2, with 44.2% of them receiving only one service and almost 15% receiving six services or more. Merging the files allowed the aggregation of the information of all services received in 2018 by each person each month, so all information about their participation in ALMPs is available for the analyses.

Third, the contract files were used to identify which jobseekers have gained a contract and which have not. The period in which this is recorded refers to the first three months of 2019 (January–March). This period was selected to prevent the jobseekers from having received another service in the first months of 2019 or being affected by another type of intervention, which would have produced various effects that would not allow the impact of participation in each programme to be captured correctly.

4.2 Definition of the treatment and comparison groups

The procedure for merging the individual information contained in the three microdata files allowed the delimitation of a treatment group (jobseekers who have received services provided by the employment offices) and a potential comparison group (similar jobseekers who have not received services).

To delimit the treatment group, we considered whether jobseekers received services in December 2018 or in the previous three months. For this reason, our sample of participants was made up of unemployed jobseekers registered in December 2018 who received job search assistance or training either that month or, at most, in the previous three months – that is, from September to December. Thus, we distinguish individuals who received treatment during: (a) September to December; (b) October to December; (c) November and December; and (d) December. Because jobseekers may receive multiple employment services, we focussed on the impact of the last type of service received. Our sample of non-participants comprised unemployed jobseekers registered in December 2018 who did not participate in any ALMP measure in that month or previously (i.e. they did not receive services in any month of 2018, nor during the first three months of 2019). This makes it possible to apply impact assessment techniques by defining a counterfactual to quantify the effect of participation in ALMPs on the probability of finding a job in the months from January to March 2019.

In selecting the sample, we followed a procedure akin to that employed by other published works using similar administrative data. For instance, Blázquez et al. (2019), using the same dataset for the region of Madrid in the period 2010–2012, selected all the individuals registered as unemployed at employment offices in December 2010 and defined as non-participants all those who did not participate in any ALMP measure during a period of three consecutive years.¹² Although our selection may potentially induce an under-representation of spells of short duration, this does not seem to have been the case. More than 40% of jobseekers had durations equal to or less than six months, and the share of long-term unemployment was around 55%, without significant differences between the treatment and comparison groups (see below). This proportion coincides with that published by the Spanish Labour Force Survey (54% in 2018).

We have deleted observations for people under 16 or over 65 years of age, as well as those with some type of disability. The final sample was thus made up of 316,769 jobseekers in December 2018, of whom 246,360 (77.8%) did not receive any service in 2018, while 70,409 (22.2%) received at least one service that year, the last of which ended between September and December 2018. In view of these data, there are enough individuals in the potential comparison group sample relative to each person included in the treatment group sample, among whom the best match was sought according to their observed characteristics. The data revealed a considerable similarity in the profiles of both participants and non-participants (see Table 5 below): the difference in means was zero or close to that value for most of the variables and categories considered. The average profile of the treated person was a male, Spanish, almost 43 years old, with a medium level of education, who had been unemployed between 1 and 6 months, had worked in the service sector, and who had been looking for a job in ‘Madrid capital’. The average profile of the untreated person was quite similar.

5 Results

Our research sought to evaluate the effectiveness of the services provided by the PES. This section presents the results of the estimation of the PSM models. First, we analyse whether this intervention improved the jobseekers’ transition from unemployment to employment. Second, we focus on the impact of participation on job quality. Third, we examine whether the effects of ALMPs differ by type of programme and socioeconomic group. Finally, we assess the quality of the matching procedure.

5.1 Job finding results

To perform the evaluation, the PSM technique discussed above was used. The application of this technique consisted of several stages. The first stage involved estimating the probability of participating in a programme provided by the employment offices based on the observations of the group of jobseekers who received services and those who did not. To determine this probability, a binomial probit model was estimated, where the dependent variable Y took the value 1 if the person participated in a programme and the value 0 if the person did not participate, using as explanatory variables those that can affect the probability of receiving a service and, therefore, the matching function, such as sex, age, nationality, educational level, area of residence, economic sector (industry) of the previous job, an indicator of whether they receive unemployment benefits and an indicator of whether the person is among the long-term unemployed.

Table 1 provides the results of the estimation of the probit model. Coefficients are presented for each category. The results of various specifications of the model are shown in each of the columns: the probability of treatment in December 2018 after having received a service in the last four months (from September to December 2018) in Column (1); in the last three months (from October to December 2018) in Column (2); in the last two months (November and December 2018) in Column (3); and in the last month (December 2018) in Column (4).

Table 1 Results of the estimation of the propensity score: probit model (Y = 1 if the person is treated, Y = 0 if the person is not treated)

While there were hardly any differences by nationality and sector of economic activity on the probability of treatment, women, young people (16–30 years old), individuals with vocational training or university education, persons who did not reside in ‘Madrid capital’, workers receiving unemployment benefits and the long-term unemployed were more likely to receive services provided by the PES.

The second stage of the PSM technique consisted in evaluating the impact of the services provided by the employment offices using the comparison groups built from the potential comparison individuals that most resemble the treatment groups in terms of estimated probability (or propensity score). To do this, after calculating the probability of treatment, we matched people from the treatment group to the potential comparison group. The algorithm used for the matching was ‘the single nearest neighbour (one-to-one) matching with replacement and without calliper’. We used this algorithm to quantify the impact of participation on the probability of having a work contract during the first three months after receiving a service provided by the PES.

Table 2 (Panel A) provides the main results of the evaluation exercise for the sample of workers who received their ‘last’ service at different times in 2018: either in the last month (December) of that year (Row (4)) or in a previous month (Rows (1) to (3)). In particular, the probability of having a contract three months after receiving a service for the treated and untreated, the ATT and the significance of these parameters are presented (‘t-stat’ indicates the t-statistic under the hypothesis that the ATT effect is null).

Table 2 Causal effect (ATT) of receiving services (job search assistance and training) on the probability of being hired during the first three months after participation, according to when the service was received

The results of the ATT estimation indicated that the impact of participation was positive and statistically significant, so that jobseekers who received services provided by employment offices were more likely to transition from unemployment to employment than similar non-treated jobseekers. The differential in probabilities between the two groups ranged between about 5 pp for those people who received a service at most four months before the moment of observation as jobseekers (between September and December 2018) and 10 pp for those people who received it within the last month (December 2018).

Focussing on this last sample (jobseekers who had received the service in the last month), the estimated probability of being employed after treatment was 24.5% for the treatment group and, according to the model, it would have been 14.6% for those treated in the absence of the service. This means that the estimated causal effect of participation in the active policies being analysed increases the probability of getting a contract by 9.9 pp, which is a large impact. The results thus indicate that the services provided by the PES are benefiting the employment of their recipients.

5.2 Job quality results

We now turn to assess whether the ALMPs provided by the employment offices helped participants to find higher quality jobs relative to the employment they would have achieved in the absence of participation. The variables used to measure employment quality were type of labour contract, working hours and contract duration. A composite quality indicator that combined these three variables was constructed. This indicator was constructed such that it presented the following categories: ‘high quality’, if the contract was permanent, full-time and lasted more than one year; ‘low quality’ if the contract was temporary, part-time and lasted one year or less; and ‘intermediate quality’ for the rest of the cases (permanent, part-time contracts of any duration, and temporary, full-time contracts of any duration). For the sample of jobseekers who received services between September and December 2018, the majority of those who signed a contract in the first three months of 2019 did so for ‘intermediate quality’ jobs (accounting for 75.9% of the total), while the rest found either ‘low quality’ jobs (14.3%) or ‘high quality’ jobs (9.8%).

Table 2 (Panel B) provides the results of the estimation using the four treatment and comparison groups previously presented with ‘the nearest neighbour’ algorithm for the three job quality indicator categories. The results of the ATT estimation showed that the effect of participation for jobseekers who received PES-provided services only had statistically significant and positive effects on the probability of transitioning from unemployment to ‘intermediate quality’ employment. These effects ranged between 4.4 pp for those people who received a service at most four months before the moment of observation (between September and December 2018) and 8.4 pp for those people who received it in the last month (December 2018). Thus, the positive effects are of greater magnitude the more recently the service was received.

We also estimated the effect of ALMPs separately for the three different job quality variables. The results of the estimation (not shown, but available upon request) showed that the services provided by the PES increase the probability of leaving unemployment for a job with a temporary contract, but do not exhibit significant effects on the probability of exiting unemployment for a permanent contract. At the same time, they showed that the services increase the probability of leaving unemployment for a full-time job, but do not have significant effects on the probability of exiting unemployment for a part-time contract. Finally, they suggested that participation in the programmes increased the probability of exiting unemployment for a contract with an effective duration greater than 6 months.

5.3 Heterogeneity

In this section, we focus on the effects of the ALMPs on the employment of the participants but disaggregated by type of service and groups of workers. For this, five categories of services were considered, which resulted from the aggregation of similar actions provided by the public employment offices (‘employment pathway’, ‘guidance and orientation’, ‘professional information’, ‘intermediation’ and ‘training’); and different groups of workers were defined according to a set of sociodemographic and labour variables (sex, nationality, age, level of education, duration of unemployment, and receipt of unemployment benefits).

The results by programme type are provided in Table 3. The ATT estimation showed that the impact of participation in almost all services had positive and statistically significant effects (except in the case of ‘employment pathway’) on the probability of transitioning from unemployment to employment based on the sample of participants whose last service was received in December 2018. These probabilities were higher for ‘intermediation’ and ‘training’ actions, whose differential effects were 14 and 11 pp, respectively. The positive effects were somewhat smaller in the case of ‘guidance and orientation’ and ‘professional information’ services, reaching 6 and 8.7 pp, respectively.

Table 3 Causal effect (ATT) of receiving services on the probability of being hired in the first three months after participation, according to when the service was received, by type of programme

An interesting result is that the effects on the probability of finding a job varied depending on the period when the considered services were received. In general, the impacts found tend to decrease when we were less restrictive in terms of the period elapsed between participation in the last service and the moment at which the possible exit towards employment begins to be measured. In the case of ‘guidance and orientation’ and ‘professional information’, the effects were positive and significant when the last service was received during December 2018, but they were positive and statistically insignificant when the last service was received earlier. The effects of the ‘employment pathway’ actions were not statistically significant in any of the cases. Finally, the impacts of ‘intermediation’, which were always statistically significant, fell from 14 to 8 pp by extending the period considered, while those of the ‘training’ actions, as their duration was longer over time, were maintained, showing differential effects of 10–11 pp regardless of the period considered.

These findings are in line with those obtained in other studies. For instance, Blázquez et al. (2019), with data also referring to the region of Madrid but in the period 2010–2012, found that among long-term unemployed jobseekers who participated in training, the likelihood of finding a ‘significant’ job in the short term (one year after participation) was 8.8 pp higher than that of non-participants, while the corresponding difference in job search and assistance programmes was 2 pp (the effects were somewhat smaller when considering a ‘non-significant’ job). This relatively large effect of participation in training and qualification programmes has also been observed in other empirical works for Spain (see Cueto et al. 2010).

The results by socioeconomic group are provided in Table 4. The impact of participation in any type of services on the probability of exiting from unemployment to employment after treatment was positive and significant for all groups of participants compared to those who did not participate. These probabilities were slightly higher among men than women; among natives than non-natives; among workers in the central age groups (i.e. 31–54 years) than those at the extremes of the working life (i.e. under 30 and over 55 years of age); among those with vocational training or university education than those with a maximum of primary education or general secondary education; among the short-term unemployed (less than one year unemployed) than the long-term unemployed (one year or more unemployed); and among the non-recipients of unemployment benefits than among recipients.

Table 4 Causal effect (ATT) of receiving services on the probability of being hired in the first three months after participation, by socioeconomic characteristics

Our results are, to some extent, in line with previous work, which has found higher positive effects of training among males and native workers (Clemente et al. 2014; Blázquez et al. 2019), although other studies have provided evidence of either higher impacts among women or no significant differences, especially for training programmes (Arellano 2010; Card et al. 2010). By age, the empirical literature has usually found that ALMPs for youths and older workers are less positive than for other groups (Card et al. 2018). The differences might be partially explained by the fact that young workers not only have less work experience, but also fewer job search skills because they have entered the labour market more recently than older jobseekers.

Regarding older workers, the interaction of the variable for receiving unemployment benefits with the age group of 55–65 years (the category of those most likely to be disengaged from the labour market, among other reasons because they are receiving special assistance benefits for older workers) in our estimation yielded no significant results. This means that participation in activation programmes does not produce any positive and significant differential effect on the employment of this group.

5.4 Matching quality assessment

One of the keys to the validity of PSM (estimated with a probit model) is the ‘overlap’ or ‘common support condition’. The set of explanatory variables included in the probit regression makes it likely that the labour outcome of the treated and comparison groups, given the propensity score, differs only due to treatment, and hence, the CIA holds. To ensure that this condition is met, we performed a graphical analysis and plotted the propensity score distribution for the treated and comparison groups to assess the overlap achieved. The results are displayed in Fig. 1. These suggest that the overlap is quite good in the four different specifications depending on the time of receipt of the service provided, although there are differences in the density of the propensity score that can be observed in the tails of the distribution.

Fig. 1

Evaluation of the quality of the match: distribution of the probability of receiving services (treated) and of not receiving them (not treated). Results of specifications (1) to (4) of the probit model shown in Table 2, Panel A. Source: PES-CM administrative records. Note: Own elaboration based on the data provided by the PES-CM

Moreover, when performing PSM, it is crucial to assess the quality of the match. This means that we should determine whether the ‘balancing’ or equilibrium property is satisfied or not. That is, if a good balance has been achieved, the marginal distribution of each variable would be similar in the treatment and comparison groups. There are a few methods to test this ‘balance’ assumption. One of the most commonly used is the standardized percentage bias proposed by Rosenbaum and Rubin (1985), which is the percentage difference of the sample means in the subsamples of the (complete or matched) treatment and comparison groups as a percentage of the square root of the mean of the sample variances of both groups. A reduction of the standardized bias after matching could prove that the balance of variables improves with matching. Rosenbaum and Rubin (1985) consider a standardized mean difference with an absolute value less than 20 as acceptable.

We used the command pstest in Stata to perform the calculations for this method. For example, Table 5 reports the results after PSM estimation using ‘one-to-one matching with replacement and without calliper’ (specification that corresponds to Row (1) of Table 2, Panel A). The results are quite similar for alternative matching estimators. This table also presents the unmatched (U) and matched (M) means of the variables for treated and untreated individuals, the percentage of bias, and the t-statistic, which tests the hypothesis that the mean value of each variable is the same in both groups. The t-statistic has been calculated before and after matching.

Table 5 Unmatched (U) and matched (M) means of the variables for the treatment and comparison groups, percentage bias, and t-test on the hypothesis that the mean value of each variable is the same in the treatment and comparison groups

The results indicate that the levels of standardized bias are very low, as they are always below 20 (note the bias percentage for M). In addition, the t-statistic indicates that the null hypothesis cannot be rejected at the 5% level of significance for each variable after matching. The same holds true when looking at Fig. 2, which allows one to assess the quality of the match by showing the percentage of bias before and after matching. In summary, all the information just presented leads to the conclusion that the balance of the variables and the quality of the matching achieved in the present study is quite good.

Fig. 2

Source: PES-CM administrative records. Note: Own elaboration based on the data provided by the PES-CM

Evaluation of the quality of the matching. Results of specifications (1) to (4) of the probit model shown in Table 2, Panel A.

Table A.1 of the Appendix shows the estimates of the ATT with different algorithms, such as the single nearest neighbour with and without replacement, with or without calliper (0.01, 0.05 or 0.1), and matching with multiple neighbours (5, 10 or 15). The results correspond to the model specification shown in Row (1) of Table 2, Panel A. The magnitudes of the ATT with alternative matching algorithms are very similar to those obtained with the algorithm we used.

Finally, our strategy assumed that the results are independent of the treatment once we control for the measured characteristics and are dependent on the available observable attributes. This implies that all existing selection bias is assumed to be determined by the attributes used as variables in the propensity score estimate. However, unobservable characteristics may play a relevant role. Any characteristic associated with the programme participation and the outcome variable, conditional on the observable explanatory variables, can induce bias (the so-called hidden bias). In our context, unobservable features could be potentially relevant.¹³ As it is not possible to test directly whether the PSM estimates are free of hidden bias, a sensitivity test was carried out to assess the robustness of our results to the presence of this bias, using the Rosenbaum bounds strategy.¹⁴

This test considers that the probability of being treated is a function of observed and unobserved factors (Rosenbaum 1987). If there is no hidden bias, then the effect of the unobserved factors (γ) takes the value zero, so it has no effect on the probability of participation. This means that the unobserved factors are the same in the treated group as they are in the non-treated group. Assuming that the probability of being treated follows a logistic distribution, the only case where treated and untreated individuals have the same probability is when e^γ = 1. In this case, no hidden bias is present, and the conditions of CIA are maintained. Higher values of e^γ would indicate that there is a hidden bias. Becker and Caliendo (2007) developed a Stata routine for implementing this sensitivity analysis by exploiting the Mantel–Haenszel (QMH) statistical test. Rosenbaum (2002) has shown that for values of γ > 1, the QMH test is limited by two distributions QMH + and QMH − . These represent the case in which the ATT has been overestimated or underestimated, respectively.

Table A.2 of the Appendix provides the results of this test with the limits of QMH and its level of significance for different values of e^γ for ‘the nearest neighbour’ PSM estimator. The QMH bounds for e^γ = 1 yielded a scenario where the ATT estimation was free from hidden bias, and both were statistically significant. The higher values of e^γ represented the effect that an unobserved factor would have on the probabilities of receiving treatment to justify the estimated ATT. QMH − was always significant, and for QMH + the results were only sensitive to the existence of an unobserved factor that would increase the probability of being treated for one specific value. Therefore, we can safely conclude that our estimates are quite robust to the existence of hidden bias. This result aligns with previous ones pointing out that, although unobservable variables may play a role for selection into ALMPs, they do not make a significant difference in estimating treatment effects on employment prospects if rich administrative data are available (Lechner and Wunsch 2013; Caliendo et al. 2017).

6 Conclusions

This paper focussed on the study of two broad types of ALMPs used to combat unemployment: job search assistance and training. Specifically, we analysed the effect of participation in these programmes on the employment opportunities and the job quality of unemployed jobseekers. Jobs were classified as high, intermediate and low quality depending on features related to the type of contract, working hours and duration of the contract. Employment success was measured as having obtained a contract during the first three months after participating in an active action.

We merged microdata from the PES administrative records for the Community of Madrid in Spain covering the period January 2018–December 2019; selected those people who were registered as unemployed jobseekers in December 2018; defined several treatment groups consisting of participants in active programmes in that month or, at most, in the three previous months, with their most recent service received from the employment offices in 2018; and constructed a potential comparison group consisting of non-participants. PSM techniques were used to quantify the causal effect of participation in ALMPs on employment and job quality.

Our evaluation analyses revealed that the overall participation of jobseekers in ALMPs exerted a positive and significant impact on their probability of finding a job compared to non-participants. The impacts ranged from a differential probability of 5 pp for those who received a service at most four months prior to observation as jobseekers (between September and December 2018), to 10 pp for people who had received a service in the last month (December 2018). These effects concentrated on the likelihood of getting an intermediate quality job, which reflects the positive impact of participation on the probability of finding a job with a temporary, full-time contract lasting at least six months.

Participation in almost all of the programmes considered had positive and significant effects when using the sample of participants whose last service was received in December 2018. These effects were higher for intermediation and training actions and smaller for orientation and job search services. The effects also varied depending on the period when services were received. These impacts tended to decrease as time elapsed between participation in the last service and the moment of measurement increases, except for training actions. The job search assistance services did not produce significant effects when considering longer participation periods. Finally, the positive influence of participation appeared to be more intense for specific groups: men, natives, middle-aged people, individuals with vocational training or college studies, the short-term unemployed and non-recipients of unemployment benefits.

Summing up, three main findings can be derived from this evaluation. First, the participation of jobseekers in ALMPs could allow them to enhance their employment rates and increase their chance of finding a job of a certain quality. Second, the treatment effect associated with participation of some disadvantaged groups in the labour market, such as the young and the long-term unemployed, although smaller than other groups, is still positive. Third, the influence of training on the probability of a jobseeker’s transitioning to employment is more effective than participation in job search assistance programmes.

Thus, our empirical results support the consensus of prior research with respect to the need to help unemployed workers improve their chances of returning to employment. Despite this, more effort should be directed at prevention measures. In this context, profiling tools could help PES to identify jobseekers who are more likely to become long-term unemployed and provide them with early activation. Although our results showed that the positive impact of training is higher than that of job search assistance services, the effects of the latter are usually also positive. Given its low unit cost, orientation and job search measures could also be a useful tool to improve the employability of jobseekers, especially for the long-term unemployed.

Appendix

See Tables 6, 7

Table 6 Causal effect (ATT) of receiving services (job search assistance and training) on the probability of being hired during the first three months after participation: services received in the last 4 months (during September to December 2018)

Table 7 Sensitivity to the presence of hidden bias