Is your interview Valid?

Are you sick of a Hiring Managers arbitrary biases being put ahead of your data driven insights?

We've all been there. You want to set up a really good technical & behavioural interview. Questions linked to specific success criteria. Review guidelines all written up. Interviewers with the right training.

Then the hiring managers says :

"But only if they have 10 years experience and got a 1st from a good college"

Then you cry inside... knowing that this offhand comment all but guarantees an increased cost and time to hire, as well as kill any progress towards your diversity goals.

What's more, you have little negotiating power. This is how they've done it for ten years, how can you fight against that?

Here's the good news. People have been studying how well different interview formats and criteria predict future success for decades.

There are hundreds of studies and multiple meta-analyses on this topic and the results are entirely in our favour.

So, how can we quantify how well something predicts future success? Validity coefficients.

[Book mark this bad boy so you can use the references to your advantage]

What is a Validity Co-Efficient?

A validity coefficient is a statistical measure that indicates the degree to which a test or assessment accurately predicts a specific criterion.

It shows how well a particular assessment measures what it is intended to measure.

The validity coefficient is typically represented as a correlation coefficient, ranging from -1.0 to +1.0.

+1.0 = a strong positive relationship, meaning the test is a good predictor of the criterion.

-1.0 = a strong negative relationship.

Around 0 suggests no relationship, indicating that the test does not predict the criterion effectively.

In the context of recruitment, a high validity coefficient for a selection method (like an interview or test) suggests that it is a good predictor of job performance. This is important for making informed hiring decisions, ensuring that the candidates selected are likely to perform well in the job.

Validity Co-Efficients of Common Assessment Formats

Let's explore the effectiveness of years of experience, college degrees, behavioural interviews, and in-person skills tests in predicting job success:

Years of Experience

The predictive validity of years of experience for job performance is relatively low, with a validity coefficient of 0.18. This suggests that relying heavily on years of experience in recruitment can be misleading.

Overemphasis on years of experience might also introduce biases and overlook candidates with high potential but less experience.

College Degree

Interestingly, the validity coefficient for years of education is only 0.10. This indicates that educational qualifications, such as possessing a college degree, are not strong predictors of job performance (Massive caveat here depending on specific role. Completing a medicine degree... HUGE indicator of success for doctors of course).

This finding challenges the traditional emphasis on formal education in recruitment processes.

Behavioural Interviews

Behavioural interviews, where candidates are asked to describe past behaviour and experiences, are a common part of many recruitment processes. While specific validity coefficients for behavioural interviews aren't as available as I'd like, we know that structured interviews, which often include behavioural questions, have a higher validity coefficient of around 0.51.

This suggests that behavioural interviews, especially when structured, can be effective in predicting job performance.

Situational Skills Tests

Skills tests, which can include practical tasks like pair programming, or simulations relevant to the job, have a high predictive validity.

Work sample tests have a validity coefficient of 0.54, making them one of the more effective methods in predicting job performance. These tests provide a direct assessment of a candidate's ability to perform job-related tasks.

This is data from multiple studies and sites i'll reference below, but it's worth noting that you can measure the validity coefficient of interviews in your own business using criterion-related validation.

Calculating your own Company's Validity Co-Efficients

This process requires correlating the scores from your interviews and assessments with an objective measure of job performance.

Determine the Criterion (Job Performance Measure): First, you need to define what constitutes successful job performance in your organisation. This could be sales figures for a sales role, quality of coding for a software developer, or customer satisfaction scores for a customer service position. Collect Assessment Data: Gather data from your interviews and assessments. This could include interview scores, test results, or ratings from assessment centres. Collect Performance Data: After a certain period (e.g., six months or a year), collect performance data for the same set of employees. This should be quantifiable data that reflects their job performance, aligned with the criteria you defined earlier. Correlate the Two Sets of Data: Assuming Column A has your assessment scores and Column B has your job performance data. Click on the cell where you want the correlation coefficient to appear.Enter the formula: =CORREL(A:A, B:B)EG, if your data is from row 1 to row 100 in both columns, the formula would be: =CORREL(A1:A100, B1:B100) Interpreting the Coefficient: A high positive coefficient (close to +1) means a strong relationship between the assessment scores and job performance, indicating your assessments are good predictors of job success. A low or negative coefficient suggests that the assessments are not effective predictors of job performance. Consider the Context: Remember that the validity of an assessment can vary based on job roles, the organisation's culture, and other situational factors. It's beneficial to conduct this analysis separately for different roles or departments. Legal and Ethical Considerations: Ensure that your methodology and the use of performance data comply with relevant employment laws and ethical standards. Continuous Improvement: Use the findings to refine your assessment processes. If the validity is low, consider revising your interview questions, assessment methods, or the criteria used for measuring job performance.

This approach requires a good understanding of statistical analysis and additionally, the reliability of this process depends on having a decent sample size and accurate, unbiased performance data.

So, that was a pretty heavy one. But if you can take this information and use it to shape your interviews and success criteria to have the highest possible chance of predicting future success, your business will have fewer probation fails, higher retention, lower cost per hire. its huge.

References

recruitingtoolbox.com

Book Update

Progress is coming along at a good clip. I've been hammering through the theoretical sections at light speed as I write extensively on this anyway. I'm also adding any formulas or technical bits in a resources section and writing up common scenarios you can use your new found data.... drivenliness to work through.

Im trying to stay away from decisions related to style as (A) I'm terrible at design and (B) it's just procrastinating from writing the actual content.

That being said, if anyone knows of a good tool to layout a non fiction E-book other than Canva, please let me know!

Well... that's issue thirteen. It's a meaty one so bookmark it and use it the next time a hiring manager comes down with a mean case of Elon Musk syndrome.

If you have any questions about it, or any feedback on this issue of The Data Driven Recruiter, grab me on LinkedIn for a chat.

I'll see you next week!