Home  /  Publications  /  Audits of Recruitment and Selection Activity  /  Appointment Process for the position of Executive Officer in Revenue
 

Audit Findings-Appointment Process

Overview of the appointment process

In March 2014 Revenue advertised an internal competition for promotion to Executive Officer (EO) in the organisation.  The selection process consisted of on-line tests at Stage 1 to include a verbal reasoning and a numerical reasoning assessment.   These tests were conducted by an independent service provider (Cut-E, a professional company involved in the design and implementation of on-line tests and questionnaires for recruitment, selection and development) selected, following a competitive tender process by Revenue, to conduct a series of web-based aptitude tests as part of the assessment process for the purposes of shortlisting.  1407 candidates sat the Stage 1 on-line tests in 21 locations around the country which took place between 19 and 28 March 2014. 

The top 340 candidates in order of merit who met the required standards in the online ability tests were invited to Stage 2 of the competition.  322 were interviewed.  A panel of 202 candidates was formed and all successful candidates were invited to sit further psychometric tests to verify their performance at the first stage.  All 202 candidates reached the qualifying standards in these tests.   

Competition Circular

A competition circular inviting applications for internal promotion to EO issued to staff in Revenue on 5 March 2014.  The circular advised that a panel would be established from this competition to fill posts in strict order of merit.  The competencies required for the role were set out as well as details on the application process and general requirements.

Selection Process

The circular advised that the selection process would consist of two stages

·         Stage 1 – On-line tests comprising of (i) verbal reasoning and (ii) numerical reasoning to be undertaken at a designated Testing Centre.

·         Stage 2 – Competency based competitive interviews

Candidates were advised that the tests measure ability to draw logical conclusions from complex verbal and numerical information set in a finance context.  The ability to retrieve relevant information from various sources was also measured.  Each assessment would be timed for twelve minutes.

Candidates were also advised that there would be advance access to a practice test site from 5 – 18 March which would prepare them for completing the online assessments. 

It was also stated in the circular that the number of applicants to be invited to Stage 2 would be determined by Revenue from time to time having regard to the number of vacancies to be filled.

Information on the Psychometric Tests

(i)     Facilitators

There were 25 Facilitators – Higher Executive Officers nominated by local senior management - selected to oversee the sessions in each of the testing centres.  They attended a briefing session on 28 February 2014 at which their roles and responsibilities were discussed.  They were provided with written instructions on administration of the test sessions including practical advice about their role on the day and how to deal with any issues arising.  All facilitators were also provided with login passwords to the live test system which were to be treated as highly confidential. 

Each Facilitator was instructed to read a standard text to all candidates before the testing commenced that included advice on logging in to the website and completing the two tests.  The advice provided to candidates included an instruction to ‘work quickly and accurately’.     

(ii)    Candidates

Candidates who applied for the competition received an email on 6 March 2014 containing important information in relation to preparing for the next stage of the assessment process.  The email provided details on how to access familiarisation material online for the psychometric tests and set out information regarding the real online tests. 

Candidates were advised that there would be 12 minutes to complete the 49 questions in the verbal reasoning test and another 12 minutes to complete the 37 questions in the numerical reasoning test; a timer would be displayed on the screen to help them manage their time.  They were also advised that there was only one correct answer to each question, i.e. True, False or Cannot Say based on the information given.

A further email to candidates in relation to the date, time and designated testing centre for the online tests advised that

“Our best advice is to work quickly and accurately through the assessments remaining focused and avoiding any distraction.  Please keep in mind there is a number of questions as part of each assessment and most candidates do not complete all the questions in the time allowed”.  

 

At the commencement of the psychometric tests candidates received instructions online on how to proceed.  Included in these instructions candidates were asked to note:

  • Most people cannot complete all questions in each test within the allotted time
  • The test requires you to work both quickly and accurately
  • If you are not sure about a question, select you best choice or skip this question.  Avoid guessing the answer. 

Assessment of the on-line psychometric tests

The service provider was responsible for evaluating the on-line tests which had been developed to assess the key skills and attributes necessary to perform effectively at the grade of Executive Officer.

Revenue has advised that all candidates were ranked on the basis of their scores subject to achieving the qualifying standard in both tests.  The minimum qualifying mark was only determined after the tests were completed based on the standard set by the norm group, i.e. the candidate pool, using a complex statistically based calculation (which cannot be released due to intellectual property rights of the service provider). 

It has been confirmed by Revenue that candidates overall performance was based on the volume and accuracy of the answers given to the questions in comparison to the standard established by the norm group comprised of all candidates who sat the tests.  This meant that the final score achieved by a candidate was based on a number of factors including the number of questions answered correctly and the level of accuracy involved.  It has also been confirmed that the test scores of candidates who reached the qualifying standard had no bearing on the outcome as all candidates who qualified at Stage 1 then had to participate in a competitive interview process which determined the final order of merit.       

Feedback

Candidates who requested feedback on their performance in the verbal and numerical ability tests were advised that statistics were used to calculate performance in order to understand how each individual performed relative to others, i.e. performance was compared with other applicants for the EO role.  Candidates were told that in order to progress to the next stage of the process they were required to meet the minimum qualifying result in the verbal test, the numerical test, and also to meet the minimum qualifying combined result.  Candidates were provided with a document that described their processing style in the tests in relation to the comparison group together with information on the number of questions answered and how many of these answers were correct.

Candidates who requested further feedback on their performance were advised by the service provider that further information in relation to the minimum qualifying result could not be provided.  They were told that the feedback provided should help candidates to understand their own performance in the tests and different ways to develop capability in these areas.    

Investigation into alleged breach of security

Following completion of the psychometric tests MPRU received a phone call on 14 April 2014 from the Assistant General Secretary, Civil & Public Service Union (CPSU) advising it had been alleged that some Revenue staff had access to the test site in advance of the tests being available to all candidates.  A subsequent letter from the CPSU requested clarification concerning the alleged access of some candidates in Dundalk to the psychometric tests for up to a week in advance of sitting the exam.  The CPSU also sought clarification, on behalf of its members, on whether negative marking had been applied in the assessment of candidates’ tests.

Revenue contacted the service provider immediately seeking an activity report from the test site to ascertain if there had been advance access to the tests.  The report indicated activity beyond what would have been expected of system checking by those with authorised access, i.e. facilitators and IT personnel. 

Having considered the matter Revenue decided to introduce an additional battery of psychometric tests for all those who passed the interview at stage 2 of the process to verify that those who achieved the requisite standard in the first test had numerical and verbal reasoning skills identified as necessary for the role and to minimise the chance of any candidate gaining an advantage as a result of the purported breach in security arrangements.

On 20 May 2014 an email issued to all staff in Revenue advising that it had been brought to the attention of Corporate Services Division (CSD) there was a potential isolated incident of advance access to the Stage 1 tests in the EO competition.  Based on preliminary enquiries it had been decided to launch a full investigation into the matter.  Furthermore CSD advised staff that, in order to ensure the integrity of the competition and fairness to all candidates, verification tests would be introduced at a later stage for those successful at interview.

Revenue has conducted a full comprehensive internal investigation into the allegations of advance unauthorised access to the test site for the EO competition.  The investigation has concluded that a small number of isolated instances of unauthorised access occurred in one location. This is being pursued further and in the event of misconduct being established, Revenue will take appropriate disciplinary action in accordance with the terms of the Civil Service Disciplinary Code.  

Stage 2 Interviews

Candidates placed in the top 340 places after the first stage psychometric tests were invited to interview.  There were two boards each consisting of an Assistant Principal and two Higher Executive Officers who attended a training/briefing session and were provided with information on their responsibilities under the Codes together with interview scoring guides for each competency to assist in the assessment process.  The structured interviews were based on the four competencies identified for the role and notified to candidates at the outset of the competition. 

Candidates were asked to provide feedback comments on the interview process.  A review of a random sample of these comments show that candidates were generally satisfied with the process.

202 candidates were deemed suitable for the position and were placed on a panel. 

Verification tests

Candidates who were invited to the Stage 2 interview were advised that all those successful at interview would be required to undertake, at a later date, parallel versions of the two on-line tests completed at Stage 1 as a verification measure.  They were advised that it was expected that performance in these tests would be broadly comparable to performance in the Stage 1 test and that results must be validated in order to be considered for placement on the panel.  The order of merit from the interviews would not be affected by the verification exercise.  

The verbal reasoning and numerical reasoning tests took the same format as the Stage 1 tests however each test was timed for a shorter period of six minutes and included 18 questions.  As before the verification tests were completed online in a supervised test centre.

All 202 candidates on the panel reached the qualifying standard in the verification tests.

Requests for review

MPRU has advised that a large volume of both formal and informal complaints were received from candidates following the Stage 1 psychometric tests and the notification of results from the tests.  

At this time MPRU also received notification of the alleged breach of security in relation to the tests and was considering its options on how to progress with the competition.  As a result of this unexpected development and its serious implications Revenue has acknowledged that there were delays in processing and responding to complaints and requests for review.  It has advised that it was not in a position to respond to complaints until it had consulted with the service provider and fully considered the issues that had been brought to its attention. 


Review and Evaluation

Revenue has advised the Commission that it will be reviewing its procedures in light of their experience in this instance.  While it is satisfied that there were measures in place to protect the integrity of the process, it accepts that there are lessons to be learnt and steps will be taken to avoid any re-occurrence of this situation, including introducing additional security measures.