Our client is a well-known provider of financial aid and advice, drawing upon a limited fund. It employs a set of rules to determine eligibility and level of support.
The customer needed to model how changes to the its eligibility rules might affect levels of support provided. At the time, the customer was considering significant changes to its eligibility rules and had therefore built an analytical model to predict the effects of changes to eligibility rules.
The client had used the SAS programming language to develop its eligibility model. It consisted of six major (SAS project) components; each component features around six (SAS program) sub-components; and each sub-component typically contains tens of (SAS) data steps or procedure calls plus associated data definitions, macros and support elements. The result was a relatively large and complex model.
Given the large sums of money granted by the client each year, and the broad catalogue of changes under consideration, the eligibility model was business-critical. In order to build decision-maker confidence in this new model, the client commissioned us to provide an independent analytical model quality assurance review prior to its use.
A senior consultant from Hartley McMaster undertook a quality assurance (QA) review of the eligibility model, with the aim of assessing the extent to which the model met its design objectives.
The review consisted of seven stages:
Stage 1 – Role Confirmation
Involved discussions with the model’s owners, and development and maintenance teams to determine the ultimate customer for the model’s outputs, the nature of these outputs and the proposed uses to which they would be put.
Stage 2 – Intended Model Operation
Entailed the collection and review of those laws and regulations defining the current processes by which an applicant’s eligibility is assessed.
Stage 3 – Actual Model Operation
This step represented the key focus of the work. It commenced with a detailed review of the model’s source code, input data sets and existing documentation. It also encompassed: the execution of individual steps of the model; the exercising of separate model components; the examination of intermediate data sets created during the model’s operation; the design implementation and use of SAS diagnostic code to analyse intermediate data sets.
Stage 4 – Model Management
Included an assessment of the version, configuration and change control procedures in place, the level of adherence to these procedures, plus reviews of the model’s Assumptions and Data Logs.
Stage 5 – Conclusions
In this step we compared the model’s calculations with those defined in the client’s corresponding procedures, considered its design and management given its intended role, then assessed the model within the broader context of its proposed use.
Stage 6 – Recommendations
Here we formulated a set of actions to improve the model and its usage. These recommendations included advice on version and configuration control arrangements, highlighting areas for further testing and suggestions for additional documentation.
Stage 7 – Report
We delivered a comprehensive report on the findings of the review, largely based on the consultant’s written log of how the model performs its calculation, the flow of control through the model.
The Hartley McMaster consultant delivered a formal, written report on his findings to the client, along with updated versions of the model’s Assumptions and Data Logs. He also liaised with those members of the development team tasked with following up on the review’s recommendations.
The review increased confidence in the correct operation of the Eligibility Model and reduced the exposure of the client to unexpected costs as a result of an ill-advised set of changes to eligibility rules.