Skillbuilding: A Look at BCG's Evaluation System
Alison Hare, CG, “A Look at BCG’s Evaluation System,” OnBoard 23 (May 2017): 9--10, 15.
A completed portfolio thuds into a mailbox. A mouse click transmits a finished portfolio to BCG electronically. Once an applicant’s work is done, it is time for BCG’s to begin. Three or four judges will evaluate whether the candidate’s submission is sufficient for certification. The system they use reflects a half-century of evolution and involves much work behind the scenes.
Early Evaluations
BCG evaluations initially looked little like they do now. For almost two decades after the board’s 1964 founding, they were a rudimentary affair. Applicants received no ratings for defined evaluation criteria, simply written remarks about the quality of their work, along with each judge’s recommendation for or against certification.
By 1982 the board introduced rating forms to supplement comments. Broad evaluation criteria for initial applications directed judges to assess knowledge of records, reading of early handwriting, competence in abstracting, ability to use original and printed sources, use of the now-abandoned preponderance of evidence principle, placement of family in historical perspective, grasp of genealogy format, citation skills, and clarity of writing.
Refinements in the 1990s sharpened evaluation of work characteristics that affect research accuracy. New criteria focused on thoroughness of research, reliability of sources, handling of conflicting evidence, and soundness of conclusions. Some single criteria were broken down into several. For example, thoroughness of documentation and completeness of individual citations were now appraised as separate items. Then as always, renewal applications were evaluated against more general criteria.
Major Advances
The board’s publication of genealogy’s first standards manual in 2000 powered the next advance. This codification made standards readily accessible to all genealogists while also useful for evaluation purposes. Evaluation criteria remained largely the same, but The BCG Genealogical Standards Manual now captured in writing the qualities necessary for sound genealogical conclusions. These same attributes being the key to certification, the manual’s seventy-four standards established clear benchmarks against which portfolios could be judged. In preceding years, judges had sometimes referred to standards in their comments, but only now were those standards explicitly defined and only now did evaluation forms mention them. In subsequent years, their influence on evaluations grew as judges became increasingly experienced with their use.
Following several years of development, BCG implemented rubrics in 2009. Another game changer, these scoring guidelines communicate expectations for each portfolio requirement using brief descriptions of performance levels for each skill being evaluated. Rubrics did not change the types of things judges evaluate but articulate them more clearly for the benefit of both judges and applicants. One or more standards underpin each rubric, and seven to eleven rubrics address each requirement in a new application, for a total of forty-eight. Broader rubrics address renewals. For both types of applications, one overarching rubric specifies the factors that should determine the pass-fail recommendation. The work should meet most standards, and lapses must be remediable without concentrated study and practice.
Eight years later rubrics remain a key element of BCG evaluations. Since 2014 they have been used in tandem with a revised standards manual, Genealogy Standards. Although the old and new standards are the same at heart, the new publication helpfully clarifies, updates, expands, and reorganizes them. Benefits include improved evaluation of research planning, documentation, proofs, and other areas.
By this route, BCG evaluations have acquired important characteristics they initially lacked. Modern-day evaluations weigh applications for certification against established criteria corresponding with authoritative standards. The standards and rubrics have both been honed several times, and their publication has increased transparency, eliminating previous unknowns about how applications for certification were measured. Together they have changed evaluations from an activity where judge preferences could have sometimes interfered to one where agreed-upon principles prevail.
Behind the Scenes
Now as in the beginning, portfolios are evaluated by a pool of judges that changes from year to year and is chosen from among BCG associates based on the strength of their skills and their geographic, language, ethnic, or other areas of expertise. The job requires long hours with no pay other than a small token honorarium, which most judges decline. As judge identities are kept confidential to allow unpressured decisions, judges also forego much-deserved public recognition. Even so, when asked to serve, most associates accept appointment, embracing it as an opportunity to promote standards and growth of the genealogical field.
BCG judges, currently about forty in number, together assess on average seventy portfolios a year. With three or four judges reviewing each initial application and two or three per renewal, more than 200 evaluations are generated annually. The time required to complete a single evaluation varies, depending on a judge’s skill and thoroughness, the complexity and quality of the submitted material, and the type of evaluation. Initial applications, typically more time-consuming than renewals, might take several days to a week to evaluate and may result in as many as three pages of comments supporting the rubric ratings and final recommendation.
BCG judges have always worked independently. They do not consult or collaborate. Except for arbiters who settle cases of disagreement, they do not know how their fellow judges have evaluated a portfolio until after all evaluations are complete. Independence does not mean, however, that BCG judges operate with complete freedom and no accountability or in the absence of checks and balances.
Internal Controls
Some restraints are built into the process. Most notably, applications for certification have never been decided by only one individual. The number needed to decide success or failure has varied over time, but at least two judges must now approve before an applicant is certified. As another control, judge selection for a particular portfolio contains an element of randomness. Using a comprehensive list of each judge’s areas of knowledge, BCG’s executive director recruits the first judge, the first judge enlists the second, and the second selects the third. The same process applies to renewals but with one less judge. Timing and judge availability affect choices as well as each judge’s determination about what expertise would be useful to an evaluation. Because applications often involve various specialties, a judge might have several good options. From BCG’s inception until the present, these variables have prevented any one judge from unduly influencing an application’s outcome.
Establishing more direct oversight of its judges, BCG in 1999 appointed a judge coordinator. Until becoming a separate position, this job was one of the BCG president’s many responsibilities. The judge coordinator’s supervision guarantees smooth transitions during times of change and safeguards applicant interests by ensuring judging policies are properly followed and evenly applied. The judge coordinator also monitors evaluations for impartiality, quality, and with a view to identifying and addressing systemic problems. Solutions have ranged from the major reforms such as rubrics to smaller measures such as the recent caps on portfolio size meant to eliminate material that unnecessarily increased judge workloads and fatigue.
As the title suggests, the judge coordinator also works closely with the judges, seeking to improve judging results through individual and group discussion and training. New judges receive one-on-one guidance, and all judges can seek advice on any aspect of an evaluation, including the best way to handle unusual or difficult situations. To help promote evaluation quality and consistency, the judge coordinator also prepares and maintains judging manuals and other instructional materials.
Further increasing the system’s rigor, a committee of experienced judges reviews all evaluations before they are returned to applicants. Begun in 2005 as an experimental initiative by three senior judges wanting to boost evaluation quality, the committee has since doubled in size while gradually acquiring more responsibility. Commonly referred to as the review committee, this group screens each completed set of evaluations for correctness, clarity, and tone. Evaluations must properly reflect The BCG Application Guide requirements, the standards, and the rubrics. Comments are expected to respect the applicant, focus on substantive rather than minor concerns, and omit personal opinions. Judges are also required to evaluate rather than teach, which means identifying a portfolio’s strengths, weaknesses, and worthiness for certification, but not explaining concepts or delivering instruction about how to remedy flaws or improve genealogical skills. Problems found by the committee are resolved with judge input.
Ongoing challenges
BCG’s evaluations are complex. The BCG Application Guide sets common requirements, but within these parameters applicants have considerable freedom in the work they may submit. As a result, two portfolios are never alike, presenting judges with many evaluation challenges. At the same time, judge independence brings differing viewpoints, influenced by each judge’s distinctive blend of experience and further shaped by various degrees of strictness and leniency. As a result, evaluations are also never identical---and never will be despite the critical framework established by rubrics and standards. Even so, BCG recognizes that judging differences can be confusing and minimizing these remains a priority, with recent gains achieved.
Conclusion
Much has changed since the first applicant dropped a completed portfolio into a mailbox. BCG has codified genealogical standards and developed rubrics, allowing focused assessments of genealogical work based on clearly defined criteria. Internal monitoring has further enhanced quality, with scrutiny of evaluations by a judge coordinator and review committee helping to ensure sound and fair outcomes. Except for the administrative duties of BCG’s executive director, all of the work related to evaluations is freely performed by volunteers, keeping certification affordable. Some improvements had to await technological advances such as e-mail, but all of them represent hard work by many BCG associates. The future will see BCG build on these efforts.
Alison Hare, CG
This article was originally published in OnBoard, BCG’s educational newsletter and is protected by copyright. Individuals may download and print copies for their personal study. Educators are granted permission to provide copies to their students as long as BCG, OnBoard, and the appropriate author are credited as the source of the material. Republication elsewhere is not permitted.