All BA Assessments Are Not Created Equal

by Kitty Hass and Lori Lindbergh, PhD

Think all BA assessments are created equal? Maybe not.... Assessments, in general, are used to measure observable events (i.e., behavior, actions, processes, etc.) that represent some particular, unobservable attribute, state, or condition. What does this mean? Well, unlike measuring blood pressure with a sphygmomanometer or temperature with a thermometer, you can’t technically measure BA competency or BA practice maturity directly. However, by examining the application of business analysis practices and supporting processes and comparing these against valid models and standards, you can infer a level of BA competence or BA practice maturity. Therefore, when you use a high-quality assessment, the findings will more closely reflect true differences in the unobservable attribute (BA competency or practice maturity) and not only lead to stronger inferences about the current state of your workplace, but lead to better decisions about how to achieve BA performance improvement.

Reliable, but not Valid



Two important characteristics should be examined when evaluating the quality of BA assessment solutions: Reliability and Validity. When a BA assessment is reliable, this means that the assessment yields consistent results every time it is used. Think of a yardstick when measuring fabric. A yardstick is a fairly reliable device for measuring fabric length. Every time a person is asked to use the yardstick to cut one yard of fabric from a bolt, the cut fabric piece will be fairly consistent to 36-inches long. When a BA competency assessment exhibits strong reliability, you can be confident the assessment tool is appropriate for measuring BA competency, participants will respond consistently to the items in the assessment, and participants with higher BA competency will answer more questions correctly and/or apply more skills associated with each competency than those who are less competent.

When a BA assessment is valid, this means that the assessment accurately measures what it is supposed to measure: the competency of a BA or the current state of an organization’s BA practices. At face value, a BA assessment may appear to measure what it is supposed to measure because it is based on industry standards. However, examining assessment validity requires additional testing and onsite examination of the actual application of practices and production of deliverables. This may involve conducting interviews, project artifacts reviews, process reviews, and stakeholder focus groups. Furthermore, a valid, interpretive frame of reference or research-based model is required for comparison and interpretation of the assessment scores. (Click here for information on reliability and validity.)

What does this mean for you as a business professional? Know that a BA assessment can be both reliable and valid, one or the other, or neither. When you are evaluating potential assessment solutions, ask your providers for strong evidence of the assessment’s reliably and validity. If the providers are unable to provide evidence of BOTH, the providers might be well-intentioned “assessment hacks.” (See the January 16^th blog entry.) Don’t waste your investment! Remember, a high-quality assessment will show strong evidence supporting the use of the assessment tool and the intended interpretation and use of the findings. You need to feel confident you can use your assessment findings to help you make accurate decisions about your BA competency and BA practice maturity and to guide the development of your action plan and roadmap for BA improvement.

How important do you think it is to have evidence that the BA assessment you plan to use exhibits strong reliability and validity? We look forward to your comments and questions.