Not Your Grandmother’s Assessment.

by Kitty Hass

I’ll never forget my grandmother’s cooking. No matter what she prepared, she was always able to pull together the right ingredients to create a mouthwatering meal or desert for everyone to enjoy. She didn’t use recipes, simply her experience and intuition: a little bit of this, a pinch of that, and a handful of the other. She didn’t care about counting carbs, measuring fiber content, or how many omega-3s and antioxidants were in her meals. As long as her food tasted good and satisfied everyone’s appetite, she was content.

In today’s complex business world, applying grandma’s cooking methods of experience and intuition to BA performance improvement will only take you so far. You need more. This is where science enters the picture. Because an organization is a complex system made up of interrelated subsystems, you need to be sure the BA assessments you use contain the exact mix of ingredients and science that your organization needs.

So what does a world class BA assessment look like? After more than 20 years in the IT, PM, and BA space, I think I have honed in on the most important characteristics of a truly superior assessment, one that produces a roadmap that leads to success in terms of value to your customers and wealth to your organization. (Grandma would be impressed.)

Mature business management practices focus on alignment with and achievement of business strategies, goals and objectives. Mature and capable business practices are directly correlated with higher organizational performance, value added to the customer, and wealth to the bottom line. With this in mind, then, world-class BA assessments should:

• Appraise both BA/PM organizational maturity and individual/workforce BA capability based on validated reference models
• Align with industry professional standards for BAs and PMs and with standards and best practices for quality and fairness in educational and psychological assessment
• Incorporate the dimension of work complexity based on the skills and knowledge needed to work successfully on the complexity of current project assignments 
• Benchmark results against a global data base of BAs and PMs performing comparable work, against our models, and against the individual peer group results
• Examine critical relationships between competency, project complexity, organizational culture, and project outcomes.
• Focus analysis and findings on critical recommendations and produce a roadmap to fill the gaps.

(In future postings, we will examine each of these characteristics in more detail. See the monthly blog theme tab for more information.)

To provide value, assessments need to balance experience with science and offer a more holistic, systems perspective versus examining competency, practices, and outcomes in isolation. By sharing this blog with you, we want to help you evaluate your assessments with a critical eye and not believe everything you read. Even though the Sara Lee package read, “It tastes like homemade,” I could always taste the difference between grandma’s homemade apple pie and the store-bought version. I think a balanced, scientific assessment is a recipe for success that even grandma would be proud of; kind of like grandma’s recipe on steroids.

Please tell us about your recipe for BA assessment success. As always, we look forward to your comments and questions.

All BA Assessments Are Not Created Equal

by Kitty Hass and Lori Lindbergh, PhD

Think all BA assessments are created equal? Maybe not.... Assessments, in general, are used to measure observable events (i.e., behavior, actions, processes, etc.) that represent some particular, unobservable attribute, state, or condition. What does this mean? Well, unlike measuring blood pressure with a sphygmomanometer or temperature with a thermometer, you can’t technically measure BA competency or BA practice maturity directly. However, by examining the application of business analysis practices and supporting processes and comparing these against valid models and standards, you can infer a level of BA competence or BA practice maturity. Therefore, when you use a high-quality assessment, the findings will more closely reflect true differences in the unobservable attribute (BA competency or practice maturity) and not only lead to stronger inferences about the current state of your workplace, but lead to better decisions about how to achieve BA performance improvement.

Reliable, but not Valid



Two important characteristics should be examined when evaluating the quality of BA assessment solutions: Reliability and Validity. When a BA assessment is reliable, this means that the assessment yields consistent results every time it is used. Think of a yardstick when measuring fabric. A yardstick is a fairly reliable device for measuring fabric length. Every time a person is asked to use the yardstick to cut one yard of fabric from a bolt, the cut fabric piece will be fairly consistent to 36-inches long. When a BA competency assessment exhibits strong reliability, you can be confident the assessment tool is appropriate for measuring BA competency, participants will respond consistently to the items in the assessment, and participants with higher BA competency will answer more questions correctly and/or apply more skills associated with each competency than those who are less competent.

When a BA assessment is valid, this means that the assessment accurately measures what it is supposed to measure: the competency of a BA or the current state of an organization’s BA practices. At face value, a BA assessment may appear to measure what it is supposed to measure because it is based on industry standards. However, examining assessment validity requires additional testing and onsite examination of the actual application of practices and production of deliverables. This may involve conducting interviews, project artifacts reviews, process reviews, and stakeholder focus groups. Furthermore, a valid, interpretive frame of reference or research-based model is required for comparison and interpretation of the assessment scores. (Click here for information on reliability and validity.)

What does this mean for you as a business professional? Know that a BA assessment can be both reliable and valid, one or the other, or neither. When you are evaluating potential assessment solutions, ask your providers for strong evidence of the assessment’s reliably and validity. If the providers are unable to provide evidence of BOTH, the providers might be well-intentioned “assessment hacks.” (See the January 16^th blog entry.) Don’t waste your investment! Remember, a high-quality assessment will show strong evidence supporting the use of the assessment tool and the intended interpretation and use of the findings. You need to feel confident you can use your assessment findings to help you make accurate decisions about your BA competency and BA practice maturity and to guide the development of your action plan and roadmap for BA improvement.

How important do you think it is to have evidence that the BA assessment you plan to use exhibits strong reliability and validity? We look forward to your comments and questions.

BA Assessments, No Big Deal….Right?

by Kitty Hass & Lori Lindbergh, PhD

That’s what we used to think. Yes, years ago, we too were assessment hacks—AKA assessment nescience—we didn’t know what we didn’t know. You may be thinking, “What’s the big deal about assessments anyway; just compose a bunch of questions, administer, and analyze. How difficult can that be?” And a funny thing, back then, that’s what most of our clients thought, as well. Unfortunately, many clients and business professionals we encounter today still think the same way. This thinking fosters the perception that all assessments are created equal, and most assessments will provide limited value.

We have created this blog for the purpose of educating business professionals about assessments and changing their perceptions of the value the right assessments can provide. Specifically for this blog, we will be focusing on business analysis assessments. Just as all cars are not the same; the same is true for BA assessments. You need to look under the hood to see what you are really getting for your investment.

If you are thinking about conducting a business analysis assessment, this is the blog you should be reading. With so many consulting organizations offering business analyst competency assessments and business analysis practice maturity assessments, as a business professional, how do you know which one to choose? How do you know whether the findings of an assessment will accurately reflect the current state of your BA workforce or the BA practices in your organization? How do you know whether the findings and results will be meaningful and relevant? And, finally, how confident should you be in using the assessment findings to guide your performance improvement efforts? These are all very important questions that should be answered before you conduct an assessment. This blog will answer these critical questions so you can maximum the value of your assessment investment.

We believe that Kitty’s incredible depth of experience as a former Malcolm Baldridge assessor and SEI assessor, as well as her considerable business analysis and project management expertise and Lori’s post-graduate focus on creating and delivering effective assessments make us the leading experts on research-based BA assessments. We have conducted numerous client assessments in multiple industries and have completed research studies examining our BA assessments. From this, we have compiled our own lessons learned and best practices that we would like to share with you in this blog. We hope that this information will dispel any myths and misconceptions you may have about BA assessments and will help you become a more discerning consumer of assessments in general. This is consistent with our overall mission: Raising the level of professionalism related to BA assessments and helping individuals and organizations use assessments to achieve deliberate BA improvement. Remember, The Right BA Assessment Can Be a Big Deal; it has the potential to provide the intelligence you need for sustained BA performance improvement.

Tell us about your BA assessment experiences. We will be posting to our blog weekly and look forward to your comments and questions.

Welcome to our blog. Look for our first blog posting on January 16th. We look forward to communicating and collaborating with you.

Kitty & Lori