What the Numbers Say: BA Assessment Findings for Dummies.

by Kitty Hass and Lori Lindbergh, PhD

Numbers and graphs and correlations, Oh My! Are you one of those people who become squeamish when presented with statistical findings, or do your eyes glaze over when you gaze for too long at spreadsheets and presentations containing statistical findings? As much as most people hate statistics, our society is churning out more numbers and statistics than ever before in the form of spreadsheets, survey results, research studies, and financial statements. Quantitative information pervades just about every aspect of our professional and personal lives. If you step back and think about it for a minute, maybe it’s not that you fear the statistics. Your fear may be rooted more in your inability to decipher what the numbers mean (and if you should trust them) and what to do with the numbers after all is said and done.

How do you know if you should trust the numbers from a BA assessment? Because the consultant says you can, right? Well, maybe yes and maybe no. We think the answer to this question has a two-part answer. The first part relates to the questionnaire used for the assessment, your sample size and type (random or nonrandom), and how the data were collected. The second part of the answer relates to the statistical tests that were used to analyze the data.

First of all, remember, statistical findings are never 100% accurate.

Some degree of measurement error is natural and unavoidable. In our January 23^rd post, we discussed reliability as an important characteristic of an assessment questionnaire. Using a questionnaire with strong reliability evidence, answers the first part of the trust question. Sample size and type are less of a concern with BA Workforce Assessments because most organizations typically include 100% of its analysts in the assessment. Sample size and type are more of an issue when you want to make meaningful comparisons to external assessment and survey findings and industry benchmarks. The sampling and data collection that others used should be scrutinized to determine the degree to which you should “trust” the comparison. In other words, should you bet the farm or not?

Statistical programs don’t have a brain.

The programs do what we command them to do. These programs treat a number as a number as a number and do not know what each number represents and how the numbers were obtained. The programs trust us to enter the data correctly and trust that we understand the appropriate statistical tests to use for the types of data we collect. Common statistics presented in a BA Workforce Assessment include the mean, median, and mode; and statistical tests used include correlations and group comparison tests, such as t-tests and tests for variance analysis. These tests ensure the findings of your BA Workforce Assessments present a descriptive snapshot of your organization’s analyst workforce and have to potential to identify statistically significant relationships and group differences to guide your BA performance improvement efforts.

What do the numbers mean and how should you use them?

Trusting the numbers serves as the foundation for your interpretation and use of any BA assessment findings. The bottom line: If you don’t trust the numbers, your assessment won’t really help you very much. This raises the question of assessment validity that we also discussed in our January 23^rd entry. If you feel you CAN trust the numbers, then next ask yourself, “Do the numbers make sense in your own organization?” Do the findings reflect the current state of your BA workforce and its capabilities? (See the supporting article: BA Assessments: Is That What’s Really Happening in my Organization?).

Conducting a BA Workforce Capability Assessment should provide the information you need to baseline your BA Workforce competencies and prepare your group’s professional development program, and can serve as input to your individual BAs’ professional develop plans.  The results should provide a basis for BA:

·       workforce adjustments and/or realignment;

·       training requirements;

·       professional development activities; and

·       specific mentoring, support, and coaching needs.

Additionally, you should be provided with a basic snapshot to understand the demographic information about each BA on your team, e.g., years of experience, time spent on BA activities versus project management or more technical tasks, and amount of BA education, etc. The state of your BA workforce capabilities should be compared to available industry standards, as available. This cumulative information provides a view as to your BA’s actual capacity to deliver new business solutions.

A BA Workforce Assessment should provide BA managers the opportunity to participate in the assessment. This is not possible with typical multiple-choice type BA assessments and most online self-assessments. We have found that valuable and interesting insights typically result when we have the opportunity to compare manager ratings and analyst ratings.

An important component of any BA Workforce Assessment is that the assessment approach provides an interpretive frame of reference, similar to the model presented in our March 5^th entry, to analyze your assessment responses. The frame of reference should be evaluated for the degree of alignment and mapping to your organization’s BA career path. This alignment will enhance the relevance of the assessment findings and help you clearly visualize the workforce adjustments necessary to enhance your organization’s BA capacity.

Remember, take time to develop a trusting relationship with the numbers before jumping into any BA Assessment for your organization. You get what you pay for. Your first goal should be to become a skilled quantitative thinker. This means that you develop the ability to use quantitative information effectively when drawing conclusions or making decisions. Correctly using your numbers as an additional form of evidence will help you focus and create compelling support for your BA improvement recommendations.