by Lori Lindbergh, PhD and Kitty Hass, PMP
Last week, we stumbled across an informal BA survey conducted by an IIBA chapter. The purpose of the member survey was to examine the status of business analysts and business analysis practices in organizations in the chapter’s service area. Interestingly, one of the items asked the participant to speculate and choose from one of four options that most closely described the characteristics of the BA practices in his or her organization. The four descriptions contained broad generalizations of each maturity level similar to the four levels in our BA practice maturity model. (See the February 13th entry.)
Ah, wouldn’t life be grand if assessing anything, including BA practice maturity, would be this easy? Some may say, “What’s harm in asking a simple question?” The harm may not come from asking the question, but more from the interpretation of the findings. Any time you reduce a complex phenomenon down to one question, you introduce speculation. Speculation is a conjectural consideration of a matter; a conjecture or surmise that may be impossible to verify. Such is the case in the survey we found. The BA practice maturity reported by each participant could not be verified without a full organizational assessment.
Accurately assessing BA practice maturity by simply asking one question to measure one person’s perception/opinion of his/her organization is NOT (and usually never) representative of the overall organization, unless you are an organization of one. Assessing BA practice maturity involves choosing an appropriate sample of representative projects to include in the assessment. It is more complex than one survey question and requires a multidimensional approach including a reliable and valid questionnaire, onsite interviews, focus groups, and deliverables inspections. When we conduct our assessments, we typically find that organizations are more mature in some of the nine dimensions of BA practice maturity and less mature in others. Therefore, a general blanket description of BA practice maturity is never an accurate measure.
This violates an important characteristic of a psychometrically sound measure: Validity. A valid assessment measures what it is supposed to measure and provides strong evidence to support the interpretations made from the assessment. Therefore, using one question to evaluate BA practice maturity, and thus presenting the survey findings as reflective of an organization’s actual BA practice maturity level would be similar to using a sphygmomanometer (blood pressure cuff) to speculate on the status of a person’s overall health. A sphygmomanometer is a valid measurement instrument for blood pressure only, and is simply one of a many measurements that are necessary to evaluate fully one’s health.
We don’t want to rain on anyone’s parade; there is no harm in conducting a fun, informative survey. Remember, however, your survey respondents have demonstrated interest in your topic by taking the time to complete your survey. They are looking to you as the survey provider to arm them with accurate findings and interpretations that they can use to enhance their knowledge and improve their performance. Be sure to meet their expectations.
