Ask an Expert
Q: I am working with client leaders using the LPI® and have administered two assessments 14 months apart. In looking at the difference in results between the initial LPI and the second, we are hoping to present data that indicates that our leaders have “statistically improved”, but I’m stumped on exactly how to determine that. Can you help?
A: There is no "absolute" answer to the question of what is a "significant" change in terms of comparing two (or more) administrations of the LPI®.
The empirical problem with determining that a statically significant change has occurred is due to a limited or restricted sample size. A sample involving one individual leader is generally so small that it’s impossible to assess with any certainty if any change is random (due to a chance within a normal range of distribution). Working with individual leaders, we're often just trying to help the leader move the needle, to see a change in frequency in one or more behaviors in a leadership practice.
At the unit or small group level, there are several ways to provide feedback on changes that may have occurred between LPI administrations. For a recent group of executive MBA students, for example, I analyzed the students’ results over a 16-month period (knowing, by the way, that the observers were very likely not the same in the two administrations). I considered a change in the average score of 1.5 (plus to be favorable and negative to be unfavorable, and +/- at 2.0 or greater to be meaningful). From this perspective, comparing Self, Managers, and Direct Reports, we calculated that there was positive movement in four of the leadership practices (with one showing little change) for the group of leaders, that there were meaningful changes for all five leadership practices from the viewpoint of their Managers, and that three of the leadership practices showed favorable movement as perceived by their Direct Reports (with the other two remaining relatively the same in frequency). There were still areas for developing their leadership skills at an individual level but clearly some things had changed from the time the group entered the program to the point that they were concluding it.
There are statistical methods available to test whether the mean (average) scores from two samples are the same or different (e.g., men versus women or time 1 versus time 2). This involves calculating a "t" or "z" statistic, and the formula is, again generally: compute the difference between the two mean scores and divide that result by the square root of the standard deviation squared of one mean divided by the sample size plus the standard deviation squared of the second mean divided by the sample size. You can look up in a t-table or z-table the probability of finding such a "result" given the degrees of freedom (which is roughly the smaller of the two sample sizes, minus 1).
For example, let’s say the average score on Model the Way was 41.9 and standard deviation 5.3, with a sample size of 14, in the initial administration and in the subsequent administration the average score was 49.6 and standard deviation was 3.9, with a sample size of 11. The formula above would yield the following calculation: (41.9-46.4)/the square root of [(5.3 squared/14) + (3.9 squared/11)] = 2.44. With 10 degrees of freedom the likely of finding such a difference by chance would be very small (only about 2% and hence you would say with 98% confidence that such a difference is "significant"). For more details, I'd refer you to various statistics books or information you may be able to find on various websites.
Barry Posner, Ph.D., is the Accolti Endowed Professor of Leadership at the Leavey School of Business, Santa Clara University, where he served as Dean for 12 years. Together with Jim Kouzes, he is author of over 30 books and workbooks on leadership and leadership development, including the just-released Stop Selling & Start Leading (with additional co-author Deb Calvert), fully-revised and updated sixth edition of the international bestseller, The Leadership Challenge, and Learning Leadership, selected by Strategy+Business as one of the 2016 Best Business Books of Year.