Extended DISC – Extending People

Posts tagged ‘validity and reliability’

DISC Tests – how to identify valid and reliable products

Hi there,

 The most common question I get asked about DISC tests and reports is, “aren’t DISC results just about how I was feeling or thinking about at the time?”

My reply, as it often is, would be Sharon’s standard answer number one, “it depends.”  Question number two is, in this case, as again, it often is:  “What’s the context?” 

If the questionnaire is a simple tick and flick on paper, my answer would be simple, Yes, the results would most likely only be about how I was feeling or thinking about at the time, and I would expect you to conclude that you can easily consciously and/or unconsciously influence the outcome/results. I would recommend this type of DISC test for personal entertainment and a very basic introduction to the four fundamental human behavioural styles, only.

 If the context is using the results for personal development one, or in a workplace environment, which is where we at Talent Tools operate, I would have a more in depth answer conversation.  Let’s recap. What was the original question?

“Aren’t DISC results just about how I was feeling or thinking about at the time?”DISC instruments at Talent Tools

My response this time would be, “it depends on the validity and reliability of the questionnaire.”

In this article, I am going to discuss “reliability”.  To find out about “validity“, please see my previous post discussing validity in psychometric instruments.

First up, let’s go to Wikipedia for an unbiased definition of the term “reliability” as it refers to psychometric tests, such as DISC the source is http://en.wikipedia.org/wiki/Reliability_(psychometrics) on 25 September 2013:

I have a very “C” comment to make at this point. I would not have gotten away with this type of blatant reliance on references/quotations in my undergraduate days. Not only did we have to identify the reference, we had to justify the stance/perspective of the author and their credibility in regard to the statements we quoted (all within the word limit constraints). Now, with “wiki” if the statements/information provided remains unchallenged/unchanged on “wiki” it is accepted – pending ongoing research, henceforth updates. The researcher/scientist (“C” style) in me, likes this, however, I do wonder how today’s students verify information and defend its relevance to their studies/assignments. But, that, ladies and gentlemen, is how the mind of a  true “C” style works. That said, back to 2013 …

Back to “wiki” http://en.wikipedia.org/wiki/Reliability_(psychometrics) sourced 25 September 2013 – In the psychometrics, reliability is used to describe the overall consistency of a measure. A measure is said to have a high reliability if it produces similar results under consistent conditions. For example, measurements of people’s height and weight are often extremely reliable.[1][2]  This means that if you complete a DISC questionnaire on one day (or measure your height) if you took it again the next day (or measured your height) we would expect the result to be very similar, if not the same, as the previous day. 

Of course, the world of psychometrics (where any decent DISC instrument resides) has to put more conditions around this for the sake or rigor.  The following is brought to you compliments of Wiki:

There are several general classes of reliability estimates:

  • Inter-rater reliability assesses the degree of agreement between two or more raters in their appraisals.
  • Test-retest reliability assesses the degree to which test scores are consistent from one test administration to the next. Measurements are gathered from a single rater who uses the same methods or instruments and the same testing conditions.[2] This includes intra-rater reliability.
  • Inter-method reliability assesses the degree to which test scores are consistent when there is a variation in the methods or instruments used. This allows inter-rater reliability to be ruled out. When dealing with forms, it may be termed parallel-forms reliability.[3]
  • Internal consistency reliability assesses the consistency of results across items within a test.[3]

All of the above are actually relevant to determining the reliability of the DISC test you are considering using. It might be a bit obsessive to expect you to have expectations in regard to each of these, however, you certainly should expect your DISC  product provider to be able to provide coherent answers to your questions in this regard and to provide evidence of how their product complies/scores on each of these reliability elements.

As you might expect, the factors can be broken down further and examined. To do so, please go to :  http://en.wikipedia.org/wiki/Reliability_(psychometrics) to source more information to assist you in this regard.

However, put simply (or as simply as a “C” style can) – The goal of estimating reliability is to determine how much of the variability in test scores is due to errors in measurement and how much is due to variability in true scores,  true score is the replicable feature of the concept being measured. It is the part of the observed score that would recur across different measurement occasions in the absence of error.

Again, to muddy the waters a bit more for you – do expect variances in the profile which depicts the person’s perception of the behaviour they need to demonstrate to be successful in their current role and environment. This profile has to, by definition, change with the role and environment of the subject person. If it remains same over a range of time, role and work environments an alarm bell should ring very loudly. And this provides the basis for another post because the reasons could be found in the instrument or the person.

However, it is reliability in the profile which depicts the person’s natural unconscious/natural style, which is the test of “reliability” in DISC reports. And, here is where I do make a bold statement – If you are using a reliable instrument, the results for your natural style will alter very little in depicting your DISC  traits distribution. There are a few exceptions to the rule (again, the topic for a future post). But they are few and uncommon. If the product you are considering/using has a “flexibility zone” it is unlikely, over your lifetime, that you the profile of your natural style will move outside your flexibility zone over your lifetime.

In conclusion. 

Style Descriptors on the Extended DISC Diamond

Style Descriptors on the Extended DISC Diamond

There are many products built on the open-source theory of DISC. Some are very reliable, others are not. Unfortunately price point, aesthetically appealing reports nor the level of customer service is an accurate gauge of the reliability (or validity) of the product you are using/considering.

So, when you are sourcing a DISC based assessment product (if the product you are looking at is based on a four quadrant model it is almost certainly based on DISC theory regardless of whether it has DISC in its name) ask  questions about the validity and reliability of the product, and about the  testing to which  it has been subjected. Be wary if you do not receive confident responses or published evidence in reply.

At Talent Tools, we will be delighted to assist you to determine reliability and validity. Or, play it safe and just use one of our DISC products.

Sharon can be contacted at Talent Tools on 1800 768 569 in Au or elsewhere on 61 7 3103 0177 or simply email:  sharon@talenttools.com.au