![\"Screenshot](\"https:\/\/openbooks.library.unt.edu\/quantitativeanalysisinexerciseandsportscience\/wp-content\/uploads\/sites\/3\/2021\/06\/5.1.png\")
Figure 5.1. Screenshot of synonyms of the word \"significant\" from www.thesaurus.com[\/caption]\r\n\r\nWe have heard the phrase statistical significance used quite a bit already when we are discussing a statistical finding that has a p<\/em> value of less than or equal to 0.05. Often in research and academic journal articles, the phrase statistical significance is shortened to just state that it was a \"significant finding.\" This is problematic for many who don\u2019t understand that the word significant in a statistical perspective may be different than the more straightforward and widely used definition for significance that most people use. For example, consider a study that demonstrates a statistically significant difference between weight loss outcomes in two groups. One that took a weight loss supplement and one that did not. This may be described as a \u201dsignificant\u201d difference in outcomes between the groups because the p value was less than 0.05. Think about how the supplement company might use that for marketing purposes. They might say something like \u201close significantly more weight with our product\u201d or any of the other synonyms shown above that comes from Chapter Learning Objectives<\/span><\/h2>\r\n<\/header>\r\n\r\n\r\n \t- \r\nDiscuss meaningfulness as it applies to quantitative analysis<\/div><\/li>\r\n \t
- \r\nLearn how to quantify and interpret effect size estimates<\/div><\/li>\r\n<\/ul>\r\n<\/div>\r\n<\/div>\r\n
Effect Sizes<\/h1>\r\nEffect sizes, or more appropriately effect size estimates, help us statistically quantify practical significance. Effect sizes have also been described as a measure of \"meaningfulness.\"[footnote]Ellis, P. (2010). The Essential Guide to Effect Sizes: Statistical Power, Meta-Analysis, and Interpretation of Results. Cambridge, NY. Cambridge University Press.[\/footnote] From a purely practical and applied perspective, the effect size should be the primary outcome of research inquiry.[footnote]Cohen J. (1990). Things I have learned (so far) Am Psychol. 45:1304\u20131312.[\/footnote] No matter what we are measuring, we are likely most interested in the size of the difference or relatedness between variables. The reliability of our finding or how confident we are in the finding is important also, but it shouldn\u2019t be the only product.\r\n\r\nThis has become such a problem in many fields of research that more than 800 researchers recently published a paper condemning the usage of p<\/em> values since many rely too heavily on it and not enough on practical measures of significance that demonstrate realistic effects.[footnote]Nature 567:305-307 (2019). https:\/\/www.nature.com\/articles\/d41586-019-00857-9?fbclid=IwAR1jzbGpWu9wsHIwBdOu3byOielCLEQxPZMvHJ-3X4GW2gvy4eD98a7a9EU<\/a>[\/footnote] Completely abandoning measures of statistical significance like p<\/em> values is probably a little too extreme, but their point is well taken. We should be publishing measures of both statistical and practical significance or meaningfulness.\r\n\r\nSo, let\u2019s define effect size. An[pb_glossary id=\"682\"] effect size[\/pb_glossary] is a measure that describes the magnitude or size of the difference or relatedness between the variables we are measuring. This means that it is describing the practical\/meaningful significance. As we will see in this chapter, it really all boils down to how much the data overlaps between our groups or variables. If we are comparing groups for differences, we would likely not expect to see much overlap in the data. If we were evaluating our data for similarity or association, we are looking for more overlap.\r\nTypes of Effect Sizes<\/h1>\r\nAs mentioned above, effect sizes can be created for both relatedness and difference tests. This essentially means that there are 2 \u201cfamilies\u201d of effect sizes; r<\/em> for relatedness and d<\/em> for difference. Conversions can also be completed between the two if it was necessary to make comparisons across different study types. That won't be demonstrated in this chapter, but a quick internet search for \"r<\/em> to d<\/em> effect size conversion,\" or vice versa, will bring up several online calculators and formulas.\r\nCohen's d<\/em> Effect Size Estimates<\/h2>\r\nLooking on the difference side of effect sizes (ES), Cohen\u2019s d<\/em> effect size estimates are most commonly seen. They are named after Jacob Cohen. They can be quickly calculated when both means and the pooled standard deviation is known. The means can be from independent or paired groups. The pooled standard deviation is the standard deviation calculated from both groups as one.\r\n[asciimath]\"Cohen's d ES\" = (M_1 - M_2)\/\"sd\"_\"pooled\"[\/asciimath]<\/p>\r\n
[asciimath]M = mean, sd = \"standard deviation\"[\/asciimath]<\/p>\r\nWith this data, we simply subtract one mean from the other and divide by the pooled standard deviation. If the value is negative, that simply means that the second mean or the one that was subtracted from the other was larger than the original mean. This is dependent on how the dataset was ordered or how they were called into the formula. So, the direction doesn\u2019t really matter here since the magnitude is the largest concern. For this reason, the negative sign is usually not included. Effect size estimates can be interpreted with the scale provided. Anything less than 0.2 is considered a trivial difference, 0.2 to 0.49 is a small difference, 0.5 to 0.79 is a medium difference, and anything \u2265 0.8 is a large difference.\r\n\r\n[caption id=\"attachment_686\" align=\"aligncenter\" width=\"446\"]![\"Figure](\"https:\/\/openbooks.library.unt.edu\/quantitativeanalysisinexerciseandsportscience\/wp-content\/uploads\/sites\/3\/2021\/06\/Screen-Shot-2021-07-15-at-2.35.52-PM.png\")
Figure 5.2 T-shirt size description for Cohen's d effect size estimate interpretation[\/caption]\r\n\r\nAs mentioned previously, effect sizes really all boil down to the amount of overlap in the data. Figure 5.3 demonstrates this with 2 examples. The first one (on top) is a comparison of 2 groups. One distribution has a mean of 50, the other has a mean of 55, and the pooled standard deviation is 10. Solving for the Cohen\u2019s d<\/em> effect size, we find that there is a medium difference between these 2 groups with a value of 0.5 (d<\/em>=(55-50)\/10). The second example has a distribution with a mean of 50, a distribution with a mean of 110, and a pooled standard deviation of 60. Here we find a large difference with a Cohen\u2019s d<\/em> effect size of 1.0 (d<\/em>=(110-50)\/60).\r\n\r\nTake a look at the amount of overlap between the distributions in each example. Can you see why the Cohen\u2019s d<\/em> effect size estimate is larger in the 2nd example? It has a lot less overlap between the distributions. What if the data distributions were completely separate with no amount of overlap? What would happen to the effect size then? It would increase quite a bit. What if nearly 100% of the data were overlapping? The effect size would shrink quite a bit and likely be trivial. This is of course if we are still using Cohen\u2019s d<\/em> as our measure of effect size, since it is looking for difference. But what if we wanted to look for similarity?\r\n\r\n[caption id=\"attachment_688\" align=\"aligncenter\" width=\"1214\"]![\"Figure](\"https:\/\/openbooks.library.unt.edu\/quantitativeanalysisinexerciseandsportscience\/wp-content\/uploads\/sites\/3\/2021\/06\/5.3.png\")
Figure 5.3 Two density plots demonstrating effect size magnitude with the amount of overlap between distributions. The top plot demonstrates an effect size (d) of 0.5 and the bottom demonstrates 1.0.[\/caption]\r\nr Effect Size Estimates<\/em><\/h2>\r\nWhen looking for similarity, the PPM r <\/em>value should be used as an effect size. The fact that r <\/em>is an effect size is a good thing, since we already know how to quantify it and use it from Chapter 3. Recall that r<\/em> values can range from -1 to 0 to +1 and we have already seen how to interpret them in Chapter 3.\r\n\r\n[caption id=\"attachment_551\" align=\"aligncenter\" width=\"750\"]![\"Venn](\"https:\/\/openbooks.library.unt.edu\/quantitativeanalysisinexerciseandsportscience\/wp-content\/uploads\/sites\/3\/2021\/06\/3.12.png\")
Figure 5.4 A reproduction of the Venn diagram representation of a correlation seen in Chapter 3[\/caption]\r\n\r\nWhen interpreting effect sizes for relationships or associations, we are still concerned with the amount of overlap in the data, and we\u2019ve seen this before in Figure 3.12 (now 5.4 in this chapter). When utilizing a bivariate correlation or simple linear regression with only 2 variables (which are represented by the ovals above), we are mostly concerned with the area where they have shared variance. That is represented as the area of overlap in this figure. Remember the coefficient of determination r2<\/sup><\/em> helps us explain the amount of shared variance. Please review Table 3.4 (reproduced below) and go back to Chapter 3 if you need a refresher.\r\n
- \r\n \t
- \r\nDiscuss meaningfulness as it applies to quantitative analysis<\/div><\/li>\r\n \t
- \r\n
Learn how to quantify and interpret effect size estimates<\/div><\/li>\r\n<\/ul>\r\n<\/div>\r\n<\/div>\r\nEffect Sizes<\/h1>\r\nEffect sizes, or more appropriately effect size estimates, help us statistically quantify practical significance. Effect sizes have also been described as a measure of \"meaningfulness.\"[footnote]Ellis, P. (2010). The Essential Guide to Effect Sizes: Statistical Power, Meta-Analysis, and Interpretation of Results. Cambridge, NY. Cambridge University Press.[\/footnote] From a purely practical and applied perspective, the effect size should be the primary outcome of research inquiry.[footnote]Cohen J. (1990). Things I have learned (so far) Am Psychol. 45:1304\u20131312.[\/footnote] No matter what we are measuring, we are likely most interested in the size of the difference or relatedness between variables. The reliability of our finding or how confident we are in the finding is important also, but it shouldn\u2019t be the only product.\r\n\r\nThis has become such a problem in many fields of research that more than 800 researchers recently published a paper condemning the usage of p<\/em> values since many rely too heavily on it and not enough on practical measures of significance that demonstrate realistic effects.[footnote]Nature 567:305-307 (2019). https:\/\/www.nature.com\/articles\/d41586-019-00857-9?fbclid=IwAR1jzbGpWu9wsHIwBdOu3byOielCLEQxPZMvHJ-3X4GW2gvy4eD98a7a9EU<\/a>[\/footnote] Completely abandoning measures of statistical significance like p<\/em> values is probably a little too extreme, but their point is well taken. We should be publishing measures of both statistical and practical significance or meaningfulness.\r\n\r\nSo, let\u2019s define effect size. An[pb_glossary id=\"682\"] effect size[\/pb_glossary] is a measure that describes the magnitude or size of the difference or relatedness between the variables we are measuring. This means that it is describing the practical\/meaningful significance. As we will see in this chapter, it really all boils down to how much the data overlaps between our groups or variables. If we are comparing groups for differences, we would likely not expect to see much overlap in the data. If we were evaluating our data for similarity or association, we are looking for more overlap.\r\n
Types of Effect Sizes<\/h1>\r\nAs mentioned above, effect sizes can be created for both relatedness and difference tests. This essentially means that there are 2 \u201cfamilies\u201d of effect sizes; r<\/em> for relatedness and d<\/em> for difference. Conversions can also be completed between the two if it was necessary to make comparisons across different study types. That won't be demonstrated in this chapter, but a quick internet search for \"r<\/em> to d<\/em> effect size conversion,\" or vice versa, will bring up several online calculators and formulas.\r\n
Cohen's d<\/em> Effect Size Estimates<\/h2>\r\nLooking on the difference side of effect sizes (ES), Cohen\u2019s d<\/em> effect size estimates are most commonly seen. They are named after Jacob Cohen. They can be quickly calculated when both means and the pooled standard deviation is known. The means can be from independent or paired groups. The pooled standard deviation is the standard deviation calculated from both groups as one.\r\n
[asciimath]\"Cohen's d ES\" = (M_1 - M_2)\/\"sd\"_\"pooled\"[\/asciimath]<\/p>\r\n
[asciimath]M = mean, sd = \"standard deviation\"[\/asciimath]<\/p>\r\nWith this data, we simply subtract one mean from the other and divide by the pooled standard deviation. If the value is negative, that simply means that the second mean or the one that was subtracted from the other was larger than the original mean. This is dependent on how the dataset was ordered or how they were called into the formula. So, the direction doesn\u2019t really matter here since the magnitude is the largest concern. For this reason, the negative sign is usually not included. Effect size estimates can be interpreted with the scale provided. Anything less than 0.2 is considered a trivial difference, 0.2 to 0.49 is a small difference, 0.5 to 0.79 is a medium difference, and anything \u2265 0.8 is a large difference.\r\n\r\n[caption id=\"attachment_686\" align=\"aligncenter\" width=\"446\"]
Figure 5.2 T-shirt size description for Cohen's d effect size estimate interpretation[\/caption]\r\n\r\nAs mentioned previously, effect sizes really all boil down to the amount of overlap in the data. Figure 5.3 demonstrates this with 2 examples. The first one (on top) is a comparison of 2 groups. One distribution has a mean of 50, the other has a mean of 55, and the pooled standard deviation is 10. Solving for the Cohen\u2019s d<\/em> effect size, we find that there is a medium difference between these 2 groups with a value of 0.5 (d<\/em>=(55-50)\/10). The second example has a distribution with a mean of 50, a distribution with a mean of 110, and a pooled standard deviation of 60. Here we find a large difference with a Cohen\u2019s d<\/em> effect size of 1.0 (d<\/em>=(110-50)\/60).\r\n\r\nTake a look at the amount of overlap between the distributions in each example. Can you see why the Cohen\u2019s d<\/em> effect size estimate is larger in the 2nd example? It has a lot less overlap between the distributions. What if the data distributions were completely separate with no amount of overlap? What would happen to the effect size then? It would increase quite a bit. What if nearly 100% of the data were overlapping? The effect size would shrink quite a bit and likely be trivial. This is of course if we are still using Cohen\u2019s d<\/em> as our measure of effect size, since it is looking for difference. But what if we wanted to look for similarity?\r\n\r\n[caption id=\"attachment_688\" align=\"aligncenter\" width=\"1214\"] Figure 5.3 Two density plots demonstrating effect size magnitude with the amount of overlap between distributions. The top plot demonstrates an effect size (d) of 0.5 and the bottom demonstrates 1.0.[\/caption]\r\nr Effect Size Estimates<\/em><\/h2>\r\nWhen looking for similarity, the PPM r <\/em>value should be used as an effect size. The fact that r <\/em>is an effect size is a good thing, since we already know how to quantify it and use it from Chapter 3. Recall that r<\/em> values can range from -1 to 0 to +1 and we have already seen how to interpret them in Chapter 3.\r\n\r\n[caption id=\"attachment_551\" align=\"aligncenter\" width=\"750\"]
Figure 5.4 A reproduction of the Venn diagram representation of a correlation seen in Chapter 3[\/caption]\r\n\r\nWhen interpreting effect sizes for relationships or associations, we are still concerned with the amount of overlap in the data, and we\u2019ve seen this before in Figure 3.12 (now 5.4 in this chapter). When utilizing a bivariate correlation or simple linear regression with only 2 variables (which are represented by the ovals above), we are mostly concerned with the area where they have shared variance. That is represented as the area of overlap in this figure. Remember the coefficient of determination r2<\/sup><\/em> helps us explain the amount of shared variance. Please review Table 3.4 (reproduced below) and go back to Chapter 3 if you need a refresher.\r\n - \r\n
![\"Image](\"https:\/\/openbooks.library.unt.edu\/quantitativeanalysisinexerciseandsportscience\/wp-content\/uploads\/sites\/3\/2021\/06\/5.5.png\")
Figure 5.5 The top density plot from Figure 5.3 showing a Cohen's d of 0.5 along with the same data depicted as a scatter plot below (r = 0.053) demonstrating no relationship.[\/caption]\r\n\r\nThis also demonstrates why we use specific data visualization techniques with each statistical test. Whenever you are testing for difference, you should likely use a box and whisker plot, density plot, or distribution plot. Whenever you are testing for similarity, you should use a scatterplot. If the wrong plot is used, you may confuse or mislead your audience.\r\n