51Quant. Methods 1 (M. Messner)
Concepts and their measurement
Pixabay.com
52Quant. Methods 1 (M. Messner)
Concepts and their measurement
• Research is usually interested in relationships between different phenomena or concepts.
– E.g., Intelligence, organizational performance, customer satisfaction, strategic position, etc.
– Some phenomena are directly observable and can thus be (rather) easily measured (e.g., age, industry affiliation, revenues, etc.)
– Other phenomena are much harder to capture.
• Construct validity is an important concern in quantitative research
53Quant. Methods 1 (M. Messner)
Concepts and their measurement
Concept measurement depends on the method of data collection:
• Archival research: Use of secondary data obtained from firms, public sources, databases, etc.
• Survey studies: Collection of data through questionnaires
• Experiments: Collection of data through laboratory replication of a real-world situation
54Quant. Methods 1 (M. Messner)
Concepts and their measurement
Examples for archival proxies:
• “To proxy and control for the firm’s innovative capability or capacity, we include the variable cumulative number of patents, which is the total number of patents that the firm has generated since the year of its establishment until the year before the observation year.” (Jia et al. 2019, AMJ)
• “As a proxy for project complexity, I used the total number of electrical engineering hours on a project”. (Young-Hyman, 2017, ASQ)
55Quant. Methods 1 (M. Messner)
Concepts and their measurement
Using surveys to measure concepts
Example
56Quant. Methods 1 (M. Messner)
Concepts and their measurement
• Surveys often rely on so-called Likert scales to measure the extent to which the respondent agrees or disagrees with a statement.
– Likert-scales can have different degrees of detail, but most common are 7-point scales.
– End-points represent the most extreme positions on a question, e.g.:
• 1 = definitely false; 7 = definitely true; 1 = strongly disagree, 7 = strongly agree, etc.
• The different questions represent “items“. The construct (representing the concept of interest) is then a combination of these items.
– Items either taken from prior research or newly developed scale
– # of items: balance length of questionnaire with measurement quality; at least 3 items for a (reflective) construct
57Quant. Methods 1 (M. Messner)
Concepts and their measurement
Compare the two regressions:
Manifest variables (items) x1-x5:
y = ß0 + ß1* x1 + ß2 * x2 + ß3 * x3 + ß4 * x4 + ß5 * x5 + u
One latent construct z1:
y = ß0 + ß1* z1 + u
58Quant. Methods 1 (M. Messner)
Concepts and their measurement
There are two types of multi-item (latent) constructs:
• Reflective: Items represent manifestations of the construct
• Formative: Items represent elements (components) of the construct
This distinction is important, because construct validity and reliability of these constructs must be assessed differently.
59Quant. Methods 1 (M. Messner)
Concepts and their measurement
Monitoring difficulty
It is not possible to supervise the salesperson’s activities closely.
It is difficult for us to evaluate how much effort this salesperson really puts into her/his job.
It is relatively easy for this salesperson to turn in falsified sales call reports.
Our evaluation of this salesperson cannot be based on his/her activity and sales call reports.
Salesperson’s ability
This salesperson has a high degree of competence in tailoring his/her sales approach to the specific situation on hand.
This salesperson has been very creative in designing relevant solutions to customers’ problems.
This salesperson is a skilled and persuasive negotiator.
This salesperson is capable of closing a deal in a tough selling situation.
This salesperson is able to learn from past experiences and adapt them to current circumstances.
This salesperson is skilled in extracting the unique problems faced by and the requirements of his/her customers.
Lo et al. (2011, JMR)
60Quant. Methods 1 (M. Messner)
Concepts and their measurement
Construct validity (Do we really measure what we seek to measure?) and reliability (Do we measure reliably?) are a frequent concern in survey research.
Construct validity and reliability should be approached from a judgmental and a technical perspective:
• Judgmental (for both reflective and formative constructs):
– Do the questions really reflect what I think the phenomenon is about? (often referred to as “face validity”)
61Quant. Methods 1 (M. Messner)
Concepts and their measurement
• Technical (for reflective constructs only):
– Internal consistency and convergent validity: do the items converge on the construct?
– Discriminant validity: are the constructs sufficiently different from each other?
62Quant. Methods 1 (M. Messner)
Concepts and their measurement
Reflective constructs Formative
constructs
(composite
constructs)
Face validity Items should all reflect the underlying construct (in a similar way)
Items should together add up to the construct (in a comprehensive way)
Internal consistency & convergent validity
Factor analysis:• High factor loadings of each item• High Average Variance Extracted
(AVE)• High composite reliability• Good goodness-of fit criteria
Discriminant validity Factor analysis:• Small cross-loadings• Square root of AVE >bivariate
correlations between constructs (Fornell-Larcker Test)
63Quant. Methods 1 (M. Messner)
Concepts and their measurement
The basis for assessing construct validity and reliability of reflective
constructs is typically factor analysis.
• Factor analysis:
– Identifies (or confirms) the relationship between items and underlying constructs (factors)
– Assumes that the correlation between items can be explained by underlying factors (of which the items are manifestations)
– Exploratory factor analysis (EFA): no ex ante specification of factor structure: see which factors emerge from the data
– Confirmatory factor analysis (CFA): factors and their relationships to items are specified a priori and then tested. Confirm if items load as they should.
• Different uses in studies:
– Only EFA or only CFA
– First an EFA, then a CFA
64Quant. Methods 1 (M. Messner)
Exploratory factor analysis
Example
We are interested in 4 (reflective) constructs and, to this end, collected 16 variables through a survey.
Our hypothesized factor structure is as follows:
Factor1: v2 + v9 + v13 + v16 + v17Factor2: v6 + v7 + v8 + v11Factor3: v1 + v3 + v5 + v20Factor4: v4 + v12 + v14
65Quant. Methods 1 (M. Messner)
Exploratory factor analysis
v1 v2 v3 v4 v5 v6 v7 v8 v9 v11 v12 v13 v14 v16 v17 v20v1 1,00v2 -0,05 1,00v3 0,44 0,08 1,00v4 0,00 -0,07 0,10 1,00v5 0,41 0,00 0,40 0,13 1,00v6 0,02 0,02 0,07 0,02 0,19 1,00v7 0,01 -0,09 0,00 0,13 0,08 0,34 1,00v8 0,05 -0,02 0,03 0,02 0,09 0,31 0,37 1,00v9 -0,05 0,25 -0,06 0,03 -0,03 -0,12 -0,06 -0,04 1,00v11 0,04 -0,10 0,08 0,04 0,16 0,47 0,48 0,37 -0,10 1,00v12 0,11 0,06 0,08 0,29 0,17 0,12 0,03 0,06 0,03 0,03 1,00v13 -0,13 0,31 -0,03 0,05 0,05 -0,08 0,02 -0,09 0,32 -0,08 0,08 1,00v14 0,02 -0,04 0,04 0,34 0,11 -0,05 0,00 0,02 0,11 -0,07 0,25 0,10 1,00v16 -0,05 0,28 0,03 0,08 0,00 -0,13 0,00 -0,04 0,36 -0,15 0,08 0,38 0,03 1,00v17 -0,01 0,28 0,01 -0,04 -0,01 -0,14 -0,06 -0,07 0,36 -0,18 0,04 0,33 -0,01 0,29 1,00v20 0,35 0,01 0,40 0,04 0,36 0,00 0,09 -0,06 -0,05 0,00 0,10 0,00 0,03 0,03 0,07 1,00
Let‘s first look at the correlations between the variables
66Quant. Methods 1 (M. Messner)
Exploratory factor analysis
Fa
Fb
x1
x2
x3
x4
Latent factors Manifest variables
a1
a2
b1
b4
X1 = a1*Fa + b1*Fb + u1
X2 = a2*Fa + b2*Fb + u2
etc
All relationships possible Exploratory factor analysis:
a1, a2, b1, etc.: factor loadings
Must specify either:- Number of factors (in line with your theory)- Minimum explanatory value of each factor (typically:
Eigenvalue > 1).
Two main “extraction methods“ for factors:- Principal component analysis: factors try to explain
entire variance of variables- Principal axis factoring: factors explain only shared
variance of variables
EFA: finding a good factor structure
67Quant. Methods 1 (M. Messner)
Exploratory factor analysis
Illustration of factor structure
Factor 1
Factor 2
1
0.5
– 0.5
– 0.5 10.5
– 1
– 1
Variable x1 = 0.6 * F1 + 0.6 * F2 + u
68Quant. Methods 1 (M. Messner)
Exploratory factor analysis
SPSS: Analyze Dimension Reduction Factor
69Quant. Methods 1 (M. Messner)
Exploratory factor analysis
Output from SPSS
70Quant. Methods 1 (M. Messner)
Exploratory factor analysis
Output from SPSS
71Quant. Methods 1 (M. Messner)
Exploratory factor analysis
• Eigenvalues: an indicator for how much variance in the variables the factor explains
• E.g., when Eigenvalue of each Factor = 1 each Factor explains the same amount of variance
• Mathematically: the sum of the squared loadings across all variables for a given factor (see next page)
• Variance Extracted: how much % of the shared variance in all variables is explained by the factor.
• E.g., when Eigenvalue (Factor 1) = 1, and 5 Factors Factor 1 explains 20% of variance
• Factor loadings for each variable-factor pair: correlations
Output from EFA
72Quant. Methods 1 (M. Messner)
Exploratory factor analysis
Output from SPSS
Factor Matrixa
Factor1 2 3 4
v1 0,336 0,419 -0,350 -0,130v2 -0,281 0,292 0,207 -0,220v3 0,319 0,508 -0,280 -0,133v4 0,120 0,259 0,122 0,559v5 0,388 0,499 -0,135 -0,036v6 0,497 -0,018 0,330 -0,103v7 0,444 0,016 0,438 -0,032v8 0,387 -0,026 0,338 -0,049v9 -0,389 0,304 0,280 -0,044v11 0,622 -0,070 0,424 -0,126v12 0,107 0,303 0,100 0,308v13 -0,370 0,364 0,345 -0,048v14 -0,002 0,244 0,049 0,515v16 -0,363 0,380 0,270 -0,066v17 -0,388 0,342 0,175 -0,180v20 0,226 0,447 -0,257 -0,118Extraction Method: Principal Axis Factoring.a. 4 factors extracted. 12 iterations required.
73Quant. Methods 1 (M. Messner)
Exploratory factor analysis
Factor analysis first provides an “unrotated” solution.
• First factor is defined in such a way that it explains most of the shared variance of all variables almost any variable will load on this factor
• Residual variance to be explained then by second, third, etc. factor
Rotation changes the factor structure such that variance is explained more equally among factors.
• The most frequently used method is “Varimax“, which defines factors in such a way that they have high or low loadings, but not medium ones.
Unrotated and rotated factor structure
74Quant. Methods 1 (M. Messner)
Exploratory factor analysis
Unrotated and rotated factor structure
Hair et al., p. 111
75Quant. Methods 1 (M. Messner)
Exploratory factor analysis
Output from SPSS
76Quant. Methods 1 (M. Messner)
Exploratory factor analysis
Output from SPSS:
rotated solution
Rotated Factor Matrix
Factor1 2 3 4
v1 -0,002 -0,089 0,648 0,012v2 -0,019 0,496 0,045 -0,083v3 0,033 0,009 0,672 0,051v4 0,065 -0,007 0,040 0,635v5 0,166 0,013 0,604 0,165v6 0,593 -0,095 0,078 -0,007v7 0,619 -0,008 0,010 0,083v8 0,514 -0,047 0,006 0,031v9 -0,073 0,553 -0,073 0,084v11 0,750 -0,141 0,058 -0,022v12 0,082 0,083 0,140 0,418v13 -0,016 0,613 -0,051 0,114v14 -0,063 0,027 0,020 0,567v16 -0,060 0,584 0,003 0,093v17 -0,123 0,559 0,039 -0,042v20 -0,016 0,034 0,572 0,038Extraction Method: Principal Axis Factoring. Rotation Method: Varimax with Kaiser Normalization.a. Rotation converged in 5 iterations.
77Quant. Methods 1 (M. Messner)
Exploratory factor analysis
Output from SPSS:
rotated solution
Rotated Factor Matrix
Factor1 2 3 4
v1 -0,002 -0,089 0,648 0,012v2 -0,019 0,496 0,045 -0,083v3 0,033 0,009 0,672 0,051v4 0,065 -0,007 0,040 0,635v5 0,166 0,013 0,604 0,165v6 0,593 -0,095 0,078 -0,007v7 0,619 -0,008 0,010 0,083v8 0,514 -0,047 0,006 0,031v9 -0,073 0,553 -0,073 0,084v11 0,750 -0,141 0,058 -0,022v12 0,082 0,083 0,140 0,418v13 -0,016 0,613 -0,051 0,114v14 -0,063 0,027 0,020 0,567v16 -0,060 0,584 0,003 0,093v17 -0,123 0,559 0,039 -0,042v20 -0,016 0,034 0,572 0,038Extraction Method: Principal Axis Factoring. Rotation Method: Varimax with Kaiser Normalization.a. Rotation converged in 5 iterations.
Guide:
Loadings > | 0.5 | or | 0.6 |
Cross-loadings < | 0.4 | or | 0.5 |
79Quant. Methods 1 (M. Messner)
Confirmatory factor analysis
Fa
Fb
x1
x2
x3
x4
Latent factors Manifest variables
a1
a2
b3
b4
X1 = a1*Fa + u1
X2 = a2*Fa + u2
X3 = b3*Fb + u3
X4 = b4*Fb + u4
Relationship specified Confirmatory factor analysis:
a1, a2, b3, etc.: factor loadings
CFA tests how well the specified model fits the data
80Quant. Methods 1 (M. Messner)
Confirmatory factor analysis
CFA produces one hypothesis test and various “fit indicators“
Indicator Description “good“ fit
GFI Goodness of fit GFI >= 0.95
RMSEA Root mean square error of approximation
RMSEA < 0.08
CFI Comparative fit index CFI >= 0.90
SRMR (Standardized) root meansquare residual
(S)RMR < 0.08
Hypothesis test (chi-squared):
H0: “The model fits perfectly.”P-value > 0.05 so as not to reject this H0
See also: https://www.cscu.cornell.edu/news/Handouts/SEM_fit.pdf
Typically these indicators are calculated for each factor separately and for the entire model as a whole
81Quant. Methods 1 (M. Messner)
Confirmatory factor analysis
Confirmatory factor analysis output
Model Test User Model:
Test statistic 95.847
Degrees of freedom 98
P-value (Chi-square) 0.543
cfi rmsea gfi rmr1.000 0.000 0.954 0.113
82Quant. Methods 1 (M. Messner)
Confirmatory factor analysis
Confirmatory factor analysis output
Latent Variables:Estimate Std.Err z-value P(>|z|) Std.lv Std.all
factor1 =~ v2 1.000 0.771 0.474v9 1.216 0.229 5.302 0.000 0.938 0.579v13 1.259 0.233 5.395 0.000 0.971 0.605v16 1.361 0.254 5.363 0.000 1.050 0.596v17 1.123 0.215 5.219 0.000 0.866 0.559
83Quant. Methods 1 (M. Messner)
Additional validity indicators
Additional checks for convergent and discriminant validity
Convergent validity (for each factor separately):
• High Average Variance Extracted (guide: > 0.5)• How much of the variance in the items (averaged across all items of the given
factor) is explained by the given latent factor• E.g. AVE = 0.4 Factor explains 40% of the variation in item values
• High Composite reliability (guide: > 0.7)• Compares the variance extracted to the error variance (not explained).
• [Cronbach’s alpha / Coefficient alpha (guide: > 0.6 or 0.7)]
84Quant. Methods 1 (M. Messner)
Additional validity indicators
AVE and Composite Reliability
Squared loading = % variance of this variable explained by the factor
Sum of squared loadings = 0.5792 + 0.6052 + 0.5962 + 0.5592 = 1.36
Average Variance extracted = 1.36 / 4 = 0.34
factorloadings
squaredloadings
1- squared loadings
0.579 0.335 0.6650.605 0.366 0.6340.596 0.355 0.6450.559 0.312 0.6882.339 1.369 2.631
Composite reliability = 2.3392 / (2.3392 + 2.631) = 0.68
85Quant. Methods 1 (M. Messner)
Additional validity indicators
Additional checks for convergent and discriminant validity
Discriminant validity of factors:
Is a given factor more strongly correlated with its own items than with other factors?
Criterion: √ AVE > cross-correlation with other factors (Fornell-Larcker test)
Why?
• Remember: correlation2 = R2 = % of variance explained
• AVE = average variance of items explained (by the given factor)
86Quant. Methods 1 (M. Messner)
Concepts and their measurement
Typical procedure for technical reliability and construct validity (for reflective
constructs)
1. Exploratory factor analysis:
– Number of factors with Eigenvalue > 1: in line with theory?
– Sufficiently high factor loadings and low cross-loadings
– Possibly delete items that do not load correctly (and re-do EFA)
2. Confirmatory factor analysis:
– Test hypothesized factor structure: Different Goodness of fit criteria
3. Further criteria to ascertain convergent validity of factors: Composite Reliability, Average Variance Extracted
4. Further criterion to ascertain discriminant validity of factors: √ AVE > cross-correlation with other factors (Fornell-Larcker test)
87Quant. Methods 1 (M. Messner)
Example
Ramani & Kumar (2008), Journal of Marketing
“Composite reliability is an indicator of the shared variance among the set of observed variables used as indicators of a latent construct (Fornell and Larcker1981; Kandemir, Yaprak, and Cavusgil 2006). As Table 2, Panel A, shows,construct reliabilities for all the seven latent constructs ranged from .82 to .97, well above the recommended value. In addition, the coefficient alpha values were well above the threshold value of .7 that Nunnally (1978) recommends. The seven-factor CFA model exhibited a good fit with the data (χ2 = 253.95, d.f. = 131; CFI = .95; GFI = .82; TLI = .93; IFI = .95; and SRMR = .07). The standardized factor loadings ranged from .62 to greater than .90 and were statistically significant at the α = .95 level (see Table 2, Panel B). This provided the necessary evidence that all the constructs exhibited convergent validity. We examined discriminant validity using a procedure suggested by Fornell and Larcker (1981) and widely followed by other researchers (e.g., De Wulf, Odekerken- Schroder, and Iacobucci 2001; Kandemir, Yaprak, and Cavusgil 2006). We computed the average variance extracted by the indicators corresponding to each of the seven factors and compared it with the highest variance that each factor shared with the other factors in the model. The average variance extracted for each factor was always greater than the highest shared variance (see Table 2, Panel A).“
88Quant. Methods 1 (M. Messner)
Example (cont’d)
Ramani & Kumar (2008), Journal ofMarketing
89Quant. Methods 1 (M. Messner)
Concepts and their measurement
How to aggregate the item responses to a value of the construct?
• When used in further regression analysis: • Add up the values from the items (“summated scale”) or average
them• Use so-called “factor scores” (there are different ways to calculate
such factor scores), which take into account the loadings
• Example: “We construct our composite variables using factor scores. The correlations between our variables using factor scores and summated scales range between 0.98 and 0.99 for our four composite variables (not tabulated).” (Abernethy, Bouwens, & Kroos, 2017)
• When part of a structural equations model:• Measurement model and structural model are estimated at the same
time “weights” used depend on structural model
