Cultural Audit: Final Report

Nichols and Associates, Inc. Washington, D.C. October 2002

Survey Development

Scale Development. The survey instrument was developed using the following methodology:

1. The survey development process began by identifying relevant content areas derived from 27 focus groups of SCSU personnel. Additional information was gathered by reviewing process, procedures, reports, records and observations to ascertain cultural climate.
2. Nichols and Associates, Inc. developed an initial set of content domains summarizing the core issues related to personnel and management. These content domains represented broad areas of concern and served as the underlying dimensions to be assessed by the survey instrument.
3. Individual items were then derived to fit into the domains of the conceptual framework
   1. Items were selected based on the following criteria:
      1. Comprehensiveness.
      2. Avoiding obvious redundancy.
      3. Balance of domains represented. Once a sizeable pool of items was accumulated, the items were categorized as representing one or more domains. The initial item selection aimed to be as comprehensive as possible. Subject matter experts then reduced the pool of items through a series of reviews. When two items were deemed essentially redundant, one item was selected and the other excluded. Every effort was made to derive multiple items for each of the selected domains. To balance the survey, items were eliminated so that no one particular domain would predominate.
   2. The working group used simple, commonly accepted rules for the construction and evaluation of items. These rules included selecting items that present a single issue, use everyday language, and are brief enough to administer orally. The result of this process was a pool of 135 survey items distributed across 15 scales.
4. The initial testing of this instrument included a pilot test. Pilot testing was completed with faculty, staff, and students at SCSU. Corrections and/or edits to the survey instrument were based on input from faculty, staff, administration, and students.
5. The survey instrument was then uploaded to an Internet website where faculty, staff and students could complete and submit the survey to Nichols and Associates, Inc. for analysis. The website and password was advertised to SCSU staff and students prior to and during the course of the active survey via flyers and the SCSU website.

Reliability. Reliability refers to a measure's consistency (i.e., "If I repeatedly measure the same thing, under similar conditions and with no true change in the level of the measured attribute, will I obtain the same results?"). There are a number of methods for assessing the reliability of a survey instrument. Perhaps the most commonly used is Cronbach's alpha coefficient, a statistic that examines the consistency of scales used in a survey. Alphas are calculated separately for each scale and it may range from 0 to 1. Higher values indicate greater reliability. While there are no independent standards for judging an acceptable level of reliability, many behavioral research scientists use a "rule of thumb" value of .7 as a reasonably high alpha for most research efforts.

Limitations. The current survey is not without its limitations and critics. Sometimes respondents may be uncomfortable responding to the survey, because sensitive areas are presented. Nichols and Associates, Inc. recognizes the concern, but believe we do not get the "hard" answers unless we ask the "hard" questions. Some leaders believed some of the concerns were not adequately addressed. Without unduly lengthening the survey (thereby decreasing the response rate and increasing resistance to the process), we could not cover every potential culture/climate issue. Based on our research, we focused on the key climate concerns, which generates an accurate and comprehensive picture of how SCSU members view/perceive/interpret the university's cultural climate.

We also received some complaints/concerns from white males who believed the survey issues assumed they are responsible for all climate, EEO or diversity difficulties. Regarding this issue, Nichols and Associates, Inc., reiterates the discussion above on survey length and adds the following: Nichols and Associates, Inc, in developing and designing the instrument conducted numerous focus groups and assessed the organizational climate involving approximately 300 faculty, staff, and students [See Appendix A -Focus Group Report].

Using these methods, university leadership can obtain a valid indication of whether discrimination is a major issue in the university. If so, the university can gather additional information (using techniques other than a survey) to determine where the concerns are. SCSU may use other techniques including:

• Individual interviews – Consider conducting semi-structured interviews to maintain consistency through all interviews. This approach and structure will assist in validating survey findings.
• Focus groups – A trained facilitator should conduct focus groups. Recorders should be able to summarize the general themes (e.g., recurring viewpoints) of focus group discussions.
• Record/documentation review – Reviewing archived records will assist in determining possible trends and the total number of reports, complaints or cases as well as validate alleged cases or reports (e.g., discrimination).
• Direct observations – While directly observing interpersonal behaviors provides qualitative information, one cannot observe all situations all the time. Keeping findings in context is a cardinal principle of observational analysis.

The Culture Survey, within its design methodology, is a sound instrument offering accurate information to SCSU management and leadership as well as student organizations.

<< Previous

Table of Contents

Next >>