2025 ASEE Annual Conference & Exposition

Submission Type: Practice
Strategic Priority: Assess

Connectedness is a key aspect of participant retention in academic programs as well as in the workplace. Achieving high levels of connectedness is often considered difficult in a virtual or hybrid environment. By producing supervisor development programs that create feelings of connection and community to both the learning environment and the participants within it, program managers can ensure better outcomes for participants and the organization.

In this mixed-methods study, a pre/post participation implementation of the Rovai Classroom Community Scale as well as post-participation interviews will be used to assess levels of participants perceived learning and connectedness to the cohort communities established in three AFRL/711 HPW leadership development programs. Using these methods, we hope to determine which of the three educational delivery models have the greatest effect on participants perceived learning and connectedness to their cohort community and the broader organization.

The outcomes of this investigation will be used to shape future Organizational Health and Development programs and can be used to make decisions on the delivery models used in other adult learning initiatives across AFRL and the broader Air Force community,

Problem Statement/Knowledge Gaps:

This study aims to determine which of three educational delivery models have the greatest effect on participants’ perceived learning and connectedness to their cohort community and the broader organization.

Hypothesis/Specific Aims:

H1: Perceived connectedness and learning will increase over the course of the program amongst members of all three cohort groups regardless of the instructional delivery model as measured by the Rovai Classroom Community.

Specific Aims: To determine if perceptions of connectedness and learning increased amongst members of all three cohort groups, we will use descriptive statistics such as the mode and median to make comparisons of pre/post course submissions of the Rovai Classroom Community Scale. Thematic analysis of the post-participation interview transcripts will add additional insight into potential differences in experiences of all participants represented in the cohorts.

H2: Military members and government civilians will exhibit similar increases in perceived connectedness and learning over the course of the programs as measured by the Rovai Classroom Community Scale.

Specific Aims: To determine if perceptions of connectedness and learning increased amongst members of military and civilian groups, we will use descriptive statistics such as the mode and median to make comparisons of pre/post course submissions of the Rovai Classroom Community Scale. Thematic analysis of the post-participation interview transcripts will add additional insight into potential differences in experiences of all participants represented in the cohorts.

H3: Regardless of instructional delivery model, Military members and government civilians will exhibit similar increases in perceived connectedness and learning over the course of the programs as measured by the Rovai Classroom Community Scale.

Specific Aims: To determine if perceptions of connectedness and learning increased amongst members of military and civilian groups, we will use descriptive statistics such as the mode and median to make comparisons of pre/post course submissions of the Rovai Classroom Community Scale. A two-way ANOVA will also be used to make comparisons across the three instructional delivery models. Thematic analysis of the post-participation interview transcripts will add additional insight into potential differences in experiences of all participants represented in the cohorts.

H4: All genders surveyed will exhibit similar increases in perceived connectedness and learning over the course of the programs as measured by the Rovai Classroom Community Scale.

Specific Aims: To determine if perceptions of connectedness and learning increased across gender groups, we will use descriptive statistics such as the mode and median to make comparisons of pre/post course submissions of the Rovai Classroom Community Scale. Thematic analysis of the post-participation interview transcripts will add additional insight into potential differences in experiences of all genders represented in the cohorts.

H5: Regardless of instructional delivery model, all genders will exhibit similar increases in perceived connectedness and learning over the course of the programs as measured by the Rovai Classroom Community Scale.

Specific Aims: To determine if perceptions of connectedness and learning increased across gender groups, we will use descriptive statistics such as the mode and median to make comparisons of pre/post course submissions of the Rovai Classroom Community Scale. A two-way ANOVA will also be used to make comparisons across the three instructional delivery models. Thematic analysis of the post-participation interview transcripts will add additional insight into potential differences in experiences of all genders represented in the cohorts.

Background:

Several delivery models are currently used to accommodate participants in Organizational Health and Development sponsored educational programs. All programs are nine months in length. One includes four, weeklong, in-person sessions. Another includes one session monthly alternating between in-person and virtual meetings. The final includes one, week-long session in month one and another in month nine, with one-day virtual sessions in months two through eight.

Recognizing that supervisors are instrumental in shaping organizational climate and more directly the employee experience, AFRL’s Organizational Health & Development (OHD) team committed to the design and delivery of programming that would grow our supervisors into people focused managers (Bersin, 2015). Traditionally, supervisor training is focused on competency and skill development, both critical to success on the job, but this content is often delivered unidirectionally, in an asynchronous model, not allowing for group reflection, feedback, collaboration nor community building. AFRL supervisory development programs differ from this traditional format by not only focusing on competencies and skills, but by also focusing on developing learners vertically, in an environment in which learners are embedded within peer cohorts over a nine-month span. Vertical development refers to the stages that adult learners progress through as they advance in their cognitive sophistication (Cook-Greuter & Miller, 1994; Kegan, 1982; Torbert, 1987; Jones et al., 2020). An important aspect of cognitive sophistication is widening one’s perspective through interaction with others in order to establish an inclusive viewpoint (Jones et al., 2020). A learner cohort provides a community in which diverse members bring their unique challenges and perspectives to bear on group learning.

As evidenced in educational theory, AFRL supervisor programs take into account that adults (andragogy) learn differently from children (pedagogy) by bringing unique life experiences, a significant knowledge base and well-formed mental schema to their learning (Knowles, 1980,1984; Fenwick & Tennant, 2004; Mukhalalati & Taylor, 2019). Foundationally, AFRL’s supervisor programs consider adult learning principles including self-concept, experience, motivation and relevance (Knowles, 1980, 1984). In order for educational programs to hold value for adults they must meet the identified needs of the adult learner. Adult learners’ needs may be defined “as those that motivate the learners and substantially enhance their learning, the lack of which will lead to demotivation and failure to achieve major learning and personal goals” (Diep, et al, 2019). AFRL’s supervisory development programs ensure learning needs are met by requiring learners to enumerate their individualized goals and objectives at the onset of programming. Coaching support ensures that learners focus on their self-identified learning needs in order to maintain motivation for reaching their goals. Throughout the duration of AFRL supervisor programs, learner feedback is encouraged and collected continuously. Previous cohort feedback informs modifications to instructional design and delivery. Beyond having a say in what they learn, adults must also see the relevance in material they have not weighed in on (Knowles, 1980, 1984). AFRL supervisor programs allow learners to bring their real-life challenges to bear on program content and group discussions.

AFRL supervisor programs are further grounded in social, constructivist and social-constructivist learning theories. The cohort model is social by design. Social learning theory acknowledges that the learner is part of a community, and the teacher is responsible for creating a learning conducive environment where learners observe desired behaviors modeled by the teacher and peers (Mukhalalati & Taylor, 2019). The OHD team, academic partners and instructors are intentional in creating the sense of community amongst all those involved. Within this community knowledge is exchanged. How this knowledge is assimilated by a learner is best described when viewed with a constructivist lens. “The constructivist theory of learning, whose philosophical origins are frequently ascribed to Kant and whose educational origins to Piaget, is based on the premise that the act of learning is based on a process which connects new knowledge to pre-existing knowledge” (Dennick, 2016). Vertical development, for example, has both social and constructivist underpinnings (Jones, et al., 2020). Social-constructivism derives its roots from the social development theory of Vygotsky (1978), where he describes a “zone-of-proximal development” in which a learner can advance their problem-solving ability by relying on support from more capable peers. The interdependence of social and individual processes in the co-construction of knowledge is the focus of social constructivist theory (Saleem, et.al, 2021).

Cohort Based Learning
Cohort models have become increasingly popular as a method for grouping students throughout the completion of an academic program, especially for adult learners (Imel, 2002). Researchers discuss the use of cohorts as a means to engage older adult learners (Spaid & Duff 2009), facilitate deeper classroom discussion (Teitel, 1997), and provide support and connectedness throughout the learning experiences (Swayze & Jakeman, 2019). The organization and structure of these cohorts is also critical to the overall success of the venture (Imel, 2002).

With adult learners, the needs are diverse and often quite different than the traditional K-12 or undergraduate student, especially when considering the working adult (Spaid & Duff, 2009). This is one reason why learning cohorts are a valuable structural method for adult learners. Spaid and Duff (2009) define a learning cohort as “a group of 10-30 students that enroll at one time and advance through a program taking the same courses at the same time” (p. 104). These cohort groups offer stability and continuity that is often not readily provided in a traditional collegiate format (Spaid & Duff, 2009). Cohort models that allow students to move through courses and the program together aid in establishing a bond between adult students that persist well beyond the date of formal program graduation (Spaid & Duff, 2009).

In her analysis of group work among adult learners, Imel (1999) examines the best methods for ensuring student receive maximum impact in a group-based learning environment. She specifically addresses the importance of relationships in this space. Imel (1999) stresses that the structure of the learner-facilitator relationship is an important aspect of group learning. The facilitator is responsible for the establishment and maintenance of learning environments and the dissemination of relevant course and program related information (Imel, 1999). Additionally, they will structure the groups in a purposeful manner that furthers the learning of all members of the group (Imel, 1999). While Imel’s (1999) work applies to group work broadly, there are still lessons to learn when considering cohort models. Activities and group composition remain a critical piece of cohort construction as does the role of the facilitator.
In her later work, Imel (2002) indicates facilitators must structure cohorts deliberately and effectively to ensure success since it is not always guaranteed. Teaching and facilitation must be structured to develop relationships, balance group and individual development, provide supports and challenges, and acknowledge and address tensions between learners and instructors (Imel, 2002).

Tietel (1997) shares Imel’s perspective. In his writing on “The Power of the Cohort Model,” Teitel (1997) stressed there are benefits and drawbacks to this model of teaching and learning. Benefits include a gain in support and connection between students as well as increased depth of discussion (Tietel, 1997, p. 68-69). Some unexpected drawbacks include change in dynamics and power relationships between students and faculty and among student groups (Teitel, 1997). Faculty must be prepared to address these changes to maintain a healthy learning environment, especially when they are familiar with more traditional classroom dynamics.

Swayze and Jakeman (2014) further explore the deep relationships, communication, and connectedness fostered through a cohort model. In their qualitative study the authors examined student perceptions of their cohort experience through a social constructivist framework using saturation theory as a guide (Swayze & Jakeman, 2014). They found students in a cohort reported an increase in all types of communication as well as greater depth of communication (Swayze & Jakeman, 2014). The students also felt the learning environment facilitated a “strong academic and personal support network…among members” (Swayze & Jakeman, 2014, p.105). Students also emphasized shared leadership as an underlying tenant of a successful cohort group, emphasizing aspects of a team with a shared goal centered around learning and mutual respect (Swayze & Jakeman, 2014). This team dynamic is an important aspect of what makes a cohort model powerful for programmatic learning.

Connectedness
Connectedness is defined as a “feeling of relatedness and identification of differences” (Laux, et al., 2016). A sense of community and connection with the organization are key factors in improving retention of learning program participants (Laux, et al., 2016). Research also links perceived learner connectedness and overall academic success, specifically in virtual learning environments (Jamison & Bollinger, 2020). Lee and Robbins (1998) assert that those with higher perceived levels of connectedness to the learning group are better able to “manage their own needs and emotions through cognitive processes.” This creates individuals that possess higher levels of trust with others which enables them to take advantage of social opportunities that in turn, continue to increase their sense of connectedness (Lee & Robbins, 1998).

Sense of community is central to creating feelings of connection within an academic cohort (Jamison & Bollinger, 2020). In a virtual setting, extra effort must be expended to ensure time for learners to participate in activities that encourage a sense of community and connection (Jamison & Bollinger, 2020). Research suggests that opportunities for collaboration and interaction with cohort peers and instructors over the course of the learning session are considered valuable element of perceived student connectedness (Yu & Zadorozhnyy, 2023).
To effectively meet these needs, opportunities for small group and partner discussion and collaboration are encouraged across all Organizational Health and Development sponsored development programs. Instructors are also carefully chosen for their use of interactive and engaging teaching methods. Social time is provided between sessions, and participants are encouraged to attend lunches and dinners together during in-person sessions.

Connectedness and Belonging
Connectedness is a key component of belonging (Laux, et al., 2016) which is an organizational construct we strive to provide for all employees in AFRL. When employees feel as if they belong, organizations experience multiple benefits (Carr, et al., 2019). Specifically, research by Carr et al. (2019) linked belonging with a “56% increase in job performance, a 50% drop in turnover risk, and a 75% reduction in sick days.” The financial benefits of these figures are staggering at more than $52 million in savings for a 10,000-person organization (Carr, et al., 2019).

Conversely, exclusionary behaviors take a significant toll on both the organization and the individual. Research by Eisenberger, et al. (2003) showed that social exclusion or pain and physical pain induce similar responses in the brain. In short, social rejection does, in fact, hurt. From an organizational perspective, additional research indicates exclusionary behaviors also reduce motivation to work as a team (Carr, et al., 2019). As we pursue the endeavor of an employer of choice, it is critical that we create educational experiences for our supervisors that encourage connectedness and induce a sense of belonging to both the cohort and the organization.

This study will add to the body of literature related to both cohorts and in-person versus hybrid delivery models of education. Intended/potential use of study findings: As AFRL pursues the endeavor of an employer of choice, it is critical that we create educational experiences for our supervisors that encourage connectedness and induce a sense of belonging to both the cohort and the organization. Results of the study will be used to inform the design of future Organizational Health and Development programming. Results can also be generalized to other areas of training and development across the 711 HPW, AFRL, AFMC communities.

Study Design and Methodology:

a. Overview: Mixed-methods, within-subjects design will be used through preparticipation/post-participation administration of the Rovai Classroom Community scale and post-participation interviews conducted by a research assistant not affiliated with any of the three participating programs. These methods map to our research questions by garnering the change in participant perspectives before and after completing the described programs using the three different delivery models.

• This is a recognized study design and methodology used in the field of social psychology.
• This is not a multicenter study.

b. Study population: The study population consists of government civilian and military members that are actively participating in one of the following programs: AFRL Advanced Development Program for Supervisors (ADPS), ARFL Supervisor Acculturation Program (ASAP), and 711 HPW Next Level Leaders Program (711 HPW NLLP). Participants will voluntarily self-select to participate in the surveys by scanning a presented QR code. Study PI and/or Co-PI will recruit participants by presenting information regarding the study and QR code for potential participants to scan on the morning of their first programmatic session. The same process will be used to solicit participation in the post program survey on the afternoon of their last programmatic session. Volunteers for one-on-one interviews will be solicited via email. Individuals will self-select to participate in interview sessions via Teams.

• We plan to make inferences about the results of our study to learning environments throughout the Air Force and DOD construct.
• Those not enrolled in one of the above programs will be ineligible to participate.
• Potential participants will be given the option to scan a QR code and opt-in to the pre/post implementation survey and interview portion of the study.
• Within subjects sampling will be used for the pre/post survey.

c. Variables/Interventions:

• The validated Rovai Classroom Community scale will be used to survey participants on their programmatic experiences (Rovai, 2002). No identifying data will be collected.
• Survey will be implemented the first day of class for each program and the last day of class for each program.
• Post-participation interview consisting of the Rovai Classroom Community Scale questions with prompts to elaborate will be conducted via Teams by a research assistant. The interview protocol has not been validated but uses questions from a validated scale with opportunity to further elaborate on why individuals chose specific responses

Statistical Analysis Plan/Sample Size Justifications:

a. Overview: Outline statistical analysis plan and test procedures. This will include describing the calculation of relevant quantitative measures for tests and instruments, such as sensitivity and specificity. Describe the statistical tools that will be employed for the study.

• We will use descriptive statistics including median and mode to draw comparisons between our three groups of participants.
• Additionally, two-way ANOVA will be used to make comparisons across demographic groups such as military and civilian participants and genders and specific program delivery models.
• Wilcoxen Signed Rank Test will be used to determine differences in the median of the three delivery model groups before and after the completion of the course.
• Thematic analysis of interview transcripts will also be used.
• Researchers as instrument is the primary bias of the study. Both investigators have a vested interest in programs and their outcomes. We also have our own prior beliefs related to the effectiveness of cohort models and online learning.
• Thematic analysis of the qualitative data will be conducted by the third-party research assistant. Results from this analysis will be used to garner additional information related to the specific constructs of the instructional delivery models that impacted participant learning and connectedness to the cohort and organization.