The Big Ideas Fest was envisioned as a starting point for a movement that supports innovation in education. The goal of the inaugural Fest, held in Half Moon Bay, CA from December 6-9, 2009, was to bring together education innovators and creative doers and thinkers in order to spotlight, challenge and change the ways in which education could be made more relevant to learning. Keynotes and rapid-fire presentations of innovation in practice were integrated into a unique experience, in which participants worked together in Action Collabs with facilitators to model a design process that included identifying challenges and opportunities, prototyping examples, and then presenting solutions that might scale.

In designing the Fest, ISKME drew upon prior experience in assessing, developing and implementing new models for knowledge sharing and processes to support collaborative community engagement and social learning. Rather than steering the educational community toward any single topical goal or focus on any particular segment or system within K-20 education, the Big Ideas Fest was geared toward establishing an initial platform of awareness and understanding about educational design and innovation. Transcending conventional silos, this platform invited participants to collaborate with the broadest possible diversity of perspectives comprising a mix of strong biases and complex interdependencies involved in K-20 education.

The three-day immersion format supported multiple learning styles and mediums while guiding participants as they engaged in a process of applying the Action Collab model for knowledge collaboration. The small, facilitated Action Collabs focused on outcomes and action, incorporating brainstorming activities, improv and new ways of thinking about familiar challenges. The labs were structured around four components, including identifying opportunities for change, designing possible solutions, testing and prototyping creations, and presenting ideas back to the group at large.

An important part of ISKME‘s work for the Big Ideas Fest involved capturing data from participants before, during and after the Fest, both to inform design components of the Fest and to assess outcomes and participant learnings during and after the Fest. Specifically, ISKME‘s internal research team evaluated the event with the goals of informing the design of next year‘s Fest and assessing the learning impact of the Fest on individual participants, both personally and also as a mechanism to support change in local contexts, with an eye toward identifying new opportunities for education projects that could better meet current challenges.

This memo presents an overview of the evaluation findings. The first section presents the methodology used to collect and analyze data. The subsequent section presents findings regarding participants‘ background, overall Fest experiences, and specific components of the Fest. The final section draws implications from the findings and presents plans for Big Ideas Fest 2010.

Evaluation Methodology

ISKME‘s research team used a variety of methods to collect data in order to comprehensively evaluate the Fest. Data collection methods included participant pre and post surveys, a facilitator survey (for the Action Collab facilitators), participant observation of Action Collabs and facilitator meetings, an analysis of Action Collab artifact outputs, and questions posed directly to participants during the Fest through an interactive art installation called ―EduTrees.

Surveys

ISKME administered a survey before the Fest to 163 registered participants (excluding ISKME staff, facilitators and performers). The survey aimed to assess participants‘ backgrounds and related educational interests, and to facilitate the division of participants into groups for the Action Collabs. Of the 137 participants who completed the survey, approximately one-half completed it online through SurveyMonkey, and the remaining one-half completed the survey during registration upon arrival at the Fest. The pre-Fest survey elicited information on participants‘ current roles, which Action Collab group focus they were interested in (designing for the student, institution, or enterprise) at the Fest, the changes they deemed most important for education today, and the roadblocks currently preventing those changes.

A post survey was administered after the Fest to gather participants‘ overall perceptions of the Fest and self-reported learnings. The survey was disseminated through email using SurveyMonkey and resulted in 78 participants (60 percent of participants). The survey included questions to assess participant take-aways and learnings from the Fest, and recommendations for improving the Fest going forward. All questions were open-ended with the exception of a final question inquiring whether participants would like to attend the Fest the following year.

After the Fest, ISKME also administered a survey to all eight of the Action Collab facilitators through SurveyMonkey. Seven of the eight facilitators completed the survey. The survey sought to assess facilitators‘ perspectives on the content, structure, and supports for the Fest, including their recommendations on the Action Collab environment, the resources provided, the design of the labs, group dynamics, and their training. Participant observation.

ISKME researchers were participant observers of the Action Collabs during the Fest. Three ISKME researchers participated in three different Action Collabs representing one of each of the Action Collab areas of focus (designing for the student, institutional, or enterprise) using an observation protocol. The protocol sought to collect information specifically on the Action Collab process and facilitation, participant perceptions about benefits and challenges to the Action Collab methodology, and indications of how they would put their learnings into practice. ISKME researchers also observed facilitator meetings (four total) before and during the Fest to assess facilitators‘ concerns going into their Action Collab work, as well as learnings, successes and challenges in working with participants and moving them toward the development of innovations and ideas.

Action Collab artifact analysis

After the Fest, ISKME researchers assessed all artifacts produced by each of the eight Action Collab groups. These included flip charts, post it notes, models and prototypes. The aim of the analysis was to assess the varied tools and techniques used in the Action Collabs, the outcomes/prototypes produced, and, by triangulating the artifact data with the facilitator survey data and post-Fest survey data, gain further insight into any variations in the Action Collab process and context by group.

EduTrees analysis

Data were also captured from the EduTrees interactive art installation at the Fest (see image to the right), in which participants wrote their thoughts about education on leaves and attached them to four different tree sculptures. Participants were prompted through four questions (one question per tree):

1. What is your important wish for education?
2. What does ―making education relevant mean to you?
3. What is vital to learning?
4. What sparks of inspiration did you encounter at the Fest?

In total, 99 responses were collected from all four of the EduTrees combined. By examining the responses as a whole and independently for each tree/question, the analysis was intended to assess participants‘ perspectives on central educational issues and emergent ideas during the Fest.

Findings

Going into the Fest: Participants and their perspectives on educational challenges

Diversity of participants

The Fest engaged stakeholders across multiple communities. As shown in Table 1 below, 24 percent (35) of Fest participants worked at for-profit organizations, 23 percent (34) at non-profit organizations, and another 23 percent (34) worked for K-12 schools and districts. The remaining participants worked for higher education institutions, foundations, government, and media.

Table 1. Participants' affiliated organizations (N=145)

Furthermore, the pre-Fest survey revealed that 93 percent (114) of those who responded to a question about their country of residence lived in the United States, with the majority (57 percent, n=70) residing in California. The Fest drew participants from five additional countries, including Canada, South Africa, the United Kingdom, Finland and Mexico. Organization type Percent (n)For-profit organizations 24% (35), Non-profit organizations 23% (34) K-12 schools and districts 23% (34) Higher education institutions 14% (21) Foundations 6% (8) Government 6% (8) Media 3% (5) 4

A focus on student-centered education as a necessary change

The pre-Fest survey analysis showed that, when asked what change they think needs to happen in education today, 49 percent (51) of participants suggested changes placing students and their learning at the center of education. As shown in Table 2, other changes mentioned by participants included enhanced use or support for technology, data use and digital content (20 percent, n=21) and overall transformation in education (19 percent, n=20). Table 2. Changes participants think need to happen today (N=104)

Recognition of existing culture and policy as central roadblocks to desired changes

Typical roadblocks perceived as preventing the changes described above were reported in the pre-Fest survey as ranging from change-resistant culture and mindsets, to policy, organizational, and budgetary constraints. Table 3 below provides a categorical list of roadblocks identified by participants as obstacles to desirable educational change. The primary roadblock, reported by 50 percent (38) of participants to this survey item, was mindset or a culture of resistance to change. Other top responses related to unaligned policies and politics (37 percent, n=28), and the current organization and structure of education (30 percent, n=23).

Table 3. Roadblocks to educational change identified by participants (N=76)

In sum, participants coming into the Fest represented a diversity of perspectives, and an interest in educational changes that move beyond current mindsets and policies toward enhanced student learning. As discussed below, participants coming out of the Fest reported takeaways that underscored these initial interests—with evidence of cross-silo sharing, idea generation and personal transformation among participants.

Participant Takeaways and Learnings from the Fest

The post-Fest survey, which elicited participants‘ takeaways and learnings, revealed specific anticipated next steps for making change—from increased networking and knowledge-sharing, to personal and professional changes and increased focus on active implementation of ideas. The Fest as a mechanism for networking and sharing. The post-Fest survey revealed networking and sharing as key takeaways from the Fest. Table 4, below, provides a summary of participants‘ anticipated next steps, listed according to percentages.

Table 4. Next steps participants plan to take after the Fest (N=69)

As indicated in Table 4, continuing established connections emerged foremost as an anticipated next step, with 39 percent (27) of participants reporting plans to connect with other participants or ISKME staff. Several survey participants, for example, reported appreciation for the diversity of perspective and open mindedness of fellow participants, and for the opportunity to interact more deeply than at a typical education-related conference. One post-Fest survey participant reportedly valued ―hearing diverse viewpoints,‖ and another ―enjoyed the variety of people in our lab and their different perspectives and viewpoints. Others noted the importance of contacts with whom they intend to follow up in relation to common work.

Sharing ideas, knowledge, message and process, also emerged as a next step, with 23 percent (16) of post-Fest survey participants indicating plans to share out learnings gleaned from the Fest.

The Fest as a mechanism for ideas and change

The analysis revealed that for participants, the Fest resulted in plans to develop and/or implement new ideas and innovations. Twenty-two percent (15) of post-Fest survey participants reported plans to make a change as a result of the Fest (see Table 4 above), including moving forward with a big idea and redirecting a job search. One participant wrote,"I‘ve been rolling the idea in my head of opening my own school, and I think this was the push I needed to make that a reality."Furthermore, when post-Fest survey participants were asked to create a title to summarize their experience at the Fest, 22 percent (16) reported titles related to change, reform, or exploration into new educational territories. Examples of these titles reported on the survey include ―Big ideas for education without boundaries‖ and ―Big ideas, broadly shared. Other titles included ―Inspiring action in changing education and ―Education revolution...now is the time!

Finally, as reported through pre-Fest surveys, the participants‘ shared focus on student-centered learning as a necessary change for education also extended through to their Action Collab work. The Action Collab artifact analysis revealed that the prototypes designed did not vary depending on the level of the Action Collab group (student, enterprise, institution). Each of the Action Collab prototypes developed during the Fest were found to have focused on individualized learning and/or models for increasing student engagement. Action Collab prototypes included, for example, a ―Learning Navigator‖ to give visibility to students‘ learning needs, opportunities, and progress, and to enable personalized learning paths, as well as an ―Urban mall‖ for educational resources and delivery.

The Fest as a mechanism for personal transformation

The post-Fest survey also revealed that 21 percent (14) of participants experienced a personal transformation including improved listening, receptivity and mindset. One participant indicated learning ―the importance of conquering fear, or the silliness of fear [and] the value of flexibility.‖ Another participant wrote,"I REALLY appreciated the notion of ‗don‘t rush to find a solution.‘ Important learning for me… and one that I want to apply more consistently to my current practice!"

This finding was underscored in data gathered through the facilitator observations. Specifically, in the course of meetings during and after the Fest, several facilitators remarked that individual participants seemed to ―transform‖ over the course of the labs. One participant, for example, who had reportedly started the Action Collabs with comments displaying discomfort with improv and the Action Collab process overall, had, by then end of the last Collab, earned a ―took biggest risk award.

Participant Perceptions of the Fest Model

The analysis revealed that the Fest model as a whole was met with an overwhelmingly positive response, as revealed through the fact that 96 percent (67) of post-Fest survey participants indicated they would like to attend next year‘s Big Ideas Fest. The analysis further provided an understanding of what worked best, including pointed suggestions for improving Fest components. Specifically, the analysis underscored successes and challenges among the following Fest components: Rapid Fire and Keynote presentations, Action Collabs, integrated arts and entertainment, and logistical matters such as scheduling, location and setting.

Action Collabs

Analysis of the artifacts produced by the Action Collabs revealed that, though groups produced different volumes of artifacts during each step of the labs, all groups used similar methods for idea development. Each group‘s materials indicated the use of improvisation exercises based on the tenants of theatrical improvisation; the use of post-its and flip charts to weave together individual, small group, and large group brainstorms; prototyping through role play, diagrams, and physical creations such as board games and puppets; voting on ideas to carry through to the next stage of thinking; and the creation of concept maps and story boards to synthesize ideas before moving into planning for presentation of prototypes.

When asked what should be kept for next year‘s Fest on an open-ended survey question, 18 percent (13) of post-Fest survey participants specifically mentioned Action Collabs. Additionally, when asked about Action Collab takeaways, 38 percent (25) of post-Fest survey participants included some aspect of the process, with improvisation and the Action Collab process listed most frequently. Similarly, 32 percent(21) of participants discussed process when asked what worked well for them in the Action Collabs. One participant, for example, noted a plan to ―take some of the techniques I learned to brainstorm curriculum, work with my instructors, etc. One participant commented,"I loved the way the Action Collabs broke down the process into four steps. I think it makes it manageable, doable, and accessible to all those involved."

Another participant responded,"[I like] the culture that was created by this structure: egalitarian, spontaneous, team-oriented, and process-focused." This positive assessment of the culture fostered by the Action Collabs was confirmed by 17 percent (11) of participants. Finally, 23 percent (15) of participants mentioned facilitation when asked what worked well in the Action Collabs. For example, one participant appreciated ―having a skilled facilitator there to guide us.

When asked what should be changed for next year‘s Fest, 47 percent (34) of participants suggested changes to the Action Collabs. Twenty-four percent (16) of post-Fest survey participants further reported the need for more focus in the Action Collabs. One participant, for example, indicated that ―at times they [the Action Collabs] felt directionless.‖ Additionally, changes to the final Action Collab group presentations were recommended by 13 percent (9) of post-Fest survey participants. On the facilitator survey, two facilitators likewise indicated that their group presentations did not accurately reflect their work.

Personal challenges emerged as an important theme in the responses to the question ―What was a challenge for you in the Action Collabs‖: 22 percent (15) of participants indicated a personal challenge, including resisting the urge to control (6 percent, n=4) and having trouble being heard or feeling too shy to contribute (4 percent, n=3). One participant wrote, "The vulnerability was a challenge, but it was also a strength and provided the opportunity for growth." Across all post-Fest survey questions, 11 participants discussed feeling rushed or needing more time in the labs. However, three felt the time allotted was adequate, and four commented that the labs were too long.

An analysis of the facilitator meetings and facilitator survey uncovered several additional recommendations. During a post-Fest facilitator debrief meeting, several facilitators agreed that integrating newcomers to the labs was challenging, and all participants to the facilitator survey recommended that only participants committed to attending all labs should participate. Facilitators were generally satisfied with the training they received but noted they would have preferred more training.

Keynotes and Rapid Fires

The evaluation revealed overwhelmingly positive feedback about the keynote and rapid fire speakers. One-half (8) of the leaves on the EduTree asking about sparks of inspiration at the Fest referred to a specific speaker, and 72 percent (53) of post-Fest survey participants listed speakers or rapid fires when asked what should be kept for next year‘s Fest. One post-Fest survey participant commented,"The talks really surprised me – they were nearly all excellent and refreshing in their content and delivery."Eighteen percent (13) of post-Fest survey participants suggested changes to the speakers such as reducing the number of speakers and having them include more visuals in their presentations.

Scheduling, location and setting

The analysis of post-Fest survey data revealed an interest on behalf of participants in changing the Fest program or scheduling. Specifically, 25 percent (18) of post survey participants requested changes to the schedule. Ten percent (7) desired more time to network with other participants, and eight percent (6) simply requested more free time. Seven percent (5) of survey participants suggested not having speakers during meals next year. Five percent (4) of participants requested mechanisms for talking with people outside of their Action Collabs.

Finally, the data revealed that a few participants had mixed perceptions about the location and setting. Five percent (4) of survey participants discussed the setting or location when asked what should be kept next year, while 10 percent (7) expressed concern about the remoteness of the location when asked what should be changed for next year‘s Fest.

Conclusions and Implications

The 2009 Big Ideas Fest fostered cross-silo immersive participation in an empowering process of designing for change, while affording opportunities for networking, knowledge sharing and transformation. On the whole, participants‘ nearly universal interest in attending next year‘s Fest suggests a success for the Fest model in its first iteration. Furthermore, the fact that nearly 40 percent of participants plan to continue the connections made at the Fest, and that almost 25 percent plan to share out new ideas or make a change prompted by their participation in the Fest provides early evidence that the Fest‘s theory of action is one worth continuing.

Next year‘s Fest will retain and refine the qualities and components that made 2009 successful, with consideration to specific changes necessary to move the Fest model forward even further. These include:

• Refinement of the Action Collab Model – The Action Collab groups will be organized around pre-determined specific design challenges related to stakeholder needs. This preliminary self-organizational step at Fest registration will replace the selection of groups into categories of student, individual or enterprise—shifting to a slightly more directive basis for group assignation. Furthermore, the Action Collabs will be expanded to include a more explicit research component in the first segment of the process. The research component may include, for example, access to video interviews with experts or students, or access to the Internet or specific educational databases. The research component will be incorporated as a way to support the Action Collabs in the identification of opportunities early on in the solution-building process.
• Enhancement of keynotes and Rapid Fires – Next year‘s Fest will likewise engage dynamic speakers in keynotes and Rapid Fires, but will more closely facilitate thematic connections, specifically between the Rapid Fires and individual steps of the Action Collab process. In light of the 2009 evaluation findings that sharing and networking were central takeaways for participants and that participants would like more time for these activities, next year‘s Fest will feature fewer speakers to allot for this time.
• Further supporting scale and spread – Preliminary to the Fest, an online space could draw on tools to share-out their collaborative problem-solving activities and conversations related to selected design challenges. Ideally, this space would available during the Fest, as well as after it to support participants‘ efforts to build relationships and move conversations forward into action. The Fest model overall will enable scalability and support for moving the Fest into local contexts. In short, to help enrich participants‘ reflection on their post-Fest experience, next year‘s Fest will focus more explicitly on takeaways and learnings related to the Action Collab framework as a method transferable and applicable to a wide array of problem-solving contexts.
• Continued location in Half Moon Bay – The Ritz Carlton in Half Moon Bay will continue to serve as the location for next year‘s Fest due to its relative ―resort‖ atmosphere that is aligned to the Fest‘s goals to inspire creativity, interaction, and hard work with few other distractions. As such, the further promotion of the Fest as a three-day immersion will be highlighted, as opposed to an event that participants attend for only a drop-in day. Through the refinements suggested above, the 2010 Big Ideas Fest seeks to further build on a platform that helps to not only spark new ideas to address intractable education challenges, but to support the design, doing, and scaling of those ideas – at the personal, institutional and process level.

BIF 2009 Evaluation PDF