Starting a new arts organization brings a jumble of challenges. Leaders of startups find themselves dealing with a crazy-quilt of issues every day, from details like the nutritional content of snacks, to advocacy presentations for parents, and data collection for funders—all in the same afternoon. Amid that swirl, it is difficult to focus on things consultants like to talk about, like: “conceptual models that organize the various tasks into categories that respond best to different kinds of treatment.” It sounds almost like a joke to urge that kind of abstract thinking when so many practical problems are clamoring. As my grandmother used to say, “When you are up to your armpits in alligators, it is hard to remember your original intent was to drain the swamp.” Looking at lengthy to-do lists across the world, I hear sighs from program leaders who know they should find more time to reflect strategically on the swirl of tasks, but they just can’t get the focus on this amid the more pressing demands.
For decades, I have worked as a consultant and program designer with non-Sistema arts organizations who undertake innovative projects that seek to do things that few or none have done before. This resonates, of course, with El Sistema-inspired (ES-i) programs that seek to create music learning environments that differ from traditional norms, that aspire outrageously high: intending to disrupt entrenched cycles of poverty, creating ensembles with unusual ambition and intensity, forging intimate connections to communities, building toward public financial support for expensive music programs where few have accomplished such funding before. ES-i work builds on deep traditions, but its aspirations reach beyond the known into pioneering work that is beyond established precedent.
Recently, I have been working with a new consultant’s tool to help innovating arts organizations, and they have found it to be particularly helpful. This essay introduces that tool to El Sistema-inspired programs. Of course consultants prefer to work face to face with tools like this, but perhaps this written description can contribute even without that luxury of side by side planning.
This framework clarifies the kinds of challenges El Sistema programs encounter. The categorizing is not just an intellectual exercise; it divides the work into different kinds of problems that are best addressed in different ways. Currently we tend to lump problems together, and end up dealing with some of our most important problems in the wrong ways, which muddies or damages progress.
The tool is called the Cynefin Framework—(pronounced ku-nev-in) which derives from the Welsh word for the sense of the incomprehensible number of factors influencing the evolution of systems. It was proposed in 1999 by the Welsh scholar David J. Snowden and has been used in businesses, and recently in other fields as well.
I will first explore the conceptual framework itself, and then we will walk through its application to El Sistema-inspired programs. In a last section, I will suggest its implications for evaluation in our programs. If these ideas interest you, I strongly recommend that you read this article which introduces the Cynefin Framework in much more detail: “A Leader’s Framework for Decision Making,” by David J. Snowden and Mary E. Boone , which appeared in the Harvard Business Review, November 2007 issue. [David J Snowdon is the founder and chief scientific officer of Cognitive Edge (http://cognitive-edge.com/tour) an international research network, and he is based primarily in Lockeridge, England. Mary E. Boone (www.maryboone.com) is the president of Boone Associates, a consulting firm in Essex, Connecticut, and the author of numerous books and articles, including Managing Interactively (McGraw-Hill, 2001).] Also, my thanks to Jamie Gamble (Imprint, Inc, a consulting group he leads: http://www.imprintinc.ca/Jamie.html) who helped me understand its use, enabling me to begin to apply its value to El Sistema work, and to Richard Evans of EMCarts (http://emcarts.org/programs/innovation-labs) who thinks with great clarity about these issues. Some of the examples below come from them.
Imagine a box divided into four quadrants, a graphic distillation of the Cynefin Framework—if I were smarter with graphic layout I would have placed it here. It has four quadrants, and each quadrant represents a different kind of problem for an organization. Each box has a label. Bottom right: Simple. Bottom left: Chaos. Top right: Complicated. Top left: Complex. This essay focuses particularly on the distinctions between those top two quadrants, discriminating between Complicated and Complex challenges. Let's define all four:
Simple. Simple problems, like baking cookies, have clearly defined goals, have established “recipes” to follow, and have easy ways to determine successful outcomes. Cause and effect work reliably, and best practices should be used. Because the nature of a challenge makes its category “Simple,” that doesn’t mean it is quick or easy to accomplish. An example of a simple problem in an El Sistema-inspired program might be acquiring instruments.
Complicated. Complicated problems, like getting a rocket to land on the moon, are basically simple problems ratcheted up many times. Success vs. failure is clear, expertise and tested models are essential, cause and effect relationships are available for analysis (although less immediately related than in Simple problems), factors influencing the work tend to be stable and predictable, and established good practices should be used, tested, and should produce reliable results. An example of a complicated problem in an ES-i program might be planning a seminario or developing a strong Board of Directors.
Complex. Complex problems, like raising a child, do not have clearly defined goals or definitions of success; they deal with many conflicting, interdependent factors that influence the process, and that change regularly; cause and effect are not clearly apparent and change; and models of prior success do not reliably apply. Expertise is essential, but it must flexibly adapt to bring improvement through careful observation of emergent practices. An example of a complex problem in an ES-i program might be developing public funding of the program.
Chaotic. Well, we know what that feels like. The chaos when, for example, a key element in a program suddenly disappears. Cause and effect no longer apply, and the need is to establish stability again. There are opportunities embedded in the situation, but stability is the main need. An example of a chaos problem in an ES-i program might be the unexpected exit of a major funder.
These distinctions can prove valuable because the most effective approaches to one kind of problem don’t work well with the others—the recipe strategies that work for baking cookies will fail in the Complex demands of uncharted territory like childrearing. The crucial distinction for our programs is between Complicated and Complex problems, which we tend to blur, treating both like Complicated problems. This is natural enough—Complex challenges are more uncertain, harder to document and measure, more “uncomfortable” to both the people and the support institutions involved. So we push Complex problems to the right, and address them as if they were Complicated problems. Therein lies a serious problem that can be alleviated by discriminating leadership.
In my experience of El Sistema-inspired programs, we do more than blur that crucial Complex/Complicated distinction, we “lean to the right”—we tend to pull Complex problems to the right, treating them as if they were Complicated problems. The reasons why we do this are the subject for a different essay, but let me just say that many of the pressures of startups—from funders’ demands, to Board member expectations (since many come from careers that excel in Complicated problem solving), to the time pressures that scrunch any focus on ambiguous and uncertain Complexity issues—all contribute.
Using the wrong tool for a challenge has consequences. Before we look into ES-i programs, let’s think about the cost of leaning to the right in other arenas. The way one succeeds in rocket science is not the way one succeeds in parenting—the classic Complex challenge. If you follow Complicated challenge strategies like the advice in a “how to” book on parenting, the results will be mixed at best, more likely damaging for the child. And even if the strategies happen to miraculously work fairly well with a first child, they are not going to apply as well to a second child. Models and established best practices are solid ground in Complicated problem solving—that’s how you build a rocket that gets to the moon. Certainly models and best practices should be studied for application to Complex problems—it is helpful to read many parenting books to inform your decisions—but you can’t rigidly impose a model and expect success, as you can in engineering. You can’t guide childrearing by precise measures; you can’t make decisions based on quantitative data alone; you can’t be entirely objective and impersonal to make good choices. If you drag parenting “to the right,” dealing with this Complex work in ways of Complicated problem solving, you damage the child, limit the child’s potential (or at least limit your relationship with the child), and in many cases, you make a mess.
How do you get better in the Complex challenge of parenting? The process is highly relational with close communication between parent, child and others involved. The process is made of ongoing experiments with quick feedback loops that help guide the parenting choices. There are multiple angles of input. And when some parts develop reliable patterns (let’s say household chore routines), they become Complicated issues that respond well to certain tools of evaluation, like a checklist on the refrigerator. These tools work with Complex challenges: intentional experimentation, quick feedback loops from those experiments to guide ongoing development; vigilant observation to identify emerging ideas, questions, understandings; trustworthy personal partnerships; and attention to patterns.
THE COST OF LEANING TO THE RIGHT
To further demonstrate the cost of “leaning to the right,” let’s play this out in an imagined arts presenting organization. Getting out a brochure with a season’s offerings is a Simple problem (although those who have done this know it isn’t so simple). Developing a marketing plan for the season is a Complicated problem. Creating new audiences who become committed to your organization is a Complex problem. And having your Executive Director leave suddenly at a crucial time invokes Chaos.
Here is what happens when that organization mistakenly treats that Complex problem as if it were a Complicated problem. The organization faces the longterm Complex challenge of evolving new, mutually valuable relationships to potential audiences in their neighboring communities. When they pull to the right, they turn that profound survival challenge into a marketing problem, a Complicated challenge. Sound familiar? As if better advertising, smarter public relations, or a spiffier website can awaken the discovery of the relevance of those art offerings for those who feel no relationship to the organization. Failure, or at least small success is guaranteed—yet this is one of the most entrenched mistakes made in the arts. We even know those strategies won’t work—ask the Executive Director if cleverer marketing is going to activate engagement by the communities in the area that don’t care about her organization, and she will say she knows that doesn’t work. And yet, the discomfort and difficulty of dealing with Complex challenges, pulls the problem to the right into marketing and dooms it to failure. This mistake alone largely explains why arts organizations struggle so hard to turn community engagement experiments into new audiences—a conceptual framework error that leads to the wrong experiments that deplete resources and the potential of the discoveries. Whereas, if they could carve out Complex experiments, and organize around the resulting learning, they could begin to change the entrenched status quo of dwindling audience.
In recent years, major grants have been given to incubate innovation, particularly by EMCarts [http://emcarts.org/index.cfm?PAGEPATH=Programs_Services/Innovation_Lab&ID=20278]—explicitly to support arts organizations dealing with Complex problems with the tools and processes that work best for them. My work in the Community Engagement Lab [http://celvt.org] focuses on exactly this challenge—addressing the Complex challenges of developing new audiences.
EL SISTEMA-INSPIRED PROGRAMS
Let’s turn to the work in El Sistema inspired programs. Here are some examples of common challenges our programs face, and the categories I propose they should be framed within. Bear in mind that categorizing a challenge as Simple doesn’t mean it is easy; it means that it adheres to the qualities of that kind of problem as described above.
Simple
Developing a volunteer schedule to provide snacks
Acquiring instruments
Starting a chorus within a program
Creating musical arrangements
Complicated
Instituting good classroom management
Raising funds
Developing a Board of Directors
Designing a seminario
Complex
Nurturing community “ownership” of program
Creating public funding streams
Developing intrinsic motivation in learners
Creating “citizenship” in young people
Making peer-to-peer learning an organic habit of mind
Chaos
Hitting a funding crisis
Grappling with community divisiveness
Healing a schism on the Board
Coping with a sudden leadership change
Let’s clarify the way these different quadrants appear in the real-world work of El Sistema-inspired programs. I will lay out a few examples that apply to different parts of our work and how the Cynefin framework sorts out different kinds of challenges that are addressed best in different kinds of ways. Note that each area includes, or could include, all four kinds of challenge. It isn’t that, say, classroom management is a Complicated challenge and community engagement is a Complex challenge; our significant endeavors work on multiple levels at the same time. This essay urges that we distinguish which challenges are inherently Complex challenges, and that we engage with them in the ways they require.
The Challenges of Community Engagement
Simple: Creating a basic calendar of program events that align with/connect to existing community events. Creating and deploying a volunteer committee of families and community members.
Example of processes to address the challenge: Communicate with local and partner organizations about scheduling and possible synergies. Hold group and individual meetings to recruit volunteers, write up agreements and clarify roles and responsibilities.
Chaotic: Responding to an important community leader who makes a strong public objection to a crucial part of the program.
Example of processes to address the challenge: Find effective methods to address the speaker and those who are influenced by that person. Assure program family. Address the issue directly.
Complicated: Building strong and sustainable partnerships with stakeholders who feel a resonance with the program. Nurturing greater involvement and “ownership” among families and volunteers.
Example of processes to address the challenge: Apply known best practices in partnership-building from community arts and community development fields. Nurture family and volunteer cohorts to learn and lead together. Establish goals and benchmarks to make progress visible.
Complex: Exploring partnership-building with public agencies, organizations and leaders who haven’t worked with arts groups. Build a sense of “ownership” in the public sector, with an eye toward sustainable funding from public monies.
Example of processes to address the challenge: Catch interest and engagement of public officials and atypical arts-partnering organizations to invest in deep dialogue about shared aspirations. Move toward “collective impact” projects, with shared goals. Draw them into experiments to discover mutually-beneficial larger projects.
The Challenge of Classroom Management
Simple: Establish learning-conducive atmosphere.
Example of processes to address the challenge: Establishing behavioral agreements, consistent classroom practices, a learner-friendly schedule.
Chaotic: Gaining control of a disruptive or unsafe atmosphere in which students cannot learn.
Example of processes to address the challenge: Stop the unworkable status quo; take time to start fresh procedural agreements, address particularly problematic individuals, offer new kinds of activities.
Complicated: Establishing working norms that are exciting and positive for the learners, and that minimize wasted time and energy lulls.
Example of processes to address the challenge: Faculty/staff regularly discuss and refine working processes, expanding on what they know from experience or study, including input from students and those who know the students well. Data about the working processes are gathered and studied; and if a priority in the program, are scrutinized through evaluation instruments.
Complex: Creating an atmosphere that nurtures the intrinsic-motivation of the individual learners and of the group, fostering wholehearted commitment. Nurturing an environment in which learners naturally and consistently help one another, to raise the success of the whole endeavor.
Example of processes to address the challenge: After studying what research offers regarding the development of intrinsic-motivation, and what experience reports about the development of peer-to-peer instructional habits, faculty and staff undertake experiments, observe impact, and regularly share with one another (and ideally with a developmental evaluator, or at least a developmental evaluation perspective in their discussions) what they learn from experimentation.
The Challenges of Funding
Simple: Getting adequate funds to meet budget, using sources that respond to early inquiries. Put clear budgeting processes in place.
Example of processes to address the challenge: Appeal to the most immediate sources of support available, including grants from likely organizations, and personal contacts with individuals who are interested. Adopt a transparent and user-friendly budgeting process.
Chaotic: Rebounding after a major funder leaves, or an essential partner changes a key relationship with financial ramifications.
Example of processes to address the challenge: All hands on deck to identify replacement support.
Complicated: Gain a solid financial footing for reliable, sustainable funding that allows for growth. Get strong personnel and financial practices in place.
Example of processes to address the challenge: Gain mastery of a set of primary fundraising strategies, such as special events; grant application processes; relationship-deepening with existing donors and cultivating new donors and new partnerships. Set strategic information-gathering processes, establish short- and longer-term development plans.
Complex: Building longterm-relationships with public funding agencies and channels; creating local collective impact endeavors that bring a variety of organizations together to address urgent local needs.
Example of processes to address the challenge: Commitment to deep partnership building over time. Identify shared goals, distribute leadership, build trust in collective action, institute some shared evaluation, escalate mutual reliance.
I find El Sistema-inspired program leaders to be impressively strong in addressing the Simple challenges—perhaps because there are so many of them, these muscles get super-fit. Even though some Simple challenges, like learning to play the violin, aren’t easy or quick to accomplish, they remain Simple because we have so much experience and expertise to bring to the process. The kitbag of best practices and tested models work reliably; cause and effect is evident. We know what to do, and evaluation is reliable and clear—we can readily tell if things are working or not.
Chaos visits most programs at some time or other. Some leaders are good in a crisis, while others flail—we all get stressed. Strategies for dealing with Chaos aren’t the focus of this essay, and we can leave this quadrant unaddressed with the reminder that the goal is to bring about stability, to re-establish patterns, and with great leadership, use the ferment as a way to initiate new patterns and experiments, to be attended to as Complicated and Complex patterns.
The distinction between Complicated and Complex problems is the one that matters. I alert us all to the tendency to pull Complex work to the right, into the Complicated quadrant. This tendency appears in many aspects of our work—in teaching, in program design, and perhaps most crucially in evaluation practices.
For example, look above at the challenge of Classroom Management that has aspects that fall into all four categories. If you apply only the known tools of good classroom management (a Complicated problem solving process) to the Complex challenge of nurturing intrinsic motivation in young musicians, you are likely to develop compliant rather than heart-hungry students. That distinction is critical because passionate motivation is exactly what it takes for students to achieve the goals of Sistema-inspired work: to have the courage, drive, and confidence to take on challenges that few in their family have ever tried before, to break the cycles of poverty. In the heat of the teaching action, we default to what we know, and our music education traditions have not prepared us to be intentional and effective in nurturing intrinsic motivation in our learners.
I have seen this particular pull to the right in action—teachers who come out of traditional music education “control” minded backgrounds (that are necessary in schooling that deals with Complicated challenges), may resist placing their highest priority on motivation-nurturance because it calls preferred practices into question, requires inquiry and experimentation to achieve different results and not just implementation of known models of control. They insist on dealing with behavior as a Complicated problem rather than the Complex one it is in ES-i work. Our shared highest goal of redirecting the trajectory of young lives requires passionate intrinsic motivation, and if we can’t wholeheartedly explore how to manage a room as a means to incubate the learner’s intrinsic drive to excel, without defaulting to tools of control, we cannot succeed in our central mission.
I don’t suggest that controlling the room is a bad idea! Rather, the potential of ES-i lives in the inquiry into building motivation without relying on unexamined traditional practices, however expedient and automatic, that are known to dampen intrinsic-motivation. The ongoing evolution of ES-i work lives in the Complex experimentation into creating hyper-effective learning environments, which of course means they must be focused and controlled, but they must be more. As the adage says: if you always do what you’ve always done, you’ll always get what you always got. And if teachers don’t know which of their practices are known to suppress intrinsic motivation, then there is research to be added to the Complex inquiry. [There is a large body of research on pedagogy and intrinsic motivation. I recommend starting with Daniel Pink’s readable book Drive.]
Similarly in fundraising, if you apply only good grant writing skills (developed in the Complicated practices of our field) to the Complex challenge of tapping government funding for the sustainability of a program, you’re not going to find the new kinds of relationships that must be carefully forged with public agencies that can lead to reliable financial support. Yes, Complicated thinking may get a modest grant from a city agency if it happens to offer a program that fits. But sustainable public funding requires unprecedented partnerships, personal relationships and trust building, and innovative experiments that find their way forward with many adjustments—this is unprecedented Complex exploratory work.
The pull to the right appears even in establishing work schedules. For Complicated challenges, you hire experts and let them perform. For Complex challenges, you hire experts and let people of different backgrounds mix and discuss and experiment as part of a learning team. These two approaches appear in the program budget and weekly schedule. Are the teachers paid to meet and think and plan together as a regular and essential part of their job (and to make sure the talk is substantive and not just about logistics)? Do hiring processes focus on flexibility and deep curiosity? If yes to both those questions, that program recognizes it is addressing Complex learning challenges. If the budget and schedule minimize meeting and discussion time, and meetings become mostly about logistics, then that program sees the creation of the learning environment as a Complicated challenge—just trust the experts, and you will get your results.
A pattern I have observed in the U.S. movement argues for the importance of this framing distinction: the programs that structurally include the most faculty planning time, including retreats and regular thinking that go beyond the practical and logistical, tend to be the programs that grow the fastest in terms of quality. The budget allocation that supports this investment in collective learning is really a commitment to treating Complex challenges in the ways that Complex challenges require.
For years I have proclaimed that El Sistema is neither a pedagogy nor a program; it is an inquiry. Another way of saying that is that El Sistema-ness lives in the upper left quadrant of the Cynefin Framework. The Complex challenges in our programs, and the ways we address them, provide the heart and guts of our true work. If El Sistema programs were just the same as traditional youth orchestra programs with added hours per week, then El Sistema programs would live well in the upper right quadrant, comprised only of Complicated problems. That’s why I flinch when I hear U.S. ES-i educators say that they already know how to do the work. They do, if we reduce the potential we are inspired by in Venezuela to known models and questions to which we have established answers. But the Venezuelan example raises deeply Complex challenges with extraordinary potential. It is that extraordinariness of the potential that drew us to this work, that has inspired its historically unprecedented proliferation in the U.S. and around the world, that requires that we manage the challenges of Complex learning without pulling to the right.
Indeed, our Venezuelan friends function superbly in the Complex quadrant, and have done so for many years. I think of Maestro Abreu as the master of the Complex quadrant. This is why the Venezuelans have learned their way to unprecedented, almost inconceivable, accomplishments. This is why they continue to improve and exceed their already-unprecedented successes. Maestro Abreu famously describes the nature of El Sistema as “ser no ser todavia”—existing and yet still becoming—which is the perfect poetic description of the mindset of the upper left Complex quadrant.
To be a part of the El Sistema movement means learning to live in, to love the feel of, the Complex quadrant of our work, while we also function extremely well in challenges appropriate to the Complicated and Simple realms. If we pull to the right, we undermine our potential.
We have more Complex challenge experimentation areas than just motivation, public sector partnerships, and work schedules. Few of us know in our bones how to master the Complex task of teaching groups to get fast, fun, focused results like the Venezuelans do. We don’t really know how to foster peer-to-peer instruction so that it becomes a natural habit of mind in young musicians. We also have a lot of Complex learning to do to create learning environments like those Venezuelan nucleos—so safe, joyful, loving and charged, that they provide a river current of learning commitment so strong that those who enter are swept into its fervent aspirations.
Complex challenges respond to intentional experimentation, with short-term feedback loops. Faculty members need regular thinktank time to share their sense of how things are going, to respond to others’ observations and questions, to feel their way forward, together. It may not sound like an impressive strategy, but it is what works. Probably the only thing that works.
The psychologist Abraham Maslow said, “If the only tool you have is a hammer, everything begins to resemble a nail.” If we cannot become effective innovators by working well with Complex tools, we apply the hammers of Complicated problem solving, and flatten our students’ potential by flattening the reach of our inquires. Just as rocket scientist protocols imposed on parenting damage the child’s life. Music educators who aren’t drawn to Complex learning may be good contributors to youth music education, but they are not true El Sistema leaders. Being a leader in the El Sistema-inspired movement means a commitment to, a pleasure in, learning in and through the Complex challenges.
EVALUATION IS ESSENTIAL
Evaluation lies at the heart of honoring Complex challenges as the innovative opportunties they are. The following sketch reminds us that ES-i programs face a variety of evaluation challenges. Let’s look at the basic evaluation tools we use for each kind of challenge.
The Challenges of Evaluation
Simple: Establish data collection and valid analysis processes that answer needs of funders and partners and that shed light on program impact. E.g.: How is our volunteer network growing? Are the kids improving in their violin playing?
Example of processes to address the challenge: You identify existing and easily-attainable relevant data and build a plan for its use. For those two examples: data collection practices on volunteer participation, consistent use of quality indicators for violin performance.
Chaotic: Program is too busy to address the evaluation, has too little expertise, or has no money to get someone to do any evaluation. Funder demands data and program has to produce it when not ready to do so.
Example of processes to address the challenge: Find a few already-available kinds of data, work with helpers who can do basic analytic work to create some evaluative findings. Ask help from funders or local experts. Begin a simple evaluation practice.
Complicated: Generate data that reliably and usefully informs the program and stakeholders about how its doing in three areas: musical learning, social development, other areas of particular interest like academic, character, and learning skill advancement. Make visible how the program is doing.
Example of processes to address the challenge: Develop evaluation tools and processes that illuminate essentials of the program, including aspects that lend themselves to evaluation and some of the more elusive-yet-essential aspects of impact. Establish longitudinal evaluation process.
Complex: Explore areas of growth that are particularly important to the program’s ambitious El Sistema-inspired goals; learning well as an organization; seek ways to evaluate how program practices can better deliver the results that change kids life-trajectories and communities.
Example of processes to address the challenge: Work with developmental evaluator and/or researcher to learn about what is really happening in students and others; experiment with different kinds of data collection to find those that are eloquent and practicable. Foster a culture of evaluation that fosters new observational and assessment thinking and habits in faculty, staff, learners and families.
We have good strong tools for determining success in Simple problems—you can count the number of instruments in the instrument drive, can use a variety of tools to demonstrate improvement on the violin, can calculate volunteer hours, etc. And for problems of Chaos, the task is beginning in a useful and sustainable way, and responding intelligently when a demand from the outside requires evaluation data.
In Complicated work, the primary tools for determining success are summative and formative evaluation. Summative evaluation judges the effectiveness of a program at the end of a project or period of time, with an external eye that is informed by reliable models, and in relation to stated goals. Formative evaluation judges the accomplishments of a program while the program activities are in progress, and focuses on the process, enabling program designers, learners, and instructors to monitor how well the instructional goals and objectives are being met (in relation to known models), enabling them to adjust along the way. Both methods develop and test models, both rely on data and objective input. However, they don’t work well with Complex problems, and to apply them to Complex problems pulls that problem to the right.
The evaluation process that best serves Complex challenges is the emerging field of developmental evaluation. It includes an evaluator in the process, who helps clarify the particular experiments underway and shapes the feedback loops for immediacy and effectiveness, without imposition of known models (but with reference to them when helpful). Developmental evaluation helps clarify the issues; it distinguishes and keeps returning to key questions that are being explored; it helps track discoveries as the action unfolds; it helps the practitioners understand the variables they are dealing with and recognize emerging patterns in the work.
Here’s a little more about “the feel” of developmental evaluation. Traditional evaluation takes an outsiders eye, provides judgment of impact and quality using pre-determined criteria. Developmental evaluators are integrated into the work; they focus on discovery and on the issues that leaders and stakeholders hold dear. They help create or adapt evaluation instruments and processes to bring understanding, to clarify patterns, and to refine experimentation. Summative evaluation tries to control, using reliable instruments; developmental stays close to what is happening, responding to what appears and devising tracking tools as opportunities appear. Traditional evaluation produces reports; developmental evaluation devises customized user-friendly feedback mechanisms. Formal evaluation seeks to produce valid models, best practices, and conclusions that can transfer reliably to other settings; developmental seeks to guide and maximize learning.
Formative and summative evaluation are the main tools used in ES-i programs, and they are invaluable. Not only ES-i program leaders, but our funders and stakeholders too, need to learn from our summative evaluation results. Developmental evaluation seeks to guide exploratory work, and, over time, surface patterns in the work that can evolve into models; these models can eventually respond well to formative and summative evaluation to produce reliable judgments. Developmental evaluation helps us understand our innovations so that, over time, we can move parts to the right, from Complex to Complicated stages, and apply summative evaluation effectively.
Few El Sistema programs can afford to hire an additional evaluator to guide their Complex learning. But there are two things program leaders can do without that designated expert; they can, in essence, develop practices that serve some of the necessary developmental evaluation functions. We can do a lot of this ourselves, and the importance is so great that programs have to honor their Complex mission in this way. “We’re too busy,” or “we can’t afford” another evaluator are just not good enough reasons to acquiesce to pulling Complex challenges to the right.
1) You can identify the Complex problems that are most crucial to your program, and support that learning intentionally. This probably means dedicated time for a key team and lead individuals taking on a champion’s role for each issue identified as Complex. You can approximate the developmental evaluator’s process by naming specific experiments, trying out feedback mechanisms, providing for speedy and regular feedback/refinement discussions. You can make everyone aware, including all faculty, students, parents, Board, and community stakeholders, of the Complex issues you are dedicated to studying, and invite their input. This keeps the program’s eyes on the right prizes.
For example, if fostering peer-to-peer learning is so important that it becomes an organic aspect of your program, as it is in Venezuela, then name the experiment, and organize experimentation in the ways this habit of mind of peer responsibility and generosity can be “taught.” Try different strategies and attend to what happens. Share the observations regularly, discussing implications, and refine and extend the experimentation as a result.
Another example. If nurturing intrinsic motivation is a key goal of your program (as I hope it is, since it is the sine qua non of achieving our youth development goals), it must be handled as a Complex area of program development. This means that the faculty studies what we know about nurturing it and discusses ways to experiment in increasing its development in students. They sharpen their observation of ways in which students demonstrate self-driven advances, and in ongoing discussion faculty refine their experiments in response to what is being learned. Over time, if all goes well, several ways of working become established and seemingly effective enough, to move them to the right, as Complicated practices to determine their impact with more formal evaluation tools.
2) You can connect with other program leaders about the Complex learning at your site. This supports and reinforces everyone’s determination and excitement for learning our way forward together, even in the face of so many pressing concerns that fill our days. Regional collaboration is increasing in the U.S. ES-i ecosystem; this would be a perfect focus to add to the mix of what we do together. Currently we come together around performances and seminarios, and sometimes around professional development opportunities; imagine how useful it would be to have a handful of programs focused on the same specific issue for a year, sharing their learning, exchanging and experimenting, testing out hypotheses in an ongoing way. If a regional consortium could take on such a process together, serving a kind of developmental evaluation function together, the level of all boats rises, we grow as a movement.
Also, taking this responsibility provides a solid foundation for professional development conversations with other music educators. As I have argued in other writing , ES-i programs must develop a healthy relationship with other contributors in the music education ecosystem. I see our role as designated learners with a fortunate situation that enables us to learn about new practices that we are obliged to share with our music education colleagues. The habits of mind of developmental evaluation will enable us to fulfill this function naturally and effectively.
Let me close with one additional concept. Developmental evaluators are skilled at following and illuminating the learning processes of Complex work. They note the questions, the patterns and experiments, and the breakthroughs. Many of the breakthroughs seem to emerge by chance—we all know this to be true in artistic work, and it is as true in the growth of our organizations. Louis Pasteur said, “Chance favors only the prepared mind.” A phrase used in developmental evaluation is “planned serendipity.” With attention and planning, serendipitous discovery arises more often. When we respect the Complex challenges of our work for what they are, and treat them with the approaches they require, we get more discoveries in our work. We get better faster. And getting better faster means we make a bigger difference in the lives of children and communities, means we draw deeper sustainable support to our work, and we grow to become a movement that truly does change cycles of poverty in a way no other social program has ever accomplished before.