Applying Disruptive Innovation within the Public Sector and Community Governance

In this post an argument for applying lessons learned from Professor Clayton Christensen’s Theory of Disruptive Innovation to the public sector and in particular community governance will begin to be developed. This has been on the back-burner for a good while and most of the work that resulted in this has come from discussions and readings provided through colleagues in the Disruptive Innovation group and more recently Disruption by Design group on LinkedIn.

In the previous Innovation Through Community; Innovation By Community post, the difficulty in generating more meaningful innovation within the public sector due to structural system problems with our current form of local institutions of government was raised. A means needs to be found that not only implements the change being sought but that also disrupts the elements of the system working to stop that change. Unlike past politically based forms of disruption which were usually disrupt first then innovate after, a way needs to be found by which the innovation and disruption occur simultaneously, that shifts the balance of influence through a process of innovation that entrenched institutions of government have minimal means to stop. The post following after made the case against Entrenched City Halls and why they can fail communities despite having the appearance of being democratic.

Developing a theory of disruptive innovation within the public sector will be difficult because disruptive innovation has become such a misused term, especially within the public sector. Most examples of disruptive innovation cited by the public sector are mislabeled either purposely or inadvertently, usually to make a more persuasive sounding argument for something. We are not looking for the next disruptive product or disruptive service or disruptive technological advancement but a continual means of innovation (which could include technical or management processes) by some form of disruptor, be it an entrepreneur or other change agent, that drastically bends the growth curve up, substantially shifting the value inherent within the system from one of scarcity to abundance and thereby disrupts the current system.

So it needs to be made clear that we are endeavoring to utilize the concepts of disruptive innovation as developed by Professor Clayton Christensen of Harvard University. A section of the New Community Paradigms wiki has been set aside to gather additional resources related to disruptive innovation. This will not be a comprehensive examination though. The stories of companies such as Apple, as a disruptor or Kodak, as the disrupted, are known and this post will leave telling the specifics of such stories to others. The intricate details of the theory will also be left to others. This post will focus on laying a foundation how disruptive innovation can be applied to the public sector by describing elements of the process and how and why it can disrupt certain components of a system in question and at the same time transform the relationship between scarcity and abundance for other components of the system in question.

Once that is established, we will then need to later determine boundaries as to where the theory can identify applicable factors that can be applied across different systems. As Professor Clayton devised his theory based on the workings of private free-market economics, its application to the public sector will be done by finding the most robust analogies between the two systems. Even if the application of disruptive innovation to the public sector is primarily figurative, it should still have the potential to have an impact upon the system and therefore able to cause meaningful change.

Another problem with explaining the concept of disruptive innovation as a process which could be purposely applied is that on the surface it is not intuitive and how it gets one from point A to point B is not readily apparent until one digs deeper. This is in part why it is effective because the industries or the companies within them being disrupted do not realize the threat and the resulting disruption seems to come out of nowhere. It is all the more effective though because after a certain point, along the path of innovation discerned by Christensen, little can be done to prevent the disruption and its inevitability becomes clearer in hindsight. This is also one of the ways the public sector misses the essential aspects of disruptive innovation. It is not just a matter of creating a brand new mousetrap for the world and everyone beating a path to your door. Someone or something finds itself being prone to being disrupted in the process putting organizations of all types in the position of being the guy at the poker table who does not know who the sucker is.

Stories of disruptive innovation can start off as the David and Goliath sort. Disruptors, entrepreneurs or startup firms, play the role of David with seemingly meager weapons and the incumbent, as in the well established, undisputedly in power, market leader, sometimes even iconic business with a long history of financial success and domination over the market assumes the role of Goliath. In some ways disruptive innovation may be more like stories of Popeye and Bluto when Popeye finds the spinach but what the spinach is changes with each new and different innovation. (I am a Baby Boomer, Google it if it’s not familiar)

The incumbent market leaders or the Goliaths and Blutos, once they are established, appear to remain unbeatable because they are the ones coming up with sustaining innovations, the ones proclaimed as new and improved in the advertising. This is when one most often hears empty claims of disruptive innovation being made as in not only new and improved but super new and improved. This means that their customers and especially high-end users continually get better service through features with better quality and improved reliability making them the safe bet. These improvements can be small and incremental or significant but they are targeted to the already defined customer base with the intention of making the base population larger at the price point set by the business or to raise the price point high enough so that even if less customers are added in numbers the overall profit for the company goes up. Customers that show loyalty by being able and willing to pay the price for the standard offering or better yet move up to the premium offering are rewarded. Those not included within this relationship are left behind to no obvious detriment to either the chosen customers or incumbent business. People get use to Goliath or Bluto being the biggest badass in town and assume it will remain that way.

Disruptive innovations or DIs, led by the Davids and Popeyes, are seen as overall inferior or underperforming solutions compared to those being provided by the market or sector leaders. Where DI agents are able to begin to compete is on a different set of benefits, such as simplicity, convenience, accessibility, significantly lower price, or ease of use that satisfy a need either not met or underserved by the incumbent market leaders. By doing so, they create what Clayton Christensen termed “asymmetries of motivation” and a new dimension of value.

This new dimension of value can be created in two ways, first through low-end disruption which means offering lower quality or performance, perhaps missing features, though at substantially lower costs making it appealing to users who can’t afford or can’t access, or new users, along with others, that don't need the full features of the incumbent market solution.

The new dimension of value can also be created through new market disruptions that provide the means to accomplish a new category of previously unrealized objectives or goals with which the incumbent has decided or defaulted to not compete. It is good enough though to meet the particular customer’s goal or what has been termed in disruptive innovation literature as the Job-To-Be-Done or JTBD. A DI directly addresses some JTBD of the customer at an acceptable price making them willing to put up with the lack of features or quality as compared to the incumbent offering.

The incumbents ability to offer sustaining innovations to its defined customer base only cements its position as long as it offers them at a pace that keeps it ahead of whatever competition it may have. Incumbents create the products or services to be bought and the market reacts within the confines of this solution choosing between better or worse alternatives. Customers adjust themselves to optimize their use of the solution offered, not the other way around. Incumbents over time mold customers expectations and train them to become dependent upon the incumbents offerings and stop seeing the possibility of alternative solutions. At some point though these sustaining innovations stop making a difference and buying decisions become based solely on price, the one providing the best offering at the lowest price becomes king of the hill.

There is, however, still those who are not being served, or are being underserved and sometime even over-served by the incumbent business. If the DI agent goes after these customers then the incumbent sees no reason to compete. Initial offerings to these customers can be seen as mere toys flying below the radar of the incumbent market leaders.

When David or Popeye start off, they and their customers may have no idea of the full potential of the innovation that they are offering. This is where the DI agent finds his sling or spinach. What it does is provide a track for continual, while still sustaining, innovation has the very important difference that it is within a noncompetitive arena with the incumbent business allowing Popeye to pop open the can of spinach unhampered.

This is where disruptive innovation has its greatest impact because this new dimension of value is what moves the value inherent in the system from a state of scarcity to abundance. It is at this point that truly meaningful change could start to be implemented.

Entrenched City Halls Defined then Disrupted

In the last blog post, the idea of combining disruption and innovation was suggested as a strategy for implementing New Community Paradigms in communities in which politically controlled bureaucratic government institutions are entrenched.

This post will provide clarification as to what is meant by an entrenched city hall or a politically controlled bureaucratic government institution that is no longer serving the community by reviewing past arguments made over the course of this effort.

The potential push for disruption against an arguably entrenched city hall is based first on the philosophical premise that the democratic basis for the power of governance is based not on the authority of government institutions but on civil society.

The idea of ‘entrenched city hall’ started taking form back in 2011 with Why is this so hard? It's complicated and it's complex but that's OK which promoted the idea of a community creating its own community paradigms despite the complexity of doing so.

There has been an unstated assumption throughout the posts of this blog that ‘City Hall’ has failed to adequately address the need for an economically and environmentally sustainable and livable community. That is obviously not going to be true for all cities and the degree to which it is will be different from city to city. In some cases, the effort will find a willing partner and the collaboration will create a new more expansive form of community governance. In other cases, there will be pushback from the incumbent institutional government. The key issue is whether City Hall truly represents the community as a whole or only special interests or privileged key community members. If it is later, then there are pathways that can be taken to weaken and subsequently disrupt that control in a sustained and innovative manner.

For this to happen requires greater Community Governance - Third Path to City Government, a move away from current entrenched forms of closed city government to more open forms of community-based governance. It also requires recognizing, as did Governance by the Community or Not - Getting with or past City Hall that:

There is a difference though between governance and government. New community paradigms do not assume that people will take turns being the city planner or a city council member for a day. It does work under the premise that the relationships with community leaders and city employees (maybe better would be community-based employees) need to change and become different. How community needs are met and community standards enforced would also likely be different. How to make this work is another challenge for new community paradigms.

The idea is that the community itself makes more of the decisions directly through some system of direct deliberative democracy or a hybrid with the current system rather than indirectly through a mayor and city council system. Support for such an approach will be considered in greater detail in the future but for now, the focus for this post is on the factors involved when Community Governance Takes On City Hall in determining the balance of influence between political leaders, public sector and the community at large. 

The standard argument is the degree to which the process of running communities should be by politicians or by the public sector. This can, however, be a false choice as politicians and the public sector or bureaucrats often come together in a disjointed form of control in which city council members use their political power to force short-term decisions that make poor long-term sense for the community and city management works to protect the political self-interests of the city council members. Even when this is not overtly excessive, it can still significantly define the culture of city hall.

Each component ends up working to support the dysfunction of the other to maintain its own survival and the entire system becomes more closed. This does not always happen, perhaps not even as often or as significantly as my experience leads me to believe but when it does, it creates a culture of entrenched institutional government control and despite appearances to the contrary discourages community participation. 

The second problem with the standard approach to institutionalized community government is that it can discourage ongoing community participation in city government as A Ladder of Citizen Participation by Sherry R. Arnstein explains.

The usual claims that change can come about through elections every four years have little merit as there may be in reality minimal opportunities for true community participation between elections and elections are always under processes controlled by the status quo power culture. Not necessarily in mechanical terms of voter suppression or such but by the culture already instilled by the daily process of city hall control that results effectively in disenfranchisement. Worse, any successful election of a new slate of officials often times only changes the players, not the game. There may be superficial changes in policy when one political clique replaces another but it swings back a few years later and the deeper culture of the existing status quo power system stays in place.

Again, it should be stated that there is no claim that this situation applies to all cities or even in the fullest extent to a substantial number. It does happen though and one of the worst examples has been the City of Bell though on the positive side Bell can also be seen as An Example of Civic Renewal From Broken City.

The City of Bell was an example of what was the worst of local municipal governance with both politicians and top public sector management taking self-serving advantage of the community. Yet, how many California cities looked at the City of Bell with derision when the difference between Bell and those cities is not really one of kind but in truth only a matter of degree. Not that there is any claim that there are numerous city councils or administrations currently breaking the law. Rather, how many city halls adopted policies whether official or not that in the words of newly elected Bell City Council person Ana Maria Quintana, instituted a lack of inclusion by the public in decisions having an impact on the public, whether intentional or through negligence.

This argues that new alternative methods of community transformation need to be developed to change the balance of power in favor of a broad-based community vision and away from entrenched city hall bureaucratic politics when the supposedly democratic process has become corrupted not through blatant illegalities but through the small corruptions of a system turned to protecting itself at all costs and turning away from the real needs of the community.

It should be recognized though that arguing against the existing system is different from arguing for a system proposed to replace it. That will still continue to be needed to be done in future posts. There are though additional challenges beyond getting past an entrenched city hall.

This blog has advocated for ‘radical’ community engagement and through it the far greater inclusion of Non-Experts to Create New Community Innovations.

This has its own set of challenges:

This brings up the at first seemingly unattainable chasm created arguably as a symptom of the current system of local political and economic power. Most people are not involved enough in their communities to make the necessary changes. Many currently have zero involvement because they have been turned off by the current system of power which becomes all the more entrenched because people are not involved. This is a matter of learned helplessness which can be unlearned.

Others continue to try to make changes but are locked out by the current benefactors of politics occupying city hall which often has the appearance of a democratic system but the culture of entrenched power.

In future articles, it will be asserted that this is not as unattainable as may be seemingly apparent. While members of a community may have different degrees of involvement in their community, it is still possible to organize to generate the necessary community will for change.

Finally, many city halls constituted under the current form of local government structure are becoming increasingly unable to adequately address the ‘wicked problems’ facing our communities because it means adopting a complex approach to finding solutions as opposed to the more traditional, complicated, machine-based approach as was argued in Complexity as Cradle for Creativity and Innovation.

Public Sector organizations though, particularly at a local level, are often not able to do this as well. Instead, reacting to protect their status they end up imposing unseen additional layers of complicated or closed processes that are essentially subconscious from an organizational perspective that in time become culturally implicit. It is a means of maintaining power by becoming entrenched as a system both operationally and culturally. This is why so many local city halls or other public sector organizations prefer the more traditional organization as machine approach in which complexity is seen as chaos and why community engagement in some communities remains stuck at the bottom rungs of Arnstein’s Ladder.

Complexity as Cradle for Creativity and Innovation

This is the third post concerning a 2013 HBR Article Why Managers Haven't Embraced Complexity by Richard Straub on addressing complexity as discussed in the Harvard Business Review LinkedIn group.

The first look Complexity Addressed From On High taking its perspective from the air was limited because it was too abstract and conceptual in its approach. The perspective of the second post, Categorizing, Controlling, Conquering Complexity as Chaos (Wrong Choice) was from behind the barricades of day to day, under fire, management but again was limited because it failed to realize both the inescapable reality of complexity and the creative potential of complexity. So some synthesis of these two perspectives or a third alternative would seem to be warranted.

One alternative perspective on complexity, besides the one previously provided by the Economist Intelligence Unit (pdf), was provided by past HBR articles including Embracing Complexity An Interview with Michael J. Mauboussin by Tim Sullivan and Learning to Live with Complexity by Gökçe Sargut and Rita Gunther McGrath. Both of these HBR articles from 2011 helped to set the framework for the question asked by Richard Straub. Their perspectives did embrace complexity.

These articles moved away from a more traditional organization-as-machine approach to one that was more organic and less top down hierarchical in managing institutions, particularly ones that dealt with complex environments. The question considered by this blog in the previous Categorizing, Controlling, Conquering Complexity as Chaos (Wrong Choice) post was the same as Richard Straub's, why some managers had failed to learn the lessons put forward by Mauboussin, Sargut and McGrath. The answer is that they chose to do so, presumably because embracing complexity was not seen as being in their self-interest.

It is also suspected to be the same reason why the current common form of local community governance, with city council, mayor and city manager in some format, has so often become increasingly unable to function effectively in meeting the needs of the community. The problem is that most people don’t recognize this, can’t imagine anything being different or even the possibility of changing anything.

As mentioned before on this blog, the need to revisit a previously presented concept and revise it so it becomes clearer arises from time to time. This happened after having nearly completed the first draft of this post concerning an idea initially stated in the New Community Paradigms Thinking Requires Systems Thinking post that government institutions address the complex challenges facing communities by developing complicated processes.

What may be a more precisely stated concept is that government institutions use complicated processes to address the challenges arising from complex environments. There is the community, which is a complex entity, and the larger environment in which it exists, which is also complex, and then there is the government institution, which is at best complicated but in a fashion designed to meet the needs of the community within its larger environment. Any change to the complex environment, which is difficult to predict and even more difficult to control, would create a corresponding change in the challenges facing the community to which the government institution’s complicated organizational process must then adapt.

There are private sector companies that succeed in doing this. They reconfigure to move along with their target audience, not only changing when they do but also anticipating changes in their users’ environment. These companies are feeding off of the complexity arising from their two way interaction with their customers to create new innovative solutions.

Public Sector organizations though, particularly at a local level, are often not able to do this as well. Instead, reacting to protect their status they end up imposing unseen additional layers of complicated or closed processes that are essentially subconscious from an organizational perspective that in time become culturally implicit. It is a means of maintaining power by becoming entrenched as a system both operationally and culturally. This is why so many local city halls or other public sector organizations prefer the more traditional organization as machine approach in which complexity is seen as chaos and why community engagement in some communities remains stuck at the bottom rungs of Arnstein’s Ladder.

Taking the complexity as chaos approach, many managers endeavor to control or conquer it using Occam's razor, the principle of parsimony as their first principle. The more organic approach to management espoused by the HBR articles recognizes that this does work for linear processes in a world constricted by smaller budgets and shorter deadlines for a time. Real community building, however, which is becoming as much a requirement for consumer businesses as for governance of our communities, is not a linear process.

Still keeping with Occam’s razor, while it may be true that the solution with the fewest assumptions could be the most viable this is not absolute. It certainly does not mean that the first such solution will always be the most optimal especially when the needs of a large group of people must be considered. An Online Stanford University course on creativity demonstrated that the resonance and viability of a solution of the first order or iteration usually turned out to be less optimal than that of a third or fourth order solution. A similar Stanford course on Design Thinking emphasizes creating an over abundance of solutions, even imaginary ones, to arrive at the most effective and optimal. Depending unquestioningly upon the simplest first solution only because it has the fewest assumptions may not go deep enough into a situation and the environment in which it occurs to create the best or most innovative approach to the challenge.

A second principle of complexity as chaos approach is that the authority of wielding Occam’s razor should be given to only a few supposedly visionary leaders with a cadre of obedient keeping-in-line managers behind them. This is very much in keeping with an autocratic city manager style of community management found in many communities. This is the management-as-machine Maginot line of defense against complexity, uncertain conditions and risks all of which left unchecked could mean chaos. An organic approach to addressing complexity asks whether these factors are really interrelated on a causal basis or merely correlated, therefore requiring a different approach?

Uncertainty and risk are undoubtedly a part of the business landscape, and therefore also for the public sector, even though we too often pretend that they are not, but they are not the same as complexity. Instead they arise because of a lack of knowledge concerning a complex situation or environment. Any resulting gap in relevant information could be a problem and could create a chaotic situation but complexity in reality exists at the edge of chaos providing a potential means of change. (This does not mean that omniscient powers will eliminate complexity or make it precisely predictable.)

Complexity is admittedly difficult to manage because it is difficult to see the myriad of interrelationships within the environment or ecosystem that makes complexity a reality from any specific singular perspective. Individual managers, using complicated processes which includes systems of bureaucratic management, may be able to know specific components in a defined span of control or over a defined period of time.

It is impossible though for anyone to know the total information of a substantially complex system. The complexity as chaos approach wrongly assumes that the uppermost levels of an organization are unilaterally able to dictate the organization’s response to complexity. The organic approach asks whether it is actually necessary for any individual manager, at any level, to always know the entire system or is it the system that needs to 'know' the entire system? This means far less top down control and a far greater need for trust which makes some very uncomfortable.

This brings up other differences in approaches between restricted complicated systems and systems adopting practices that are more coherent regarding complexity. With a complicated process the cumulation of different managerial perspectives happens more in the aggregate over time with top management then making decisions about taking a specific approach usually after the fact, while with an organic approach to complexity there is often some form of synthesis between the community and its environment to which the organization adapts more in line with real time with managers facilitating the process.

A third principle of complexity as chaos is that complexity is seen as not having any potential value either for the institution or for the individual manager except in its removal. Individual managers taking a complexity as chaos approach are incentivized to only propose safe known solutions or predetermined approaches in addressing complexity to avoid career suicide. Office politics then becomes a more important game than competition on the field, a situation that can be made all the worse in the public sector arena where the presumed leadership can be based on political gamesmanship.

The truth according to the more organic managerial approach is that complexity properly addressed through innovation, particularly disruptive innovation, can add tremendous value. It is a source of creativity and innovation in an advanced economy. The more traditional’ approach, particularly in the public sector, provides no real path towards innovation in any sense of the word as even forms of sustaining innovation can be hampered and more disruptive forms of innovation have little chance of occurring.