Big data offers leaders unparalleled opportunities to create and deliver value in a constantly changing world – if they can achieve instancy. 

The past few centuries of economic history have shown that market economies typically experience extended periods of stability that are punctuated with occasional short, disruptive periods of instability that often fundamentally change the global economic outlook.

Technological innovations, such as the printing press and the steam engine, were catalysts in creating step changes in the societal, political and economic landscapes of their respective eras.

Today, the world finds itself in the midst of a particularly prolonged – some argue permanent – period of instability as international adoption of disruptive innovations such as the web-browser, social media and the smart-phone have converged to create a global digital nervous system that is profoundly redefining how people connect, communicate, coordinate, collaborate and take collective action.

Over the past 20 years, the internet and its social and mobile technological cousins have wrapped our planet in a digital blanket that is creating information and connecting people at a pace that was unfathomable just two decades ago.

According to Don Peck, deputy editor of The Atlantic, more than 98% of the world’s information is now stored digitally and the volume of that data has quadrupled since 2007.

The impacts resulting from this data-rich and far-reaching digital platform are well documented: from creating personalized digital marketplaces to facilitate consumption; to optimizing traffic patterns to ease congestion and reduce pollution; to creating social revolutions to eradicate oppression. The story of the past two decades is a story of how this evolving and increasingly pervasive digital nervous system has permeated what we do professionally and socially, to such an extent that we have become somewhat desensitized to the incredible potential that it holds to create value on a global scale.

Digital Darwinism

History has also taught us that the periods of economic discontinuity emerging from the mass adoption of revolutionary technologies also usher in swift and significant changes in how organizations create, deliver and capture value. As the electronic exoskeleton continues to evolve and expand without bound, it is ushering in an era of “digital Darwinism” where markets shift in the blink-of-an-eye, disruptive technologies wipe out age-old industries at unprecedented rates, and previously successful products and services are rendered obsolete almost overnight.

In the connected, complex and constantly changing business context, leaders are increasingly challenged to enable their organizations to become more responsive to unanticipated market shifts, more resilient to unpredicted disruptions and more adaptive to unforeseen fall-offs in market adoption. To create more responsive, resilient and adaptive enterprises requires leaders who are more agile and adept in deciding how to optimally allocate scarce resources and capabilities against the ever-expanding array of opportunities and threats that face their business.

Under normal circumstances, this decision making challenge would be considerable. Under the current circumstances, it sometimes appears to border on becoming impossible. In the 20 years since the web browser arrived on the scene, the world has weaved the web into an electronic exoskeleton that is moving through time at a rate of change humans are ill-equipped to detect, let alone respond to or anticipate. Today, leaders are facing a situation where the mean-time between  “bet-the-farm” decisions is shrinking at an alarming rate and the amount of data surrounding each of those critical decisions is expanding at an even more alarming one.

To survive and thrive in the era of digital Darwinism, leaders must not only make more informed and timely decisions, but they must ensure that the decisions they make better balance the age-old organizational tension of exploiting their core business to deliver the required returns in the short term and exploring new value frontiers that will become sources of sustainable long-term competitive advantage for the firm.

This analysis will investigate the promise and peril that big data presents for leaders seeking to create, deliver and capture value in a world where technology is proliferating, information is exploding, time is compressing, change is accelerating and competition is intensifying.

The quest for instancy

The digitally interconnected global market economy places a premium on innovation, fresh business models and new ways of organizing and virtualizing work. In this highly competitive and increasingly transparent business context, organizations that cannot change as fast as the environment within which they operate are destined to regress to a mean of mediocrity. This regression pattern has a biological equivalent: organisms that cannot develop the capacity to assimilate data regarding their surroundings to make decisions in the interest of their own survival simply recede into the ecosystem from whence they came.

Put differently, organisms within volatile ecosystems must learn to adapt – or die. Modern-day enterprises could well benefit from perceiving themselves as value-creating organisms within a digitally-mediated economic ecosystem where data is their source of sustenance and analysis, learning and adaptation are the metabolic mechanisms that enable them to survive and thrive.

In the business landscape, the only constant is change and the path to maintaining sustainable competitive advantage lies in developing the ability to intuit, innovate and adapt in perpetuity. Today, insights drive innovation and innovation drives profitable growth. These insights are generated from serendipitous knowledge accidents; that magic moment where expertise collides with opportunity and whole new industries are born.

The predominant challenge facing business  leaders lies in cultivating the capacity for instancy within their enterprises. Instant enterprises focus on developing a perpetual state of readiness for the unexpected.

Readiness is the state of being able to creatively adopt and adapt what you know and what you can do under a varying and unknown set of environmental circumstances. Within the instant enterprise, value is created by groups of individuals who can collectively sense impending shifts in their ecosystem and adapt to harmonize with these shifts in real-time.

As these people connect, communicate, coordinate, collaborate and take collective action, work and learning become synonymous and the organization develops the capacity to ride the waves of creative destruction rather than run the risk of becoming crushed by them.

The reality of rigidity

The history of organization design suggests there is always a lag-time between the identification of an opportunity to create value, the construction of a mechanism through which that value can be delivered and the optimization of an organization structure to ensure that maximal value is captured by the firm.

An apple falling on Sir Isaac Newton’s head leads to the creation of the laws of physics, which leads to the invention of the internal combustion engine, that is followed by the growth of the automobile industry, that culminates in Alfred Sloan’s creation of the modern-day bureaucracy and Henry Ford’s assembly line.

As leaders pursue their quest for enterprise instancy, they encounter an age-old organizational reality: the existing structure and culture within the enterprise is optimized to exploit the old rather than explore the new. As organizations grow in response to positive market acceptance, they develop structures, practices and routines that are geared to improve the operational efficiency of their existing business model. These structures, routines and practices calcify into a set of core-rigidities that undermine the organization’s capacity for instancy.

The arrival of digital Darwinism has not altered the perennial organization challenges of structural-lag and core-rigidities that leaders face as they seek to create instancy within their enterprises. Instead, it has accelerated and amplified these challenges to such a degree that the need to find innovative ways to address them has become unavoidable.

The reformation of transformation

In today’s dynamic and unfamiliar business context, leaders are increasingly called upon to field decisions that the current enterprise structure and culture cannot address. As the volume and importance of these decisions increase exponentially, leaders are recognizing that they are contributing to the lack of responsiveness, adaptability and resilience they seek to cultivate within their own organizations. In short, they have come to the stark realization that they are their own worst enemy in that they have become the bottleneck to productivity and growth.

As a result, leaders are actively seeking out fresh ways to reconfigure their resources and capabilities to be more responsive, adaptive and resilient in dealing with the increased number of unanticipated decisions that need to be attended to within their organizations. Unfortunately, history is replete with organizations that have struggled mightily to transform their organizations from one structure to another in a quest for instancy. Most of these transformational attempts have been unsuccessful for the simple reason that businesses that are geared to exploit their core business differ in almost every respect from organizations that are geared to explore new value horizons. Transforming an exploit organization into an explore organization is at least as difficult as trying to turn chalk into cheese.

To address this unpalatable reality, innovative leaders are reframing the transformation challenge from one of “from-to” to one of “both-and”. Rather than falling into the typical trap of trying to get the organizational leopard to change its spots, or trying to coax the enterprise elephant to dance, these leaders are recognizing that change is no longer something to be managed in and of itself. Rather, change itself is a constant that resides at the core of every business and the tensions and trade-offs it throws off are what need to be attended to in a more focused and timely manner.

These leaders have realized that transformation is not about moving the organization from one structural and cultural state to another. Instead, they have come to understand Transformation 2.0 is about developing the capability to create a two-speed organization that can simultaneously handle the pedantic and predictable pace of exploitation and the fickle and frenzied pace of exploration.

On one hand, organizations need to do everything in their power to extract the most value they can from the business they operate. On the other hand, they need to ensure that once the existing  business meets its inevitable demise, there are new sources of value to draw upon to sustain the business over the long term. Two-speed organizations are required to deal with the environmental reality of simultaneity that every business faces today.

The leadership challenge associated with running two-speed organizations is that they are inherently and perpetually at odds with one another. Organizational operating systems that look to exploit focus on evidence and best practice and seek to apply deductive logic to optimize processes, structures and routines within the existing enterprise. Organizational operating systems that look to explore focus on intuition and trial and error and seek to apply inductive and abductive logic to improve the odds of finding fertile ground within which to sow the seeds of future value for the enterprise.

Promise and peril of big data

The promise of big data is that it will allow leaders to make more informed decisions on how to create, deliver and capture more value in both the short and long term. The peril in this promise is that, often, the biggest sources of new value creation are not derived from data-based extrapolations of the past, but from collective and intuitive insights derived from explorations into a set of possible futures.

In examining the behaviour of organizations adopting new technologies, leadership expert Peter Drucker popularized the notion of the Routinization Trap: a recurring pattern where organizations apply radically new technologies to automate existing business models rather than exploring new business models that the technology itself might enable. The printing press was used for many decades to print bibles until someone had the bright idea that it could be used to print other books too. Similarly, the steam engine was first used to power cotton-gins in the UK before someone had the bright idea to use it to power a locomotive. In both these cases, the transformational power of the technology was not fully realized until decades after it was initially applied to automate the existing model at play.

In analyzing the emergence of big  data, a similar pattern transpires. The field of data analytics was born in the 1950s with the advent of computing technologies that used algorithms to analyze large quantities of information. As the information technology industry flourished and enterprise systems focused on stream-lining supply chains, optimizing inventory on factory floors and improving customer relationships were implemented, managers were provided with the data they needed to make more informed decisions on how to more efficiently run the business.

By the mid-2000s, the era of big data was officially declared. The emergence of the digital nervous system had broadened the field of data analysis to extend well beyond the domain of internal enterprise transnational systems.

This new era ushered in analysis of massive data sets on both the back end and the front end of the enterprise. Want to shave more pennies off the Supply Chain dollar? Simply pool the data sets from the multiple entities and torture the aggregated data set to reveal its cost-saving secrets. Want to find out how many teenagers there are in Singapore who like Bon Jovi? Just ask Facebook. Want to know what three items are most likely to indicate credit card fraud? Just crunch the transaction data the right way and, presto, Champagne, razors and diapers are the top three items bought with stolen cards. Want to know what promotions to offer to a customer who walks in the door? Simple, just ask Four-Square.

Question by question and answer by answer, the promise of big data as the technological salve for all organization efficiency woes began to take hold. Pretty soon, most enterprise leaders began to buy into the belief that the thorough analysis of any sufficiently large pool of data poop would almost certainly reveal an efficiency pony.

As the amount of data that surrounds our private and professional lives has grown from a trickle to a torrent over the past few decades, the ability to discover correlations through computation and analytics clearly holds the potential to deliver value. Moreover, as the volume, velocity, variety and need for veracity of data increase within and across enterprises, the demand for increasingly high-powered computational crunching and increasingly sophisticated  algorithmic alchemy appears to be growing by leaps and bounds.

So, if demand for big data services is increasing based on sound evidence that it yields predictable and reliable efficiency results, what seems to be the problem? The problem is that – in its first incarnation at least – big data analytics is primarily focused on the exploit side of the enterprise equation. Data is being crunched and analyzed to improve the firms operations much more than it is being applied to define new offerings. It is being leveraged to optimize and maximize the current business model, not to define new ones that will sustain the business over time.

In short, the application of big data has fallen prey to the routinization trap where it is primarily being applied to automate the past at the expense of creating the future.

The balancing of attention

At the end of the day, the most important asset that leaders can contribute to the continued success of their organizations is their focused attention. Each and every day, leaders around the world have to attend to urgent decisions regarding the optimization of their existing business and important decisions regarding an ever-expanding array of opportunities and threats that their organizations will face in the future. The promise of big data is that it can help leaders break the tension in the need for attention at both ends of the exploit/explore spectrum. The peril in this promise is the paradoxical insight that a wealth of information often creates a poverty of attention.

In its current incarnation, the application of big data runs the risk of diverting leaders’ attention in unfair measure towards the exploitation of the current business. With the ever-increasing pressure to deliver short-term results and the high likelihood that increased analysis of growing piles of data will reveal tangible opportunities to improve operational efficiency, leaders may be unconsciously lured into spending too much of their time attending to improving their current business.

Taking this pattern of attention its logical extension reveals an undesirable outcome: the law of diminishing returns will set in and ever larger data sets will have to be mined with ever more sophisticated algorithms to yield increasingly incremental operational gains at the expense of missing a fundamental shift in the business where the opportunity to create new value is squandered or the threat of undermining the existing business is missed.

To successfully run a two-speed organization that requires one operating system to exploit the past and another one to explore the future, leaders must learn to not only effectively balance their attention between the present and the future, they also must recognize that the electronic exoskeleton that blankets our planet can play a critical dual role of crunching information to optimize productivity for one operating system and connecting people to create value for the other.

On the crunching side of the equation, leaders must recognize that mining the past for efficiencies can only be effective in slowing the inevitable decline of the current business. On the creating side of the equation, leaders must recognize that the same set of technologies being used to crunch data – if perceived and applied differently – could be leveraged to intuit new value horizons for the organization by enabling people to connect, communicate, coordinate, collaborate and take collective action around their intuition of what the future may hold.

In the era of digital Darwinism, leaders have come to realize that the true sweet spot for organizations that want to survive and thrive in an increasingly complex and connected environment lies in constantly striving to find the right balance between exploiting what we know and exploring what we don’t.

The first step in achieving this illusive state of equilibrium lies in avoiding the trap of mining the past at the expense of making the future.

Tony O’Driscoll is Duke CE’s Regional Managing Director for Asia.

An adapted version of this article appeared on the Dialogue Review website.