Use Agile Methodologies to Profit from the 99% of Your Data that’s Going to Waste

by John Lang – 

There is no doubt that we have entered the age of Big Data. Vast quantities of data are flooding into every organization at astonishing rates.

More data is expected to be generated this year than in any previous year in human history. The Internet of Things, wearable devices, connected devices, smartphones, smart homes, smart cars — all are adding to the flood of actionable data available to companies.

That’s sort of a good news, bad news thing.

It’s good because all that data represents immense opportunity to improve services, enhance processes, develop new products, save lives, and — the ultimate driver behind everything — increase profits.

But here’s the bad news part: companies are struggling mightily to make use of all of that data. It’s very much like the old analogy of trying to drink from a firehose. The result is that a lot of valuable data is just being wasted. A recent article in App Developer Magazine¹ reports that less than half of one percent of available data is currently utilized in operational decision making.

And that’s the epitome of squandered opportunity.

Unlimited Potential for Those Who Can Capitalize

The age of Big Data is providing massive benefits to companies worldwide — even in the face of so much squandered opportunity.

The Worldwide Semiannual Big Data and Analytics Spending Guide by International Data Corporation predicts that “…worldwide revenues for big data and business analytics will reach $150.8 billion in 2017, an increase of 12.4% over 2016.” The report predicts that those revenues will top $210 billion by 2020².

And less than 1% of all available data is being utilized!

Imagine the opportunity available to companies that are able to faster and more fully utilize the data available to them. That opportunity can be transformed into reality by borrowing from the software development toolset.

It’s time to go Agile with your Big Data.

Big Data and the Agile Manifesto

Many of the twelve foundational principles³ of the Agile Manifesto can be applied directly to efficiently and effectively harnessing massive volumes of data. Consider, for example, how the following Agile Manifesto principles can relate directly to the process of utilizing Big Data:

• Agile Manifesto principle: “Business people and developers must work together daily throughout the project.”

• Big Data tie-in: Business drivers must define how data will be utilized. Business leadership must define and prioritize capabilities and processes. IT must then determine how those goals can best be achieved.

• Agile Manifesto principle: “Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale.”

• Big Data tie-in: Making use of the incoming flood of data requires starting small with multiple projects that are executed fast, getting quick wins and demonstrating value to the organization. Short cycles are particularly important in the age of Big Data, given that much of the data flooding in has only short-term value; it must be used quickly, or it won’t be usable at all.

• Agile Manifesto principle: “Welcome changing requirements, even late in development. Agile processes harness change for the customer’s competitive advantage.”

• Big Data tie-in: The data itself can and should steer the process of determining how best to use the data. Quite simply, the data will not always align with preconceived assumptions or expectations, and the ability to change accordingly, and quickly, is crucial.

Best Practices for Applying Agility to Big Data

The process of effectively harnessing Agile methodologies to the utilization of Big Data can be enhanced by following a few best-practice guidelines:

• Executive leadership must steer the process: We’re talking about data, so this all belongs to the IT department, right? Wrong. Any type of Big Data solution must be focused upon business success; it’s not technology for technology’s sake. Company leadership must guide the process with an eye toward organizational goals, and while tracking metrics along the way. It’s not IT’s job to conceptualize that vision; IT’s job is to make leadership’s vision a reality.

• Lose the one-and-done mindset: The traditional concept of project lifecycles — conception, execution, completion — won’t work here. The continuous incoming flood of new data should drive ongoing tweaks to projects. The business must always be evaluating many different data-driven analytics to determine how they impact business drivers. And project requirements should be revised and refined as needed.

• Start small, win quick: It’s important to start small with scalable solutions, and quickly be able to show positive impacts to the company. That helps to solidify organizational buy-in, and may even be key in securing project funding.

It’s a Mindset

Perhaps the most important element of agility is the mindset that it encourages, and the culture it helps foster. Making the most of Big Data requires an attitude of openness: open to new ideas, new concepts, and open to exploring ways of using entirely new forms of information.

After all, venturing into the age of Big Data is somewhat akin to pioneering new and unexplored territory. Fortune will favor the bold and the agile. And the not-so-agile, well, they’ll just be left behind.

#IdeasRealized

Works Cited

1. App Development (2016, December 23) Retrieved from https://appdevelopermagazine.com/4773/2016/12/23/more-data-will-be-created-in-2017-than-the-previous-5,000-years-of-humanity-/

2. Businesswire (2017, March 14) Retrieved from https://www.businesswire.com/news/home/20170314005262/en/Big-Data-and-Business-Analytics-Revenues-Forecast-to-Reach-150.8-Billion-This-Year-Led-by-Banking-and-Manufacturing-Investments-According-to-IDC

3. Agile Manifesto http://agilemanifesto.org/principles.html