Big Data & Decision Making Survey

6/13/2012

The Economist Intelligence Unit surveyed over 600 business leaders, across the globe and industry sectors about the use of Big Data in their organizations. The research confirms a growing appetite for data and data-driven decisions and those who harness these correctly stay ahead of the game.

The report provides insight on their use of Big Data today and in the future, and highlights the advantages seen and the specific challenges Big Data has on decision making for business leaders.

Key Findings:

75% of respondents believe their organizations to be data-driven

9 out of 10 say decisions made in the past 3 years would have been better if they’d had all the relevant information

42% say that unstructured content is too difficult to interpret

85% say the issue is not about volume but the ability to analyze and act on the data in real time

More than half (54 percent) of respondents cite access to talent as a key impediment to making the most of Big Data, followed by the barrier of organizational silos (51 percent)

Other impediments to effective decision-making are lack of time to interpret data sets (46 percent), and difficulty managing unstructured data (39 percent)

71 percent say they struggle with data inaccuracies on a daily basis

62 percent say there is an issue with data automation, and not all operational decisions have been automated yet

Half will increase their investments in Big Data analysis over the next three years

The report reveals that nine out of ten business leaders believe data is now the fourth factor of production, as fundamental to business as land, labor, and capital. The study, which surveyed more than 600 C-level executives and senior management and IT leaders worldwide, indicates that the use of Big Data has improved businesses' performance, on average, by 26 percent and that the impact will grow to 41 percent over the next three years. The majority of companies (58 percent) claim they will make a bigger investment in Big Data over the next three years.

Approximately two-thirds of 168 North American (NA) executives surveyed believe Big Data will be a significant issue over the next five years, and one that needs to be addressed so the organization can make informed decisions. They consider their companies as 'data-driven,' reportingthat the collection and analysis of data underpins their firm's business strategy and day-to-day decision-making.

Fifty-five percent are already making management decisions based on "hard analytic information." Additionally, 44 percent indicated that the increasing volume of data collected by their organization (from both internal and external sources) has slowed down decision-making, but the vast majority (84 percent) feel the larger issue is being able to analyze and act on it in real-time.

The exploitation of Big Data is fueling a major change in the quality of business decision-making, requiring organizations to adopt new and more effective methods to obtain the most meaningful results from their data that generate value. Organizations that do so will be able to monitor customer behaviors and market conditions with greater certainty, and react with speed and effectiveness to differentiate from their competition.

the_deciding_factor__big_data___decision_making.pdf
File Size:	3379 kb
File Type:	pdf

Download File

23 Comments

Fred Baer link

6/13/2012 10:05:01 am

Data mining process: sampling, exploring, modifying, modeling and assessing data.

Sampling Data:

Sample the data by creating one or more data tables or data marts that represent the target data set(s).

Since data mining can only uncover patterns already present in the data, the sample should be large enough to contain the significant
information, yet small enough to process.

Fred Baer link

6/13/2012 10:08:43 am

Data mining process: sampling, exploring, modifying, modeling and assessing data.

Exploring Data:

Start by exploring the data by searching for anticipated relationships, unanticipated trends, and anomalies in order to gain understanding and ideas.

For example, clustering discovers groups or structures in the data that are similar, beyond the structures known in the data. Classification generalizes a known structure to apply to new data, such as classifying a customer as a good or poor credit risk.

Regression works to find a function that models the data with the least error.

Association rule learning searches for relationships among variables, such as products frequently bought together, known as market basket analysis.

Fred Baer link

6/13/2012 01:34:15 pm

Data mining process: sampling, exploring, modifying, modeling and assessing data.

Modifying Data:

Modify the data by creating, selecting and transforming the variables to focus the model selection process.

Fred Baer link

6/13/2012 01:35:31 pm

Data mining process: sampling, exploring, modifying, modeling and assessing data.

Modeling Data:

Model the data by using analytical tools to search for a combination of the data that reliably predicts a desired outcome.

Fred Baer link

6/13/2012 02:36:36 pm

Data mining process: sampling, exploring, modifying, modeling and assessing data.

Assessing Data:

Assess the data and models by evaluating the usefulness and reliability of the findings from the data mining process.

Not all patterns found by the data mining algorithms will be valid. The algorithms might find patterns in the training data set that are not present in the general data set. This is called over-fitting.

To address this concern, patterns are validated against a test set of data. The patterns learned on the training data will be applied to the test set, and the resulting output is compared to the desired (or known) output.

For example, a data mining algorithm that had been trained to distinguish fraudulent credit card transactions from legitimate ones would then be applied to the test set of transactions on which it had not been trained. the accuracy of the patterns can then be measured from how many credit card transactions are correctly classified.

John Harris

6/13/2012 03:12:14 pm

The large volumes of data collected from many different sources make the data-cleaning process more difficult.

The lack of proper data management and data-quality tools may completely derail what you can achieve with the faster and
advanced analytics tools available in the market today.

Gil Nabon link

6/14/2012 12:48:57 am

While data management professionals have a well-defined set of expertise around managing and organizing highly structured data and modeling and creating reports in SQL, those conventional skill sets don't translate well to the unstructured, flat-file world of big data, where command lines and NoSQL technologies are the core building blocks of most of the emerging platforms.

Larrry Wallace

6/14/2012 01:38:12 am

You need a good set of system administrators and solid practices around how to build out data analytics environments.

noSQL database is a must

Michael Walker link

6/14/2012 02:07:06 am

The discipline of developing the data analytics models (ie: predictive analytics) is not within the skill set of the average business user or even the traditional business intelligence (BI) data analyst.

A lot of the data is in a raw form, be it in documents, CRM, social media, Web logs or from sensor data, thus companies need access to experts (data scientists) who are versed in statistical and mathematical principles to build the advanced models that uncover patterns and actually make big data useful.

After the models are built and tested it can be distributed to business users in a format that is useful and productive.

Wendy Dorn link

6/14/2012 04:25:27 am

Most of the data is raw -- it's not something the average business user can read and get value out of.

There will always be a need for a skill set of people who know what to do with the raw information.

You have to build the acquisition of talent into the business case.

Michael Walker link

6/14/2012 04:45:05 am

To get the best insights, use and productivity gains from data analytics, I suggest hiring a team of data scientists - people with PhDs in statistics and mathematics - either full time employees of organization or as professional consultants.

Pablo Ortiz

6/14/2012 05:46:04 am

At many organizations, some users are trained in predictive modeling. Others, however, don't possess this level of expertise.

David Davis link

6/14/2012 06:07:25 am

Training was an integral part of the big data analytics strategy for our bank. We deployed the technology to do modeling and risk management for various loan portfolios.

Yet the training wasn't just about learning Hadoop skills or serving as a crash course in statistical science.

Rather, a considerable amount of time and energy went into acclimating the technical team so they were able to comfortably transition to a totally new way of managing data.

Peter Fox link

6/14/2012 06:12:57 am

Advanced data analytics is new technology that traditional and very conservative IT shops may be reluctant to implement.

You have systems administrators or database administrators who've built an entire career around a particular skill set, and then you thrust some new technology at them and say they have to learn it.

There are cultural challenges you have to deal with in terms of supporting the new model.

John Bender link

6/14/2012 06:23:33 am

I would strongly recommend data virtualization for any analytics program.

Benefits of Data Virtualization

http://www.rosebt.com/benefits-of-data-virtualization.html

Data virtualization is the process of offering data consumers a data access interface that hides the technical aspects of stored data, such as location, storage structure, API, access language, and
storage technology. Consuming applications may include: business intelligence, analytics, CRM,enterprise resource planning, and more across both cloud computing platforms and on-premises.

Data Virtualization Benefits:

● Decision makers gain fast access to reliable information
● Improve operational efficiency - flexibility and agility of integration due to the short cycle creation of virtual data stores without the need to touch underlying sources
● Improved data quality due to a reduction in physical copies
● Improved usage through creation of subject-oriented, business-friendly data objects
● Increases revenues
● Lowers costs
● Reduces risks

Data virtualization abstracts, transforms, federates and delivers data from a variety of sources and presents itself as a single access point to a consumer regardless of the physical location or nature of the various data sources.

Chris Mullins link

6/14/2012 06:30:31 am

Major cultural challenges to overcome to support the new data anlytics model.

The big benefits and rewards from better decisions at all levels of the firm - based on data and evidence rather than intuition - are clear to all of us in top management.

Getting IT and most in the firm to learn new ways and change their way of thinking is a tough nut to crack.

Jed Hopkins link

6/14/2012 06:56:00 am

Data Quality Tools Magic Quadrant 2011
http://www.rosebt.com/data-quality-tools-magic-quadrant-2011.html

Selecting the right set of data-quality tools is absolutely essential, because data quality is also a critical factor in data governance.

Some of the basic attributes of data-quality tools in traditional IT are:

Profiling
Parsing
Matching
Cleansing

These attributes are also critical in the big data world, but they also require tools to scale horizontally (though vertical scaling still matters to a certain degree) and that are fast enough to process large volumes of data.

Roger Miles link

6/14/2012 07:00:33 am

Many data-quality projects fail because organizations spend a lot of time cleaning up only their internal data.

However, big data encompasses both internal and external data, including public data sets used in certain cases (for example, government data used by the oil industry).

It is important to take a holistic view and select the right set of data-quality tools to meet the data-quality needs of the organization.

Many IT managers think the most expensive tools will help them reach their data-quality goals.

This thinking is flawed, and it is another reason for the failure of data-quality projects.

Bob Dent link

6/14/2012 07:13:18 am

We suggest you think very carefully about data quality and tools for data analytics projects. Unlike data quality issues in traditional enterprise IT, big data requires processes that are highly automated and also requires careful planning.

Unlike the traditional approaches to data quality, where it was always considered a project, big data requires data quality to be considered a foundational layer, which could determine whether an organization’s wealth of large data sets can act as a trusted business input.

If you are the person responsible for implementing big data platforms at your organization, we strongly suggest you give data quality a huge priority in your strategy.

Even though large vendors like IBM, SAP and Oracle are offering data-quality tools, there are some visionaries emerging:

Trillium Software System
Informatica Platform
DataFlux Data Management Platform
Pervasive DataRush
Talend Platform for Big Data

The above list is by no means exhaustive, but it offers a feel for the type of innovative solutions available to solve data-quality issues in the big data world.

Data Quality Tools Magic Quadrant 2011

http://www.rosebt.com/data-quality-tools-magic-quadrant-2011.html

Michael Walker link

6/14/2012 07:26:36 am

Data obesity is defined as the indiscriminate accumulation of data without a proper strategy to shed the unwanted or undesirable data.

At this time the cost of data storage very cheap. However, as we go further into the data-driven world, the rate of data acquisition will increase exponentially.

Even though the cost of storing all of that data may be affordable, the cost of processing, cleaning and analyzing the data is going to be prohibitively expensive.

Once we reach this stage, data obesity is going to be a big problem for all organizations.

The following are some of the problems organizations will face regarding data obesity:

Cost for processing, cleaning and analyzing the data. The additional overhead for data that doesn’t offer any insight to the organization will end up becoming a severe drag on financials in the future.

Data-governance issues. Data obesity makes data governance expensive, and it could lead to unnecessary headaches for the organization.

Data obesity coupled with poor data quality could have a devastating effect on the organization (data cancer).

Michael Walker link

6/14/2012 07:30:36 am

It is critical for organizations to have a data-obesity strategy in the early part of their big data planning, as it will help organizations save
valuable resources in the future.

Left undetected for a long time, data obesity could be deadly for organizations

Tom Leeds link

6/14/2012 07:49:12 am

Master Data Management (MDM)
http://www.rosebt.com/master-data-management-mdm.html

One partial solution to data obesity is carefully planning a rock solid Master Data Management (MDM) system in the beginning of the data analytics program.

MDM comprises a set of processes and tools that defines and manages data. MDM lies at the core of many organizations’ operations, and the quality of that data shapes decision making.

MDM can be defined as a set of policies, procedures, applications and technologies for harmonizing and managing the system of record and systems of entry for the data and metadata associated with the key business entities of an organization.

Recent developments in business intelligence and data analytics aid in regulatory compliance and provide more usable and quality data for smarter decision making and spending. Virtual master data management (Virtual MDM) utilizes data virtualization and a persistent metadata server to implement a multi-level automated MDM hierarchy.

MDM provides processes for collecting, aggregating, matching, consolidating, quality-assuring, persisting and distributing data throughout an organization to ensure consistency and control in the ongoing maintenance and application use of this information.

MDM seeks to ensure that an organization does not use multiple (potentially inconsistent) versions of the same master data in different parts of its operations and solves issues with the quality of data, consistent classification and identification of data, and data-reconciliation issues.

MDM solutions include source identification, data collection, data transformation, normalization, rule administration, error detection and correction, data consolidation, data storage, data distribution, and data governance.

MDM tools include data networks, file systems, a data warehouse, data marts, an operational data store, data mining, data analysis, data virtualization, data federation and data visualization.

Harry Moore link

6/14/2012 09:52:06 am

Virtual master data management is an automated, fast and easy approach to master data management - without requiring legacy tools such as SQL.

Customer, product, vendor, human relations, business unit, geography - any way you want to approach managing your data is easy and consistent. One system that will align all of these types of data.

Big Data & Decision Making Survey

Leave a Reply.

Rose Technology

Archives

Categories