In-Memory Data Grids (IDG) allow organizations to collect, store, analyze and distribute large, fast-changing data sets in near real-time. Organizations are increasingly using IDG's for the efficient sharing of fast-changing data across multiple sites. IDG's provide the scalability and low latency required to enable applications to handle large workloads with fast responsiveness.
Data scientists need to be able to access and analyze fast-changing data quickly and easily - without regard to where it originates. The difference between high-value data science and good data science is increasingly about the ability to analyze larger amounts of data at faster speeds. Speed kills in data science and the ability to provide valuable, actionable insights to the client in a timely fashion can mean the difference between competitive advantage and no or little value-added.
For example, Rose financial clients use IDG's to collect and store fast-changing trading data for quickly analyzing and responding to emerging market trends. The significant reduction in time (from 60 minutes to 60 seconds) to access and analyze data improves decision-making, profitability and market competitiveness through increased performance in trading. Rose retail clients use IDG's for fast analysis of buyer patterns, purchase data and call-center communications to understand trends to improve marketing competitiveness, decision-making and profitability. Detecting and acting on consumer trends and competitors marketing and pricing immediately is critical in retail, especially the online retail space.
Organizations are demanding faster and easy access to information to make better decisions. IDG's enables immediate access to the right information which results in more informed decisions. Traditional Business Intelligence (BI) technology loads data onto the disk in the form of tables and multi-dimensional cubes against which queries are run. In-memory data is loaded into Random Access Memory (RAM) instead of hard disks. Thus, staff spends less development time on data modeling, query analysis, cube building and table design - and more time on high-value data science and business analysis.
Research shows that organizations using IDG's are able to analyze larger amounts of data at faster speeds than competitors. With in-memory tools, data available for analysis can be as large as data mart or small data warehouse which is entirely in memory. This can be accessed within seconds by multiple concurrent users at a detailed level and offers the potential for excellent analytics. The improvement in data access may be 10,000 to 1,000,000 times faster than from disk. It also minimizes the need for performance tuning and provides faster service for end users.
IDG's provide the following benefits:
1. Competitive Advantage. Organizations can make better decisions faster.
2. Speed. Improves time to find and analyze data to obtain valuable, actionable insights.
3. Better Decision-making. Organizations can improve the quality of their decision-making.
4. Productivity. Improved knowledge and business process efficiency increases profitability and reduces waste.
5. Customer Experience. Provides faster, more reliable service which can mean the difference between success and failure, especially in online transactions.
IDG's have the following characteristics:
1.The data model is distributed across many servers in a single location or across multiple locations. This distributed model is known as a "shared nothing" architecture and distribution is known as a "data fabric".
2. All servers can be active in each site.
3. All data is stored in the RAM of the servers.
4. Servers can be added or removed non-disruptively, to increase the amount of RAM available.
5. The data model is non-relational and is object-based.
6. Distributed applications written on the .NET and Java application platforms are supported.
7. The data fabric is resilient, allowing non-disruptive automated detection and recovery of a single server or multiple servers.
The Data Supply Chain: A Different Approach to Managing Your Company's Data
There's no argument that data is a corporate asset, but there's often disagreement on how to manage and address everyone's needs. Traditional data strategies assume that data is created, distributed, and consumed within a company's four walls. Today, companies are moving toward external applications and information providers to support their growing business demands. It's no longer sufficient to manage and track where data is created and consumed—we must also know how it moves and migrates.
In this keynote, Evan Levy will introduce the concept of the data supply chain, a new approach to managing a company's data assets. This approach expands the traditional corporate information life cycle to include the numerous data sourcing, provisioning, and logistical activities that are required to successfully manage a company's data.
The goal of Data Analytics (big and small) is to get actionable insights resulting in smarter decisions and better business outcomes. How you architect business technologies and design data analytics processes to get valuable, actionable insights varies.
It is critical to design and build a data warehouse / business intelligence (BI) architecture that provides a flexible, multi-faceted analytical ecosystem, optimized for efficient ingestion and analysis of large and diverse datasets.
There are three types of data analysis:
Descriptive (business intelligence and data mining)
Prescriptive (optimization and simulation)
Predictive analytics turns data into valuable, actionable information. Predictive analytics uses data to determine the probable future outcome of an event or a likelihood of a situation occurring.
Predictive analytics encompasses a variety of statistical techniques from modeling, machine learning, data mining and game theory that analyze current and historical facts to make predictions about future events.
In business, predictive models exploit patterns found in historical and transactional data to identify risks and opportunities. Models capture relationships among many factors to allow assessment of risk or potential associated with a particular set of conditions, guiding decision making for candidate transactions.
Three basic cornerstones of predictive analytics are:
Decision Analysis and Optimization
An example of using predictive analytics is optimizing customer relationship management systems. They can help enable an organization to analyze all customer data therefore exposing patterns that predict customer behavior.
Another example is for an organization that offers multiple products, predictive analytics can help analyze customers’ spending, usage and other behavior, leading to efficient cross sales, or selling additional products to current customers. This directly leads to higher profitability per customer and stronger customer relationships.
An organization must invest in a team of experts (data scientists) and create statistical algorithms for finding and accessing relevant data. The data analytics team works with business leaders to design a strategy for using predictive information.
Descriptive analytics looks at data and analyzes past events for insight as to how to approach the future. Descriptive analytics looks at past performance and understands that performance by mining historical data to look for the reasons behind past success or failure. Almost all management reporting such as sales, marketing, operations, and finance, uses this type of post-mortem analysis.
Descriptive models quantify relationships in data in a way that is often used to classify customers or prospects into groups. Unlike predictive models that focus on predicting a single customer behavior (such as credit risk), descriptive models identify many different relationships between customers or products. Descriptive models do not rank-order customers by their likelihood of taking a particular action the way predictive models do.
Descriptive models can be used, for example, to categorize customers by their product preferences and life stage. Descriptive modeling tools can be utilized to develop further models that can simulate large number of individualized agents and make predictions.
For example, descriptive analytics examines historical electricity usage data to help plan power needs and allow electric companies to set optimal prices.
Prescriptive analytics automatically synthesizes big data, mathematical sciences, business rules, and machine learning to make predictions and then suggests decision options to take advantage of the predictions.
Prescriptive analytics goes beyond predicting future outcomes by also suggesting actions to benefit from the predictions and showing the decision maker the implications of each decision option. Prescriptive analytics not only anticipates what will happen and when it will happen, but also why it will happen.
Further, prescriptive analytics can suggest decision options on how to take advantage of a future opportunity or mitigate a future risk and illustrate the implication of each decision option. In practice, prescriptive analytics can continually and automatically process new data to improve prediction accuracy and provide better decision options.
Prescriptive analytics synergistically combines data, business rules, and mathematical models. The data inputs to prescriptive analytics may come from multiple sources, internal (inside the organization) and external (social media, et al.). The data may also be structured, which includes numerical and categorical data, as well as unstructured data, such as text, images, audio, and video data, including big data. Business rules define the business process and include constraints, preferences, policies, best practices, and boundaries. Mathematical models are techniques derived from mathematical sciences and related disciplines including applied statistics, machine learning, operations research, and natural language processing.
For example, prescriptive analytics can benefit healthcare strategic planning by using analytics to leverage operational and usage data combined with data of external factors such as economic data, population demographic trends and population health trends, to more accurately plan for future capital investments such as new facilities and equipment utilization as well as understand the trade-offs between adding additional beds and expanding an existing facility versus building a new one.
Another example is energy and utilities. Natural gas prices fluctuate dramatically depending upon supply, demand, econometrics, geo-politics, and weather conditions. Gas producers, transmission (pipeline) companies and utility firms have a keen interest in more accurately predicting gas prices so that they can lock in favorable terms while hedging downside risk. Prescriptive analytics can accurately predict prices by modeling internal and external variables simultaneously and also provide decision options and show the impact of each decision option.