The "big data" technology vendor market is ripe for consolidation. The myriad of vendors and technologies is causing market confusion. Matt Turck created a nice visualization entitled "Big Data Landscape v. 3.0" (see picture above) showing the tangled mess. Further, the vague definition of "big data" and how to measure return on investment (ROI) is creating skepticism about the true value of this new data technology.
While many vendors received ample venture capital, it is unknown how many are profitable or have a realistic chance of becoming profitable in the future. Also unknown are sales growth rates and whether customers actually attain or perceive strong ROI. If customers perceive they have wasted millions of dollars on technology with little demonstrable value, most of these vendors and technologies are doomed.
Considering we are in the pre-industrial age of sophisticated data management and analytical technology, it is reasonable to surmise that only a few of the vendors will become profitable on their own and most will either be acquired or die a slow death. Predicting winners at this time is difficult if not impossible considering a high causal density environment with numerous variables.
The data science question is: how can we collect and analyze data to measure vendor technology effectiveness and true ROI for customers?
Dimensional Research conducted a survey of business and IT leaders on the drivers of cloud adoption. Key findings include:
• Cloud adoption is driven by multiple factors
- CIOs report a wide range of reasons for adopting cloud applications, including compliance requirements
(58%), better value (53%), and competitive advantage (51%)
- Business executives cite one key driver for choosing cloud applications, better value (80%)
• Employees like using cloud applications
- 79% of all participants report that employees believe experience with cloud applications is beneficial
- 95% of CIOs say that IT employees want to gain expertise with cloud applications
- 83% of CIOs have no problem finding technical help for cloud applications
• Outdated on-premise software is common
- 61% report critical applications that have not been updated recently
- 14% have business critical software that has not been updated in over four years
- 28% of CIOs with current maintenance requirements for business-critical software lack confidence that they
are in compliance
Mobile Devices Battles: Cearley said companies need to prepare for heterogeneity, with lots of different devices, including Bring Your Own Device (BYOD) policies. He talked about a client that had 18,000 PCs but 33,000 mobile devices. No one platform, no technology will dominate, he said. Microsoft won't disappear, but it won't lead as much as it has in the past few years. Microsoft is making a big bet with Windows 8, he said, setting up a "new era" that will take some time to get traction in the enterprise. Gartner expects Microsoft will gain about 20 percent share in smartphones and tablets, mostly coming out of Android market share, but Cearley said the jury is still out on that depending in part on how well Microsoft executes over the next few years.
Mobile Applications & HTML5: Both native and HTML5 applications will be important, as well as hybrid and special apps. Companies need to understand what works best for them and their target users. Cearley expects HTML5 will be increasingly more important for a wide variety of needs, but "no client" (browser-only or Citrix-like solutions) are still significant in some cases where security is important but native apps may be better in consumer-facing applications.
He talked about the importance of having experiences that flow to where you are, and new ways of thinking about how you design applications, using new features and cloud-client architecture as opposed to client-server environments.
Personal Cloud: This is new on the list this year, as Cearley expects that the notion of a personal computer will be replaced by a notion of a personal cloud, including all the services an individual uses. He said users will increasingly be focused on personal services as well as business services. Consumers will be increasingly centering their digital lives in a personal cloud, defined by features such as cloud storage, sync across devices, and….
Enterprise App Stores: These will become strategic for governing cloud and mobile use in a world driven by consumers who pick their own devices. Users want choice, while IT needs license management and distribution, as well as reporting. He talked about how many of the mobile device management (MDM) vendors have added such features, but said portals are becoming increasingly important for governing cloud as well as mobile use. IT will not have complete control, he said, but will instead offer more capabilities individual users and departments can choose from. He expects tactical enterprise app stores will become more mainstream in 2014 and later with packaged and portal options. Even later, mobile and cloud solutions will converge.
Internet of Things: This is "here now," Cearley said. Over half of all Internet connections today already are "things," ranging from remote sensors to cameras and building and infrastructure management. There will be tens of billions of permanent connections and hundreds of billions of intermittent connections between devices and the Internet. It soon will cost only pennies to add connectivity to any device.
As part of this, the concept of "operational technology" and IT "information technology" may well be converging. In many organizations, especially those with lots of devices, there will be a single executive who will oversee all Internet-connected entities. This will result in new business models, such as a digital supply chain, usage-based insurance models or taxes, or a "smart city" with devices informing municipal services. This will impact more organizations than are currently thinking about it, and eventually, he says, the Internet of things will become "the Internet of everything."
Hybrid IT & Cloud Computing: Cloud computing has been on the list for years, but now is augmented by the concept of "hybrid IT," where cloud will increasingly be hybrid, managed, and brokered. Cearley said the key issues are: strategic models for cloud service consumption; understanding when you should adopt cloud technology; building private cloud environments; managing hybrid cloud; adopting new application design and architecture to the cloud; and turning externally facing services into cloud services. The big new focus here is on the third area: taking private clouds and building a management platform for them, and then using that to manage both internal and external services, which helps IT become a broker for such services.
Strategic Big Data: Cearley continued to define big data as dealing not just with volume, but also with variety, velocity, and the complexity of data an organization is dealing with. He talked about managing both internal and external data, and talked about how technologies such as Hadoop may be a big part, but only a part, of it. Going forward, this will move beyond isolated projects and he suggested that IT departments start viewing it as a transformational architecture, replacing the homogenous relational database model with a heterogeneous fabric. In the long run, big data needs to be incorporated into an overall information strategy, he said.
Social data will be an important component of this data for many organizations. Cearley promoted the concept of a logical data warehouse that federated data from lots of sources, as opposed to the "enterprise data warehouse" concept that was widely held as a model in previous years.
Actionable Analytics: The key focus of big data is to provide actionable ideas to the business, Cearley said. Driven by mobile, social, and big data forces, analytics will need to change to embrace concepts such as moving from offline analytics to in-line embedded analytics, and from explanatory analytics to predictive analytics. In the next year, a big focus will be on developing real-time operational intelligence, perhaps delivered to mobile devices out in the field, he said. He talked about query and analytical engines such as IBM's Watson and Apple's Siri, where the analytical engine is just part of day-to-day operations and invisible to the consumer.
Mainstream In-Memory Computing: In-memory enabled applications of all sorts are becoming more mainstream, and in-memory features will be embedded within many of the applications we buy, such as SAP's HANA and other applications, ranging from websites to risk assessment. This seems particularly important for real-time analytics. One client is building an in-memory situation with 200TB to do real-time risk analysis and fraud detection, Cearley said.
Integrated Ecosystems: Finally, integrated systems will be more important, driven by needs for simplification, optimization and security. This is being seen in three areas, he said. Appliances are easier to deploy, manage, and configure, but no one appliance does it all, so they are always balanced with other solutions. Driven by cloud computing and things like Salesforce.com's AppExchange and Microsoft's marketplace, brokerages and marketplaces will become more important, with more vendors adding such services. Finally, end-to-end ecosystems, such as Apple's control of apps and the App Store, are becoming more important in the mobile world.
The bottom line, Cearley said, is that mobile, social, information, and cloud will combine to impact all of these trends and IT organizations as a whole.
Gartner's David Cappuccio outlined 10 key IT trends for the next five years at the 2012 Gartner Symposium/ITxpo.
Organizational entrenchment and disruption: By 2014, 30% of organizations using SasS will revert back to on-premise due to poor service levels. There’s significant growth in IT complexity. Faster change cycles. Shorter development timelines. Reduced budgets. We need 24/7/365 global IT support. End users are driving IT; end users demanded access to iPads. Same with iPhone, and other smart phones. End-users are driving IT to make change. There’s also a “skills shift,” with many people retirees, and new set of skills required of newer employees.
Software defined networks: A new way to operate networks, in which control of the networks moves into an OS. It moves control from individual devices to a central controller. Allows configuration of the network from one place. “Think of it as virtualizing your network,” he says. Location of physical data center no longer is relevant, creating completely virtual environment. It reduces the time require to provision new resources. Work loads are crossing data-center boundaries. There is potential for significant organizational disruption.
Bigger data and storage: By 2015, big data demand will generate 1 million jobs in the Global 1000, but only a third will get filled due to shortage of talent. Seeing 30%-60% compounded growth in data depending on the organization. Audit, archive and recovery are increasingly complex. Analytics and pattern recognition are key. Seeing new specialized ARM-based servers to do specialty analytics. Get more performance in smaller footprint, with reduced power requirements. He notes that 15%-20% of current servers are doing nothing at all. He says most of the storage growth has been on premise.
Hybrid cloud services: Composed of services from multiple providers. combination of private and public clouds. Use cloud as extension of IT. Gartner thinks private clouds improve agility and will dominate. People are looking at the cloud as a way to accelerate business growth, particularly mobile apps. You could end up with hybrid environment with dozens of specialty providers. It’s about increasing capability and/or capacity. He says that “hybrid data centers will be in your future.” You can move non-critical work to the cloud to free up space. Result can be incremental operating expense growth, but long-term capital spending deferral.
Client and server architectures: One size does not fit all. One OS does not fit all. Form factors are not static. You have to let tablets in. Forced end-user standardization does not work; let me people do what they want within reason. Windows 8 will in your organization, but will not be full replacement for Windows 7 or XP or whatever you are using now. Office on tablets makes no sense unless you like typing on glass. There are other apps that make no sense on notebooks. Users force IT to do wireless networks, and instant messaging, and now tablets and smartphones. You need to decide what to do about Office; “The days of the monolithic suite are going away.
Internet of things: Cheap, small devices. Everything will have a radio and GPS capability. Self-assembling mesh networks. Location aware. This all creates the always on society. All of these things have an IP address and can be tracked. Most new cars being enabled for social. Street lights are being networked. Devices proliferating everywhere. It’s not a single technology, it’s a concept. Driving the trend are things like embedded sensors, image recognition, augmented reality, near field communication. The result is situational decision support, asset management, more transparency. Many, many business opportunities with the Internet of things. But it all adds to complexity of IT; brings more fo the business into IT.
IT appliance madness: Proliferation of point solutions, which are easy to deploy, with embedded OS, and locked down environments. They contribute to the complexity issue. Now seeing more virtualized appliances, which again adds to complexity. Some appliances are for specific workloads. Sometimes with entire embedded software stack. Inventory monitoring. Security monitoring. Easy to deploy, easy to forget about.
Operational complexity: By 2014, employee devices will be compromised by malware at 2x the rate of corporate-owned devices. For every 25% increase in functionality of a system, there is 100% increase in complexity. Cisco 6500 Switch has 2,390 pages of installation and reference information. Oracle 10g database has 1,677 parameters. With Exchange on VMware there are 115 performance/capacity settings.
Virtual data centers: Ratio of virtual to physical servers is now about 11-to-1. Virtualization creates inexpensive resources. Can provision sever in minutes. Resources are distributed to workloads. Can be distributed across data centers and geographies. New focus on distributing specific workloads, rather than focus on physical servers. Seeing segmentation of legacy workloads and new applications. Complexity increases resolution times and masks ownership of issues. Workloads and issues are no longer confined to a known environment. Enable staff innovation, the help drive staff retention – give them more to do. Cloud services and hybrid environments exacerbate the support issue.
IT Demand: By 2017, 40% of enterprise contact information will have leaked on to Facebook via employee mobile devices. Server workloads growth 10% a year. Network bandwidth demand growing 35%. Storage capacity, 50%. Power costs growth, 20%. Throwing more capacity at demand is not the solution; you need to optimize capacity in new ways: virtualization, data deduplication, etc. Over 1.5 billion Web pages are accessible, 450,000 iPhone apps, over 200,000 Android apps, 10,500 radio stations, 5,500 magazine. All drives demand for IT.
Gartner’s recommendations include: