Print

Managing Big Data: Solutions for storage and cost optimization

Trends and Developments , November 29, 2013

A data centre is the most important asset for an enterprise as it affects every aspect of decision making and workload management. However, establishing and maintaining a data centre facility has significant financial implications. Further, rising real estate costs, especially in metros and major cities, pose a challenge for organisations that are looking to set up data centres. Besides the huge investment in physical infrastructure, rising energy costs are the biggest challenge for the data centre industry. Company executives are, therefore, faced with issues related to demanding audit and regulatory requirements for data security; asset inventory; and the financial and environmental implications of running the data centre.

Data centre usage is currently undergoing a change due to the increasing use of cloud infrastructure (that is, outsourced or external), services and applications. According to Cisco, nearly two-thirds (64 per cent) of all data centre traffic will be processed in cloud facilities by 2016, rather than in traditional (that is, on-premise) data centres. In 2011, the estimated split was 61 per cent traditional and 39 per cent cloud. As companies weigh their IT options, the cloud offers an attractive alternative to building capital-intensive data centres. However, centralisation of the cloud with a few providers – as opposed to highly distributed data centres contributing smaller service capabilities – is an issue.

Another trend which has impacted data centre usage is “big data”. While big data has largely benefited internet service providers and application developers, it has led to privacy-related worries for both individuals and corporates. Besides, providing storage facilities for the large volume of information is a concern.

As enterprises struggle to resolve these issues by using various security and storage solutions, they need to undertake a cost-benefit exercise. In order to address the most important data centre inefficiencies, three types of costs need to be reduced. These are related to the low utilisation levels of servers resulting in server sprawl, software licences and management of these systems. The common practice to reduce the first two types of costs has been to consolidate and virtualise the servers, while the third aspect of the cost is mostly taken care of by simplified management and control. While these solutions are effective, they involve several challenges.

Enterprises have realised cost savings and flexibility benefits by virtualising their servers. Virtualisation enables organisations to make more efficient use of the data centre’s hardware. For the majority of the time, physical servers in a data centre are lying idle. By implementing virtual servers on the hardware, the organisation can optimise the use of its CPUs. In an event where an organisation decides to change the hardware, the data centre administrator could simply move the virtual server to the newer, advanced hardware, achieving improved performance at a small cost. However, regardless of the benefits of virtualisation, the increasing adoption of virtual servers is putting pressure on traditional data centre infrastructure and storage devices.

The previous virtualisation models used local storage in the physical server, making it impossible for administrators to move a virtual machine from a physical server to the one with a more powerful CPU. The introduction of shared storage  – network-attached storage or a storage area network – to the virtualisation hosts solved this problem, thereby introducing the ability to stack on several virtual machines. This configuration eventually evolved to today’s server virtualisation scenario, where all physical servers and virtual machines are connected to a unified storage infrastructure. This has led to data congestion in data traffic.

Moreover as, the number of virtual machines and data intensity increases going forward, the storage architecture must be improved.

 Emerging solutions and technologies

IT service providers will face challenges in achieving lower construction costs, higher computing power per watt, lower latency for users, and a smaller carbon footprint, besides in ensuring improved management of these complex systems.

Design and storage

The moves towards virtualisation and cloud computing and the emergence of big data have been impacting the way servers are being designed for various storage requirements. Some of the emerging solutions are as follows:

Microservers: For certain types of high-volume, low-compute power workload – such as web page serving, search engine query execution, or parallelised data processing tasks – a new type of server, the microserver, may occupy less data centre space and consume less power than the traditional Xeon- or Opteron-based enterprise server.

Solid-state storage: Fast, low-power, physically resilient solid-state storage is common in client devices – the flash memory in smartphones, the solid-state drives (SSDs) in tablets and ultrabooks, etc. SSD arrays are also increasingly incorporated in the data centre, where their performance and power efficiency have been high. However, as with microservers, solid-state storage is not universally applicable in this environment. While currently this technology is highly priced, economies of scale will drive down costs with a large number of companies entering this space and increasing research.

Software-defined data centres and OpenFlow: Software-defined networking (SDN) is a new field, within which OpenFlow has emerged as a leading specification for enabling it. As SDN gains traction, vendors and enterprises will adopt a three-tiered architecture. The first tier will involve the physical network equipment, including Ethernet switches and routers. The middle tier consists of controllers that initiate and terminate traffic, leveraging a library of enterprise-wide information about capacity and demand from the networking gear that shuttles the traffic. The top tier will involve applications to direct security, management and other specific functions through the controller. A controller-based load balancing application could automate the movement of workload among virtual machines by using the controller’s library of data on the capacity of individual network devices.

Modular data centres: Establishing traditional custom-built data centres involves high cost and time requirements (and expand when capacity is limited). Pre-fabricated modular data centres – often fitted into standard shipping containers containing servers, storage, networking and cooling equipment – allow companies to rapidly build the required capacity with the ability to deploy a higher number of modules.

 Automation

Data centres are increasingly relying on automation as this can facilitate service provision and significantly reduce the system management time and costs. Integrated systems should be included at the factory based on the best practices. One of the key benefits of this is that the system once delivered to the customers would need only a few hours rather than weeks to become operational. This translates to faster provisioning and lesser time-to-market for customers. Furthermore, ordering pre-integrated systems can reduce the number of months from a typical procurement cycle. It will also help businesses put their IT staff to more important work.

 DCIM tools

In order to gather and make sense of the large volume of data, and convert it into plausible information for decision making, the industry is turning to data centre infrastructure management (DCIM) tools.

DCIM tools go beyond the outdated method of storing information in separate, vendor-proprietary, software silos that offer limited reporting capabilities. While overseeing all parts of the data centre’s operations may be complex, DCIM is designed to make device and system monitoring easier, flexible and more customisable across a range of user requirements. These can help in real-time monitoring, performance management, capacity planning, and energy efficiency.

In conclusion, while both the vendor and operator communities have progressed in terms of providing data centre solutions and services, the proliferation of applications, data and platforms such as mobile devices will continue to pose new challenges. As the volume of data will continue to grow, it will strain the data centre’s processing power and capacity. Therefore, the coming years are likely to witness major technology innovations in the data centre space both in terms of cost optimisation as well as efficiency enhancement.

 
 

Copyright © 2010, tele.net.in All Rights Reserved