The Evolution of Cost-Effective, Efficient Distributed Computing Systems: Where We Stand Today


by Paddy Krishnamoorthy, October 28, 2016
The internal combustion engines that power modern automobiles are wonders of efficiency and complexity. They’re capable of producing a maximum of power with a minimum of fuel.

It wasn’t always so.

The very first internal combustion engines relied upon a single cylinder to produce power. But it didn’t take long for engineers to realize the value of teaming together multiple cylinders in a single engine. Distributing the process of generating power across multiple cylinders enabled a single engine to produce more power with greater efficiency.

Producing More Computing Horsepower

Early attempts to engineer more computing capability followed a course of evolution similar to that of the early internal combustion engines. Just as the early engines relied upon a single large cylinder to produce power, pioneering computer scientists relied upon a single, massive computer to produce compute power – a very expensive approach, and very limited in top-end capability.

But grouping together a network of computers offered the promise of far more capability for much less expense. And thus the modern distributed computing systems have started gaining momentum.

The Problems of Distributed Systems

There’s no such thing as a free ride; every engineering concept comes with some unique problems to accompany the unique benefits offered. Distributed computing systems are no exception.

Distributed systems are inherently susceptible to three common problems:

  1. Compute power: Generating enough compute power to maximize performance can be a problem
  2. Storage: Teaming together a network of servers with locally attached storage creates its own complications
  3. Memory: Building up enough memory to have a low latency respones and to sustain the compute workloads can be also problem

Hadoop and MapReduce programming models provided solutions to the compute and storage problems. As a result, Hadoop became the go-to distributed solution for large enterprises.

But there was still a strong need for huge memory requirements and push for overall resource utilization.

Spinning a Better YARN

YARN (Yet Another Resource Negotiator) has helped to address the three problems with Hadoop. According to Hortonworks, “YARN is the architectural center of Hadoop that allows multiple data processing engines such as interactive SQL, real-time streaming, data science and batch processing to handle data stored in a single platform, unlocking an entirely new approach to analytics.”

In essence, YARN has made Hadoop more capable and efficient, and has helped to maximize the benefits of distributed systems.

Cluster Management with Mesos

Mesos offers another approach for solving the problems inherent with distributed systems. Developed at the University of California Berkeley, Mesos is open-source software designed with the goal of simplifying the process of managing applications in large-scale clustered environments.

According to the Apache Software Foundation, “Mesos is built using the same principles as the Linux kernal, only at a different level of abstraction. The Mesos kernal runs on every machine and provides applications (e.g. Hadoop, Spark, Kafka, Elasticsearch) with API’s for resource management and scheduling across entire datacenter and cloud environments.”

I think of Mesos as a data operating system. It has a distributor system kernel, just like any other operating system, but provides a different level of abstraction. Each server  the cluster has a number of CPUs, a certain amount of storage, and X amount of memory. And each server is registered to the Mesos cluster as an available resource.

So now we can build a cluster that has compute power, storage power, and memory power along with the ability to schedule the utilization of those resources with out any conflicts.

In essence, Mesos provides built-in resource management. You can run any workload on the cluster as needed. You simply tell Mesos how much storage, memory, and computing you need, and Mesos will allocate those resources for you.

From a resource utilization perspective, Mesos enables an enterprise to utilize a single large operating system upon which any number of applications can be run.

YARN or Mesos?

YARN and Mesos are both relative newcomers to the world of distributed computing; both have been available for just a couple of years. Together, YARN and Mesos represent the current pinnacle of the evolution of distributed computing systems.

But they also represent somewhat different approaches to solving the common problems of distributed systems. So which would be the better approach for your enterprise?

There’s really only one primary difference between the two: YARN has data storage built into it; Mesos does not. YARN’s HDFS provides a distributed file system. So your current data storage set-up may indicate whether YARN or Mesos would be more adaptable to your system.

It’s also important to note that a new trend in storage is emerging: cloud-based storage options, such as Amazon S3 or  Object based strorage like CEPH. The advent of cloud-based storage has signaled a new era in the evolution of distributed computing. Organizations can now use YARN or Mesos as a compute platform, and rely upon S3 for all storage needs. The latency impact of using S3 is debatable, but for certain batch-related use cases it could be a cost-effective alternative.

An Ongoing Evolution…

After more than a century, internal combustion engines continue to evolve at a rapid pace. Though distributed computing systems have evolved greatly over the last few decades, there’s no doubt that they, too, will continue to evolve at a rapid pace.

And there can also be no doubt that today’s distributed systems offer an important alternative to mainframe/super computers by providing competitive performance through more economical, easier-to-obtain hardware and software packages.

Only the passage of time will reveal what the future will hold for distributed computing systems. But thanks to the evolution of Hadoop, YARN and Mesos, today’s solutions represent quite a leap over the early days of distributed computing systems.


Subscribe to receive more blogs like this from RCG

Read more about our Big Data solutions.