The internal combustion engines that power modern automobiles are wonders of efficiency and complexity. They’re capable of producing a maximum of power with a minimum of fuel.
It wasn’t always so.
The very first internal combustion engines relied upon a single cylinder to produce power. But it didn’t take long for engineers to realize the value of teaming together multiple cylinders in a single engine. Distributing the process of generating power across multiple cylinders enabled a single engine to produce more power with greater efficiency.
Early attempts to engineer more computing capability followed a course of evolution similar to that of the early internal combustion engines. Just as the early engines relied upon a single large cylinder to produce power, pioneering computer scientists relied upon a single, massive computer to produce compute power – a very expensive approach, and very limited in top-end capability.
But grouping together a network of computers offered the promise of far more capability for much less expense. And thus the modern distributed computing systems have started gaining momentum.
There’s no such thing as a free ride; every engineering concept comes with some unique problems to accompany the unique benefits offered. Distributed computing systems are no exception.
Distributed systems are inherently susceptible to three common problems:
Hadoop and MapReduce programming models provided solutions to the compute and storage problems. As a result, Hadoop became the go-to distributed solution for large enterprises.
But there was still a strong need for huge memory requirements and push for overall resource utilization.
YARN (Yet Another Resource Negotiator) has helped to address the three problems with Hadoop. According to Hortonworks1, “YARN is the architectural center of Hadoop that allows multiple data processing engines such as interactive SQL, real-time streaming, data science and batch processing to handle data stored in a single platform, unlocking an entirely new approach to analytics.”
In essence, YARN has made Hadoop more capable and efficient, and has helped to maximize the benefits of distributed systems.
Mesos offers another approach for solving the problems inherent with distributed systems. Developed at the University of California Berkeley, Mesos is open-source software designed with the goal of simplifying the process of managing applications in large-scale clustered environments.
According to the Apache Software Foundation2, “Mesos is built using the same principles as the Linux kernal, only at a different level of abstraction. The Mesos kernal runs on every machine and provides applications (e.g. Hadoop, Spark, Kafka, Elasticsearch) with API’s for resource management and scheduling across entire datacenter and cloud environments.”
I think of Mesos as a data operating system. It has a distributor system kernel, just like any other operating system, but provides a different level of abstraction. Each server the cluster has a number of CPUs, a certain amount of storage, and X amount of memory. And each server is registered to the Mesos cluster as an available resource.
So now we can build a cluster that has compute power, storage power, and memory power along with the ability to schedule the utilization of those resources with out any conflicts.
In essence, Mesos provides built-in resource management. You can run any workload on the cluster as needed. You simply tell Mesos how much storage, memory, and computing you need, and Mesos will allocate those resources for you.
From a resource utilization perspective, Mesos enables an enterprise to utilize a single large operating system upon which any number of applications can be run.
YARN and Mesos are both relative newcomers to the world of distributed computing; both have been available for just a couple of years. Together, YARN and Mesos represent the current pinnacle of the evolution of distributed computing systems.
But they also represent somewhat different approaches to solving the common problems of distributed systems. So which would be the better approach for your enterprise?
There’s really only one primary difference between the two: YARN has data storage built into it; Mesos does not. YARN’s HDFS provides a distributed file system. So your current data storage set-up may indicate whether YARN or Mesos would be more adaptable to your system.
It’s also important to note that a new trend in storage is emerging: cloud-based storage options, such as Amazon S3 or Object based strorage like CEPH3. The advent of cloud-based storage has signaled a new era in the evolution of distributed computing. Organizations can now use YARN or Mesos as a compute platform, and rely upon S3 for all storage needs. The latency impact of using S3 is debatable, but for certain batch-related use cases it could be a cost-effective alternative.
After more than a century, internal combustion engines continue to evolve at a rapid pace. Though distributed computing systems have evolved greatly over the last few decades, there’s no doubt that they, too, will continue to evolve at a rapid pace.
And there can also be no doubt that today’s distributed systems offer an important alternative to mainframe/super computers by providing competitive performance through more economical, easier-to-obtain hardware and software packages.
Only the passage of time will reveal what the future will hold for distributed computing systems. But thanks to the evolution of Hadoop, YARN and Mesos, today’s solutions represent quite a leap over the early days of distributed computing systems.
1. Hortonworks https://www.cloudera.com/products/hdp.html
2. Apache Software Foundation https://mesos.apache.org/
3. CEPH https://www.ceph.com/en/