A Comparative Study on Performance and Resource Utilization of Real-time Distributed Messaging Systems for Big Data

Abstract

In the past few years, data continuously increase and in various forms called Big Data. Besides, data analytics play an important role more and more in most organizations. From these reasons, many efficiently distributed messaging systems have been introducing to handle Big Data in real-time. However, choosing the appropriate and efficient methods and tools to transfer Big Data is still challenging. Therefore, this paper purposes of comparing the architecture, performance, and resource utilization between Apache Kafka which is one of the favorite tools for Big Data and Apache Pulsar which is similar to Kafka and become one of the latest tools for big data. After we implemented both systems in the same environment, the results show that Pulsar outperforms in throughput, latency, and average resource utilization especially when the size of messages is small (such as 1 KB and 1MB).