It aims to solve most of the pain points of Kafka making it easier to scale. It claims to be faster than Kafka and hence cheaper to run. It has many features and it is very flexible. Yahoo, Verizon, Twitter among other companies use it in production to process millions of messages. Pulsar is now an Apache top level project. Pulsar was created by Yahoo in 2013 and donated to the Apache foundation in 2016. Also, it is not quite as flexible and resilient as it could be. Although, it is relative easy to setup up, Kafka is difficult to manage and tune. Companies such Uber have created their own solution to overcome these issues.ДЄs you can see, most of the problems are related to the operational aspects. MirrorMaker Geo replication mechanism is problematic.Cluster re-balancing can impact the performance of connected producers and consumers.Working with offsets could be complicated if you just need a messaging system.You must plan and calculate number of brokers, topics, partitions and replicas ahead of time (that fits planned future usage growth) to avoid scaling problems, this is extremely difficult.It is possible to lose messages in case replicas are out of sync.Storage can become quite expensive, and although you can store data for a long period of time, it is rarely used because of the cost implications.No native multi-tenancy with complete isolation of tenants.Spinning off another broker means it has to replicate the topic partitions and replicas, which is time consuming. Scaling Kafka is tricky, this is due to the coupled architecture where brokers also store data.So, letsвЂ™ review some the pain points of Kafka. While Apache Kafka has always been friendly from a developerвЂ™s point of view, it has been something of a mixed bag operationally. Kafka is fast, easy to setup, extremely popular and can be used for a wide range or use cases. It has also many connectors to many system, check Confluent Platform for more details. Created by LinkedIn in 2011, it has spread widely thanks to the support of Confluent who has released to the open source community many new features and add-ons such as Schema Registry for schema evolution, Kafka Connect for easy streaming from other data sources such as databases to Kafka, Kafka Streams for distributed stream processing, and most recently KSQL for performing SQL-like querying over Kafka topics and much more. But letвЂ™s be clear, in terms of production usage, support, community, documentation, etc Kafka clearly surpasses Pulsar, and I would only consider Pulsar if most of the advantages discussed in this article hold true to your use case.

In this article, I will focus on the Pulsar advantages and give you some reasons why you should consider it over Kafka. So finally, I managed to have some time to play around with it and I did quite a lot of research. A quick search will show you that there is a current вЂњ warвЂќ between the two most famous open source messaging systems.ДЄs a Kafka user, I do struggle with some of the issues with Kafka and IвЂ™m very exited about Pulsar. Recently, IвЂ™ve been looking at Pulsar and how it compares to Kafka. Photo by Mikael Kristenson on Unsplash Introduction

Shards of war pulsar