What is Apache Pulsar

In 2013, Yahoo felt the need for a platform where several tasks such as streaming, PubSub messaging, microservices, real-time data processing, and other high-performance tasks could be done. However, we already had Apache Kafka before the inception of Pulsar, but to overcome some of its limitations. Yahoo developed Pulsar. In 2016, it was open-sourced and contributed to the Apache Software Foundation (ASF); since then, it has become the default choice of many users.

What is Apache Pulsar

Apache Pulsar is an open-sourced, PubSub, that is, a Public subscribe messaging system. Its biggest advantage is that its architectural structure is designed in such a way that it can handle hundreds of billions of events daily. As it is a streaming platform too. It provides high-quality streaming; also, it is a multi-tenant, high-performance solution. Pulsar is highly scalable and can manage the most demanding data movement out there.

Apache Pulsar is unique because it runs with the support of two technologies. Apache BookKeeper and Apache ZooKeeper. Together they can provide high performance, low latency, and high throughput. Pulsar can easily simplify most complex tasks.

Features of Apache Pulsar

Because of its features, Pulsar is gaining popularity like a shot. Some of its key features are:

• Built-In schema registry

The biggest challenge faced by any messaging system is that producers and consumers are able to communicate in the same language. Well, because it decouples the producer and consumer, it helps them to change the format of the message they are sending or receiving. However, eventually, the application ends up broken! 

The solution to this problem is a schema registry. This helps producers and consumers to use messages with a compatible schema. Pulsar has a schema registry, and you just have to register the schema with a Pulsar topic. And it takes care of enforcing schema rules. 

• Built-in Geo-Replication

While traveling or to support disaster recovery, you need an option to replicate messages to remote locations. This will help you to work on a global level. And even when you are traveling, you can have the same working experience with Geo-replication. It can help you connect to the local cluster and still receive or send it to clusters around the world.

Pulsar comes with an in-built Geo-replication of messages. This will help you when you want to publish a message to a topic, and Pulsar will automatically replicate it to the configured remote geo-location. Moreover, no additional settings or configuration is needed.

• IO Connectors

To glue together the data-intensive systems such as databases, stream-processing engines, and other such messaging systems. It is one of the most important functions of a messaging system. However, as we know, it's common, so it makes sense to provide a common framework and connectors to make this easy. And that's what happens when Pulsar deals with its IO connectors. 

However, there are several ready-made connectors of Pulsar. Some of them are MySQL, Cassandra, Kafka, etc.

Benefits of Using Multi-Layered Apache Pulsar

Because Apache Pulsar has a segment-based architecture, therefore, at times, when the memory is maxed out, your incoming messages will not disrupt, and it will help you to work seamlessly without worrying about the space. Unlike other messaging platforms, where you have to either delete your old messages or replicate data to other discs. And replicating data is expensive and error-prone. 

In pulsar, things are pretty easy; when space is maxed out, you just have to add a new bookie, and it will automatically redistribute data to it. No rebalancing is required. 

Moreover, its architecture is designed with a multi-layer, in which each layer is scalable. Scaling in a pulsar is a simple and non-disruptive task. 

Best Uses of PubSub Messaging with Pulsar

These uses will help you to get the most out of Pulsar:

• Queries on high-data storage

You can store a lot of data on Pulsar as it will be very useful to run queries on it. And that too, when Pulsar is doing its job of sending and receiving messages. Pulsar can do such high-speed tasks. It is possible with Pulsar when it leverages the SQL query engine; Presto. To make it easier for you to perform SQL queries on the data stored in your topic. It does this by integrating itself with Presto. Even when it is offloaded into tiered storage, you can query the data. They bypass the queries of the broker, so it won't impact the ability of the Pulsar cluster to send and receive messages in real time.

• Coordinate Partitions with Performance

Apache supports both Partitioned and Non-Partitioned based messaging. In case you want it to perform a lower use case, you can go for Non partitioned, and when you want a high-performance use case, that is, stream heavy data or process heaving data on a single topic; you can go for Partitioned topic to take advantage of likeness in the processing. As requirements grow, you can add more partitions easily.

• Utilize non-performance messages as necessary

Regular messages are sent to Apache BookKeeper for storage on disk. And these messages are guaranteed to be delivered once regardless of the fact that the network is not working or even the failure of Pulsar itself. In some cases, these guaranteed deliveries are not required, and just delivery once is sufficient. And in such cases, Pulsar uses Non-persistent messages. These messages help in reducing the resources as we don't have to store data in discs. At the same time, it can still deliver high throughput and low latency.

Summing Up

As you have seen it's useful features, and that's the reason that it is gaining popularity rapidly. However, compared to other streaming and PubSub messaging platforms such as RabbitMQ and Apache Kafka, Pulsar is relatively new, which is why it is still in its growing stage. While its popularity has been growing exponentially, however, it still has some limitations, such as lesser documentation and a small community. 

Moreover, the Pulsar is better at performance. Although you should keep your requirements in check, it could be possible that you don't have to process heavy data, then using Kafka will be okay as per your requirements.