What is Idempotency?

Think of your paycheck. You don't want a communications error between your employer and the payroll processor to prevent you from getting paid. If an error happens in that process, you want the communication to be re-sent, and when the re-send occurs it shouldn't cause you to get paid twice. Effectively, idempotency is the ability to re-send messages when needed without causing problems from duplicated processing. While you may not mind receiving your paycheck twice, you would feel differently if you were charged twice for a plane ticket.

We live in a world of interconnected systems. Our systems need the ability to re-send messages in the event of a failure, and systems receiving the communications should be capable of receiving the same message more than once without duplicating transactions. In the most general of terms, handling messages in an idempotent manner comes down to two concepts; knowing when a message needs to be re-sent, and knowing if an incoming message has already been processed. First let's look at some of the message flows that can occur in a REST-based system.

Point-to-Point Message Flows via REST:

Happy path:

When things go well, the message is sent once, processed once, and acknowledged once.

The code to handle the happy path is simply a single line of code invoking an async Send method.

Unfortunately, given enough time and enough traffic, you can be assured that something will eventually fail.

Ways it can go wrong:

The original message could fail during the send, because of a communication error or because the Consumer is down.

The message might be sent but have a failure in processing.

The Consumer may successfully report back a failure, or there may be a communication failure preventing the Producer from receiving a notification of success or failure.

In a point-to-point communication pattern like this, the message producer now has to be responsible for re-sending when anything other than the happy path occurs.

Point-to-point communication gets even more challenging when multiple consumers are interested in the same message.

Once the concept of multiple consumers is introduced, the logic for sending and re-sending messages reliably gets exponentially more difficult. What should the Producer do when a message fails for a single consumer. Does it repeatedly re-send the message to all the consumers until it's acknowledged by the failing consumer? Does it stop sending to the other consumers until the failing consumer finally acknowledges the message? Or, does the Producer keep track of which consumers were successful and which failed and send different content to each?

The problem gets even worse if you introduce the concept of ordered message delivery.

With this type of setup, you can quickly find yourself writing more code dealing with message delivery than with providing actual business features. Instead of doing that, we can solve the problem by changing our communication paradigm by using a system designed specifically for handling messaging and the related complexities.

A Better Approach:

If we go back to basics and consider the Separation of Responsibilities, our business code should be focused on providing business functionality. The capabilities around handling messaging and delivery rules have nothing to do with our core business logic. It is critical to the proper functioning of interconnected systems, but it's not part of our core competencies. Instead of writing message plumbing, we should instead focus on our business deliverables and offload message delivery concerns to a system designed specifically for that purpose. An additional benefit of using a messaging intermediary is we can increase system reliability since Message Producers can continue functioning and sending messages even when one or more Consumers are unavailable.

Various forms of messaging systems can act as intermediaries between message producers and consumers, and they range from Queues and SNS Topics to Enterprise Service Buses. One such option is Pulsar by Apache, which is a cloud-native message broker that addresses the shortcomings of some of the solutions that came before it.

According to their website (https://pulsar.apache.org/), "Apache Pulsar is a cloud-native, multi-tenant, high-performance solution for server-to-server messaging and queuing built on the publisher-subscribe (pub-sub) pattern." Instead of having point-to-point communication where each Producer knows about each consumer, Pulsar allows Producers and Consumers to communicate only with Pulsar, and Pulsar manages the reliable delivery of messages to consumers.

With this type of model, Pulsar is responsible for deliveries to the Consumers. If Consumer 3 doesn't successfully acknowledge a message, the Message Producer won't even know there was an issue, much less have to implement logic to try a re-delivery. All of that work can be offloaded to Pulsar. It is worth highlighting that this type of approach does not have point-to-point communication between systems. In other words, it uses asynchronous communication patterns.

We've Been Working Without a Message Broker Forever. Why Change Now?

Breaking up the Monolith

When working with monolithic software, code can communicate directly with other code dependencies via in-process calls. Communication within a monolith is easy, but the disadvantages of monolithic software begin to outweigh the benefits as software projects get larger and more complex. As we break functionality out into different services, we need another way for the services to communicate with each other in a reliable fashion. The moment we extract the first micro-service out of the monolith, we introduce inter-process communications and now the stability of the system as a whole is dependent on multiple services being up and running at the same time.

Interconnected Application Domains

Applications don't exist in a vacuum and need to communicate across different Application Domains, especially in large enterprises. In the past, a single business unit might have been able to operate in isolation, but as part of a larger organization, it becomes necessary for that business unit to share messaging and events with other parts of the organization. Not only do we need a way for messages to be transmitted within an application domain, we need to be able to communicate between domains. Message brokers facilitate this type of communication and can eliminate the need for message producers to change each time a new consumer needs to be added. Martin Fowler pointed out that message-based integration patterns allow us to reverse the direction of dependencies between Producers and Consumers. Upstream message Consumers can depend on messages generated by downstream Producers as opposed to Producers needing to know about specific Consumers in order to send messages.

Idempotent Consumers

I previously mentioned that Idempotency was about the ability of messages to be re-sent when necessary AND for message consumers to handle the receipt of duplicate messages without undesired side effects from duplicate processing. Much of the discussion so far was about the logical complexities of re-sending messages. A future post will show a code example with Consumer code capable of receiving the same message more than once while processing the duplicate messages "effectively once".

This is the first in a series of blog posts related to idempotency and how to handle it effectively.

Posted 27 July 2022 by Doug Ferguson, Director, Software Engineer, WSO Software, S&P Global Market Intelligence

S&P Global provides industry-leading data, software and technology platforms and managed services to tackle some of the most difficult challenges in financial markets. We help our customers better understand complicated markets, reduce risk, operate more efficiently and comply with financial regulation.

This article was published by S&P Global Market Intelligence and not by S&P Global Ratings, which is a separately managed division of S&P Global.