Infinitic Update 3
Impact of Apache Pulsar 2.7 release on the development of Infinitic
Hi,
This update is a bit technical. I describe the impact of the recent Apache Pulsar release on the development of Infinitic. If you do not want to read all the details, note that my objective is now to have a working version by the end of the year ;)
Pulsar 2.7 release
Streamnative announced today the release of Apache Pulsar 2.7. I have to say that I’m impressed by the team behind Pulsar and how vibrant the community is. This release brings at least 3 important improvements for Infinitic:
Ability to use a custom serializer
Pulsar lets us send and receive java objects from its topics using a default Avro serializer. Why not sending just binary data? Because the schema describing the message is discovered during the serialization process, and Pulsar can check if this schema is compatible with previous ones. This guarantees your messages' type-safety and prevents errors that you discover only too late in production.
The issue for Infinitic is this serialization/deserialization process was not working for Kotlin classes (I opened an issue), forcing us to implement some dirty hacks. Now that using a custom serializer is possible, we should be able to use the recently released Kotlin serialization plugin. This will both clean our codebase (meaning fewer bugs) and make it much more efficient.
Transactions (developer-preview)
From the announcement:
Transactional semantics enable event streaming applications to consume, process, and produce messages in one atomic operation. With transactions, Pulsar achieves the exactly-once semantics for a single partition and multiple partitions as well.
For example, imagine that our engine receives a message, processes it, and as a response sends two messages on different topics. What would be the state if something breaks in the middle of this? Maybe only one message was sent; maybe the initial message won’t be processed again as it should. Having atomic operations prevents creating - hard to debug - inconsistent states in such situations. It makes Infinitic even more reliable.
Key-Shared Functions
Having functions with key-shared subscriptions were required to use functions instead of regular workers for Infinitic’s engines. Using functions was a big part of Infinitic’s initial vision to minimize the infrastructure overhead for future Infinitic users.
Still Missing in Pulsar
Function states are still buggy and in developer preview - it means that Infinitic still requires an external database today. Hopefully, this will be fixed for 2.8, where states could be stored inside Pulsar’s Bookkeeper.
It is still impossible to add indexes to topics - it means that Infinitic’dashboard still requires an external database today. I have less visibility when this could be possible, but it is a smaller inconvenience as it does not impact workflows processing - only dashboards.
What’s next?
I now have to validate those new features and integrate them into the Infinitic codebase. It’s a big step forward to realize the initial vision - and my objective is now to have a working version including those elements by the end of the year.
That’s all for today. As usual, if you have questions, do not hesitate to reply to those emails; I’d happily discuss them.