Hi,
Before anything else, I wish you all the best for 2021 to you and your loved ones. Let's hope together that 2021 will calm down the world’s pace of bad news. By the way, I'm currently reading Humankind and recommend it, perfect timing.
2021
For me, 2021 will see the birth of Infinitic. I'm working on it for months, and I'm now on the verge of releasing it.
As announced, thanks to Pulsar’s 2.7 release, I can use the native Kotlin serialization, simplifying the code base and giving a nice bump to performance. I've written this article about using Apache Pulsar with Kotlin.
For a couple of months, I've been following the Pareto principle, and the last 20% to do is much longer than expected (of course). For example, I've been working on a simple Java demo project since mid-December. One thing leading to another, I found myself working on:
Some updates on my API syntax to appear more natural in Java (reminder: everything is currently coded in Kotlin).
The process of releasing public Java packages. A process that is naturally more complex and cumbersome than it should.
An easy way for end-users to configure workers.
A state storage implementation through Redis (initially and eventually, Infinitic should use Pulsar stateful functions, but there are not production-ready yet).
An issue related to message batching interfering with key-shared subscriptions
Some reliability issues occurred when you emulated random crashes. Pulsar makes sure that no message is lost - ever - but the counterpart is that double processings can happen for some. (If you are curious, look at this article, based on Kafka, but the concepts are the same for Pulsar). These double processings led to bugs in the state machine.
The implementation of a decent internal log system.
An all in-memory implementation for end-to-end testings of the algorithm work.
I'm pleased to say that this is now all solved, and I do not have any known bugs anymore that prevent a first public release. The current features set of Infinitic is:
Solid reliability!
Task processing, with retry and cancel capabilities
Workflow processing, able to pilot:
Sequential executions of tasks and workflows;
Parallel (asynchronous) executions of tasks and workflows;
Get status, result, or waiting for the completion of any logical combination of asynchronous tasks or workflows;
Child workflows;
Inline tasks
What's next?
Now, my next focus will be on the documentation website for a first public release next week!