What’s new since 0.9.4
It has been a while since I released an Infinitic newsletter. 6 months actually - you can not say I’m cluttering your mailbox! I should have been given news more often, but I’ve been pretty busy. Infinitic reached version 0.11.2 - here is what happened since 0.9.4:
Workflows state can be stored in MySQL
This is the first significant contribution coming from external contributors :) Until now, Redis was the only store proposed to store states of workflows; now you can use MySQL. To do so, just provide this configuration to your workers:
storage:
mysql:
host: 127.0.0.1
port: 3306
user: root
password:
database: infinitic
Backward compatibility ensured
Infinitic now has tests to ensure that all versions have backward-compatible schemas since 0.9.0. This ensures that messages already in Pulsar - and binary states stored in databases - can be handled when you upgrade your Infinitic version.
As it’s a backward compatibility only, you can still have issues if you mix workers/clients of different versions or if you try to roll back Infinitic after an upgrade. I’m thinking of introducing a full-compatibility policy before the first stable release.
Timers can be manually completed
When using timers, please check that topics containing delayed messages have a TTL covering the duration of your timers. Per default, Infinitic now sets a one-year TTL in those topics (whose name contains “delay”). Make sure that your Pulsar cluster allows topic-level settings.
Be careful; if your timer’s duration is longer than the TTL, the timers won’t be fired. For those situations, Infinitic now proposes a `completeTimers` command to complete the running timers manually when needed.
Improved workers configuration
Timeout and retry policies are now easier to configure at the worker level. Those policies can now be independent of services/tasks implementation.
Services can also be manually registered to workers. This allows dependency injections if needed.
Default settings for Pulsar’s consumers and producers can now be provided.
Workflow versioning
Infinitic now lets you version your workflows. Workflow versioning is a tricky subject, and I waited a bit on this one to make sure to provide the simplest API.
The easiest way to update a workflow today is to have the new instances use the latest version while the current instances continue to run the version they were started with. This is now possible with a simple configuration of workers.
What’s next for Infinitic?
I’d like to mention here a few things that I believe are worth discussing before reaching a stable 1.0 version.
Use of protobuf - RPC framework
At its core, Infinitic is also an RPC framework, allowing distributed executions of remote services. By pure ignorance, I did not consider the ongoing works in this area (e.g. gRPC), when developing the foundations of Infinitic. I believe today that Infinitic should evolve as a generic Pulsar-based RPC framework. One way to follow this direction is to start using Protobuf for declaring services and their input/output and to code an Infinitic protobuf plugin to generate stubs.
I’m also looking at other emerging initiatives such as CloudEvents and AsyncApi - but the benefits for Infinitic seem less obvious - but I’m curious to hear your thoughts about them.
State storage
Since its beginnings, I wanted Infinitic to be usable with only a Pulsar cluster running - without having to add any other pieces of infrastructure. This vision is not fulfilled yet, as we still must have databases to store states. This should be temporary as those states were supposed - eventually - to be managed by Pulsar stateful functions. But it’s not that clear anymore:
stateful functions seem to have a hard time reaching a “production-ready” label
stateful functions would not be suitable for multi-region Pulsar deployment
I wonder if Infinitic should follow an event-sourcing pattern instead of relying on storage. These are preliminary thoughts, and I’m curious to hear if you have opinions on how (and if) to do this.
Dashboarding
Infinitic already has a dashboard focused on monitoring its workers and topics. It would be great to extend it to be able to display the state of workflows. Initially, I was hoping not to have to add dedicated databases for requesting events. I believe I should now build connectors and interfaces to work with databases such as elastic or Postgres.
By the way, the excellent library Kweb I use to build dynamic UIs without javascript just reached the 1.0 version. I encourage you to test it.
Observability
This one is probably a quick win - integrating OpenTelemetry and/or Micrometer into Infinitic. Contributions are welcome ;)