Data storage support

[xzycn]

Can we support Data storage in version 5 and open-source ?
I want to persist messages after restarting emqx.

[lafirest]

In the open source version, you can use the rule engine + webhook to do this. Data storage will be the included in the enterprise edition

[zmstone]

Hi @xzycn

The on-disk persistent session feature will be added in 5.1 (opensource edition), so there will not be a need for external databases to persist messages.

[xzycn]

@zmstone I have read the post yesterday that you will use RocksDB for this goal :)
What's the time about v5.1 to release?
Thank you for your great job 👍

[zmstone]

We aim to release it by the end of this year.
But no timeline commitment yet.

[xzycn]

@zmstone Hi, almost one year has passed from then, has this feature been released?

[chinlean]

Hi, I went through the v5.1.1 notes and didn't see this. Any updates on this feature?

[ieQu1]

Hello. Our benchmarks had showed unsatisfactory performance of the first iteration of the message persistence feature, so we went back to the drawing board.
Currently our team is working on a different approach. Some work-in-progress code can found there https://github.com/emqx/emqx/tree/master/apps/emqx_durable_storage #11592

[id]

@chinlean @xzycn we did implement the initial support last year, but unfortunately the performance was too bad to include it in any release, so we decided to drop it.

After that we have invested into new design, and right now working full speed on implementing persistence for sessions and messages based on RocksDB https://github.com/emqx/eip/blob/main/active/0023-rocksdb-message-persistence.md. Current plan is to release it in 5.3.

Apologies for lack of communications.

@ieQu1 FYI

[xzycn]

Stay tuning! Thanks for your job.

[delfme]

Also, from the website it seems that the feature would be added to the enterprise version only. Can you please change your mind on that? We would like to dismiss Vernemq in favour of Emqx but the lack of the feature is a deal breaker for us :-/

[ieQu1]

Message persistence will be available in the open-source edition.

[ennc0d3]

If I understand, the durable_storage (persistence MQTT) will be included in CE. What is the status ? Thanks, Naga

[ieQu1]

Hello,

Yes, it will be released in EMQX open source edition in v5.7 (planned release date is in May). The status is the following: session persistence has been included in 5.6 (as a hidden feature), currently we're working on the bugfixes before the public release.

[delfme]

Meantime, would it be possible for us to persist offline messages to Redis? We didnt find a plugin for the community edition, nor a doc related to it.

[zmstone]

Hi @delfme
There is no external storage support for offline message in v5.

[iosephmagno]

Hello @zmstone any way for us to test persist offline message with the opensource edition?