[feat] Mqtt v5 support

[muka]

Hi
I am wondering if anybody has started taking a look at MQTT v5 specs. Is there someone working on that or would like to work on the implementation?

Here some task as a reference mqttjs/MQTT.js#651

Thanks

[mcollina]

I have absolutely no time for that. However, yes it would be fantastic (I would start with MQTT.js first).

[muka]

OK thanks,
so I may create a sibling issue there and eventually attempt to list the new protocol features as a checklist?

[mcollina]

Go ahead.

[robertsLando]

https://github.com/mqtt/mqtt.github.io/wiki/Differences-between-3.1.1-and-5.0

[robertsLando]

@muka Are you interested to help implementing v5 support in Aedes too?

[muka]

Hi @robertsLando, yes I can commit some free time during the week. Ping me somewhere for an alignment! Thanks

[robertsLando]

@muka Nice! 😄 We can keep this issue as a track for mqttv5 progress, if you need any other info just ask

[robertsLando]

MQTT 5 specs: http://docs.oasis-open.org/mqtt/mqtt/v5.0/mqtt-v5.0.html

• Session expiry:
  Split the Clean Session flag into a Clean Start flag which indicates that the session should start without using an existing session, and a Session Expiry interval which says how long to retain the session after a disconnect. The session expiry interval can be modified at disconnect. Setting of Clean Start to 1 and Session Expiry Interval to 0 is equivalent in MQTT v3.1.1 of setting Clean Session to 1.

• Message expiry:
  Allow an expiry interval to be set when a message is published.

• Reason code on all ACKs:
  Change all response packets to contain a reason code. This include CONNACK, PUBACK, PUBREC, PUBREL, PUBCOMP, SUBACK, UNSUBACK, DISCONNECT, and AUTH. This allows the invoker to determine whether the requested function succeeded.

• Reason string on all ACKs:
  Change most packets with a reason code to also allow an optional reason string. This is designed for problem determination and is not intended to be parsed by the receiver.

• Server disconnect:
  Allow DISCONNECT to be sent by the Server to indicate the reason the connection is closed.

• Payload format and content type:
  Allow the payload format (binary, text) and a MIME style content type to be specified when a message is published. These are forwarded on to the receiver of the message.

• Request / Response:
  Formalize the request/response pattern within MQTT and provide the Response Topic and Correlation Data properties to allow response messages to be routed back to the publisher of a request. Also, add the ability for the Client to get configuration information from the Server about how to construct the response topics.

• Shared Subscriptions:
  Add shared subscription support allowing for load balanced consumers of a subscription

• Subscription ID:
  Allow a numeric subscription identifier to be specified on a SUBSCRIBE, and returned on the message when it is delivered. This allows the Client to determine which subscription or subscriptions caused the message to be delivered.

• Topic Alias:
  Decrease the size of the MQTT packet overhead by allowing the topic name to be abbreviated to a small integer. The Client and Server independently specify how many topic aliases they allow.

• Flow control:
  Allow the Client and Server to independently specify the number of outstanding reliable messages (QoS>0) they allow. The sender pauses sending such messages to stay below this quota. This is used to limit the rate of reliable messages, and to limit how many are in flight at one time.

• User properties:
  Add User Properties to most packets. User properties on PUBLISH are included with the message and are defined by the Client applications. The user properties on PUBLISH and Will Properties are forwarded by the Server to the receiver of the message. User properties on the CONNECT, SUBSCRIBE, and UNSUBSCRIBE packets are defined by the Server implementation. The user properties on CONNACK PUBACK, PUBREC, PUBREL, PUBCOMP, SUBACK, UNSUBACK and AUTH packets are defined by the sender, and are unique to the sender implementation. The meaning of user properties is not defined by MQTT.

• Maximum Packet Size:
  Allow the Client and Server to independently specify the maximum packet size they support. It is an error for the session partner to send a larger packet.

• Optional Server feature availability:
  Define a set of features which the Server does not allow and provide a mechanism for the Server to specify this to the Client. The features which can be specified in this way are: Maximum QoS, Retain Available, Wildcard Subscription Available, Subscription Identifier Available, and Shared Subscription Available. It is an error for the Client to use features that the Server has declared are not available. It is possible in earlier versions of MQTT for a Server to not implement a feature by declaring that the Client is not authorized for that function. This feature allows such optional behavior to be declared and adds specific Reason Codes when the Client uses one of these features anyway.

• Enhanced authentication:
  Provide a mechanism to enable challenge/response style authentication including mutual authentication. This allows SASL style authentication to be used if supported by both Client and Server, and includes the ability for a Client to re-authenticate within a connection.

• Subscription options:
  Provide subscription options primarily defined to allow for message bridge applications. These include an option to not send messages originating on this Client (noLocal), and options for handling retained messages on subscribe.

• Will delay:
  Add the ability to specify a delay between the end of the connection and sending the will message. This is designed so that if a connection to the session is re-established then the will message is not sent. This allows for brief interruptions of the connection without notification to others.

• Server Keep Alive:
  Allow the Server to specify the value it wishes the Client to use as a keep alive. This allows the Server to set a maximum allowed keepalive and still have the Client honor it.

• Assigned ClientID:
  In cases where the ClientID is assigned by the Server, return the assigned ClientID. This also lifts the restriction that Server assigned ClientIDs can only be used with Clean Session=1 connections.

• Server reference:
  Allow the Server to specify an alternate Server to use on CONNACK or DISCONNECT. This can be used as a redirect or to do provisioning.

[muka]

Ciao,
I started digging into it 🔨 So far I noticed some of those features are already handled by mqtt-connection and mqtt packages (and also mqtt-packet).

As of now, considering my limited knowledge on the code-base and protocol details I would start with Request / Response which seems easier to get started and provide a PR for review. Any suggestion is welcome

[robertsLando]

@muka What suggestions do you need? Info about the code base or from what feature start?

[muka]

Nothing specific, will attempt an implementation to start with.

[ccravens]

@muka I'm interested in MQTTv5 req/resp, but I didn't see a PR number. Have you submitted a PR yet? Is this something in progress? @robertsLando thanks for that comprehensive list!

[mcollina]

@ccravens I do not think this is being worked on atm.

[robertsLando]

@ccravens Actually there isn't any work in progress on Mqtt5 support

[friesendrywall]

I wonder if mqtt 5 is a good fit for aedes as presently structured. There are quite a few options that to utilize would require a more extensive middleware api. Some aspects could be simpler, like session management.

[robertsLando]

I wonder if mqtt 5 is a good fit for aedes as presently structured

What do you mean?

[friesendrywall]

Well, some things that I see..

1. Without some type of server side processing, what good do reason codes and strings do?
2. You don’t need to have perpetual sessions with 5.0, so the persist layer might need to be separated.

I would envision more of an express style middleware. I suppose this could slow down aedes.

[robertsLando]

Without some type of server side processing, what good do reason codes and strings do

I still don't get this, why should this be a problem with the actual structure?

You don’t need to have perpetual sessions with 5.0, so the persist layer might need to be separated.

That's not correct, the clean flag in MQTT 5 has been splitted in Clean start and Clean session expiry. Tha actual implementation doens't handle the session expiry but that only reqiures to add that support on persistence side.

[friesendrywall]

I thought that the server can puback the maximum session length supported? I may misunderstand that part.

It seems to me that 5 is more application oriented, where < 5 the broker is just a middleman. For example, there isn’t really a way to reject a message other than disconnecting in 3. In 5, you can reject a message for different reasons.

[robertsLando]

There shouldn't be any structure problem in Aedes to support the mqtt 5. It just requires lot work and we haven't so much contributors to work on that.

[tatuco]

I would like to contribute to the development of the project

[jonathanhaviv]

Hey if this hasn't been picked up would be happy to help out working on this support!

[mcollina]

Go for it

[robertsLando]

@mcollina Do you agree with the TODO list I wrote above? What could be the best order to implement such features in your opinion?

I think the most interesting ones are topic alias and shared subscriptions, but there are some easier ones that could be picked up before this.

I think we should identify all the issues that require changes on persistences and the ones that doesn't and start with the ones without persistences changes.

For the ones that are new to aedes when I speak about persistences I refer to:

• aedes-persistence
• aedes-cached-persistence (abstract class for on disk persistences)

And all the db specific persistences:

• aedes-persistence-redis
• aedes-persistence-mongodb

[mcollina]

The hardest thing to implement for me would be request/response.

[jonathanhaviv]

@mcollina @robertsLando I'll start on some of the easier-to-implement features this week. Session and message expiry, reason codes and strings, etc. It will also give me a chance to familiarize myself with the codebase a bit more before tackling some of the more complex changes. Let me know if y'all have any objections to that approach!

[robertsLando]

@jonathanhaviv thanks for your help! Remeber that each feature should be implemented keeping in mind that Aedes can also be run in a cluster setup

[robertsLando]

@jonathanhaviv IMO the features that doesn't require changes on persistences are:

• Reason code on all ACKs
• Reason string on all ACKs
• Message expiry: the lifetime of the Will Message in seconds
• Server disconnect
• Payload format and content type (this should be already handled by mqtt-packet we just have to forward it)
• Request / Response
• Subscription id (persistence addSubscription stores the entire sub packet so we don't have to make edits on that side)
• Flow control
• User properties (this should be already handled by mqtt-packet we just have to forward it)
• Maximum Packet Size
• Optional Server feature availability
• Enhanced authentication
• Subscription options: I think this should already be implemented, expecially the noLocal flag but we should check this
• Will delay: checking the putWill we store the entire packet in persistence so no change is needed on that part
• Server Keep Alive
• Assigned ClientID
• Server reference

The ones that require changes on persistences:

• Session expiry: we actually don't have storage support for this
• Shared subscriptions: I drafted an implementation in WIP: Shared subscriptions #449 but i never ended it
• Topic alias

[jonathanhaviv]

@robertsLando thanks for organizing this! Will start working tonight after work and let y'all know if I have any questions on the current implementation but should be able to work down this list. Anywhere you want me to track my work?

[robertsLando]

I created a traking issue and a project:

#821
https://github.com/orgs/moscajs/projects/1/views/1

If you wish I can add you as a maintainer

[yutf]

Will the version that supports MQTT V5 be released?

[robertsLando]

No work has been done yet

[liutaohz]

How soon will a version with MQTT V5 support be released？

[robertsLando]

Never without help