Control protocol transport layer.

[BenediktBurger]

First version of the control protocol transport layer.
It is a up to date state of the current discussions regarding the control protocol in different issues.
The mermaid diagrams can be seen in #39 (in the right order)
A first rendering (if it succeeds) is in https://leco-laboratory-experiment-control-protocol--38.org.readthedocs.build/en/38/control_protocol.html

• Please use different issues for content discussion.
• Please do not remark on language, until it is "ready for review", in this first stage, we have to rig down the protocol itself.

I hope I made everything right with the formatting.

Things to determine regarding content:

• How do we address the Coordinator, see Component IDs #27
• How do we handle heartbeats? Heartbeats #40 Postponed.
• How to do the coordination of Coordinators (global address book)? see Multiple Coordinators #22, my proposal: Multiple Coordinators #22 (comment)

What are your ideas/suggestions regarding formatting:

• Should we link more to the definitions?

TODO

• Add link to nodes, DMT etc. after Add Node concept and start network structure page. #24

[bklebel]

Thanks for compiling this together! I almost fully agree with the content of the protocol as you compiled it here. The only suggestion which I made here now also appears in this comment here (#27)

[BenediktBurger]

• I added a COORDINATOR placeholder for the final name of the Coordinator.
• I expanded the Coordinator flow chart to include messages from a different Coordinator, and to include Error messages.
• I renamed the commands from (DIS)CONNECT to SIGNIN/OUT, in order to differentiate better from the socket connection.

[BenediktBurger]

I updated the Coordinator-sign-in procedure according to #44 (with a few improvements). An additional, flow chart of decisions is in #39 .

I suggest, we leave heartbeats open (a TBD not is there) and finish this PR.

The heartbeat can be added in its own PR.

[bklebel]

Thanks again!
I agree with setting the heartbeats into a TBD, and get this PR through.
Apart from minor changes I have two topics to consider:

1. the message types, and reduction to CO_UPDATE, as per this comment
2. Referring to the discussion in the matrix channel regarding separation of transport and message layer, it might be worthwhile to separate the abstract transport layer protocol (which does not depend on zmq, like Directories, sign-ins and Directory-updates between the Coordinators) a bit stronger from the zmq related definitions of ROUTER and DEALER sockets. However, I think we can go through with this PR, and as one of the next steps make this separation more clear in a new PR

[bilderbuchi]

I managed to configure readthedocs builds (configured to fail on warning), and managed to activate the "list of unresolved conversations" so we have an easier time finding out how many are still open.

[bilderbuchi]

There are some problems with some references/labels, I also see them when building locally.

/home/docs/checkouts/readthedocs.org/user_builds/leco-laboratory-experiment-control-protocol/checkouts/38/control_protocol.md:17: WARNING: undefined label: 'appendix.md#router-sockets'
/home/docs/checkouts/readthedocs.org/user_builds/leco-laboratory-experiment-control-protocol/checkouts/38/control_protocol.md:29: WARNING: undefined label: 'coordinator-sign-in'
/home/docs/checkouts/readthedocs.org/user_builds/leco-laboratory-experiment-control-protocol/checkouts/38/control_protocol.md:91: WARNING: undefined label: 'directory'
/home/docs/checkouts/readthedocs.org/user_builds/leco-laboratory-experiment-control-protocol/checkouts/38/control_protocol.md:91: WARNING: undefined label: 'coordinator-coordination'
/home/docs/checkouts/readthedocs.org/user_builds/leco-laboratory-experiment-control-protocol/checkouts/38/control_protocol.md:125: WARNING: undefined label: 'signing-out'
/home/docs/checkouts/readthedocs.org/user_builds/leco-laboratory-experiment-control-protocol/checkouts/38/control_protocol.md:135: WARNING: undefined label: 'directory'
/home/docs/checkouts/readthedocs.org/user_builds/leco-laboratory-experiment-control-protocol/checkouts/38/control_protocol.md:135: WARNING: undefined label: 'coordinator-coordination'
/home/docs/checkouts/readthedocs.org/user_builds/leco-laboratory-experiment-control-protocol/checkouts/38/control_protocol.md:241: WARNING: undefined label: 'directory'
/home/docs/checkouts/readthedocs.org/user_builds/leco-laboratory-experiment-control-protocol/checkouts/38/glossary.md:32: WARNING: undefined label: 'namespace'
/home/docs/checkouts/readthedocs.org/user_builds/leco-laboratory-experiment-control-protocol/checkouts/38/glossary.md:32: WARNING: undefined label: 'component-name'

Probably we have to check in myst-parser docs how to refer to autogenerated labels in the same file. Tomorrow.. ^.^

[BenediktBurger]

I tried to fix it and somehow, sometimes the auto-slugs to not work (despite being shown with myst-anchors -l 4 control_protocol.md), sometimes the manual ones do not work (to the router socket in the appendix)...

[bilderbuchi]

sometimes the auto-slugs to not work (despite being shown with myst-anchors -l 4 control_protocol.md),

turns out, for the autoslugs you always have to give the filename too, even when it's a reference to inside the same file. So, in control_protocol.md,

{ref}`message-layer`

does not work, but

{ref}`control_protocol.md#message-layer`

works as expected.

This one: <appendix.md#router-sockets> was mixing autogenerated-slug-style (filename#anchor) and native ReST/Sphinx (directly refer the label, which has to be unique throughout the whole document), which did not work.

Thinking about it, both things make sense, just not yesterday night :D

[BenediktBurger]

Btw, here is the link to the generated documentation (also in the first post): https://leco-laboratory-experiment-control-protocol--38.org.readthedocs.build/en/38/control_protocol.html

@bilderbuchi thanks for making the rendering possible and thanks for the explanation regarding the links (I looked at myst parser and was puzzled at first, we we use {ref}..., while the myst documentation gives the normal markdown syntax for links, until I understood it).

I removed the Coordinator's addresses and Component connection ID from the local Directory, such that the Directory is independent of the Transport Layer's underlying protocol (zmq).

That way, we push the discussion regarding Directory updates to the Message Layer (here I only request, that they notify, not whether the notification happens via a diff update or a full version).

With these changes, I hope we can finish this PR.

[BenediktBurger]

In pymeasure/pyleco#1 I have a Coordinator, where you may look at the Directory implementation (there are a couple of dictionaries needed, all in the init with a comment indicating what the key and what the value is).

[bilderbuchi]

Thanks for the updates! I agree, I think it's time to wrap this up.
LGTM, I went through my remaining conversations yesterday.

[BenediktBurger]

If the conversations are resolved for you, @bklebel, you may merge.

[bklebel]

LGTM
Sorry for the longer delay, I agree, I am going to merge this now