From 238b2df83556a2c84b692f96f7b482940d7e9aff Mon Sep 17 00:00:00 2001 From: Benedikt Moneke <67148916+bmoneke@users.noreply.github.com> Date: Thu, 2 Feb 2023 12:06:00 +0100 Subject: [PATCH 01/27] Initial version of the Control protocol transport layer. --- control_protocol.md | 232 ++++++++++++++++++++++++++++++++++++++++++++ index.rst | 1 + 2 files changed, 233 insertions(+) create mode 100644 control_protocol.md diff --git a/control_protocol.md b/control_protocol.md new file mode 100644 index 0000000..1e6b26a --- /dev/null +++ b/control_protocol.md @@ -0,0 +1,232 @@ +# Control protocol + +The control protocol transmits messages via its {ref}`transport-layer` from one Component to another one. +The {ref]}`message-layer` is the common language to understand commands, thus creating a remote procedure call. + + +(transport-layer)= +## Transport layer + +The transport layer ensures, that a message arrives its destination. + + +### Sockets and Connections + +We use [zmq](https://zeromq.org/) sockets for our communication. For more details see the [zmq guide](https://zguide.zeromq.org/) or [zmq API](http://api.zeromq.org/) + + +#### Definitions + +Zmq messages consist in a series of frames, each is a byte sequence. +We indicate the separation between frames with `|`. +An empty frame is indicated with two frame separators `||`, even at the beginning or end of a message. +For example `||Second frame|Third frame||Fifth frame` consists of 5 frames. +The first and fourth frames are empty frames. + + +#### Configuration + +Each {ref}`Coordinator ` shall offer one ROUTER socket, bound to a host name (or IP address or any address of a computer with "*") and port in the DMT. + +{ref}`Components ` shall have one DEALER socket connecting to one Coordinator's ROUTER socket. + +Coordinators shall have one DEALER socket per other Coordinator in the network. +This DEALER socket shall connect to the other Coordinator's ROUTER socket. + +Messages must be sent to a Coordinator's ROUTER socket. + + +#### Particularities of ROUTER sockets + +A small introduction to ROUTER sockets, for more details see [zmq guide chapter 3](https://zguide.zeromq.org/docs/chapter3/#Exploring-ROUTER-Sockets). + +A router socket assigns a random identity to each connecting peer. +If, for example, two Components `CA`, `CB` connect to the ROUTER sockets, the socket assigns the identities, which we call `IA`, `IB` (they can be any bytes sequence). + +Whenever a message is sent to the ROUTER socket from a peer, it reads as a first frame that identity. +For example if `CA` sends the message `Request A`, the ROUTER socket will read `IA|Request A`. +That way, you can return an answer to exactly the original peer and not, for example `CB`. +If you call the ROUTER's send command with `IA|Reply A`, the socket will send `Reply A` to the peer, whose connection is `IA`, in this case `CA`. + +The following diagram shows this example communication with two Components: +:::{mermaid} +graph TD + CA([CA]) -->|"Request A"| D[recv] -->|"IA|Request A"| A([Coordinator]) + CB([CB]) -->|"Request B"| C[recv] -->|"IB|Request B"| A + subgraph ROUTER socket + B + C + D + E + end + A -. "IA|Reply A" .-> B + B[send] -. "Reply A" .-> CA + A -. "IB|Reply B" .-> E + E[send] -. "Reply B" .-> CB +::: + + +### Naming scheme + +Each Component has an individual ID, given by the user, the Component ID. +A Component ID must be a series of bytes, without the ASCII character "." (byte value 46). +Component IDs must be unique in a Node, i.e. among the Componets connected to a single Coordinator. + +As each Component belongs to exactly one Coordinator, it is fully identified by the combination of Node ID and Component ID. +This _full ID_ is the composition of Node ID, ".", and Component ID. + +The receiver of a message may be specified without the Node ID, if the receiver lives in the same Node. + +:::{note} +How to address the Coordinator? I used "no Recipient". +Or a fixed name? +::: + + +### Message composition + +A message consists in two or more frames. +1. The receiver full ID. +2. The sender full ID. +3. Message content: The optional payload, which can be 0 or more frames. + + +### Conversation protocol + +In the protocol examples, `CA`, `CB`, etc. indicate Components. +`Co1`, `Co2`, etc. indicate Coordinators. + +Here the Message content is expressed in plain English, for the exact definition see {ref}`message-layer`. + +In the exchange of messages, only the messages over the wire are shown, the connection identity of the ROUTER socket is not shown. + + +#### Communication with the Coordinator + +##### Initial connection + +After connecting to a Coordinator (`Co1`), a Component (`CA`) shall send a CONNECT message indicating its ID. +The Coordinator shall respond with an ACKNOWLEDGE, giving the Node ID and other relevant information, or with an ERROR, if the ID is already taken. +After that successful handshake, the Coordinator shall store the connection identity and correspondind Component ID in its address book. +:::{note} +publish that information +::: +Similarly, the Component shall store the Node ID and use it from this moment to generate its full ID. + +:::{mermaid} +sequenceDiagram + Note over CA,Co1: ID "CA" is still free + CA ->> Co1: ||CA|CONNECT + Note right of Co1: Connection identity "IA" + Note right of Co1: Stores "CA" with identity "IA" + Co1 ->> CA: Co1.CA||ACKNOWLEDGE: Node ID is "Co1" + Note left of CA: Stores "Co1" as Node ID + Note over CA,Co1: ID "CA" is already used + CA ->> Co1: ||CA|CONNECT + Co1 ->> CA: CA||ERROR: ID "CA" is already used. + Note left of CA: May retry with another ID +::: + + +##### Heartbeat + +We use heartbeat to know, whether a communication partner is still online. + +Every message received counts as a heartbeat. + +:::{note} +TBD: Respond to every non empty message with an empty one? +::: + + +##### Disconnecting + +A Component should tell a Coordinator, when it stops working, with a DISCONNECT message. +The Coordinator shall ACKNOWLEDGE the disconnect and remove the ID from its address book. +:::{note} +publish that information +::: + +:::{mermaid} +sequenceDiagram + CA ->> Co1: ||Co1.CA|DISCONNECT + Co1 ->> CA: Co1.CA||ACKNOWLEDGE + Note right of Co1: Removes "CA" with identity "IA"
from address book + Note left of CA: Shall not send any message anymore +::: + + +#### Communication with other Components + +The following two examples show, how a message is transferred between two components `CA`, `CB` via one or two Coordinators. +Coordinators shall send messages from their DEALER socket to other Coordinator's ROUTER socket. + +Coordinators shall hand on the message to the corresponding Coordinator or connected Component. + + +:::{mermaid} +sequenceDiagram + CA ->> Co1: Co1.CB|Co1.CA| Give me property A. + Co1 ->> CB: Co1.CB|Co1.CA| Give me property A. + Note left of CB: Reads property A + CB ->> Co1: Co1.CA|Co1.CB| Property A has value 5. + Co1 ->> CA: Co1.CA|Co1.CB| Property A has value 5. + Note over CA,Co1: The first message could be as well: + CA ->> Co1: CB|Co1.CA| Give me property A. +::: + + +:::{mermaid} +sequenceDiagram + CA ->> Co1: Co2.CB|Co1.CA| Give me property A. + Note over Co1,Co2: Co1.DEALER socket sends to Co2.ROUTER + Co1 ->> Co2: Co2.CB|Co1.CA| Give me property A. + Co2 ->> CB: Co2.CB|Co1.CA| Give me property A. + Note left of CB: Reads property A + CB ->> Co2: Co1.CA|Co2.CB| Property A has value 5. + Note over Co1,Co2: Co2.DEALER socket sends to Co1.ROUTER + Co2 ->> Co1: Co1.CA|Co2.CB| Property A has value 5. + Co1 ->> CA: Co1.CA|Co2.CB| Property A has value 5. +::: + + +The following flow chart shows the decision scheme and message modification in a Coordinator. +`IA` and `IB` are the connection identities of `CA` and `Co1.Recipient`. +Bold arrows indicate message flow, thin lines indicate decision flow. + +:::{mermaid} +flowchart TB + C([CA DEALER]) == "NodeID.Recipient|Co1.CA|Content" ==> R0 + R0[Co1 ROUTER receive] == "IA|NodeID.Recipient|Co1.CA|Content" ==> Code[remove sender identity] + Code == "NodeID.Recipient|Co1.CA|Content" ==> NS + NS -- "is None" --> Local + NS{NodeID} -- "== Co1"--> Local + Local{Recipient
is None} -- "yes" --> Self([Message for Co1
itself]) + Local -- "no" --> Local2 + Local2[add Recipient identity IB] == "IB|NodeID.Recipient|Co1.CA|Content" ==> R1[send] + R1 == "NodeID.Recipient|Co1.CA|Content" ==> W1([Wire to Co1.Recipient DEALER]) + NS -- "== connected Coordinator Co2" --> Keep + Keep[send to Coordinator Co2] == "NodeID.Recipient|Co1.CA|Content" ==> R2[send] + R2 == "NodeID.Recipient|Co1.CA|Content" ==> W2([Wire to Co2 ROUTER]) + NS -- "reachable via Coordinator CoX" --> Augment + Augment[send via CoX] == "NodeID.Recipient|Co1.CA|Content" ==> R3[send] + R3 == "NodeID.Recipient|Co1.CA|Content"==> W3([Wire to CoX ROUTER]) + subgraph Co1 ROUTER socket + R1 + end + subgraph Co1 DEALER socket to Co2 + R2 + end + subgraph Co1 DEALER socket to CoX + R3 + end +::: + + + +(message-layer)= +## Message layer + +:::{note} +TODO +::: diff --git a/index.rst b/index.rst index a6eb47c..d0acc69 100644 --- a/index.rst +++ b/index.rst @@ -13,6 +13,7 @@ introduction goals components + control_protocol glossary Hello_world From 2f14be0d1a33e8e3f4a12d4da4be58afafbef9da Mon Sep 17 00:00:00 2001 From: Benedikt Moneke <67148916+bmoneke@users.noreply.github.com> Date: Fri, 3 Feb 2023 09:33:07 +0100 Subject: [PATCH 02/27] Coordinator has ID COORDINATOR. Also: heartbeat expanded. Added Coordinator coordination (initial version). Small text refactoring. Added message without a CONNECT raises ERROR. --- control_protocol.md | 86 ++++++++++++++++++++++++++++++++------------- 1 file changed, 62 insertions(+), 24 deletions(-) diff --git a/control_protocol.md b/control_protocol.md index 1e6b26a..a9c2601 100644 --- a/control_protocol.md +++ b/control_protocol.md @@ -36,6 +36,7 @@ This DEALER socket shall connect to the other Coordinator's ROUTER socket. Messages must be sent to a Coordinator's ROUTER socket. +(router-sockets)= #### Particularities of ROUTER sockets A small introduction to ROUTER sockets, for more details see [zmq guide chapter 3](https://zguide.zeromq.org/docs/chapter3/#Exploring-ROUTER-Sockets). @@ -66,24 +67,26 @@ graph TD ::: -### Naming scheme +### Protocol basics -Each Component has an individual ID, given by the user, the Component ID. +#### Naming scheme + +Each Component has an individual ID, given by the user, the _Component ID_. A Component ID must be a series of bytes, without the ASCII character "." (byte value 46). Component IDs must be unique in a Node, i.e. among the Componets connected to a single Coordinator. +The Coordinator itself has the Component ID `COORDINATOR` +:::{note} +COORDINATOR is a placeholder for the final version. +::: As each Component belongs to exactly one Coordinator, it is fully identified by the combination of Node ID and Component ID. This _full ID_ is the composition of Node ID, ".", and Component ID. +For example `Co1.CA` is the full ID of the Component `CA` in the Node `Co1`. The receiver of a message may be specified without the Node ID, if the receiver lives in the same Node. -:::{note} -How to address the Coordinator? I used "no Recipient". -Or a fixed name? -::: - -### Message composition +#### Message composition A message consists in two or more frames. 1. The receiver full ID. @@ -91,10 +94,19 @@ A message consists in two or more frames. 3. Message content: The optional payload, which can be 0 or more frames. +(address-book)= +#### Address book + +Each Coordinator shall have a list of the Components connected to it. +This is its _local address book_. + +The _global address book_ is the combination of the local address books of all Coordinators in a Network. + + ### Conversation protocol -In the protocol examples, `CA`, `CB`, etc. indicate Components. -`Co1`, `Co2`, etc. indicate Coordinators. +In the protocol examples, `CA`, `CB`, etc. indicate Component IDs. +`Co1`, `Co2`, etc. indicate Coordinators with their Node IDs. Here the Message content is expressed in plain English, for the exact definition see {ref}`message-layer`. @@ -107,24 +119,30 @@ In the exchange of messages, only the messages over the wire are shown, the conn After connecting to a Coordinator (`Co1`), a Component (`CA`) shall send a CONNECT message indicating its ID. The Coordinator shall respond with an ACKNOWLEDGE, giving the Node ID and other relevant information, or with an ERROR, if the ID is already taken. -After that successful handshake, the Coordinator shall store the connection identity and correspondind Component ID in its address book. -:::{note} -publish that information -::: +After that successful handshake, the Coordinator shall store the connection identity and corresponding Component ID in its local address book. +It shall also publish to the other Coordinators in the network that this Component connected, see {ref}`Coordinator coordination`. Similarly, the Component shall store the Node ID and use it from this moment to generate its full ID. +If a Component does send a message to someone without having signed in via CONNECT, the Coordinator shall refuse message handling and return an error. + :::{mermaid} sequenceDiagram Note over CA,Co1: ID "CA" is still free - CA ->> Co1: ||CA|CONNECT + CA ->> Co1: COORDINATOR|CA|CONNECT Note right of Co1: Connection identity "IA" Note right of Co1: Stores "CA" with identity "IA" - Co1 ->> CA: Co1.CA||ACKNOWLEDGE: Node ID is "Co1" + Co1 ->> CA: Co1.CA|Co1.COORDINATOR|ACKNOWLEDGE: Node ID is "Co1" Note left of CA: Stores "Co1" as Node ID Note over CA,Co1: ID "CA" is already used - CA ->> Co1: ||CA|CONNECT - Co1 ->> CA: CA||ERROR: ID "CA" is already used. + CA ->> Co1: COORDINATOR|CA|CONNECT + Co1 ->> CA: CA|Co1.COORDINATOR|ERROR: ID "CA" is already used. Note left of CA: May retry with another ID + Note over CA,Co1: "CA" has not send CONNECT + Note left of CA: Wants to send a message to CB + CA ->> Co1: Co1.CB|CA|Content + Note right of Co1: Does not know CA + Co1 ->> CA: CA|Co1.COORDINATOR|ERROR:I do not know you + Note left of CA: Should send a CONNECT message ::: @@ -134,23 +152,25 @@ We use heartbeat to know, whether a communication partner is still online. Every message received counts as a heartbeat. +A Component should and a Coordinator shall send a STATUS request and wait some time before considering a connection dead. +A Coordinator shall follow the {ref}`disconnect routine` for a connected Component considered dead. + :::{note} TBD: Respond to every non empty message with an empty one? ::: +(disconnecting)= ##### Disconnecting A Component should tell a Coordinator, when it stops working, with a DISCONNECT message. The Coordinator shall ACKNOWLEDGE the disconnect and remove the ID from its address book. -:::{note} -publish that information -::: +It shall also publish to the other Coordinators in the network that this Component disconnected, see {ref}`Coordinator coordination`. :::{mermaid} sequenceDiagram - CA ->> Co1: ||Co1.CA|DISCONNECT - Co1 ->> CA: Co1.CA||ACKNOWLEDGE + CA ->> Co1: COORDINATOR|Co1.CA|DISCONNECT + Co1 ->> CA: Co1.CA|Co1.COORDINATOR|ACKNOWLEDGE Note right of Co1: Removes "CA" with identity "IA"
from address book Note left of CA: Shall not send any message anymore ::: @@ -201,7 +221,7 @@ flowchart TB Code == "NodeID.Recipient|Co1.CA|Content" ==> NS NS -- "is None" --> Local NS{NodeID} -- "== Co1"--> Local - Local{Recipient
is None} -- "yes" --> Self([Message for Co1
itself]) + Local{Recipient
==
COORDINATOR} -- "yes" --> Self([Message for Co1
itself]) Local -- "no" --> Local2 Local2[add Recipient identity IB] == "IB|NodeID.Recipient|Co1.CA|Content" ==> R1[send] R1 == "NodeID.Recipient|Co1.CA|Content" ==> W1([Wire to Co1.Recipient DEALER]) @@ -223,6 +243,24 @@ flowchart TB ::: +(coordinator-coordination)= +#### Coordinator coordination + +Each Coordinator shall keep an up to date {ref}`global address book` with the IDs of all Components in the Network. +For this, Coordinators shall tell each other regarding connecting and disconnecting Components and Coordinators. +Coordinators shall send on request the IDs of their local address book, or of their global address book, depending on the request type. + + +Necessary information: +- Event type (connect or disconnect) +- Full ID (Node ID and Component ID) of the Component + +:::{note} +TODO decide whether via Control or Data protocol. +TODO add the log in of a Coordinator +::: + + (message-layer)= ## Message layer From df1a074df2de3cf34118ffb18552474478ffc019 Mon Sep 17 00:00:00 2001 From: Benedikt Moneke <67148916+bmoneke@users.noreply.github.com> Date: Fri, 3 Feb 2023 09:33:29 +0100 Subject: [PATCH 03/27] Changing (DIS)CONNECT to SIGNIN/OUT. --- control_protocol.md | 35 ++++++++++++++++++++--------------- 1 file changed, 20 insertions(+), 15 deletions(-) diff --git a/control_protocol.md b/control_protocol.md index a9c2601..21e18b7 100644 --- a/control_protocol.md +++ b/control_protocol.md @@ -115,34 +115,35 @@ In the exchange of messages, only the messages over the wire are shown, the conn #### Communication with the Coordinator +(sign-in)= ##### Initial connection -After connecting to a Coordinator (`Co1`), a Component (`CA`) shall send a CONNECT message indicating its ID. +After connecting to a Coordinator (`Co1`), a Component (`CA`) shall send a SIGNIN message indicating its ID. The Coordinator shall respond with an ACKNOWLEDGE, giving the Node ID and other relevant information, or with an ERROR, if the ID is already taken. After that successful handshake, the Coordinator shall store the connection identity and corresponding Component ID in its local address book. -It shall also publish to the other Coordinators in the network that this Component connected, see {ref}`Coordinator coordination`. +It shall also publish to the other Coordinators in the network that this Component signed in, see {ref}`Coordinator coordination`. Similarly, the Component shall store the Node ID and use it from this moment to generate its full ID. -If a Component does send a message to someone without having signed in via CONNECT, the Coordinator shall refuse message handling and return an error. +If a Component does send a message to someone without having signed in, the Coordinator shall refuse message handling and return an error. :::{mermaid} sequenceDiagram Note over CA,Co1: ID "CA" is still free - CA ->> Co1: COORDINATOR|CA|CONNECT + CA ->> Co1: COORDINATOR|CA|SIGNIN Note right of Co1: Connection identity "IA" Note right of Co1: Stores "CA" with identity "IA" Co1 ->> CA: Co1.CA|Co1.COORDINATOR|ACKNOWLEDGE: Node ID is "Co1" Note left of CA: Stores "Co1" as Node ID Note over CA,Co1: ID "CA" is already used - CA ->> Co1: COORDINATOR|CA|CONNECT + CA ->> Co1: COORDINATOR|CA|SIGNIN Co1 ->> CA: CA|Co1.COORDINATOR|ERROR: ID "CA" is already used. Note left of CA: May retry with another ID - Note over CA,Co1: "CA" has not send CONNECT + Note over CA,Co1: "CA" has not send SIGNIN Note left of CA: Wants to send a message to CB CA ->> Co1: Co1.CB|CA|Content Note right of Co1: Does not know CA Co1 ->> CA: CA|Co1.COORDINATOR|ERROR:I do not know you - Note left of CA: Should send a CONNECT message + Note left of CA: Should send a SIGNIN message ::: @@ -153,23 +154,23 @@ We use heartbeat to know, whether a communication partner is still online. Every message received counts as a heartbeat. A Component should and a Coordinator shall send a STATUS request and wait some time before considering a connection dead. -A Coordinator shall follow the {ref}`disconnect routine` for a connected Component considered dead. +A Coordinator shall follow the {ref}`sign out routine` for a signed in Component considered dead. :::{note} TBD: Respond to every non empty message with an empty one? ::: -(disconnecting)= -##### Disconnecting +(sign-out)= +##### Signing out -A Component should tell a Coordinator, when it stops working, with a DISCONNECT message. -The Coordinator shall ACKNOWLEDGE the disconnect and remove the ID from its address book. -It shall also publish to the other Coordinators in the network that this Component disconnected, see {ref}`Coordinator coordination`. +A Component should tell a Coordinator, when it stops working, with a SIGNOUT message. +The Coordinator shall ACKNOWLEDGE the sign out and remove the ID from its address book. +It shall also publish to the other Coordinators in the network that this Component signed out, see {ref}`Coordinator coordination`. :::{mermaid} sequenceDiagram - CA ->> Co1: COORDINATOR|Co1.CA|DISCONNECT + CA ->> Co1: COORDINATOR|Co1.CA|SIGNOUT Co1 ->> CA: Co1.CA|Co1.COORDINATOR|ACKNOWLEDGE Note right of Co1: Removes "CA" with identity "IA"
from address book Note left of CA: Shall not send any message anymore @@ -209,6 +210,10 @@ sequenceDiagram Co1 ->> CA: Co1.CA|Co2.CB| Property A has value 5. ::: +Prerequisite of Communication between two Components are: +- Both Components are connected to a Coordinator and {ref}`signed in`. +- Both Components are either connected to the same Coordinator (example one), or their Coordinators are connected to each other (example two). + The following flow chart shows the decision scheme and message modification in a Coordinator. `IA` and `IB` are the connection identities of `CA` and `Co1.Recipient`. @@ -247,7 +252,7 @@ flowchart TB #### Coordinator coordination Each Coordinator shall keep an up to date {ref}`global address book` with the IDs of all Components in the Network. -For this, Coordinators shall tell each other regarding connecting and disconnecting Components and Coordinators. +For this, Coordinators shall tell each other regarding signing in and signing out Components and Coordinators. Coordinators shall send on request the IDs of their local address book, or of their global address book, depending on the request type. From fac558751e192e78798085e8145240d58b79af25 Mon Sep 17 00:00:00 2001 From: Benedikt Moneke <67148916+bmoneke@users.noreply.github.com> Date: Fri, 3 Feb 2023 10:18:39 +0100 Subject: [PATCH 04/27] Decision flow in Coordinator expanded (including errors) --- control_protocol.md | 49 ++++++++++++++++++++++++++++++--------------- 1 file changed, 33 insertions(+), 16 deletions(-) diff --git a/control_protocol.md b/control_protocol.md index 21e18b7..150c62d 100644 --- a/control_protocol.md +++ b/control_protocol.md @@ -41,7 +41,7 @@ Messages must be sent to a Coordinator's ROUTER socket. A small introduction to ROUTER sockets, for more details see [zmq guide chapter 3](https://zguide.zeromq.org/docs/chapter3/#Exploring-ROUTER-Sockets). -A router socket assigns a random identity to each connecting peer. +A router socket assigns a random _identity_ to each connecting peer. If, for example, two Components `CA`, `CB` connect to the ROUTER sockets, the socket assigns the identities, which we call `IA`, `IB` (they can be any bytes sequence). Whenever a message is sent to the ROUTER socket from a peer, it reads as a first frame that identity. @@ -216,28 +216,45 @@ Prerequisite of Communication between two Components are: The following flow chart shows the decision scheme and message modification in a Coordinator. -`IA` and `IB` are the connection identities of `CA` and `Co1.Recipient`. +`Node0`, `NodeR` are placeholders for sender and recipient Node ID. +`Recipient` is a placeholder for the recipient Component ID. +`IA` is the connection identity of `Co1.CA` or of `Node0.COORDINATOR`. +`IB` is the connection identities of `Co1.Recipient`. Bold arrows indicate message flow, thin lines indicate decision flow. +Thin, dotted lines indicate decision flow in case of errors :::{mermaid} flowchart TB - C([CA DEALER]) == "NodeID.Recipient|Co1.CA|Content" ==> R0 - R0[Co1 ROUTER receive] == "IA|NodeID.Recipient|Co1.CA|Content" ==> Code[remove sender identity] - Code == "NodeID.Recipient|Co1.CA|Content" ==> NS - NS -- "is None" --> Local - NS{NodeID} -- "== Co1"--> Local + C([Node0.CA DEALER]) == "NodeR.Recipient|Node0.CA|Content" ==> R0 + R0[receive] == "IA|NodeR.Recipient|Node0.CA|Content" ==> CN0{Node0 == Co1} + CN0-->|no| RemIdent + CN0-->|yes| Clocal{CA in
local address book?} + Clocal -->|yes| CidKnown{IA is CA's identity
in address book?} + CidKnown -->|yes| RemIdent + Clocal -.->|no| E1[ERROR: Sender did not sign in] ==>|"IA|Node0.CA|Co1.COORDINATOR|ERROR: Sender dit not sign in"| S + S[send] ==> WA([Node0.CA DEALER]) + CidKnown -.->|no| E2[ERROR: ID and identity do not match]==>|"IA|Node0.CA|Co1.COORDINATOR|ERROR: ID and identity do not match"| S + RemIdent[remove sender identity] == "NodeR.Recipient|Node0.CA|Content" ==> CNR + CNR -- "is None" --> Local + CNR{NodeR} -- "== Co1"--> Local Local{Recipient
==
COORDINATOR} -- "yes" --> Self([Message for Co1
itself]) - Local -- "no" --> Local2 - Local2[add Recipient identity IB] == "IB|NodeID.Recipient|Co1.CA|Content" ==> R1[send] - R1 == "NodeID.Recipient|Co1.CA|Content" ==> W1([Wire to Co1.Recipient DEALER]) - NS -- "== connected Coordinator Co2" --> Keep - Keep[send to Coordinator Co2] == "NodeID.Recipient|Co1.CA|Content" ==> R2[send] - R2 == "NodeID.Recipient|Co1.CA|Content" ==> W2([Wire to Co2 ROUTER]) - NS -- "reachable via Coordinator CoX" --> Augment - Augment[send via CoX] == "NodeID.Recipient|Co1.CA|Content" ==> R3[send] - R3 == "NodeID.Recipient|Co1.CA|Content"==> W3([Wire to CoX ROUTER]) + Local -- "no" --> Local2a{Recipient in Address book} + Local2a -->|yes, with Identity IB| Local2 + Local2[add Recipient identity IB] == "IB|NodeR.Recipient|Node0.CA|Content" ==> R1[send] + R1 == "NodeR.Recipient|Node0.CA|Content" ==> W1([Wire to Co1.Recipient DEALER]) + Local2a -.->|no| E3[ERROR Recipient unknown
send Error to original sender] ==>|"Node0.CA|Co1.COORDINATOR|ERROR Co1.Recipient is unknown"|CNR + CNR -- "== connected Coordinator Co2" --> Keep + Keep[send to Coordinator Co2] == "NodeR.Recipient|Node0.CA|Content" ==> R2[send] + R2 == "NodeR.Recipient|Node0.CA|Content" ==> W2([Wire to Co2 ROUTER]) + CNR -- "reachable via Coordinator CoX" --> Augment + Augment[send via CoX] == "NodeR.Recipient|Node0.CA|Content" ==> R3[send] + R3 == "NodeR.Recipient|Node0.CA|Content"==> W3([Wire to CoX ROUTER]) + subgraph Co1 ROUTER socket + R0 + end subgraph Co1 ROUTER socket R1 + S end subgraph Co1 DEALER socket to Co2 R2 From bd59eaaf18c7efbfd8418a4c368649f6f8f33874 Mon Sep 17 00:00:00 2001 From: Benedikt Moneke <67148916+bmoneke@users.noreply.github.com> Date: Fri, 3 Feb 2023 12:03:25 +0100 Subject: [PATCH 05/27] Wording and links changed to accomodate Nodes. --- control_protocol.md | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/control_protocol.md b/control_protocol.md index 150c62d..c97ad37 100644 --- a/control_protocol.md +++ b/control_protocol.md @@ -26,11 +26,11 @@ The first and fourth frames are empty frames. #### Configuration -Each {ref}`Coordinator ` shall offer one ROUTER socket, bound to a host name (or IP address or any address of a computer with "*") and port in the DMT. +Each {ref}`Coordinator ` shall offer one ROUTER socket, bound to a host name (or IP address or any address of a computer with "*") and port. {ref}`Components ` shall have one DEALER socket connecting to one Coordinator's ROUTER socket. -Coordinators shall have one DEALER socket per other Coordinator in the network. +Coordinators shall have one DEALER socket per other Coordinator in the Network. This DEALER socket shall connect to the other Coordinator's ROUTER socket. Messages must be sent to a Coordinator's ROUTER socket. @@ -73,13 +73,14 @@ graph TD Each Component has an individual ID, given by the user, the _Component ID_. A Component ID must be a series of bytes, without the ASCII character "." (byte value 46). -Component IDs must be unique in a Node, i.e. among the Componets connected to a single Coordinator. -The Coordinator itself has the Component ID `COORDINATOR` +Component IDs must be unique in a {ref}`Node `, i.e. among the Componets connected to a single Coordinator. +The Coordinator itself has the Component ID `COORDINATOR`. :::{note} COORDINATOR is a placeholder for the final version. ::: -As each Component belongs to exactly one Coordinator, it is fully identified by the combination of Node ID and Component ID. +Also every Node ID has to be unique in the Network. +As each Component belongs to exactly one Coordinator, it is fully identified by the combination of Node ID and Component ID., which is globally unique. This _full ID_ is the composition of Node ID, ".", and Component ID. For example `Co1.CA` is the full ID of the Component `CA` in the Node `Co1`. From 5b1b885c9f36d92256e7b6b92d33fa76b3c6e557 Mon Sep 17 00:00:00 2001 From: Benedikt Moneke <67148916+bmoneke@users.noreply.github.com> Date: Fri, 3 Feb 2023 13:14:41 +0100 Subject: [PATCH 06/27] Make ROUTER socket more understantable. And small diagram changes. --- control_protocol.md | 36 ++++++++++++++++++------------------ 1 file changed, 18 insertions(+), 18 deletions(-) diff --git a/control_protocol.md b/control_protocol.md index c97ad37..923e6a3 100644 --- a/control_protocol.md +++ b/control_protocol.md @@ -50,20 +50,18 @@ That way, you can return an answer to exactly the original peer and not, for exa If you call the ROUTER's send command with `IA|Reply A`, the socket will send `Reply A` to the peer, whose connection is `IA`, in this case `CA`. The following diagram shows this example communication with two Components: -:::{mermaid} -graph TD - CA([CA]) -->|"Request A"| D[recv] -->|"IA|Request A"| A([Coordinator]) - CB([CB]) -->|"Request B"| C[recv] -->|"IB|Request B"| A - subgraph ROUTER socket - B - C - D - E - end - A -. "IA|Reply A" .-> B - B[send] -. "Reply A" .-> CA - A -. "IB|Reply B" .-> E - E[send] -. "Reply B" .-> CB +sequenceDiagram + participant Code + participant ROUTER as ROUTER socket + Note over Code, ROUTER: Coordinator + CA ->> ROUTER: "Request A" + ROUTER ->> Code: "IA|Request A" + Code ->> ROUTER: "IA|Reply A" + ROUTER ->> CA: "Reply A" + CB ->> ROUTER: "Request B" + ROUTER ->> Code: "IB|Request B" + Code ->> ROUTER: "IB|Reply B" + ROUTER ->> CB: "Reply B" ::: @@ -174,7 +172,7 @@ sequenceDiagram CA ->> Co1: COORDINATOR|Co1.CA|SIGNOUT Co1 ->> CA: Co1.CA|Co1.COORDINATOR|ACKNOWLEDGE Note right of Co1: Removes "CA" with identity "IA"
from address book - Note left of CA: Shall not send any message anymore + Note left of CA: Shall not send any message anymore except SIGNIN ::: @@ -188,13 +186,15 @@ Coordinators shall hand on the message to the corresponding Coordinator or conne :::{mermaid} sequenceDiagram - CA ->> Co1: Co1.CB|Co1.CA| Give me property A. + alt full ID + CA ->> Co1: Co1.CB|Co1.CA| Give me property A. + else only Component ID + CA ->> Co1: CB|Co1.CA| Give me property A. + end Co1 ->> CB: Co1.CB|Co1.CA| Give me property A. Note left of CB: Reads property A CB ->> Co1: Co1.CA|Co1.CB| Property A has value 5. Co1 ->> CA: Co1.CA|Co1.CB| Property A has value 5. - Note over CA,Co1: The first message could be as well: - CA ->> Co1: CB|Co1.CA| Give me property A. ::: From 588e35a6216c32f664fc7e242443e6f48bcee4b6 Mon Sep 17 00:00:00 2001 From: Benedikt Moneke <67148916+bmoneke@users.noreply.github.com> Date: Mon, 6 Feb 2023 10:44:28 +0100 Subject: [PATCH 07/27] Decision diagram improved. placeholders are lowercase, actual values uppercase. Node IDs begin with N (not Co). Messages in two versions given. Multi-Hop removed. --- control_protocol.md | 60 ++++++++++++++++++++++----------------------- 1 file changed, 29 insertions(+), 31 deletions(-) diff --git a/control_protocol.md b/control_protocol.md index 923e6a3..2f9d2ef 100644 --- a/control_protocol.md +++ b/control_protocol.md @@ -216,40 +216,41 @@ Prerequisite of Communication between two Components are: - Both Components are either connected to the same Coordinator (example one), or their Coordinators are connected to each other (example two). -The following flow chart shows the decision scheme and message modification in a Coordinator. -`Node0`, `NodeR` are placeholders for sender and recipient Node ID. -`Recipient` is a placeholder for the recipient Component ID. -`IA` is the connection identity of `Co1.CA` or of `Node0.COORDINATOR`. -`IB` is the connection identities of `Co1.Recipient`. +The following flow chart shows the decision scheme and message modification in a Coordinator `Co1` of Node `N1`. +Its full ID is `N1.Coordinator`. +`n0`, `nR` are placeholders for sender and recipient Node ID. +`recipient` is a placeholder for the recipient Component ID. +`IA` is the connection identity of the Component `N1.CA` (if it is directly connected to `Co1`) or of its Coordinator `n0.COORDINATOR`. +`IB` is the connection identities of `N1.Recipient`. Bold arrows indicate message flow, thin lines indicate decision flow. -Thin, dotted lines indicate decision flow in case of errors +Thin, dotted lines indicate decision flow in case of errors. +Placeholder values are written in lowercase, while actually known values are begin with an uppercase letter. :::{mermaid} flowchart TB - C([Node0.CA DEALER]) == "NodeR.Recipient|Node0.CA|Content" ==> R0 - R0[receive] == "IA|NodeR.Recipient|Node0.CA|Content" ==> CN0{Node0 == Co1} - CN0-->|no| RemIdent - CN0-->|yes| Clocal{CA in
local address book?} + C1([N1.CA DEALER]) == "nr.recipient|n0.CA|Content
(==nr.recipient|N1.CA|Content)" ==> R0 + C0([n0.COORDINATOR DEALER]) == "nr.recipient|n0.CA|Content" ==> R0 + R0[receive] == "IA|nr.recipient|n0.CA|Content" ==> Cn0{n0 == N1?} + Cn0-->|no| RemIdent + Cn0-->|yes| Clocal{CA in
local address book?} Clocal -->|yes| CidKnown{IA is CA's identity
in address book?} CidKnown -->|yes| RemIdent - Clocal -.->|no| E1[ERROR: Sender did not sign in] ==>|"IA|Node0.CA|Co1.COORDINATOR|ERROR: Sender dit not sign in"| S - S[send] ==> WA([Node0.CA DEALER]) - CidKnown -.->|no| E2[ERROR: ID and identity do not match]==>|"IA|Node0.CA|Co1.COORDINATOR|ERROR: ID and identity do not match"| S - RemIdent[remove sender identity] == "NodeR.Recipient|Node0.CA|Content" ==> CNR - CNR -- "is None" --> Local - CNR{NodeR} -- "== Co1"--> Local - Local{Recipient
==
COORDINATOR} -- "yes" --> Self([Message for Co1
itself]) - Local -- "no" --> Local2a{Recipient in Address book} + Clocal -.->|no| E1[ERROR: Sender did not sign in] ==>|"IA|n0.CA|N1.COORDINATOR|ERROR: Sender dit not sign in
(==IA|N1.CA|N1.COORDINATOR|ERROR...)"| S + S[send] ==> WA([N1.CA DEALER]) + CidKnown -.->|no| E2[ERROR: ID and identity do not match]==>|"IA|n0.CA|N1.COORDINATOR|ERROR: ID and identity do not match
(==IA|N1.CA|N1.COORDINATOR|ERROR...)"| S + RemIdent[remove sender identity] == "nr.recipient|n0.CA|Content" ==> Cnr + Cnr -- "is None" --> Local + Cnr{nr?} -- "== N1"--> Local + Local{recipient
==
COORDINATOR?} -- "yes" --> Self[Message for Co1
itself] + Self == "nr.recipient|n0.CA|Content
(==N1.COORDINATOR|n0.CA|Content)" ==> SC([Co1 Message handling]) + Local -- "no" --> Local2a{recipient in Address book?} Local2a -->|yes, with Identity IB| Local2 - Local2[add Recipient identity IB] == "IB|NodeR.Recipient|Node0.CA|Content" ==> R1[send] - R1 == "NodeR.Recipient|Node0.CA|Content" ==> W1([Wire to Co1.Recipient DEALER]) - Local2a -.->|no| E3[ERROR Recipient unknown
send Error to original sender] ==>|"Node0.CA|Co1.COORDINATOR|ERROR Co1.Recipient is unknown"|CNR - CNR -- "== connected Coordinator Co2" --> Keep - Keep[send to Coordinator Co2] == "NodeR.Recipient|Node0.CA|Content" ==> R2[send] - R2 == "NodeR.Recipient|Node0.CA|Content" ==> W2([Wire to Co2 ROUTER]) - CNR -- "reachable via Coordinator CoX" --> Augment - Augment[send via CoX] == "NodeR.Recipient|Node0.CA|Content" ==> R3[send] - R3 == "NodeR.Recipient|Node0.CA|Content"==> W3([Wire to CoX ROUTER]) + Local2[add recipient identity IB] == "IB|nr.recipient|n0.CA|Content
(==IB|N1.recipient|n0.CA|Content)" ==> R1[send] + R1 == "nr.recipient|n0.CA|Content
(==N1.recipient|n0.CA|Content)" ==> W1([Wire to N1.recipient DEALER]) + Local2a -.->|no| E3[ERROR recipient unknown
send Error to original sender] ==>|"n0.CA|N1.COORDINATOR|ERROR N1.recipient is unknown"|Cnr + Cnr -- "== N2" --> Keep + Keep[send to N2.COORDINATOR] == "nr.recipient|n0.CA|Content
(==N2.recipient|n0.CA|Content)" ==> R2[send] + R2 == "nr.recipient|n0.CA|Content
(==N2.recipient|n0.CA|Content)" ==> W2([Wire to N2.COORDINATOR ROUTER]) subgraph Co1 ROUTER socket R0 end @@ -257,12 +258,9 @@ flowchart TB R1 S end - subgraph Co1 DEALER socket to Co2 + subgraph Co1 DEALER socket
to N2.COORDINATOR R2 end - subgraph Co1 DEALER socket to CoX - R3 - end ::: From e711d6deda7daeb46977caaf47c822ee58754a54 Mon Sep 17 00:00:00 2001 From: Benedikt Moneke <67148916+bmoneke@users.noreply.github.com> Date: Mon, 6 Feb 2023 11:16:31 +0100 Subject: [PATCH 08/27] Changing Coordinator names to Node names --- control_protocol.md | 67 ++++++++++++++++++++++++--------------------- 1 file changed, 36 insertions(+), 31 deletions(-) diff --git a/control_protocol.md b/control_protocol.md index 2f9d2ef..b1a90c7 100644 --- a/control_protocol.md +++ b/control_protocol.md @@ -80,7 +80,7 @@ COORDINATOR is a placeholder for the final version. Also every Node ID has to be unique in the Network. As each Component belongs to exactly one Coordinator, it is fully identified by the combination of Node ID and Component ID., which is globally unique. This _full ID_ is the composition of Node ID, ".", and Component ID. -For example `Co1.CA` is the full ID of the Component `CA` in the Node `Co1`. +For example `N1.CA` is the full ID of the Component `CA` in the Node `N1`. The receiver of a message may be specified without the Node ID, if the receiver lives in the same Node. @@ -105,7 +105,7 @@ The _global address book_ is the combination of the local address books of all C ### Conversation protocol In the protocol examples, `CA`, `CB`, etc. indicate Component IDs. -`Co1`, `Co2`, etc. indicate Coordinators with their Node IDs. +`N1`, `N2`, etc. indicate Node IDs and `Co1`, `Co2` their corresponding Coordinators. Here the Message content is expressed in plain English, for the exact definition see {ref}`message-layer`. @@ -127,21 +127,22 @@ If a Component does send a message to someone without having signed in, the Coor :::{mermaid} sequenceDiagram - Note over CA,Co1: ID "CA" is still free - CA ->> Co1: COORDINATOR|CA|SIGNIN - Note right of Co1: Connection identity "IA" - Note right of Co1: Stores "CA" with identity "IA" - Co1 ->> CA: Co1.CA|Co1.COORDINATOR|ACKNOWLEDGE: Node ID is "Co1" - Note left of CA: Stores "Co1" as Node ID - Note over CA,Co1: ID "CA" is already used - CA ->> Co1: COORDINATOR|CA|SIGNIN - Co1 ->> CA: CA|Co1.COORDINATOR|ERROR: ID "CA" is already used. + Note over CA,N1: ID "CA" is still free + participant N1 as N1.COORDINATOR + CA ->> N1: COORDINATOR|CA|SIGNIN + Note right of N1: Connection identity "IA" + Note right of N1: Stores "CA" with identity "IA" + N1 ->> CA: N1.CA|N1.COORDINATOR|ACKNOWLEDGE: Node ID is "N1" + Note left of CA: Stores "N1" as Node ID + Note over CA,N1: ID "CA" is already used + CA ->> N1: COORDINATOR|CA|SIGNIN + N1 ->> CA: CA|N1.COORDINATOR|ERROR: ID "CA" is already used. Note left of CA: May retry with another ID - Note over CA,Co1: "CA" has not send SIGNIN + Note over CA,N1: "CA" has not send SIGNIN Note left of CA: Wants to send a message to CB - CA ->> Co1: Co1.CB|CA|Content - Note right of Co1: Does not know CA - Co1 ->> CA: CA|Co1.COORDINATOR|ERROR:I do not know you + CA ->> N1: N1.CB|CA|Content + Note right of N1: Does not know CA + N1 ->> CA: CA|N1.COORDINATOR|ERROR:I do not know you Note left of CA: Should send a SIGNIN message ::: @@ -169,9 +170,10 @@ It shall also publish to the other Coordinators in the network that this Compone :::{mermaid} sequenceDiagram - CA ->> Co1: COORDINATOR|Co1.CA|SIGNOUT - Co1 ->> CA: Co1.CA|Co1.COORDINATOR|ACKNOWLEDGE - Note right of Co1: Removes "CA" with identity "IA"
from address book + CA ->> N1: COORDINATOR|N1.CA|SIGNOUT + participant N1 as N1.COORDINATOR + N1 ->> CA: N1.CA|N1.COORDINATOR|ACKNOWLEDGE + Note right of N1: Removes "CA" with identity "IA"
from address book Note left of CA: Shall not send any message anymore except SIGNIN ::: @@ -187,28 +189,31 @@ Coordinators shall hand on the message to the corresponding Coordinator or conne :::{mermaid} sequenceDiagram alt full ID - CA ->> Co1: Co1.CB|Co1.CA| Give me property A. + CA ->> N1: N1.CB|N1.CA| Give me property A. else only Component ID - CA ->> Co1: CB|Co1.CA| Give me property A. + CA ->> N1: CB|N1.CA| Give me property A. end - Co1 ->> CB: Co1.CB|Co1.CA| Give me property A. + participant N1 as N1.COORDINATOR + N1 ->> CB: N1.CB|N1.CA| Give me property A. Note left of CB: Reads property A - CB ->> Co1: Co1.CA|Co1.CB| Property A has value 5. - Co1 ->> CA: Co1.CA|Co1.CB| Property A has value 5. + CB ->> N1: N1.CA|N1.CB| Property A has value 5. + N1 ->> CA: N1.CA|N1.CB| Property A has value 5. ::: :::{mermaid} sequenceDiagram - CA ->> Co1: Co2.CB|Co1.CA| Give me property A. - Note over Co1,Co2: Co1.DEALER socket sends to Co2.ROUTER - Co1 ->> Co2: Co2.CB|Co1.CA| Give me property A. - Co2 ->> CB: Co2.CB|Co1.CA| Give me property A. + CA ->> N1: N2.CB|N1.CA| Give me property A. + participant N1 as N1.COORDINATOR + Note over N1,N2: N1 DEALER socket sends to N2 ROUTER + participant N2 as N2.COORDINATOR + N1 ->> N2: N2.CB|N1.CA| Give me property A. + N2 ->> CB: N2.CB|N1.CA| Give me property A. Note left of CB: Reads property A - CB ->> Co2: Co1.CA|Co2.CB| Property A has value 5. - Note over Co1,Co2: Co2.DEALER socket sends to Co1.ROUTER - Co2 ->> Co1: Co1.CA|Co2.CB| Property A has value 5. - Co1 ->> CA: Co1.CA|Co2.CB| Property A has value 5. + CB ->> N2: N1.CA|N2.CB| Property A has value 5. + Note over N1,N2: N2 DEALER socket sends to N1 ROUTER + N2 ->> N1: N1.CA|N2.CB| Property A has value 5. + N1 ->> CA: N1.CA|N2.CB| Property A has value 5. ::: Prerequisite of Communication between two Components are: From fff85bead2c0d10c64b232b7d09b2bd4fded0d70 Mon Sep 17 00:00:00 2001 From: Benedikt Moneke <67148916+bmoneke@users.noreply.github.com> Date: Tue, 7 Feb 2023 18:00:14 +0100 Subject: [PATCH 09/27] Change from ID to name. Also address book becomes directory. --- control_protocol.md | 142 ++++++++++++++++++++++++++++---------------- 1 file changed, 90 insertions(+), 52 deletions(-) diff --git a/control_protocol.md b/control_protocol.md index b1a90c7..b940326 100644 --- a/control_protocol.md +++ b/control_protocol.md @@ -51,7 +51,7 @@ If you call the ROUTER's send command with `IA|Reply A`, the socket will send `R The following diagram shows this example communication with two Components: sequenceDiagram - participant Code + participant Code as Message handling participant ROUTER as ROUTER socket Note over Code, ROUTER: Coordinator CA ->> ROUTER: "Request A" @@ -69,43 +69,45 @@ sequenceDiagram #### Naming scheme -Each Component has an individual ID, given by the user, the _Component ID_. -A Component ID must be a series of bytes, without the ASCII character "." (byte value 46). -Component IDs must be unique in a {ref}`Node `, i.e. among the Componets connected to a single Coordinator. -The Coordinator itself has the Component ID `COORDINATOR`. +Each Component has an individual name, given by the user, the _Component Name_. +A Component Name must be a series of bytes, without the ASCII character "." (byte value 46). +Component Names must be unique in a {ref}`Node `, i.e. among the Components connected to a single Coordinator. +The Coordinator itself has the Component Name `COORDINATOR`. :::{note} COORDINATOR is a placeholder for the final version. ::: -Also every Node ID has to be unique in the Network. -As each Component belongs to exactly one Coordinator, it is fully identified by the combination of Node ID and Component ID., which is globally unique. -This _full ID_ is the composition of Node ID, ".", and Component ID. -For example `N1.CA` is the full ID of the Component `CA` in the Node `N1`. +Similarly, every node has a name, the _Namespace_. +Every Namespace has to be unique in the Network. +As each Component belongs to exactly one Node, it is fully identified by the combination of Namespace and Component Name, which is globally unique. +This _full name_ is the composition of Namespace, ".", and Component Name. +For example `N1.CA` is the full name of the Component `CA` in the Node `N1`. -The receiver of a message may be specified without the Node ID, if the receiver lives in the same Node. +The receiver of a message may be specified without the Namespace, if the receiver lives in the same Node. #### Message composition A message consists in two or more frames. -1. The receiver full ID. -2. The sender full ID. -3. Message content: The optional payload, which can be 0 or more frames. +1. The protocol version. +2. The receiver full name. +3. The sender full name. +4. A content header. +5. Message content: The optional payload, which can be 0 or more frames. -(address-book)= -#### Address book +#### Directory Each Coordinator shall have a list of the Components connected to it. -This is its _local address book_. +This is its _local Directory_. -The _global address book_ is the combination of the local address books of all Coordinators in a Network. +The _global Directory_ is the combination of the local directories of all Coordinators in a Network. ### Conversation protocol -In the protocol examples, `CA`, `CB`, etc. indicate Component IDs. -`N1`, `N2`, etc. indicate Node IDs and `Co1`, `Co2` their corresponding Coordinators. +In the protocol examples, `CA`, `CB`, etc. indicate Component Names. +`N1`, `N2`, etc. indicate Node Namespaces and `Co1`, `Co2` their corresponding Coordinators. Here the Message content is expressed in plain English, for the exact definition see {ref}`message-layer`. @@ -117,27 +119,27 @@ In the exchange of messages, only the messages over the wire are shown, the conn (sign-in)= ##### Initial connection -After connecting to a Coordinator (`Co1`), a Component (`CA`) shall send a SIGNIN message indicating its ID. -The Coordinator shall respond with an ACKNOWLEDGE, giving the Node ID and other relevant information, or with an ERROR, if the ID is already taken. -After that successful handshake, the Coordinator shall store the connection identity and corresponding Component ID in its local address book. +After connecting to a Coordinator (`Co1`), a Component (`CA`) shall send a SIGNIN message indicating its Component Name. +The Coordinator shall respond with an ACKNOWLEDGE, giving the Namespace and other relevant information, or with an ERROR, if the Component Name is already taken. +After that successful handshake, the Coordinator shall store the connection identity and corresponding Component Name in its local {ref}`directory`. It shall also publish to the other Coordinators in the network that this Component signed in, see {ref}`Coordinator coordination`. -Similarly, the Component shall store the Node ID and use it from this moment to generate its full ID. +Similarly, the Component shall store the Namespace and use it from this moment to generate its full name. If a Component does send a message to someone without having signed in, the Coordinator shall refuse message handling and return an error. :::{mermaid} sequenceDiagram - Note over CA,N1: ID "CA" is still free + Note over CA,N1: Name "CA" is still free participant N1 as N1.COORDINATOR CA ->> N1: COORDINATOR|CA|SIGNIN Note right of N1: Connection identity "IA" Note right of N1: Stores "CA" with identity "IA" - N1 ->> CA: N1.CA|N1.COORDINATOR|ACKNOWLEDGE: Node ID is "N1" - Note left of CA: Stores "N1" as Node ID - Note over CA,N1: ID "CA" is already used + N1 ->> CA: N1.CA|N1.COORDINATOR|ACKNOWLEDGE: Namespace is "N1" + Note left of CA: Stores "N1" as Namespace + Note over CA,N1: Name "CA" is already used CA ->> N1: COORDINATOR|CA|SIGNIN - N1 ->> CA: CA|N1.COORDINATOR|ERROR: ID "CA" is already used. - Note left of CA: May retry with another ID + N1 ->> CA: CA|N1.COORDINATOR|ERROR: Name "CA" is already used. + Note left of CA: May retry with another Name Note over CA,N1: "CA" has not send SIGNIN Note left of CA: Wants to send a message to CB CA ->> N1: N1.CB|CA|Content @@ -153,19 +155,18 @@ We use heartbeat to know, whether a communication partner is still online. Every message received counts as a heartbeat. -A Component should and a Coordinator shall send a STATUS request and wait some time before considering a connection dead. -A Coordinator shall follow the {ref}`sign out routine` for a signed in Component considered dead. +A Component should and a Coordinator shall send a PING and wait some time before considering a connection dead. +A Coordinator shall follow the {ref}`sign out routine` for a signed in Component considered dead. :::{note} TBD: Respond to every non empty message with an empty one? ::: -(sign-out)= ##### Signing out A Component should tell a Coordinator, when it stops working, with a SIGNOUT message. -The Coordinator shall ACKNOWLEDGE the sign out and remove the ID from its address book. +The Coordinator shall ACKNOWLEDGE the sign out and remove the Name from its directory. It shall also publish to the other Coordinators in the network that this Component signed out, see {ref}`Coordinator coordination`. :::{mermaid} @@ -173,7 +174,7 @@ sequenceDiagram CA ->> N1: COORDINATOR|N1.CA|SIGNOUT participant N1 as N1.COORDINATOR N1 ->> CA: N1.CA|N1.COORDINATOR|ACKNOWLEDGE - Note right of N1: Removes "CA" with identity "IA"
from address book + Note right of N1: Removes "CA" with identity "IA"
from directory Note left of CA: Shall not send any message anymore except SIGNIN ::: @@ -188,9 +189,9 @@ Coordinators shall hand on the message to the corresponding Coordinator or conne :::{mermaid} sequenceDiagram - alt full ID + alt full name CA ->> N1: N1.CB|N1.CA| Give me property A. - else only Component ID + else only Component name CA ->> N1: CB|N1.CA| Give me property A. end participant N1 as N1.COORDINATOR @@ -222,14 +223,14 @@ Prerequisite of Communication between two Components are: The following flow chart shows the decision scheme and message modification in a Coordinator `Co1` of Node `N1`. -Its full ID is `N1.Coordinator`. -`n0`, `nR` are placeholders for sender and recipient Node ID. -`recipient` is a placeholder for the recipient Component ID. +Its full name is `N1.Coordinator`. +`n0`, `nR` are placeholders for sender and recipient Namespaces. +`recipient` is a placeholder for the recipient Component name. `IA` is the connection identity of the Component `N1.CA` (if it is directly connected to `Co1`) or of its Coordinator `n0.COORDINATOR`. -`IB` is the connection identities of `N1.Recipient`. +`IB` is the connection identity of `N1.Recipient`. Bold arrows indicate message flow, thin lines indicate decision flow. Thin, dotted lines indicate decision flow in case of errors. -Placeholder values are written in lowercase, while actually known values are begin with an uppercase letter. +Placeholder values are written in lowercase, while actually known values begin with an uppercase letter. :::{mermaid} flowchart TB @@ -242,7 +243,7 @@ flowchart TB CidKnown -->|yes| RemIdent Clocal -.->|no| E1[ERROR: Sender did not sign in] ==>|"IA|n0.CA|N1.COORDINATOR|ERROR: Sender dit not sign in
(==IA|N1.CA|N1.COORDINATOR|ERROR...)"| S S[send] ==> WA([N1.CA DEALER]) - CidKnown -.->|no| E2[ERROR: ID and identity do not match]==>|"IA|n0.CA|N1.COORDINATOR|ERROR: ID and identity do not match
(==IA|N1.CA|N1.COORDINATOR|ERROR...)"| S + CidKnown -.->|no| E2[ERROR: Name and identity do not match]==>|"IA|n0.CA|N1.COORDINATOR|ERROR: Name and identity do not match
(==IA|N1.CA|N1.COORDINATOR|ERROR...)"| S RemIdent[remove sender identity] == "nr.recipient|n0.CA|Content" ==> Cnr Cnr -- "is None" --> Local Cnr{nr?} -- "== N1"--> Local @@ -269,28 +270,65 @@ flowchart TB ::: -(coordinator-coordination)= #### Coordinator coordination -Each Coordinator shall keep an up to date {ref}`global address book` with the IDs of all Components in the Network. +Each Coordinator shall keep an up to date global {ref}`directory` with the names of all Components in the Network. For this, Coordinators shall tell each other regarding signing in and signing out Components and Coordinators. -Coordinators shall send on request the IDs of their local address book, or of their global address book, depending on the request type. +Coordinators shall send on request the Names of their local directory, or of their global directory, depending on the request type. +For the format of the Messages, see {ref}`message-layer`. -Necessary information: -- Event type (connect or disconnect) -- Full ID (Node ID and Component ID) of the Component -:::{note} -TODO decide whether via Control or Data protocol. -TODO add the log in of a Coordinator -::: +##### Coordinator sign in + +Two Coordinators shall follow a more thorough sign in procedure, than Components: + +1. Coordinator Co1 signs in to Co2 + 1. Co1 creates a DEALER socket and connects to another Coordinator Co2 (ROUTER socket). + 2. Co1 sends a CO-SIGNIN message indicating its own address (host and port) to Co2. + 3. Co2 creates a DEALER socket and connects to Co1's DEALER socket. + 4. Co2 stores the the Namespace of Co1 and references it to the corresponding DEALER socket + 5. Co2 acknowledges to Co1 the Coordinator sign in. +2. Co2 signs in to Co1 + 6. Co2 sends a CO-SIGNIN message to Co1. + 7. Co1 stores the Co2 Namespace and references it to the corresponding DEALER socket + 8. Co1 ackwnowledges to Co2 the Coordinator sign in. +3. Co1 and Co2 request the local address book of the other one. +4. If desired (see below), Co2 tells all its known Coordinators, that they shall sign in to Co1 + +In a CO-SIGNIN message, a Coordinator may indicate, that the other Coordinator shall tell all the other Coordinators, that it is present in the Network. +For example Co1 asks this from Co2 and Co2 tells Co3, Co4, etc., that they shall sign in to Co1. +That way all Coordinators are connected to all other ones. + + +##### Coordinator updates + +Whenever a Component signs in to or out of its Coordinator, the Coordinator shall send a note regarding this event to all the other Coordinators. +The note shall contain the full name of the Component and the event type (sign in or out) +The other Coordinators shall update their global directory according to this note (add or remove an entry). + + + +TODO Add full update (full local list) +TODO Problem: If a full list arrives after an update: On the other hand: One message after the other: Sequence should work out. (message-layer)= ## Message layer + + +### Messages for Transport Layer + +- SIGNIN +- SIGNOUT +- ACKNOWLEDGE +- ERROR +- PING + + + :::{note} TODO ::: From 553b37f94f16d20934b5260096f94c5492418aa3 Mon Sep 17 00:00:00 2001 From: Benedikt Moneke <67148916+bmoneke@users.noreply.github.com> Date: Tue, 7 Feb 2023 18:33:28 +0100 Subject: [PATCH 10/27] Version and Header added to message flow. --- control_protocol.md | 87 ++++++++++++++++++++++++--------------------- 1 file changed, 46 insertions(+), 41 deletions(-) diff --git a/control_protocol.md b/control_protocol.md index b940326..f75fc15 100644 --- a/control_protocol.md +++ b/control_protocol.md @@ -89,10 +89,10 @@ The receiver of a message may be specified without the Namespace, if the receive #### Message composition A message consists in two or more frames. -1. The protocol version. +1. The protocol version (abbreviated with "V" in examples). 2. The receiver full name. 3. The sender full name. -4. A content header. +4. A content header (abbreviated with "H" in examples). 5. Message content: The optional payload, which can be 0 or more frames. @@ -131,20 +131,20 @@ If a Component does send a message to someone without having signed in, the Coor sequenceDiagram Note over CA,N1: Name "CA" is still free participant N1 as N1.COORDINATOR - CA ->> N1: COORDINATOR|CA|SIGNIN + CA ->> N1: V|COORDINATOR|CA|H|SIGNIN Note right of N1: Connection identity "IA" Note right of N1: Stores "CA" with identity "IA" - N1 ->> CA: N1.CA|N1.COORDINATOR|ACKNOWLEDGE: Namespace is "N1" + N1 ->> CA: V|N1.CA|N1.COORDINATOR|H|ACKNOWLEDGE: Namespace is "N1" Note left of CA: Stores "N1" as Namespace Note over CA,N1: Name "CA" is already used - CA ->> N1: COORDINATOR|CA|SIGNIN - N1 ->> CA: CA|N1.COORDINATOR|ERROR: Name "CA" is already used. + CA ->> N1: V|COORDINATOR|CA|H|SIGNIN + N1 ->> CA: V|CA|N1.COORDINATOR|H|ERROR: Name "CA" is already used. Note left of CA: May retry with another Name Note over CA,N1: "CA" has not send SIGNIN Note left of CA: Wants to send a message to CB - CA ->> N1: N1.CB|CA|Content + CA ->> N1: V|N1.CB|CA|H|Content Note right of N1: Does not know CA - N1 ->> CA: CA|N1.COORDINATOR|ERROR:I do not know you + N1 ->> CA: V|CA|N1.COORDINATOR|H|ERROR:I do not know you Note left of CA: Should send a SIGNIN message ::: @@ -171,9 +171,9 @@ It shall also publish to the other Coordinators in the network that this Compone :::{mermaid} sequenceDiagram - CA ->> N1: COORDINATOR|N1.CA|SIGNOUT + CA ->> N1: V|COORDINATOR|N1.CA|H|SIGNOUT participant N1 as N1.COORDINATOR - N1 ->> CA: N1.CA|N1.COORDINATOR|ACKNOWLEDGE + N1 ->> CA: V|N1.CA|N1.COORDINATOR|H|ACKNOWLEDGE Note right of N1: Removes "CA" with identity "IA"
from directory Note left of CA: Shall not send any message anymore except SIGNIN ::: @@ -190,31 +190,31 @@ Coordinators shall hand on the message to the corresponding Coordinator or conne :::{mermaid} sequenceDiagram alt full name - CA ->> N1: N1.CB|N1.CA| Give me property A. + CA ->> N1: V|N1.CB|N1.CA|H| Give me property A. else only Component name - CA ->> N1: CB|N1.CA| Give me property A. + CA ->> N1: V|CB|N1.CA|H| Give me property A. end participant N1 as N1.COORDINATOR - N1 ->> CB: N1.CB|N1.CA| Give me property A. + N1 ->> CB: V|N1.CB|N1.CA|H| Give me property A. Note left of CB: Reads property A - CB ->> N1: N1.CA|N1.CB| Property A has value 5. - N1 ->> CA: N1.CA|N1.CB| Property A has value 5. + CB ->> N1: V|N1.CA|N1.CB|H| Property A has value 5. + N1 ->> CA: V|N1.CA|N1.CB|H| Property A has value 5. ::: :::{mermaid} sequenceDiagram - CA ->> N1: N2.CB|N1.CA| Give me property A. + CA ->> N1: V|N2.CB|N1.CA|H| Give me property A. participant N1 as N1.COORDINATOR Note over N1,N2: N1 DEALER socket sends to N2 ROUTER participant N2 as N2.COORDINATOR - N1 ->> N2: N2.CB|N1.CA| Give me property A. - N2 ->> CB: N2.CB|N1.CA| Give me property A. + N1 ->> N2: V|N2.CB|N1.CA|H| Give me property A. + N2 ->> CB: V|N2.CB|N1.CA|H| Give me property A. Note left of CB: Reads property A - CB ->> N2: N1.CA|N2.CB| Property A has value 5. + CB ->> N2: V|N1.CA|N2.CB|H| Property A has value 5. Note over N1,N2: N2 DEALER socket sends to N1 ROUTER - N2 ->> N1: N1.CA|N2.CB| Property A has value 5. - N1 ->> CA: N1.CA|N2.CB| Property A has value 5. + N2 ->> N1: V|N1.CA|N2.CB|H| Property A has value 5. + N1 ->> CA: V|N1.CA|N2.CB|H| Property A has value 5. ::: Prerequisite of Communication between two Components are: @@ -224,39 +224,40 @@ Prerequisite of Communication between two Components are: The following flow chart shows the decision scheme and message modification in a Coordinator `Co1` of Node `N1`. Its full name is `N1.Coordinator`. -`n0`, `nR` are placeholders for sender and recipient Namespaces. +`nS`, `nR` are placeholders for sender and recipient Namespaces. `recipient` is a placeholder for the recipient Component name. -`IA` is the connection identity of the Component `N1.CA` (if it is directly connected to `Co1`) or of its Coordinator `n0.COORDINATOR`. +`iA` is the connection identity of the incoming message, for example `IA` for Component `N1.CA`. `IB` is the connection identity of `N1.Recipient`. Bold arrows indicate message flow, thin lines indicate decision flow. Thin, dotted lines indicate decision flow in case of errors. Placeholder values are written in lowercase, while actually known values begin with an uppercase letter. +For brevity's sake, the protocol version (in front of recipient frame) and the header (in front of content) are not shown. :::{mermaid} flowchart TB - C1([N1.CA DEALER]) == "nr.recipient|n0.CA|Content
(==nr.recipient|N1.CA|Content)" ==> R0 - C0([n0.COORDINATOR DEALER]) == "nr.recipient|n0.CA|Content" ==> R0 - R0[receive] == "IA|nr.recipient|n0.CA|Content" ==> Cn0{n0 == N1?} - Cn0-->|no| RemIdent - Cn0-->|yes| Clocal{CA in
local address book?} - Clocal -->|yes| CidKnown{IA is CA's identity
in address book?} + C1([N1.CA DEALER]) == "nR.recipient|nS.CA|Content
(==nR.recipient|N1.CA|Content)" ==> R0 + C0([nS.COORDINATOR DEALER]) == "V|nR.recipient|nS.CA|Content" ==> R0 + R0[receive] == "iaA|nR.recipient|nS.CA|Content" ==> CnS{nS == N1?} + CnS-->|no| RemIdent + CnS-->|yes| Clocal{CA in
local address book?} + Clocal -->|yes| CidKnown{iA is CA's identity
in address book?} CidKnown -->|yes| RemIdent - Clocal -.->|no| E1[ERROR: Sender did not sign in] ==>|"IA|n0.CA|N1.COORDINATOR|ERROR: Sender dit not sign in
(==IA|N1.CA|N1.COORDINATOR|ERROR...)"| S + Clocal -.->|no| E1[ERROR: Sender unknown] ==>|"iA|nS.CA|N1.COORDINATOR|ERROR: Sender unknown in
(==IA|N1.CA|N1.COORDINATOR|ERROR...)"| S S[send] ==> WA([N1.CA DEALER]) - CidKnown -.->|no| E2[ERROR: Name and identity do not match]==>|"IA|n0.CA|N1.COORDINATOR|ERROR: Name and identity do not match
(==IA|N1.CA|N1.COORDINATOR|ERROR...)"| S - RemIdent[remove sender identity] == "nr.recipient|n0.CA|Content" ==> Cnr - Cnr -- "is None" --> Local - Cnr{nr?} -- "== N1"--> Local + CidKnown -.->|no| E2[ERROR: Name and identity do not match]==>|"iA|nS.CA|N1.COORDINATOR|ERROR: Name and identity do not match
(==IA|N1.CA|N1.COORDINATOR|ERROR...)"| S + RemIdent[remove sender identity] == "nR.recipient|nS.CA|Content" ==> CnR + CnR -- "is None" --> Local + CnR{nR?} -- "== N1"--> Local Local{recipient
==
COORDINATOR?} -- "yes" --> Self[Message for Co1
itself] - Self == "nr.recipient|n0.CA|Content
(==N1.COORDINATOR|n0.CA|Content)" ==> SC([Co1 Message handling]) + Self == "nR.recipient|nS.CA|Content
(==N1.COORDINATOR|nS.CA|Content)" ==> SC([Co1 Message handling]) Local -- "no" --> Local2a{recipient in Address book?} Local2a -->|yes, with Identity IB| Local2 - Local2[add recipient identity IB] == "IB|nr.recipient|n0.CA|Content
(==IB|N1.recipient|n0.CA|Content)" ==> R1[send] - R1 == "nr.recipient|n0.CA|Content
(==N1.recipient|n0.CA|Content)" ==> W1([Wire to N1.recipient DEALER]) - Local2a -.->|no| E3[ERROR recipient unknown
send Error to original sender] ==>|"n0.CA|N1.COORDINATOR|ERROR N1.recipient is unknown"|Cnr - Cnr -- "== N2" --> Keep - Keep[send to N2.COORDINATOR] == "nr.recipient|n0.CA|Content
(==N2.recipient|n0.CA|Content)" ==> R2[send] - R2 == "nr.recipient|n0.CA|Content
(==N2.recipient|n0.CA|Content)" ==> W2([Wire to N2.COORDINATOR ROUTER]) + Local2[add recipient identity IB] == "IB|nR.recipient|nS.CA|Content
(==IB|N1.recipient|nS.CA|Content)" ==> R1[send] + R1 == "nR.recipient|nS.CA|Content
(==N1.recipient|nS.CA|Content)" ==> W1([Wire to N1.recipient DEALER]) + Local2a -.->|no| E3[ERROR recipient unknown
send Error to original sender] ==>|"nS.CA|N1.COORDINATOR|ERROR N1.recipient is unknown"|CnR + CnR -- "== N2" --> Keep + Keep[send to N2.COORDINATOR] == "nR.recipient|nS.CA|Content
(==N2.recipient|nS.CA|Content)" ==> R2[send] + R2 == "nR.recipient|nS.CA|Content
(==N2.recipient|nS.CA|Content)" ==> W2([Wire to N2.COORDINATOR ROUTER]) subgraph Co1 ROUTER socket R0 end @@ -281,6 +282,10 @@ For the format of the Messages, see {ref}`message-layer`. ##### Coordinator sign in +:::{note} +TODO +::: + Two Coordinators shall follow a more thorough sign in procedure, than Components: 1. Coordinator Co1 signs in to Co2 From daee254549b2dd3d5b6d0f15e321d13dc22cdb3d Mon Sep 17 00:00:00 2001 From: Benedikt Moneke <67148916+bmoneke@users.noreply.github.com> Date: Tue, 7 Feb 2023 18:47:21 +0100 Subject: [PATCH 11/27] Graph changed to variables only. --- control_protocol.md | 36 +++++++++++++++++------------------- 1 file changed, 17 insertions(+), 19 deletions(-) diff --git a/control_protocol.md b/control_protocol.md index f75fc15..13ae98e 100644 --- a/control_protocol.md +++ b/control_protocol.md @@ -226,38 +226,36 @@ The following flow chart shows the decision scheme and message modification in a Its full name is `N1.Coordinator`. `nS`, `nR` are placeholders for sender and recipient Namespaces. `recipient` is a placeholder for the recipient Component name. -`iA` is the connection identity of the incoming message, for example `IA` for Component `N1.CA`. -`IB` is the connection identity of `N1.Recipient`. +`iA` is a placeholder for the connection identity of the incoming message and `iB` that of `N1.Recipient`. Bold arrows indicate message flow, thin lines indicate decision flow. Thin, dotted lines indicate decision flow in case of errors. Placeholder values are written in lowercase, while actually known values begin with an uppercase letter. -For brevity's sake, the protocol version (in front of recipient frame) and the header (in front of content) are not shown. :::{mermaid} flowchart TB - C1([N1.CA DEALER]) == "nR.recipient|nS.CA|Content
(==nR.recipient|N1.CA|Content)" ==> R0 - C0([nS.COORDINATOR DEALER]) == "V|nR.recipient|nS.CA|Content" ==> R0 - R0[receive] == "iaA|nR.recipient|nS.CA|Content" ==> CnS{nS == N1?} + C1([N1.CA DEALER]) == "V|nR.recipient|nS.CA|H|Content" ==> R0 + C0([nS.COORDINATOR DEALER]) == "V|nR.recipient|nS.CA|H|Content" ==> R0 + R0[receive] == "iA|V|nR.recipient|nS.CA|H|Content" ==> CnS{nS == N1?} CnS-->|no| RemIdent - CnS-->|yes| Clocal{CA in
local address book?} - Clocal -->|yes| CidKnown{iA is CA's identity
in address book?} + CnS-->|yes| Clocal{CA in
local directory?} + Clocal -->|yes| CidKnown{iA is CA's identity
in directory?} CidKnown -->|yes| RemIdent - Clocal -.->|no| E1[ERROR: Sender unknown] ==>|"iA|nS.CA|N1.COORDINATOR|ERROR: Sender unknown in
(==IA|N1.CA|N1.COORDINATOR|ERROR...)"| S + Clocal -.->|no| E1[ERROR: Sender unknown] ==>|"iA|V|nS.CA|N1.COORDINATOR|H|ERROR: Sender unknown"| S S[send] ==> WA([N1.CA DEALER]) - CidKnown -.->|no| E2[ERROR: Name and identity do not match]==>|"iA|nS.CA|N1.COORDINATOR|ERROR: Name and identity do not match
(==IA|N1.CA|N1.COORDINATOR|ERROR...)"| S - RemIdent[remove sender identity] == "nR.recipient|nS.CA|Content" ==> CnR + CidKnown -.->|no| E2[ERROR: Name and identity do not match]==>|"iA|V|nS.CA|N1.COORDINATOR|H|ERROR: Name and identity do not match"| S + RemIdent[remove sender identity] == "V|nR.recipient|nS.CA|H|Content" ==> CnR CnR -- "is None" --> Local CnR{nR?} -- "== N1"--> Local Local{recipient
==
COORDINATOR?} -- "yes" --> Self[Message for Co1
itself] - Self == "nR.recipient|nS.CA|Content
(==N1.COORDINATOR|nS.CA|Content)" ==> SC([Co1 Message handling]) - Local -- "no" --> Local2a{recipient in Address book?} - Local2a -->|yes, with Identity IB| Local2 - Local2[add recipient identity IB] == "IB|nR.recipient|nS.CA|Content
(==IB|N1.recipient|nS.CA|Content)" ==> R1[send] - R1 == "nR.recipient|nS.CA|Content
(==N1.recipient|nS.CA|Content)" ==> W1([Wire to N1.recipient DEALER]) - Local2a -.->|no| E3[ERROR recipient unknown
send Error to original sender] ==>|"nS.CA|N1.COORDINATOR|ERROR N1.recipient is unknown"|CnR + Self == "V|nR.recipient|nS.CA|H|Content" ==> SC([Co1 Message handling]) + Local -- "no" --> Local2a{recipient in directory?} + Local2a -->|yes, with Identity iB| Local2 + Local2[add recipient identity iB] == "iB|V|nR.recipient|nS.CA|H|Content" ==> R1[send] + R1 == "V|nR.recipient|nS.CA|H|Content" ==> W1([Wire to N1.recipient DEALER]) + Local2a -.->|no| E3[ERROR recipient unknown
send Error to original sender] ==>|"V|nS.CA|N1.COORDINATOR|H|ERROR N1.recipient is unknown"|CnR CnR -- "== N2" --> Keep - Keep[send to N2.COORDINATOR] == "nR.recipient|nS.CA|Content
(==N2.recipient|nS.CA|Content)" ==> R2[send] - R2 == "nR.recipient|nS.CA|Content
(==N2.recipient|nS.CA|Content)" ==> W2([Wire to N2.COORDINATOR ROUTER]) + Keep[send to N2.COORDINATOR] == "V|nR.recipient|nS.CA|H|Content" ==> R2[send] + R2 == "V|nR.recipient|nS.CA|H|Content" ==> W2([Wire to N2.COORDINATOR ROUTER]) subgraph Co1 ROUTER socket R0 end From 705aa8af82b7e33b487bfa82a8d8503f7f51afc9 Mon Sep 17 00:00:00 2001 From: Benedikt Moneke <67148916+bmoneke@users.noreply.github.com> Date: Wed, 8 Feb 2023 13:44:15 +0100 Subject: [PATCH 12/27] Coordinator-Coordinator sign in/out added. --- control_protocol.md | 79 +++++++++++++++++++++++++++++---------------- 1 file changed, 52 insertions(+), 27 deletions(-) diff --git a/control_protocol.md b/control_protocol.md index 13ae98e..31c72e6 100644 --- a/control_protocol.md +++ b/control_protocol.md @@ -90,7 +90,7 @@ The receiver of a message may be specified without the Namespace, if the receive A message consists in two or more frames. 1. The protocol version (abbreviated with "V" in examples). -2. The receiver full name. +2. The receiver full name or Component name, if sender and receiver live in the same Node (Namespace). 3. The sender full name. 4. A content header (abbreviated with "H" in examples). 5. Message content: The optional payload, which can be 0 or more frames. @@ -280,28 +280,55 @@ For the format of the Messages, see {ref}`message-layer`. ##### Coordinator sign in -:::{note} -TODO -::: +A Coordinator `Co1`joining a network follows a few steps: +1. It signs in to one Coordinator `Co2` of the Network. +2. It sends a CO_TELL_ALL message to `Co2`, to tell all other Coordinators about `Co1`s address. +3. `Co2` tells all the Coordinators signed in (`Co3`, `Co4`...) about `Co1` with a CO_NEW message. +4. These other Coordinators (`Co3`, `Co4`...) sign in to `Co1`. +5. All Coordinators are connected to all others. -Two Coordinators shall follow a more thorough sign in procedure, than Components: +Two Coordinators shall follow a more thorough sign in/sign out procedure, than Components (address is for example host an port). +The sign in might happen because of a CO_NEW message arrived or at startup. +The sign out might happen because the Coordinator shuts down. -1. Coordinator Co1 signs in to Co2 - 1. Co1 creates a DEALER socket and connects to another Coordinator Co2 (ROUTER socket). - 2. Co1 sends a CO-SIGNIN message indicating its own address (host and port) to Co2. - 3. Co2 creates a DEALER socket and connects to Co1's DEALER socket. - 4. Co2 stores the the Namespace of Co1 and references it to the corresponding DEALER socket - 5. Co2 acknowledges to Co1 the Coordinator sign in. -2. Co2 signs in to Co1 - 6. Co2 sends a CO-SIGNIN message to Co1. - 7. Co1 stores the Co2 Namespace and references it to the corresponding DEALER socket - 8. Co1 ackwnowledges to Co2 the Coordinator sign in. -3. Co1 and Co2 request the local address book of the other one. -4. If desired (see below), Co2 tells all its known Coordinators, that they shall sign in to Co1 - -In a CO-SIGNIN message, a Coordinator may indicate, that the other Coordinator shall tell all the other Coordinators, that it is present in the Network. -For example Co1 asks this from Co2 and Co2 tells Co3, Co4, etc., that they shall sign in to Co1. -That way all Coordinators are connected to all other ones. +:::{mermaid} +sequenceDiagram + participant r1 as ROUTER + participant d1 as DEALER + participant r2 as ROUTER + participant d2 as DEALER + Note over r1,d1: N1 Coordinator
at address1 + Note over r2,d2: N2 Coordinator
at address2 + Note over r1,d2: Sign in between two Coordinators + Note right of r1: shall connect
to address2 + activate d1 + Note left of d1: created with
name "temp-NS" + d1-->>r2: connect to address2 + d1->>r2: CO_SIGNIN
N1, address1,
ref:temp-NS + par + d1->>r2: GET local directory + and + Note right of r2: stores N1 identity + activate d2 + Note left of d2: created with
name "N1" + d2-->>r1: connect to address1 + d2->>r1: CO_SIGNIN
N2, address2
your ref:temp-NS + Note right of r1: stores N2 identity + Note left of d1: name changed
from "temp-NS"
to "N2" + d2->>r1: GET local directory + end + d2->>r1: Here is my
local directory + Note right of r1: Updates
global directory + d1->>r2: Here is my
local directory + Note right of r2: Updates
global directory + Note over r1,d2: Sign out between two Coordinators + Note right of r1: shall sign out from N2 + d1->>r2: CO_SIGNOUT + Note right of r2: removes N1 identity + d2->>-r1: CO_SIGNOUT + Note right of r1: removes N2 identity + deactivate d1 +::: ##### Coordinator updates @@ -311,12 +338,6 @@ The note shall contain the full name of the Component and the event type (sign i The other Coordinators shall update their global directory according to this note (add or remove an entry). - -TODO Add full update (full local list) -TODO Problem: If a full list arrives after an update: On the other hand: One message after the other: Sequence should work out. - - - (message-layer)= ## Message layer @@ -329,6 +350,10 @@ TODO Problem: If a full list arrives after an update: On the other hand: One mes - ACKNOWLEDGE - ERROR - PING +- CO_SIGNIN +- CO_SIGNOUT +- CO_TELL_ALL +- CO_NEW From b3226d29ad9a8cc4dc5c803cad2f58be7bf5d3c7 Mon Sep 17 00:00:00 2001 From: Benedikt Moneke <67148916+bmoneke@users.noreply.github.com> Date: Wed, 8 Feb 2023 17:12:09 +0100 Subject: [PATCH 13/27] Coordinator name set to COORDINATOR (ASCII). --- control_protocol.md | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/control_protocol.md b/control_protocol.md index 31c72e6..aae5ae5 100644 --- a/control_protocol.md +++ b/control_protocol.md @@ -72,10 +72,7 @@ sequenceDiagram Each Component has an individual name, given by the user, the _Component Name_. A Component Name must be a series of bytes, without the ASCII character "." (byte value 46). Component Names must be unique in a {ref}`Node `, i.e. among the Components connected to a single Coordinator. -The Coordinator itself has the Component Name `COORDINATOR`. -:::{note} -COORDINATOR is a placeholder for the final version. -::: +The Coordinator itself has the Component Name `COORDINATOR` (ASCII encoded). Similarly, every node has a name, the _Namespace_. Every Namespace has to be unique in the Network. From d4cd17f1df244c69e5d15a131550819c7afd8f05 Mon Sep 17 00:00:00 2001 From: Benedikt Moneke <67148916+bmoneke@users.noreply.github.com> Date: Thu, 9 Feb 2023 12:17:11 +0100 Subject: [PATCH 14/27] Language improvements by bilderbuchi Co-authored-by: Christoph Buchner --- control_protocol.md | 78 +++++++++++++++++++++++++-------------------- 1 file changed, 43 insertions(+), 35 deletions(-) diff --git a/control_protocol.md b/control_protocol.md index aae5ae5..5ea4292 100644 --- a/control_protocol.md +++ b/control_protocol.md @@ -7,18 +7,18 @@ The {ref]}`message-layer` is the common language to understand commands, thus cr (transport-layer)= ## Transport layer -The transport layer ensures, that a message arrives its destination. +The transport layer ensures that a message arrives at its destination. ### Sockets and Connections -We use [zmq](https://zeromq.org/) sockets for our communication. For more details see the [zmq guide](https://zguide.zeromq.org/) or [zmq API](http://api.zeromq.org/) +[Zmq](https://zeromq.org/) sockets are used for our communication. For more details see the [zmq guide](https://zguide.zeromq.org/) or [zmq API](http://api.zeromq.org/) #### Definitions Zmq messages consist in a series of frames, each is a byte sequence. -We indicate the separation between frames with `|`. +The separation between frames is indicated with `|`. An empty frame is indicated with two frame separators `||`, even at the beginning or end of a message. For example `||Second frame|Third frame||Fifth frame` consists of 5 frames. The first and fourth frames are empty frames. @@ -33,6 +33,10 @@ Each {ref}`Coordinator ` shall offer one ROUTER socke Coordinators shall have one DEALER socket per other Coordinator in the Network. This DEALER socket shall connect to the other Coordinator's ROUTER socket. +:::{note} +While the number of DEALER sockets thus required scales badly with the number of Connectors in a LECO Network, the scope of the protocol means that at most a few Coordinators will be involved. +::: + Messages must be sent to a Coordinator's ROUTER socket. @@ -42,9 +46,9 @@ Messages must be sent to a Coordinator's ROUTER socket. A small introduction to ROUTER sockets, for more details see [zmq guide chapter 3](https://zguide.zeromq.org/docs/chapter3/#Exploring-ROUTER-Sockets). A router socket assigns a random _identity_ to each connecting peer. -If, for example, two Components `CA`, `CB` connect to the ROUTER sockets, the socket assigns the identities, which we call `IA`, `IB` (they can be any bytes sequence). +If, for example, two Components `CA`, `CB` connect to the ROUTER sockets, the socket assigns the identities, which will be called `IA`, `IB` here (they can be any byte sequence). -Whenever a message is sent to the ROUTER socket from a peer, it reads as a first frame that identity. +Whenever a message is sent to the ROUTER socket from a peer, the socket prepends that identity in front of the message frames. For example if `CA` sends the message `Request A`, the ROUTER socket will read `IA|Request A`. That way, you can return an answer to exactly the original peer and not, for example `CB`. If you call the ROUTER's send command with `IA|Reply A`, the socket will send `Reply A` to the peer, whose connection is `IA`, in this case `CA`. @@ -69,13 +73,13 @@ sequenceDiagram #### Naming scheme -Each Component has an individual name, given by the user, the _Component Name_. +Each Component must have an individual name, given by the user, the _Component Name_. A Component Name must be a series of bytes, without the ASCII character "." (byte value 46). -Component Names must be unique in a {ref}`Node `, i.e. among the Components connected to a single Coordinator. -The Coordinator itself has the Component Name `COORDINATOR` (ASCII encoded). +Component Names must be unique in a {ref}`Node `, i.e. among the Components (except other Coordinators) connected to a single Coordinator. +A Coordinator itself must have the Component Name `COORDINATOR` (ASCII encoded). -Similarly, every node has a name, the _Namespace_. -Every Namespace has to be unique in the Network. +Similarly, every Node has a name, the _Namespace_. +Every Namespace must be unique in the Network. As each Component belongs to exactly one Node, it is fully identified by the combination of Namespace and Component Name, which is globally unique. This _full name_ is the composition of Namespace, ".", and Component Name. For example `N1.CA` is the full name of the Component `CA` in the Node `N1`. @@ -85,17 +89,17 @@ The receiver of a message may be specified without the Namespace, if the receive #### Message composition -A message consists in two or more frames. +A message consists of 4 or more frames. 1. The protocol version (abbreviated with "V" in examples). -2. The receiver full name or Component name, if sender and receiver live in the same Node (Namespace). -3. The sender full name. +2. The receiver Full name or Component name, as appropriate. +3. The sender Full name. 4. A content header (abbreviated with "H" in examples). 5. Message content: The optional payload, which can be 0 or more frames. #### Directory -Each Coordinator shall have a list of the Components connected to it. +Each Coordinator shall have a list of the Components (including other Connectors) connected to it. This is its _local Directory_. The _global Directory_ is the combination of the local directories of all Coordinators in a Network. @@ -117,9 +121,12 @@ In the exchange of messages, only the messages over the wire are shown, the conn ##### Initial connection After connecting to a Coordinator (`Co1`), a Component (`CA`) shall send a SIGNIN message indicating its Component Name. -The Coordinator shall respond with an ACKNOWLEDGE, giving the Namespace and other relevant information, or with an ERROR, if the Component Name is already taken. -After that successful handshake, the Coordinator shall store the connection identity and corresponding Component Name in its local {ref}`directory`. -It shall also publish to the other Coordinators in the network that this Component signed in, see {ref}`Coordinator coordination`. +The Coordinator shall indicate success/acceptance with an ACKNOWLEDGE response, giving the Namespace and other relevant information, or reply with an ERROR, e.g. if the Component Name is already taken. +In that case, the Coordinator may indicate a suitable still available variation on the indicated Component Name. +The Component may retry SIGNIN with a different chosen name. + +After a successful handshake, the Coordinator shall store the (zmq) connection identity and corresponding Component Name in its local {ref}`directory`. +It shall also notify the other Coordinators in the network that this Component signed in, see {ref}`Coordinator coordination`. Similarly, the Component shall store the Namespace and use it from this moment to generate its full name. If a Component does send a message to someone without having signed in, the Coordinator shall refuse message handling and return an error. @@ -142,13 +149,13 @@ sequenceDiagram CA ->> N1: V|N1.CB|CA|H|Content Note right of N1: Does not know CA N1 ->> CA: V|CA|N1.COORDINATOR|H|ERROR:I do not know you - Note left of CA: Should send a SIGNIN message + Note left of CA: Must send a SIGNIN message before further messaging. ::: ##### Heartbeat -We use heartbeat to know, whether a communication partner is still online. +Heartbeats are used to know whether a communication peer is still online. Every message received counts as a heartbeat. @@ -162,9 +169,9 @@ TBD: Respond to every non empty message with an empty one? ##### Signing out -A Component should tell a Coordinator, when it stops working, with a SIGNOUT message. -The Coordinator shall ACKNOWLEDGE the sign out and remove the Name from its directory. -It shall also publish to the other Coordinators in the network that this Component signed out, see {ref}`Coordinator coordination`. +A Component should tell a Coordinator when it stops participating in the network with a SIGNOUT message. +The Coordinator shall ACKNOWLEDGE the sign-out and remove the Name from its directory. +It shall also notify the other Coordinators in the network that this Component signed out, see {ref}`Coordinator coordination`. :::{mermaid} sequenceDiagram @@ -172,16 +179,17 @@ sequenceDiagram participant N1 as N1.COORDINATOR N1 ->> CA: V|N1.CA|N1.COORDINATOR|H|ACKNOWLEDGE Note right of N1: Removes "CA" with identity "IA"
from directory + Note right of N1: Notifies other Coordinators about sign-out of "CA" Note left of CA: Shall not send any message anymore except SIGNIN ::: #### Communication with other Components -The following two examples show, how a message is transferred between two components `CA`, `CB` via one or two Coordinators. +The following two examples show how a message is transferred between two components `CA`, `CB` via one or two Coordinators. Coordinators shall send messages from their DEALER socket to other Coordinator's ROUTER socket. -Coordinators shall hand on the message to the corresponding Coordinator or connected Component. +Coordinators shall route the message to the corresponding Coordinator or connected Component. :::{mermaid} @@ -214,7 +222,7 @@ sequenceDiagram N1 ->> CA: V|N1.CA|N2.CB|H| Property A has value 5. ::: -Prerequisite of Communication between two Components are: +Prerequisites of Communication between two Components are: - Both Components are connected to a Coordinator and {ref}`signed in`. - Both Components are either connected to the same Coordinator (example one), or their Coordinators are connected to each other (example two). @@ -268,14 +276,14 @@ flowchart TB #### Coordinator coordination -Each Coordinator shall keep an up to date global {ref}`directory` with the names of all Components in the Network. -For this, Coordinators shall tell each other regarding signing in and signing out Components and Coordinators. -Coordinators shall send on request the Names of their local directory, or of their global directory, depending on the request type. +Each Coordinator shall keep an up-to-date global {ref}`directory` with the names of all Components in the Network. +For this, Coordinators shall tell each other about sign-ins and sign-outs of Components and Coordinators. +On request, Coordinators shall send the Names of their local directory, or of their global directory, depending on the request type. For the format of the Messages, see {ref}`message-layer`. -##### Coordinator sign in +##### Coordinator sign-in A Coordinator `Co1`joining a network follows a few steps: 1. It signs in to one Coordinator `Co2` of the Network. @@ -284,9 +292,9 @@ A Coordinator `Co1`joining a network follows a few steps: 4. These other Coordinators (`Co3`, `Co4`...) sign in to `Co1`. 5. All Coordinators are connected to all others. -Two Coordinators shall follow a more thorough sign in/sign out procedure, than Components (address is for example host an port). -The sign in might happen because of a CO_NEW message arrived or at startup. -The sign out might happen because the Coordinator shuts down. +Two Coordinators shall follow a more thorough sign-in/sign-out procedure than Components (address is for example host and port). +The sign-in might happen because of a CO_NEW message arrived or at startup. +The sign-out might happen because the Coordinator shuts down. :::{mermaid} sequenceDiagram @@ -330,9 +338,9 @@ sequenceDiagram ##### Coordinator updates -Whenever a Component signs in to or out of its Coordinator, the Coordinator shall send a note regarding this event to all the other Coordinators. -The note shall contain the full name of the Component and the event type (sign in or out) -The other Coordinators shall update their global directory according to this note (add or remove an entry). +Whenever a Component signs in to or out of its Coordinator, the Coordinator shall send a message regarding this event to all the other Coordinators. +The message shall contain the full name of the Component and the event type (sign in or out) +The other Coordinators shall update their global directory according to this message (add or remove an entry). (message-layer)= From d938423456efed3f427c28162839be7bb6626813 Mon Sep 17 00:00:00 2001 From: Benedikt Moneke <67148916+bmoneke@users.noreply.github.com> Date: Thu, 9 Feb 2023 12:53:09 +0100 Subject: [PATCH 15/27] Naming changed and ZMQ information moved to other files. --- appendix.md | 34 ++++++++++++ control_protocol.md | 126 +++++++++++++++---------------------------- glossary.md | 10 ++++ index.rst | 1 + network-structure.md | 19 ++++++- 5 files changed, 105 insertions(+), 85 deletions(-) create mode 100644 appendix.md diff --git a/appendix.md b/appendix.md new file mode 100644 index 0000000..f833295 --- /dev/null +++ b/appendix.md @@ -0,0 +1,34 @@ +# Appendix + +## ZMQ + +Some useful hints regarding the working of the zmq package. + +(router-sockets)= +### Particularities of ROUTER sockets + +A small introduction to ROUTER sockets, for more details see [zmq guide chapter 3](https://zguide.zeromq.org/docs/chapter3/#Exploring-ROUTER-Sockets). + +A router socket assigns a random _identity_ to each connecting peer. +If, for example, two Components `CA`, `CB` connect to the ROUTER sockets, the socket assigns the identities, which will be called `IA`, `IB` here (they can be any byte sequence). + +Whenever a message is sent to the ROUTER socket from a peer, the socket prepends that identity in front of the message frames. +For example if `CA` sends the message `Request A`, the ROUTER socket will read `IA|Request A`. +That way, the answer to that message can be returned to the correct peer and not another one (a ROUTER socket can have many connected peers). +If the ROUTER's send command is called with `IA|Reply A`, the socket will send `Reply A` to the peer, whose connection is `IA`, in this case `CA`. + +The following diagram shows this example communication with two Components: +sequenceDiagram + participant Code as Message handling + participant ROUTER as ROUTER socket + Note over Code, ROUTER: Coordinator + CA ->> ROUTER: "Request A" + ROUTER ->> Code: "IA|Request A" + Code ->> ROUTER: "IA|Reply A" + ROUTER ->> CA: "Reply A" + CB ->> ROUTER: "Request B" + ROUTER ->> Code: "IB|Request B" + Code ->> ROUTER: "IB|Reply B" + ROUTER ->> CB: "Reply B" +::: + diff --git a/control_protocol.md b/control_protocol.md index 5ea4292..2d5f19e 100644 --- a/control_protocol.md +++ b/control_protocol.md @@ -10,23 +10,12 @@ The {ref]}`message-layer` is the common language to understand commands, thus cr The transport layer ensures that a message arrives at its destination. -### Sockets and Connections - -[Zmq](https://zeromq.org/) sockets are used for our communication. For more details see the [zmq guide](https://zguide.zeromq.org/) or [zmq API](http://api.zeromq.org/) - - -#### Definitions - -Zmq messages consist in a series of frames, each is a byte sequence. -The separation between frames is indicated with `|`. -An empty frame is indicated with two frame separators `||`, even at the beginning or end of a message. -For example `||Second frame|Third frame||Fifth frame` consists of 5 frames. -The first and fourth frames are empty frames. - +### Protocol basics -#### Configuration +#### Socket Configuration -Each {ref}`Coordinator ` shall offer one ROUTER socket, bound to a host name (or IP address or any address of a computer with "*") and port. +Each {ref}`Coordinator ` shall offer one {ref}`ROUTER` socket, bound to an address. +The address consists of a host (this can be the host name, an IP address of the device, or "\*" for all IP addresses of the device) and a port number, for example `*:12345` for all IP addresses and the port `12345`. {ref}`Components ` shall have one DEALER socket connecting to one Coordinator's ROUTER socket. @@ -40,51 +29,21 @@ While the number of DEALER sockets thus required scales badly with the number of Messages must be sent to a Coordinator's ROUTER socket. -(router-sockets)= -#### Particularities of ROUTER sockets - -A small introduction to ROUTER sockets, for more details see [zmq guide chapter 3](https://zguide.zeromq.org/docs/chapter3/#Exploring-ROUTER-Sockets). - -A router socket assigns a random _identity_ to each connecting peer. -If, for example, two Components `CA`, `CB` connect to the ROUTER sockets, the socket assigns the identities, which will be called `IA`, `IB` here (they can be any byte sequence). - -Whenever a message is sent to the ROUTER socket from a peer, the socket prepends that identity in front of the message frames. -For example if `CA` sends the message `Request A`, the ROUTER socket will read `IA|Request A`. -That way, you can return an answer to exactly the original peer and not, for example `CB`. -If you call the ROUTER's send command with `IA|Reply A`, the socket will send `Reply A` to the peer, whose connection is `IA`, in this case `CA`. - -The following diagram shows this example communication with two Components: -sequenceDiagram - participant Code as Message handling - participant ROUTER as ROUTER socket - Note over Code, ROUTER: Coordinator - CA ->> ROUTER: "Request A" - ROUTER ->> Code: "IA|Request A" - Code ->> ROUTER: "IA|Reply A" - ROUTER ->> CA: "Reply A" - CB ->> ROUTER: "Request B" - ROUTER ->> Code: "IB|Request B" - Code ->> ROUTER: "IB|Reply B" - ROUTER ->> CB: "Reply B" -::: - - -### Protocol basics - #### Naming scheme -Each Component must have an individual name, given by the user, the _Component Name_. -A Component Name must be a series of bytes, without the ASCII character "." (byte value 46). -Component Names must be unique in a {ref}`Node `, i.e. among the Components (except other Coordinators) connected to a single Coordinator. -A Coordinator itself must have the Component Name `COORDINATOR` (ASCII encoded). +Each Component must have an individual name, given by the user, the _Component name_. +A Component name must be a series of bytes, without the ASCII character "." (byte value 46). +Component names must be unique in a {ref}`Node `, i.e. among the Components (except other Coordinators) connected to a single Coordinator. +A Coordinator itself must have the Component name `COORDINATOR` (ASCII encoded). -Similarly, every Node has a name, the _Namespace_. +Similarly, every Node must have a name, the _Namespace_. Every Namespace must be unique in the Network. -As each Component belongs to exactly one Node, it is fully identified by the combination of Namespace and Component Name, which is globally unique. -This _full name_ is the composition of Namespace, ".", and Component Name. -For example `N1.CA` is the full name of the Component `CA` in the Node `N1`. +As each Component belongs to exactly one Node, it is fully identified by the combination of Namespace and Component name, which is globally unique. +This _Full name_ is the composition of Namespace, ".", and Component name. +For example `N1.CA` is the Full name of the Component `CA` in the Node `N1`. -The receiver of a message may be specified without the Namespace, if the receiver lives in the same Node. +The receiver of a message may be specified by Component name alone if the receiver belongs to the same Node as the sender. +In all other cases, the receiver of a message must be specified by Full name. #### Message composition @@ -100,19 +59,19 @@ A message consists of 4 or more frames. #### Directory Each Coordinator shall have a list of the Components (including other Connectors) connected to it. -This is its _local Directory_. +This is its _Directory_. -The _global Directory_ is the combination of the local directories of all Coordinators in a Network. +The _Glossary_ is the combination of the Directories of all Coordinators in a Network. ### Conversation protocol -In the protocol examples, `CA`, `CB`, etc. indicate Component Names. +In the protocol examples, `CA`, `CB`, etc. indicate Component names. `N1`, `N2`, etc. indicate Node Namespaces and `Co1`, `Co2` their corresponding Coordinators. Here the Message content is expressed in plain English, for the exact definition see {ref}`message-layer`. -In the exchange of messages, only the messages over the wire are shown, the connection identity of the ROUTER socket is not shown. +In the exchange of messages, only the messages over the wire are shown, the connection identity used by the ROUTER socket is not shown. #### Communication with the Coordinator @@ -120,14 +79,14 @@ In the exchange of messages, only the messages over the wire are shown, the conn (sign-in)= ##### Initial connection -After connecting to a Coordinator (`Co1`), a Component (`CA`) shall send a SIGNIN message indicating its Component Name. -The Coordinator shall indicate success/acceptance with an ACKNOWLEDGE response, giving the Namespace and other relevant information, or reply with an ERROR, e.g. if the Component Name is already taken. -In that case, the Coordinator may indicate a suitable still available variation on the indicated Component Name. +After connecting to a Coordinator (`Co1`), a Component (`CA`) shall send a SIGNIN message indicating its Component name. +The Coordinator shall indicate success/acceptance with an ACKNOWLEDGE response, giving the Namespace and other relevant information, or reply with an ERROR, e.g. if the Component name is already taken. +In that case, the Coordinator may indicate a suitable still available variation on the indicated Component name. The Component may retry SIGNIN with a different chosen name. -After a successful handshake, the Coordinator shall store the (zmq) connection identity and corresponding Component Name in its local {ref}`directory`. +After a successful handshake, the Coordinator shall store the (zmq) connection identity and corresponding Component name in its {ref}`directory`. It shall also notify the other Coordinators in the network that this Component signed in, see {ref}`Coordinator coordination`. -Similarly, the Component shall store the Namespace and use it from this moment to generate its full name. +Similarly, the Component shall store the Namespace and use it from this moment to generate its Full name. If a Component does send a message to someone without having signed in, the Coordinator shall refuse message handling and return an error. @@ -170,7 +129,7 @@ TBD: Respond to every non empty message with an empty one? ##### Signing out A Component should tell a Coordinator when it stops participating in the network with a SIGNOUT message. -The Coordinator shall ACKNOWLEDGE the sign-out and remove the Name from its directory. +The Coordinator shall ACKNOWLEDGE the sign-out and remove the Name from its Directory. It shall also notify the other Coordinators in the network that this Component signed out, see {ref}`Coordinator coordination`. :::{mermaid} @@ -178,7 +137,7 @@ sequenceDiagram CA ->> N1: V|COORDINATOR|N1.CA|H|SIGNOUT participant N1 as N1.COORDINATOR N1 ->> CA: V|N1.CA|N1.COORDINATOR|H|ACKNOWLEDGE - Note right of N1: Removes "CA" with identity "IA"
from directory + Note right of N1: Removes "CA" with identity "IA"
from Directory Note right of N1: Notifies other Coordinators about sign-out of "CA" Note left of CA: Shall not send any message anymore except SIGNIN ::: @@ -194,7 +153,7 @@ Coordinators shall route the message to the corresponding Coordinator or connect :::{mermaid} sequenceDiagram - alt full name + alt Full name CA ->> N1: V|N1.CB|N1.CA|H| Give me property A. else only Component name CA ->> N1: V|CB|N1.CA|H| Give me property A. @@ -228,7 +187,7 @@ Prerequisites of Communication between two Components are: The following flow chart shows the decision scheme and message modification in a Coordinator `Co1` of Node `N1`. -Its full name is `N1.Coordinator`. +Its Full name is `N1.Coordinator`. `nS`, `nR` are placeholders for sender and recipient Namespaces. `recipient` is a placeholder for the recipient Component name. `iA` is a placeholder for the connection identity of the incoming message and `iB` that of `N1.Recipient`. @@ -242,8 +201,8 @@ flowchart TB C0([nS.COORDINATOR DEALER]) == "V|nR.recipient|nS.CA|H|Content" ==> R0 R0[receive] == "iA|V|nR.recipient|nS.CA|H|Content" ==> CnS{nS == N1?} CnS-->|no| RemIdent - CnS-->|yes| Clocal{CA in
local directory?} - Clocal -->|yes| CidKnown{iA is CA's identity
in directory?} + CnS-->|yes| Clocal{CA in
Directory?} + Clocal -->|yes| CidKnown{iA is CA's identity
in Directory?} CidKnown -->|yes| RemIdent Clocal -.->|no| E1[ERROR: Sender unknown] ==>|"iA|V|nS.CA|N1.COORDINATOR|H|ERROR: Sender unknown"| S S[send] ==> WA([N1.CA DEALER]) @@ -253,7 +212,7 @@ flowchart TB CnR{nR?} -- "== N1"--> Local Local{recipient
==
COORDINATOR?} -- "yes" --> Self[Message for Co1
itself] Self == "V|nR.recipient|nS.CA|H|Content" ==> SC([Co1 Message handling]) - Local -- "no" --> Local2a{recipient in directory?} + Local -- "no" --> Local2a{recipient in Directory?} Local2a -->|yes, with Identity iB| Local2 Local2[add recipient identity iB] == "iB|V|nR.recipient|nS.CA|H|Content" ==> R1[send] R1 == "V|nR.recipient|nS.CA|H|Content" ==> W1([Wire to N1.recipient DEALER]) @@ -276,9 +235,9 @@ flowchart TB #### Coordinator coordination -Each Coordinator shall keep an up-to-date global {ref}`directory` with the names of all Components in the Network. -For this, Coordinators shall tell each other about sign-ins and sign-outs of Components and Coordinators. -On request, Coordinators shall send the Names of their local directory, or of their global directory, depending on the request type. +Each Coordinator shall keep an up-to-date {ref}`Glossary` with the Names of all Components in the Network. +For this, Coordinators shall notify each other about sign-ins and sign-outs of Components and Coordinators. +On request, Coordinators shall send the Names of their Directory, or of their Glossary, depending on the request type. For the format of the Messages, see {ref}`message-layer`. @@ -311,7 +270,7 @@ sequenceDiagram d1-->>r2: connect to address2 d1->>r2: CO_SIGNIN
N1, address1,
ref:temp-NS par - d1->>r2: GET local directory + d1->>r2: GET Directory and Note right of r2: stores N1 identity activate d2 @@ -320,12 +279,12 @@ sequenceDiagram d2->>r1: CO_SIGNIN
N2, address2
your ref:temp-NS Note right of r1: stores N2 identity Note left of d1: name changed
from "temp-NS"
to "N2" - d2->>r1: GET local directory + d2->>r1: GET Directory end - d2->>r1: Here is my
local directory - Note right of r1: Updates
global directory - d1->>r2: Here is my
local directory - Note right of r2: Updates
global directory + d2->>r1: Here is my
Directory + Note right of r1: Updates
Glossary + d1->>r2: Here is my
Directory + Note right of r2: Updates
Glossary Note over r1,d2: Sign out between two Coordinators Note right of r1: shall sign out from N2 d1->>r2: CO_SIGNOUT @@ -338,9 +297,9 @@ sequenceDiagram ##### Coordinator updates -Whenever a Component signs in to or out of its Coordinator, the Coordinator shall send a message regarding this event to all the other Coordinators. -The message shall contain the full name of the Component and the event type (sign in or out) -The other Coordinators shall update their global directory according to this message (add or remove an entry). +Whenever a Component signs in to or out of its Coordinator, the Coordinator shall send a CO_UPDATE message regarding this event to all the other Coordinators. +The message shall contain the Full name of the Component and the event type (sign in or out) +The other Coordinators shall update their Glossary according to this message (add or remove an entry). (message-layer)= @@ -359,6 +318,7 @@ The other Coordinators shall update their global directory according to this mes - CO_SIGNOUT - CO_TELL_ALL - CO_NEW +- CO_UPDATE diff --git a/glossary.md b/glossary.md index 8580c3d..a557be4 100644 --- a/glossary.md +++ b/glossary.md @@ -10,6 +10,9 @@ Actor Component A type of entity, a set of which make up the LECO communication Network, see {ref}`components.md#components`. +Component name + The individual name in a Node, under which a Component can be addressed, see {ref}`control_protocol.md#naming-scheme`. + Coordinator A Component primarily concerned with routing/coordinating the message flow between other Components, see {ref}`components.md#coordinator`. There are Control Coordinators, Data Coordinators, and Logging Coordinators. @@ -23,6 +26,10 @@ Director Driver An object that takes care of communicating with a Device. This object is external to LECO, for example coming from and instrument control library like `pymeasure`, `instrumentkit` or `yaq`. See {ref}`components.md#driver`. +Full name + The name of a Component unique in the whole name. + It consists of the {ref}`namespace` and {ref}`component-name`, see {ref}`control_protocol.md#naming-scheme`. + LECO The **L**aboratory **E**xperiment **CO**ntrol protocol framework. @@ -39,6 +46,9 @@ Message Layer Message Transport Mode (LMT/DMT) The Node-local Message Layer can have a local or distributed mode, see {ref}`network-structure.md#message-transport-mode-lmtdmt`. +Namespace + The name of a Node in the Network, see {ref}`control_protocol.md#naming-scheme`. + Node A Node is a local context in which (part of) a LECO deployment runs. This may be a single application using one or more threads or processes. diff --git a/index.rst b/index.rst index 443cf56..6c887d1 100644 --- a/index.rst +++ b/index.rst @@ -18,6 +18,7 @@ control_protocol glossary Hello_world + appendix Indices and tables ================== diff --git a/network-structure.md b/network-structure.md index 180c1d1..ae40c56 100644 --- a/network-structure.md +++ b/network-structure.md @@ -47,10 +47,25 @@ flowchart LR Control Coordinator to Control Coordinator communication always uses DMT. ## Message Transport Mode (LMT/DMT) + The Node-local Message Layer can have a local or distributed mode. -The Distributed Message Transport (DMT, using the zeromq TCP protocol) works within or across Nodes. -The Local Message Transport (LMT) only works within a Node _and_ within a process. +### Distributed Message Transport (DMT) + +The Distributed Message Transport works within or across Nodes. +It uses [Zmq](https://zeromq.org/) sockets for the communication. For more details see the [zmq guide](https://zguide.zeromq.org/) or [zmq API](http://api.zeromq.org/) + +Zmq messages consist in a series of frames, each is a byte sequence. +In this documentation, the separation between frames is indicated by `|`. +An empty frame is indicated with two frame separators `||`, even at the beginning or end of a message. +For example, the message `||Second frame|Third frame||Fifth frame` consists of 5 frames, with the first and fourth frames being empty frames. + +For some useful information see our {ref}`appendix.md#zmq`. + + +### Local Message Transport (LMT) + +The Local Message Transport only works within a Node _and_ within a process. Local Message Transport options include queues between threads/processes and zeromq inproc. :::{admonition} Warning LMT details are still notional and not to be relied upon this will be fleshed out at a later date. From 0dbb7cf42721c7312c344f9d002f19466632c6f4 Mon Sep 17 00:00:00 2001 From: Benedikt Moneke <67148916+bmoneke@users.noreply.github.com> Date: Fri, 10 Feb 2023 10:18:46 +0100 Subject: [PATCH 16/27] Language improvements by bklebel Co-authored-by: Benjamin Klebel-Knobloch <32774645+bklebel@users.noreply.github.com> --- appendix.md | 10 +++++----- control_protocol.md | 24 ++++++++++++------------ glossary.md | 2 +- network-structure.md | 4 ++-- 4 files changed, 20 insertions(+), 20 deletions(-) diff --git a/appendix.md b/appendix.md index f833295..d915c82 100644 --- a/appendix.md +++ b/appendix.md @@ -2,7 +2,7 @@ ## ZMQ -Some useful hints regarding the working of the zmq package. +Some useful hints regarding how the zmq package works. (router-sockets)= ### Particularities of ROUTER sockets @@ -10,11 +10,11 @@ Some useful hints regarding the working of the zmq package. A small introduction to ROUTER sockets, for more details see [zmq guide chapter 3](https://zguide.zeromq.org/docs/chapter3/#Exploring-ROUTER-Sockets). A router socket assigns a random _identity_ to each connecting peer. -If, for example, two Components `CA`, `CB` connect to the ROUTER sockets, the socket assigns the identities, which will be called `IA`, `IB` here (they can be any byte sequence). +If, for example, two Components `CA`, `CB` connect to a ROUTER socket, the socket assigns identities, which will be called `IA`, `IB` here (they can be any byte sequence). -Whenever a message is sent to the ROUTER socket from a peer, the socket prepends that identity in front of the message frames. -For example if `CA` sends the message `Request A`, the ROUTER socket will read `IA|Request A`. -That way, the answer to that message can be returned to the correct peer and not another one (a ROUTER socket can have many connected peers). +Whenever a message is sent to a ROUTER socket from a peer, the socket prepends that identity in front of the message frames, before handing the message to the application code. +For example, if `CA` sends the message `Request A`, the ROUTER socket will read `IA|Request A`. +That way, an answer to that message can be returned to this same peer, and not another one (a ROUTER socket may have many connected peers). If the ROUTER's send command is called with `IA|Reply A`, the socket will send `Reply A` to the peer, whose connection is `IA`, in this case `CA`. The following diagram shows this example communication with two Components: diff --git a/control_protocol.md b/control_protocol.md index 2d5f19e..7728145 100644 --- a/control_protocol.md +++ b/control_protocol.md @@ -1,6 +1,6 @@ # Control protocol -The control protocol transmits messages via its {ref}`transport-layer` from one Component to another one. +The control protocol transmits messages via its {ref}`transport-layer` from one Component to another. The {ref]}`message-layer` is the common language to understand commands, thus creating a remote procedure call. @@ -15,7 +15,7 @@ The transport layer ensures that a message arrives at its destination. #### Socket Configuration Each {ref}`Coordinator ` shall offer one {ref}`ROUTER` socket, bound to an address. -The address consists of a host (this can be the host name, an IP address of the device, or "\*" for all IP addresses of the device) and a port number, for example `*:12345` for all IP addresses and the port `12345`. +The address consists of a host (this can be the host name, an IP address of the device, or "\*" for all IP addresses of the device) and a port number, for example `*:12345` for all IP addresses at the port `12345`. {ref}`Components ` shall have one DEALER socket connecting to one Coordinator's ROUTER socket. @@ -32,7 +32,7 @@ Messages must be sent to a Coordinator's ROUTER socket. #### Naming scheme Each Component must have an individual name, given by the user, the _Component name_. -A Component name must be a series of bytes, without the ASCII character "." (byte value 46). +A Component name must be a series of bytes, without the ASCII character "." (the byte value 46 is not permitted). Component names must be unique in a {ref}`Node `, i.e. among the Components (except other Coordinators) connected to a single Coordinator. A Coordinator itself must have the Component name `COORDINATOR` (ASCII encoded). @@ -43,7 +43,7 @@ This _Full name_ is the composition of Namespace, ".", and Component name. For example `N1.CA` is the Full name of the Component `CA` in the Node `N1`. The receiver of a message may be specified by Component name alone if the receiver belongs to the same Node as the sender. -In all other cases, the receiver of a message must be specified by Full name. +In all other cases, the receiver of a message must be specified by the Full name. #### Message composition @@ -81,12 +81,12 @@ In the exchange of messages, only the messages over the wire are shown, the conn After connecting to a Coordinator (`Co1`), a Component (`CA`) shall send a SIGNIN message indicating its Component name. The Coordinator shall indicate success/acceptance with an ACKNOWLEDGE response, giving the Namespace and other relevant information, or reply with an ERROR, e.g. if the Component name is already taken. -In that case, the Coordinator may indicate a suitable still available variation on the indicated Component name. +In that case, the Coordinator may indicate a suitable, still available variation on the indicated Component name. The Component may retry SIGNIN with a different chosen name. After a successful handshake, the Coordinator shall store the (zmq) connection identity and corresponding Component name in its {ref}`directory`. It shall also notify the other Coordinators in the network that this Component signed in, see {ref}`Coordinator coordination`. -Similarly, the Component shall store the Namespace and use it from this moment to generate its Full name. +Similarly, the Component shall store the Namespace and use it from this moment on, to generate its Full name. If a Component does send a message to someone without having signed in, the Coordinator shall refuse message handling and return an error. @@ -129,7 +129,7 @@ TBD: Respond to every non empty message with an empty one? ##### Signing out A Component should tell a Coordinator when it stops participating in the network with a SIGNOUT message. -The Coordinator shall ACKNOWLEDGE the sign-out and remove the Name from its Directory. +The Coordinator shall ACKNOWLEDGE the sign-out and remove the Component name from its Directory. It shall also notify the other Coordinators in the network that this Component signed out, see {ref}`Coordinator coordination`. :::{mermaid} @@ -186,7 +186,7 @@ Prerequisites of Communication between two Components are: - Both Components are either connected to the same Coordinator (example one), or their Coordinators are connected to each other (example two). -The following flow chart shows the decision scheme and message modification in a Coordinator `Co1` of Node `N1`. +The following flow chart shows the decision scheme and message modification in the Coordinator `Co1` of Node `N1`. Its Full name is `N1.Coordinator`. `nS`, `nR` are placeholders for sender and recipient Namespaces. `recipient` is a placeholder for the recipient Component name. @@ -237,14 +237,14 @@ flowchart TB Each Coordinator shall keep an up-to-date {ref}`Glossary` with the Names of all Components in the Network. For this, Coordinators shall notify each other about sign-ins and sign-outs of Components and Coordinators. -On request, Coordinators shall send the Names of their Directory, or of their Glossary, depending on the request type. +On request, Coordinators shall send the Names of their local or global Directory, depending on the request type. For the format of the Messages, see {ref}`message-layer`. ##### Coordinator sign-in -A Coordinator `Co1`joining a network follows a few steps: +A Coordinator `Co1` joining a network follows a few steps: 1. It signs in to one Coordinator `Co2` of the Network. 2. It sends a CO_TELL_ALL message to `Co2`, to tell all other Coordinators about `Co1`s address. 3. `Co2` tells all the Coordinators signed in (`Co3`, `Co4`...) about `Co1` with a CO_NEW message. @@ -297,9 +297,9 @@ sequenceDiagram ##### Coordinator updates -Whenever a Component signs in to or out of its Coordinator, the Coordinator shall send a CO_UPDATE message regarding this event to all the other Coordinators. +Whenever a Component signs in to or out from its Coordinator, the Coordinator shall send a CO_UPDATE message regarding this event to all the other Coordinators. The message shall contain the Full name of the Component and the event type (sign in or out) -The other Coordinators shall update their Glossary according to this message (add or remove an entry). +The other Coordinators shall update their global Directory according to this message (add or remove an entry). (message-layer)= diff --git a/glossary.md b/glossary.md index a557be4..e425401 100644 --- a/glossary.md +++ b/glossary.md @@ -27,7 +27,7 @@ Driver An object that takes care of communicating with a Device. This object is external to LECO, for example coming from and instrument control library like `pymeasure`, `instrumentkit` or `yaq`. See {ref}`components.md#driver`. Full name - The name of a Component unique in the whole name. + The name of a Component unique for the whole setup. It consists of the {ref}`namespace` and {ref}`component-name`, see {ref}`control_protocol.md#naming-scheme`. LECO diff --git a/network-structure.md b/network-structure.md index ae40c56..84c4b93 100644 --- a/network-structure.md +++ b/network-structure.md @@ -53,7 +53,7 @@ The Node-local Message Layer can have a local or distributed mode. ### Distributed Message Transport (DMT) The Distributed Message Transport works within or across Nodes. -It uses [Zmq](https://zeromq.org/) sockets for the communication. For more details see the [zmq guide](https://zguide.zeromq.org/) or [zmq API](http://api.zeromq.org/) +Currently, the only defined transport layer uses [Zmq](https://zeromq.org/) sockets for the communication. For more details see the [zmq guide](https://zguide.zeromq.org/) or [zmq API](http://api.zeromq.org/) Zmq messages consist in a series of frames, each is a byte sequence. In this documentation, the separation between frames is indicated by `|`. @@ -68,6 +68,6 @@ For some useful information see our {ref}`appendix.md#zmq`. The Local Message Transport only works within a Node _and_ within a process. Local Message Transport options include queues between threads/processes and zeromq inproc. :::{admonition} Warning -LMT details are still notional and not to be relied upon this will be fleshed out at a later date. +LMT details are still notional and not to be relied upon, this will be fleshed out at a later date. The list of LMT options is not definitive yet. ::: From 90208de920465e3e277a1fbe023455e4d72752f8 Mon Sep 17 00:00:00 2001 From: Benedikt Moneke <67148916+bmoneke@users.noreply.github.com> Date: Fri, 10 Feb 2023 12:55:47 +0100 Subject: [PATCH 17/27] Information regarding connected Components is stored in the local or global Directory. --- control_protocol.md | 32 ++++++++++++++++---------------- glossary.md | 3 +++ 2 files changed, 19 insertions(+), 16 deletions(-) diff --git a/control_protocol.md b/control_protocol.md index 7728145..447b979 100644 --- a/control_protocol.md +++ b/control_protocol.md @@ -58,10 +58,10 @@ A message consists of 4 or more frames. #### Directory -Each Coordinator shall have a list of the Components (including other Connectors) connected to it. -This is its _Directory_. +Each Coordinator shall have a list of the Components (including other Coordinators) connected to it. +This is its _local Directory_. -The _Glossary_ is the combination of the Directories of all Coordinators in a Network. +The _global Directory_ is the combination of the Directories of all Coordinators in a Network. ### Conversation protocol @@ -128,8 +128,8 @@ TBD: Respond to every non empty message with an empty one? ##### Signing out -A Component should tell a Coordinator when it stops participating in the network with a SIGNOUT message. -The Coordinator shall ACKNOWLEDGE the sign-out and remove the Component name from its Directory. +A Component should send a SIGNOUT message to its Coordinator when it stops participating in the Network. +The Coordinator shall ACKNOWLEDGE the sign-out and remove the Component name from its local {ref}`directory`. It shall also notify the other Coordinators in the network that this Component signed out, see {ref}`Coordinator coordination`. :::{mermaid} @@ -137,7 +137,7 @@ sequenceDiagram CA ->> N1: V|COORDINATOR|N1.CA|H|SIGNOUT participant N1 as N1.COORDINATOR N1 ->> CA: V|N1.CA|N1.COORDINATOR|H|ACKNOWLEDGE - Note right of N1: Removes "CA" with identity "IA"
from Directory + Note right of N1: Removes "CA" with identity "IA"
from local Directory Note right of N1: Notifies other Coordinators about sign-out of "CA" Note left of CA: Shall not send any message anymore except SIGNIN ::: @@ -201,8 +201,8 @@ flowchart TB C0([nS.COORDINATOR DEALER]) == "V|nR.recipient|nS.CA|H|Content" ==> R0 R0[receive] == "iA|V|nR.recipient|nS.CA|H|Content" ==> CnS{nS == N1?} CnS-->|no| RemIdent - CnS-->|yes| Clocal{CA in
Directory?} - Clocal -->|yes| CidKnown{iA is CA's identity
in Directory?} + CnS-->|yes| Clocal{CA in
local Directory?} + Clocal -->|yes| CidKnown{iA is CA's identity
in local Directory?} CidKnown -->|yes| RemIdent Clocal -.->|no| E1[ERROR: Sender unknown] ==>|"iA|V|nS.CA|N1.COORDINATOR|H|ERROR: Sender unknown"| S S[send] ==> WA([N1.CA DEALER]) @@ -212,7 +212,7 @@ flowchart TB CnR{nR?} -- "== N1"--> Local Local{recipient
==
COORDINATOR?} -- "yes" --> Self[Message for Co1
itself] Self == "V|nR.recipient|nS.CA|H|Content" ==> SC([Co1 Message handling]) - Local -- "no" --> Local2a{recipient in Directory?} + Local -- "no" --> Local2a{recipient in local Directory?} Local2a -->|yes, with Identity iB| Local2 Local2[add recipient identity iB] == "iB|V|nR.recipient|nS.CA|H|Content" ==> R1[send] R1 == "V|nR.recipient|nS.CA|H|Content" ==> W1([Wire to N1.recipient DEALER]) @@ -235,7 +235,7 @@ flowchart TB #### Coordinator coordination -Each Coordinator shall keep an up-to-date {ref}`Glossary` with the Names of all Components in the Network. +Each Coordinator shall keep an up-to-date global {ref}`directory` with the Names of all Components in the Network. For this, Coordinators shall notify each other about sign-ins and sign-outs of Components and Coordinators. On request, Coordinators shall send the Names of their local or global Directory, depending on the request type. @@ -270,7 +270,7 @@ sequenceDiagram d1-->>r2: connect to address2 d1->>r2: CO_SIGNIN
N1, address1,
ref:temp-NS par - d1->>r2: GET Directory + d1->>r2: GET local Directory and Note right of r2: stores N1 identity activate d2 @@ -279,12 +279,12 @@ sequenceDiagram d2->>r1: CO_SIGNIN
N2, address2
your ref:temp-NS Note right of r1: stores N2 identity Note left of d1: name changed
from "temp-NS"
to "N2" - d2->>r1: GET Directory + d2->>r1: GET local Directory end - d2->>r1: Here is my
Directory - Note right of r1: Updates
Glossary - d1->>r2: Here is my
Directory - Note right of r2: Updates
Glossary + d2->>r1: Here is my
local Directory + Note right of r1: Updates
global Directory + d1->>r2: Here is my
local Directory + Note right of r2: Updates
global Directory Note over r1,d2: Sign out between two Coordinators Note right of r1: shall sign out from N2 d1->>r2: CO_SIGNOUT diff --git a/glossary.md b/glossary.md index e425401..ed13906 100644 --- a/glossary.md +++ b/glossary.md @@ -23,6 +23,9 @@ Device Director A Component which takes part in orchestrating a (i.e. LECO-controlled) measurement setup, see {ref}`components.md#director`. +Directory + Each Coordinator maintains a local Directory with all the Components connected to it, and a global Directory with all Components in the Network, see {ref}`control_protocol.md#directory`. + Driver An object that takes care of communicating with a Device. This object is external to LECO, for example coming from and instrument control library like `pymeasure`, `instrumentkit` or `yaq`. See {ref}`components.md#driver`. From 6af9c0b92c94b24b76891036478ff0bb128f4d63 Mon Sep 17 00:00:00 2001 From: Benedikt Moneke <67148916+bmoneke@users.noreply.github.com> Date: Fri, 10 Feb 2023 12:56:01 +0100 Subject: [PATCH 18/27] Router socket better explained. --- appendix.md | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/appendix.md b/appendix.md index d915c82..f6923b4 100644 --- a/appendix.md +++ b/appendix.md @@ -9,13 +9,16 @@ Some useful hints regarding how the zmq package works. A small introduction to ROUTER sockets, for more details see [zmq guide chapter 3](https://zguide.zeromq.org/docs/chapter3/#Exploring-ROUTER-Sockets). -A router socket assigns a random _identity_ to each connecting peer. +The ROUTER socket is mostly used in a server role as it can maintain connections to many peers. +In order to distinguish peers, it assigns a random _identity_ to each connected peer. +Application code does not know, which peer gets which identity, however, the identity of a peer stays the same for the lifetime of the connection. + If, for example, two Components `CA`, `CB` connect to a ROUTER socket, the socket assigns identities, which will be called `IA`, `IB` here (they can be any byte sequence). Whenever a message is sent to a ROUTER socket from a peer, the socket prepends that identity in front of the message frames, before handing the message to the application code. For example, if `CA` sends the message `Request A`, the ROUTER socket will read `IA|Request A`. That way, an answer to that message can be returned to this same peer, and not another one (a ROUTER socket may have many connected peers). -If the ROUTER's send command is called with `IA|Reply A`, the socket will send `Reply A` to the peer, whose connection is `IA`, in this case `CA`. +Consequently, in order to send such an answer, the identity has to be prepended to the frames to send: Calling the ROUTER's send command with `IA|Reply A`, the socket will send `Reply A` to the peer, whose identity is `IA`, in this case that is `CA`. The following diagram shows this example communication with two Components: sequenceDiagram From 0c693140a465503cc490bba45ce21c938cb9c80a Mon Sep 17 00:00:00 2001 From: Benedikt Moneke <67148916+bmoneke@users.noreply.github.com> Date: Mon, 13 Feb 2023 13:53:20 +0100 Subject: [PATCH 19/27] Coordinator sign-in improved. --- control_protocol.md | 73 +++++++++++++++++++++++++++++---------------- 1 file changed, 47 insertions(+), 26 deletions(-) diff --git a/control_protocol.md b/control_protocol.md index 447b979..7c288fa 100644 --- a/control_protocol.md +++ b/control_protocol.md @@ -122,7 +122,7 @@ A Component should and a Coordinator shall send a PING and wait some time before A Coordinator shall follow the {ref}`sign out routine` for a signed in Component considered dead. :::{note} -TBD: Respond to every non empty message with an empty one? +TBD: Heartbeat details are still to be determined. ::: @@ -244,17 +244,14 @@ For the format of the Messages, see {ref}`message-layer`. ##### Coordinator sign-in -A Coordinator `Co1` joining a network follows a few steps: -1. It signs in to one Coordinator `Co2` of the Network. -2. It sends a CO_TELL_ALL message to `Co2`, to tell all other Coordinators about `Co1`s address. -3. `Co2` tells all the Coordinators signed in (`Co3`, `Co4`...) about `Co1` with a CO_NEW message. -4. These other Coordinators (`Co3`, `Co4`...) sign in to `Co1`. -5. All Coordinators are connected to all others. - -Two Coordinators shall follow a more thorough sign-in/sign-out procedure than Components (address is for example host and port). -The sign-in might happen because of a CO_NEW message arrived or at startup. +A Coordinator joins a Network by signing in to any Coordinator of that Network. +The sign-in/sign-out procedure between two Coordinators is more thorough than that of Components. +During the sign-in procedure, Coordinators exchange their local Directories and shall sign in to all Coordinators, they are not yet signed in. +The sign-in might happen because the Coordinator learns a new Coordinator address via Directory updates or at startup. The sign-out might happen because the Coordinator shuts down. +These are the sign-in/sign-out sequences between Coordinators, where `address` is for example the host name and port number of the Coordinator's ROUTER socket. + :::{mermaid} sequenceDiagram participant r1 as ROUTER @@ -268,23 +265,21 @@ sequenceDiagram activate d1 Note left of d1: created with
name "temp-NS" d1-->>r2: connect to address2 - d1->>r2: CO_SIGNIN
N1, address1,
ref:temp-NS - par - d1->>r2: GET local Directory - and - Note right of r2: stores N1 identity - activate d2 - Note left of d2: created with
name "N1" - d2-->>r1: connect to address1 - d2->>r1: CO_SIGNIN
N2, address2
your ref:temp-NS - Note right of r1: stores N2 identity - Note left of d1: name changed
from "temp-NS"
to "N2" - d2->>r1: GET local Directory - end - d2->>r1: Here is my
local Directory - Note right of r1: Updates
global Directory + d1->>r2: CO_SIGNIN
N1, address1 + Note right of r2: stores N1 identity + r2->>d1: ACK: Namespace is N2 + Note left of d1: DEALER name
set to "N2" + activate d2 + Note left of d2: created with
name "N1" + d2-->>r1: connect to address1 + d2->>r1: CO_SIGNIN
N2, address2 + Note right of r1: stores N2 identity + r1->>d2: ACK: Namespace is N1 + Note left of d1: Already has
DEALER named N2 d1->>r2: Here is my
local Directory - Note right of r2: Updates
global Directory + Note right of r2: Updates global
Directory and signs
in to all unknown
Coordinators + d2->>r1: Here is my
local Directory + Note right of r1: Updates global
Directory and signs
in to all unknown
Coordinators Note over r1,d2: Sign out between two Coordinators Note right of r1: shall sign out from N2 d1->>r2: CO_SIGNOUT @@ -295,6 +290,32 @@ sequenceDiagram ::: +:::{note} +The sign-in procedure is symmetric, with exception of not creating a DEALER socket, if it already exists. +::: + +These are the decision processes for signing in: +:::{mermaid} +graph TD + COS([ROUTER receives
'CO_SIGNIN N2 address2']) --> Store[Store the identity of 'N2'] + Store --> ACK[Respond 'ACK N1'
via ROUTER] + ACK-->DEAL{DEALER socket
with name 'N2'
already created?} + DEAL -->|yes| END([End]) + DEAL -->|no| CREATE[Create DEALER
named 'N2'] + CREATE --> CONNECT[Connect to address2] + + CMD([Command to sign in
to address2])-->CREATE2[Create DEALER
named 'temp-NS'] + CREATE2 --> CONNECT + + CONNECT-->SEND[Send 'CO_SIGNIN N1 address1'] + SEND --> END + + AR(['ACK N2'
at DEALER]) --> Known{DEALER name == 'N2'?} + Known -->|no| Rename[Rename DEALER socket to 'N2'] -->SLD + Known -->|yes| SLD([Send my local Directory]) +::: + + ##### Coordinator updates Whenever a Component signs in to or out from its Coordinator, the Coordinator shall send a CO_UPDATE message regarding this event to all the other Coordinators. From c748a5928ebf8e911a34bc0d81a64d6387c4fb27 Mon Sep 17 00:00:00 2001 From: Benedikt Moneke <67148916+bmoneke@users.noreply.github.com> Date: Mon, 13 Feb 2023 13:53:31 +0100 Subject: [PATCH 20/27] Wordings improved. --- control_protocol.md | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/control_protocol.md b/control_protocol.md index 7c288fa..89afae1 100644 --- a/control_protocol.md +++ b/control_protocol.md @@ -26,18 +26,20 @@ This DEALER socket shall connect to the other Coordinator's ROUTER socket. While the number of DEALER sockets thus required scales badly with the number of Connectors in a LECO Network, the scope of the protocol means that at most a few Coordinators will be involved. ::: -Messages must be sent to a Coordinator's ROUTER socket. +Communicating with a Coordinator, messages must be sent to a Coordinator's ROUTER socket. +Only for acknowledging a {ref}`coordinator-sign-in`, it is permitted to send a message to a Coordinator's DEALER socket. #### Naming scheme Each Component must have an individual name, given by the user, the _Component name_. -A Component name must be a series of bytes, without the ASCII character "." (the byte value 46 is not permitted). +A Component name must be a series of ASCII characters, without the character ".". Component names must be unique in a {ref}`Node `, i.e. among the Components (except other Coordinators) connected to a single Coordinator. -A Coordinator itself must have the Component name `COORDINATOR` (ASCII encoded). +A Coordinator itself must have the Component name `COORDINATOR`. Similarly, every Node must have a name, the _Namespace_. Every Namespace must be unique in the Network. +It must be a series of ASCII characters, without the character ".". As each Component belongs to exactly one Node, it is fully identified by the combination of Namespace and Component name, which is globally unique. This _Full name_ is the composition of Namespace, ".", and Component name. For example `N1.CA` is the Full name of the Component `CA` in the Node `N1`. @@ -45,6 +47,8 @@ For example `N1.CA` is the Full name of the Component `CA` in the Node `N1`. The receiver of a message may be specified by Component name alone if the receiver belongs to the same Node as the sender. In all other cases, the receiver of a message must be specified by the Full name. +The sender of a message must be specified by Full name, except during SIGNIN, when the Component name alone is sufficient. + #### Message composition @@ -146,7 +150,6 @@ sequenceDiagram #### Communication with other Components The following two examples show how a message is transferred between two components `CA`, `CB` via one or two Coordinators. -Coordinators shall send messages from their DEALER socket to other Coordinator's ROUTER socket. Coordinators shall route the message to the corresponding Coordinator or connected Component. From c62844c1a60ad3b657483654c6305b0f646fc6b2 Mon Sep 17 00:00:00 2001 From: Benedikt Moneke <67148916+bmoneke@users.noreply.github.com> Date: Tue, 14 Feb 2023 08:42:06 +0100 Subject: [PATCH 21/27] Apply suggestions from @bklebel Co-authored-by: Benjamin Klebel-Knobloch <32774645+bklebel@users.noreply.github.com> --- appendix.md | 2 +- control_protocol.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/appendix.md b/appendix.md index f6923b4..51392fe 100644 --- a/appendix.md +++ b/appendix.md @@ -11,7 +11,7 @@ A small introduction to ROUTER sockets, for more details see [zmq guide chapter The ROUTER socket is mostly used in a server role as it can maintain connections to many peers. In order to distinguish peers, it assigns a random _identity_ to each connected peer. -Application code does not know, which peer gets which identity, however, the identity of a peer stays the same for the lifetime of the connection. +Application code does not know which peer gets which identity, however, the identity of a peer stays the same for the lifetime of the connection. If, for example, two Components `CA`, `CB` connect to a ROUTER socket, the socket assigns identities, which will be called `IA`, `IB` here (they can be any byte sequence). diff --git a/control_protocol.md b/control_protocol.md index 89afae1..e98b69a 100644 --- a/control_protocol.md +++ b/control_protocol.md @@ -238,7 +238,7 @@ flowchart TB #### Coordinator coordination -Each Coordinator shall keep an up-to-date global {ref}`directory` with the Names of all Components in the Network. +Each Coordinator shall keep an up-to-date global {ref}`directory` with the Full names of all Components in the Network. For this, Coordinators shall notify each other about sign-ins and sign-outs of Components and Coordinators. On request, Coordinators shall send the Names of their local or global Directory, depending on the request type. From 1c5ee094c413eb9463c03c9c3be3bdadd544d7e2 Mon Sep 17 00:00:00 2001 From: Benedikt Moneke <67148916+bmoneke@users.noreply.github.com> Date: Mon, 13 Feb 2023 16:38:08 +0100 Subject: [PATCH 22/27] Small clarity improvements. --- control_protocol.md | 16 +++++++++------- 1 file changed, 9 insertions(+), 7 deletions(-) diff --git a/control_protocol.md b/control_protocol.md index e98b69a..bf8653c 100644 --- a/control_protocol.md +++ b/control_protocol.md @@ -255,6 +255,10 @@ The sign-out might happen because the Coordinator shuts down. These are the sign-in/sign-out sequences between Coordinators, where `address` is for example the host name and port number of the Coordinator's ROUTER socket. +In this diagram, the Namespaces (`N1`, `N2`) are added for clarity. +They are transmitted in the sender frame. +For example `ACK: Namespace is N2` is transmitted as `V|N1.COORDINATOR|N2.COORDINATOR|H|ACK`. + :::{mermaid} sequenceDiagram participant r1 as ROUTER @@ -297,11 +301,11 @@ sequenceDiagram The sign-in procedure is symmetric, with exception of not creating a DEALER socket, if it already exists. ::: -These are the decision processes for signing in: +These are the decision processes of `N1`'s Coordinator (at `address1`) for signing in: :::{mermaid} graph TD - COS([ROUTER receives
'CO_SIGNIN N2 address2']) --> Store[Store the identity of 'N2'] - Store --> ACK[Respond 'ACK N1'
via ROUTER] + COS([ROUTER receives
'CO_SIGNIN address2'
from N2.COORDINATOR]) --> Store[Store the identity of 'N2'] + Store --> ACK[Respond 'ACK'
via ROUTER] ACK-->DEAL{DEALER socket
with name 'N2'
already created?} DEAL -->|yes| END([End]) DEAL -->|no| CREATE[Create DEALER
named 'N2'] @@ -310,10 +314,10 @@ graph TD CMD([Command to sign in
to address2])-->CREATE2[Create DEALER
named 'temp-NS'] CREATE2 --> CONNECT - CONNECT-->SEND[Send 'CO_SIGNIN N1 address1'] + CONNECT-->SEND[Send 'CO_SIGNIN address1'] SEND --> END - AR(['ACK N2'
at DEALER]) --> Known{DEALER name == 'N2'?} + AR(['ACK' from N2.COORDINATOR
at DEALER socket]) --> Known{DEALER name == 'N2'?} Known -->|no| Rename[Rename DEALER socket to 'N2'] -->SLD Known -->|yes| SLD([Send my local Directory]) ::: @@ -340,8 +344,6 @@ The other Coordinators shall update their global Directory according to this mes - PING - CO_SIGNIN - CO_SIGNOUT -- CO_TELL_ALL -- CO_NEW - CO_UPDATE From 509d57f580192b96ac073148d4e450aaf39c1d15 Mon Sep 17 00:00:00 2001 From: Benedikt Moneke <67148916+bmoneke@users.noreply.github.com> Date: Tue, 14 Feb 2023 19:00:14 +0100 Subject: [PATCH 23/27] Sign-in procedure improved. --- control_protocol.md | 45 ++++++++++----------------------------------- 1 file changed, 10 insertions(+), 35 deletions(-) diff --git a/control_protocol.md b/control_protocol.md index bf8653c..4275852 100644 --- a/control_protocol.md +++ b/control_protocol.md @@ -255,10 +255,6 @@ The sign-out might happen because the Coordinator shuts down. These are the sign-in/sign-out sequences between Coordinators, where `address` is for example the host name and port number of the Coordinator's ROUTER socket. -In this diagram, the Namespaces (`N1`, `N2`) are added for clarity. -They are transmitted in the sender frame. -For example `ACK: Namespace is N2` is transmitted as `V|N1.COORDINATOR|N2.COORDINATOR|H|ACK`. - :::{mermaid} sequenceDiagram participant r1 as ROUTER @@ -272,20 +268,21 @@ sequenceDiagram activate d1 Note left of d1: created with
name "temp-NS" d1-->>r2: connect to address2 - d1->>r2: CO_SIGNIN
N1, address1 + d1->>r2: V|COORDINATOR|N1.COORDINATOR|H|
CO_SIGNIN Note right of r2: stores N1 identity - r2->>d1: ACK: Namespace is N2 + r2->>d1: V|N1.COORDINATOR|N2.COORDINATOR|H|ACK Note left of d1: DEALER name
set to "N2" + d1->>r2: V|N1.COORDINATOR|N2.COORDINATOR|H|
Here is my local directory + Note right of r2: Updates global
Directory and signs
in to all unknown
Coordinators,
also N1 + Note over d1,r2: Mirror of above sign-in procedure activate d2 Note left of d2: created with
name "N1" d2-->>r1: connect to address1 - d2->>r1: CO_SIGNIN
N2, address2 + d2->>r1: V|COORDINATOR|N2.COORDINATOR|H|
CO_SIGNIN Note right of r1: stores N2 identity - r1->>d2: ACK: Namespace is N1 - Note left of d1: Already has
DEALER named N2 - d1->>r2: Here is my
local Directory - Note right of r2: Updates global
Directory and signs
in to all unknown
Coordinators - d2->>r1: Here is my
local Directory + r1->>d2: V|N2.COORDINATOR|N1.COORDINATOR|H|ACK + Note left of d2: Name is already "N1" + d2->>r1: V|N2.COORDINATOR|N1.COORDINATOR|H|
Here is my local directory Note right of r1: Updates global
Directory and signs
in to all unknown
Coordinators Note over r1,d2: Sign out between two Coordinators Note right of r1: shall sign out from N2 @@ -296,30 +293,8 @@ sequenceDiagram deactivate d1 ::: - :::{note} -The sign-in procedure is symmetric, with exception of not creating a DEALER socket, if it already exists. -::: - -These are the decision processes of `N1`'s Coordinator (at `address1`) for signing in: -:::{mermaid} -graph TD - COS([ROUTER receives
'CO_SIGNIN address2'
from N2.COORDINATOR]) --> Store[Store the identity of 'N2'] - Store --> ACK[Respond 'ACK'
via ROUTER] - ACK-->DEAL{DEALER socket
with name 'N2'
already created?} - DEAL -->|yes| END([End]) - DEAL -->|no| CREATE[Create DEALER
named 'N2'] - CREATE --> CONNECT[Connect to address2] - - CMD([Command to sign in
to address2])-->CREATE2[Create DEALER
named 'temp-NS'] - CREATE2 --> CONNECT - - CONNECT-->SEND[Send 'CO_SIGNIN address1'] - SEND --> END - - AR(['ACK' from N2.COORDINATOR
at DEALER socket]) --> Known{DEALER name == 'N2'?} - Known -->|no| Rename[Rename DEALER socket to 'N2'] -->SLD - Known -->|yes| SLD([Send my local Directory]) +Note that the DEALER socket responds with the local Directory to the received Acknowledgment. ::: From 7ecbc43982ebf8e38986e0e29798f712db7eb7f6 Mon Sep 17 00:00:00 2001 From: Christoph Buchner Date: Tue, 14 Feb 2023 22:27:18 +0100 Subject: [PATCH 24/27] Increase depth of generated level heading anchors. --- conf.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/conf.py b/conf.py index d9a9d06..88ec9fb 100644 --- a/conf.py +++ b/conf.py @@ -18,7 +18,7 @@ myst_enable_extensions = [ "colon_fence", ] -myst_heading_anchors = 3 +myst_heading_anchors = 4 templates_path = ['_templates'] exclude_patterns = ['_build', 'Thumbs.db', '.DS_Store'] From 9fc3316c53deb29e88f71001bc013be49542847b Mon Sep 17 00:00:00 2001 From: Benedikt Moneke <67148916+bmoneke@users.noreply.github.com> Date: Wed, 15 Feb 2023 10:21:21 +0100 Subject: [PATCH 25/27] Small changes and links fixed (auto-slugs to not work sometimes). --- appendix.md | 2 ++ control_protocol.md | 21 ++++++++++++++++----- glossary.md | 4 ++-- 3 files changed, 20 insertions(+), 7 deletions(-) diff --git a/appendix.md b/appendix.md index 51392fe..a556af1 100644 --- a/appendix.md +++ b/appendix.md @@ -1,9 +1,11 @@ # Appendix + ## ZMQ Some useful hints regarding how the zmq package works. + (router-sockets)= ### Particularities of ROUTER sockets diff --git a/control_protocol.md b/control_protocol.md index 4275852..27d376a 100644 --- a/control_protocol.md +++ b/control_protocol.md @@ -1,7 +1,7 @@ # Control protocol The control protocol transmits messages via its {ref}`transport-layer` from one Component to another. -The {ref]}`message-layer` is the common language to understand commands, thus creating a remote procedure call. +The {ref}`message-layer` is the common language to understand commands, thus creating a remote procedure call. (transport-layer)= @@ -12,9 +12,10 @@ The transport layer ensures that a message arrives at its destination. ### Protocol basics + #### Socket Configuration -Each {ref}`Coordinator ` shall offer one {ref}`ROUTER` socket, bound to an address. +Each {ref}`Coordinator ` shall offer one {ref}`ROUTER ` socket, bound to an address. The address consists of a host (this can be the host name, an IP address of the device, or "\*" for all IP addresses of the device) and a port number, for example `*:12345` for all IP addresses at the port `12345`. {ref}`Components ` shall have one DEALER socket connecting to one Coordinator's ROUTER socket. @@ -60,6 +61,7 @@ A message consists of 4 or more frames. 5. Message content: The optional payload, which can be 0 or more frames. +(directory)= #### Directory Each Coordinator shall have a list of the Components (including other Coordinators) connected to it. @@ -73,13 +75,19 @@ The _global Directory_ is the combination of the Directories of all Coordinators In the protocol examples, `CA`, `CB`, etc. indicate Component names. `N1`, `N2`, etc. indicate Node Namespaces and `Co1`, `Co2` their corresponding Coordinators. -Here the Message content is expressed in plain English, for the exact definition see {ref}`message-layer`. +Here the Message content is expressed in plain English and placed in the Content frame, for the exact definition see {ref}`message-layer`. + +:::{note} +TBD: How to show the encoded content in the examples? +::: + In the exchange of messages, only the messages over the wire are shown, the connection identity used by the ROUTER socket is not shown. #### Communication with the Coordinator + (sign-in)= ##### Initial connection @@ -89,7 +97,7 @@ In that case, the Coordinator may indicate a suitable, still available variation The Component may retry SIGNIN with a different chosen name. After a successful handshake, the Coordinator shall store the (zmq) connection identity and corresponding Component name in its {ref}`directory`. -It shall also notify the other Coordinators in the network that this Component signed in, see {ref}`Coordinator coordination`. +It shall also notify the other Coordinators in the network that this Component signed in, see {ref}`coordinator-coordination`. Similarly, the Component shall store the Namespace and use it from this moment on, to generate its Full name. If a Component does send a message to someone without having signed in, the Coordinator shall refuse message handling and return an error. @@ -130,11 +138,12 @@ TBD: Heartbeat details are still to be determined. ::: +(signing-out)= ##### Signing out A Component should send a SIGNOUT message to its Coordinator when it stops participating in the Network. The Coordinator shall ACKNOWLEDGE the sign-out and remove the Component name from its local {ref}`directory`. -It shall also notify the other Coordinators in the network that this Component signed out, see {ref}`Coordinator coordination`. +It shall also notify the other Coordinators in the network that this Component signed out, see {ref}`coordinator-coordination`. :::{mermaid} sequenceDiagram @@ -236,6 +245,7 @@ flowchart TB ::: +(coordinator-coordination)= #### Coordinator coordination Each Coordinator shall keep an up-to-date global {ref}`directory` with the Full names of all Components in the Network. @@ -245,6 +255,7 @@ On request, Coordinators shall send the Names of their local or global Directory For the format of the Messages, see {ref}`message-layer`. +(coordinator-sign-in)= ##### Coordinator sign-in A Coordinator joins a Network by signing in to any Coordinator of that Network. diff --git a/glossary.md b/glossary.md index ed13906..e4f4afe 100644 --- a/glossary.md +++ b/glossary.md @@ -24,14 +24,14 @@ Director A Component which takes part in orchestrating a (i.e. LECO-controlled) measurement setup, see {ref}`components.md#director`. Directory - Each Coordinator maintains a local Directory with all the Components connected to it, and a global Directory with all Components in the Network, see {ref}`control_protocol.md#directory`. + Each Coordinator maintains a local Directory with all the Components connected to it (i.e. other Coordinators and the Components of its own Node), and a global Directory with all Components in the whole Network, see {ref}`control_protocol.md#directory`. Driver An object that takes care of communicating with a Device. This object is external to LECO, for example coming from and instrument control library like `pymeasure`, `instrumentkit` or `yaq`. See {ref}`components.md#driver`. Full name The name of a Component unique for the whole setup. - It consists of the {ref}`namespace` and {ref}`component-name`, see {ref}`control_protocol.md#naming-scheme`. + It consists of the namespace and component-name, see {ref}`control_protocol.md#naming-scheme`. LECO The **L**aboratory **E**xperiment **CO**ntrol protocol framework. From 217dcefd2f744c84129bd98bacf0a206f1ef3115 Mon Sep 17 00:00:00 2001 From: Benedikt Moneke <67148916+bmoneke@users.noreply.github.com> Date: Thu, 16 Feb 2023 10:02:42 +0100 Subject: [PATCH 26/27] Links made to work and Directory overhaul. --- appendix.md | 3 +-- conf.py | 2 +- control_protocol.md | 64 ++++++++++++++++++++------------------------- 3 files changed, 31 insertions(+), 38 deletions(-) diff --git a/appendix.md b/appendix.md index a556af1..e7692f6 100644 --- a/appendix.md +++ b/appendix.md @@ -6,8 +6,7 @@ Some useful hints regarding how the zmq package works. -(router-sockets)= -### Particularities of ROUTER sockets +### ROUTER sockets A small introduction to ROUTER sockets, for more details see [zmq guide chapter 3](https://zguide.zeromq.org/docs/chapter3/#Exploring-ROUTER-Sockets). diff --git a/conf.py b/conf.py index 88ec9fb..5a1d0c4 100644 --- a/conf.py +++ b/conf.py @@ -18,7 +18,7 @@ myst_enable_extensions = [ "colon_fence", ] -myst_heading_anchors = 4 +myst_heading_anchors = 5 templates_path = ['_templates'] exclude_patterns = ['_build', 'Thumbs.db', '.DS_Store'] diff --git a/control_protocol.md b/control_protocol.md index 27d376a..248ba4b 100644 --- a/control_protocol.md +++ b/control_protocol.md @@ -1,10 +1,9 @@ # Control protocol -The control protocol transmits messages via its {ref}`transport-layer` from one Component to another. -The {ref}`message-layer` is the common language to understand commands, thus creating a remote procedure call. +The control protocol transmits messages via its {ref}`control_protocol.md#transport-layer` from one Component to another. +The {ref}`control_protocol.md#message-layer` is the common language to understand commands, thus creating a remote procedure call. -(transport-layer)= ## Transport layer The transport layer ensures that a message arrives at its destination. @@ -15,7 +14,7 @@ The transport layer ensures that a message arrives at its destination. #### Socket Configuration -Each {ref}`Coordinator ` shall offer one {ref}`ROUTER ` socket, bound to an address. +Each {ref}`Coordinator ` shall offer one {ref}`ROUTER ` socket, bound to an address. The address consists of a host (this can be the host name, an IP address of the device, or "\*" for all IP addresses of the device) and a port number, for example `*:12345` for all IP addresses at the port `12345`. {ref}`Components ` shall have one DEALER socket connecting to one Coordinator's ROUTER socket. @@ -28,7 +27,7 @@ While the number of DEALER sockets thus required scales badly with the number of ::: Communicating with a Coordinator, messages must be sent to a Coordinator's ROUTER socket. -Only for acknowledging a {ref}`coordinator-sign-in`, it is permitted to send a message to a Coordinator's DEALER socket. +Only for acknowledging a {ref}`control_protocol.md#coordinator-sign-in`, it is permitted to send a message to a Coordinator's DEALER socket. #### Naming scheme @@ -61,13 +60,14 @@ A message consists of 4 or more frames. 5. Message content: The optional payload, which can be 0 or more frames. -(directory)= #### Directory -Each Coordinator shall have a list of the Components (including other Coordinators) connected to it. +Each Coordinator shall have a list of the Components connected to it. This is its _local Directory_. -The _global Directory_ is the combination of the Directories of all Coordinators in a Network. +They shall also keep a list of the addresses of all Coordinators, they are connected to. + +Additionally, they shall maintain a _global Directory_, which is a Coordinator's copy of the union of the local Directories of all Coordinators in a Network. ### Conversation protocol @@ -75,7 +75,7 @@ The _global Directory_ is the combination of the Directories of all Coordinators In the protocol examples, `CA`, `CB`, etc. indicate Component names. `N1`, `N2`, etc. indicate Node Namespaces and `Co1`, `Co2` their corresponding Coordinators. -Here the Message content is expressed in plain English and placed in the Content frame, for the exact definition see {ref}`message-layer`. +Here the Message content is expressed in plain English and placed in the Content frame, for the exact definition see {ref}`control_protocol.md#message-layer`. :::{note} TBD: How to show the encoded content in the examples? @@ -88,16 +88,15 @@ In the exchange of messages, only the messages over the wire are shown, the conn #### Communication with the Coordinator -(sign-in)= -##### Initial connection +##### Signing-in After connecting to a Coordinator (`Co1`), a Component (`CA`) shall send a SIGNIN message indicating its Component name. The Coordinator shall indicate success/acceptance with an ACKNOWLEDGE response, giving the Namespace and other relevant information, or reply with an ERROR, e.g. if the Component name is already taken. In that case, the Coordinator may indicate a suitable, still available variation on the indicated Component name. The Component may retry SIGNIN with a different chosen name. -After a successful handshake, the Coordinator shall store the (zmq) connection identity and corresponding Component name in its {ref}`directory`. -It shall also notify the other Coordinators in the network that this Component signed in, see {ref}`coordinator-coordination`. +After a successful handshake, the Coordinator shall store the Component name in its {ref}`control_protocol.md#directory` and shall ensure message delivery to that Component (e.g. by storing the (zmq) connection identity with the local directory). +It shall also notify the other Coordinators in the network that this Component signed in, see {ref}`control_protocol.md#coordinator-coordination`. Similarly, the Component shall store the Namespace and use it from this moment on, to generate its Full name. If a Component does send a message to someone without having signed in, the Coordinator shall refuse message handling and return an error. @@ -131,19 +130,18 @@ Heartbeats are used to know whether a communication peer is still online. Every message received counts as a heartbeat. A Component should and a Coordinator shall send a PING and wait some time before considering a connection dead. -A Coordinator shall follow the {ref}`sign out routine` for a signed in Component considered dead. +A Coordinator shall follow the {ref}`control_protocol.md#signing-out` for a signed in Component considered dead. :::{note} TBD: Heartbeat details are still to be determined. ::: -(signing-out)= ##### Signing out A Component should send a SIGNOUT message to its Coordinator when it stops participating in the Network. -The Coordinator shall ACKNOWLEDGE the sign-out and remove the Component name from its local {ref}`directory`. -It shall also notify the other Coordinators in the network that this Component signed out, see {ref}`coordinator-coordination`. +The Coordinator shall ACKNOWLEDGE the sign-out and remove the Component name from its local {ref}`control_protocol.md#directory`. +It shall also notify the other Coordinators in the network that this Component signed out, see {ref}`control_protocol.md#coordinator-coordination`. :::{mermaid} sequenceDiagram @@ -194,7 +192,7 @@ sequenceDiagram ::: Prerequisites of Communication between two Components are: -- Both Components are connected to a Coordinator and {ref}`signed in`. +- Both Components are connected to a Coordinator and {ref}`signed in`. - Both Components are either connected to the same Coordinator (example one), or their Coordinators are connected to each other (example two). @@ -214,7 +212,7 @@ flowchart TB R0[receive] == "iA|V|nR.recipient|nS.CA|H|Content" ==> CnS{nS == N1?} CnS-->|no| RemIdent CnS-->|yes| Clocal{CA in
local Directory?} - Clocal -->|yes| CidKnown{iA is CA's identity
in local Directory?} + Clocal -->|yes| CidKnown{iA is CA's identity?} CidKnown -->|yes| RemIdent Clocal -.->|no| E1[ERROR: Sender unknown] ==>|"iA|V|nS.CA|N1.COORDINATOR|H|ERROR: Sender unknown"| S S[send] ==> WA([N1.CA DEALER]) @@ -245,22 +243,17 @@ flowchart TB ::: -(coordinator-coordination)= #### Coordinator coordination -Each Coordinator shall keep an up-to-date global {ref}`directory` with the Full names of all Components in the Network. -For this, Coordinators shall notify each other about sign-ins and sign-outs of Components and Coordinators. -On request, Coordinators shall send the Names of their local or global Directory, depending on the request type. - -For the format of the Messages, see {ref}`message-layer`. +Coordinators are the backbone of the Network and need to coordinate themselves. -(coordinator-sign-in)= ##### Coordinator sign-in A Coordinator joins a Network by signing in to any Coordinator of that Network. The sign-in/sign-out procedure between two Coordinators is more thorough than that of Components. -During the sign-in procedure, Coordinators exchange their local Directories and shall sign in to all Coordinators, they are not yet signed in. +During the sign-in procedure, Coordinators exchange their local Directories and addresses of all known Coordinatos. +They shall sign in to all Coordinators, they are not yet signed in. The sign-in might happen because the Coordinator learns a new Coordinator address via Directory updates or at startup. The sign-out might happen because the Coordinator shuts down. @@ -283,7 +276,7 @@ sequenceDiagram Note right of r2: stores N1 identity r2->>d1: V|N1.COORDINATOR|N2.COORDINATOR|H|ACK Note left of d1: DEALER name
set to "N2" - d1->>r2: V|N1.COORDINATOR|N2.COORDINATOR|H|
Here is my local directory + d1->>r2: V|N1.COORDINATOR|N2.COORDINATOR|H|
Here is my local directory
and Coordinator addresses Note right of r2: Updates global
Directory and signs
in to all unknown
Coordinators,
also N1 Note over d1,r2: Mirror of above sign-in procedure activate d2 @@ -293,7 +286,7 @@ sequenceDiagram Note right of r1: stores N2 identity r1->>d2: V|N2.COORDINATOR|N1.COORDINATOR|H|ACK Note left of d2: Name is already "N1" - d2->>r1: V|N2.COORDINATOR|N1.COORDINATOR|H|
Here is my local directory + d2->>r1: V|N2.COORDINATOR|N1.COORDINATOR|H|
Here is my local directory
and Coordinator addresses Note right of r1: Updates global
Directory and signs
in to all unknown
Coordinators Note over r1,d2: Sign out between two Coordinators Note right of r1: shall sign out from N2 @@ -305,20 +298,22 @@ sequenceDiagram ::: :::{note} -Note that the DEALER socket responds with the local Directory to the received Acknowledgment. +Note that the DEALER socket responds with the local Directory and Coordinator addresses to the received Acknowledgment. ::: ##### Coordinator updates -Whenever a Component signs in to or out from its Coordinator, the Coordinator shall send a CO_UPDATE message regarding this event to all the other Coordinators. -The message shall contain the Full name of the Component and the event type (sign in or out) +Each Coordinator shall keep an up-to-date global {ref}`control_protocol.md#directory` with the Full names of all Components in the Network. +For this, whenever a Component signs in to or out from its Coordinator, the Coordinator shall notify all the other Coordinators regarding this event. The other Coordinators shall update their global Directory according to this message (add or remove an entry). +On request, Coordinators shall send the Names of their local or global Directory, depending on the request type. + +For the format of the Messages, see {ref}`control_protocol.md#message-layer`. -(message-layer)= -## Message layer +## Message layer ### Messages for Transport Layer @@ -330,7 +325,6 @@ The other Coordinators shall update their global Directory according to this mes - PING - CO_SIGNIN - CO_SIGNOUT -- CO_UPDATE From 59298113b2f3a7abe465d7d56ea0bc7b0f66e621 Mon Sep 17 00:00:00 2001 From: Benedikt Moneke <67148916+bmoneke@users.noreply.github.com> Date: Thu, 16 Feb 2023 12:10:19 +0100 Subject: [PATCH 27/27] Restrict Component name and Namespace to printable ASCII, closes #47 --- control_protocol.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/control_protocol.md b/control_protocol.md index 248ba4b..e38ce96 100644 --- a/control_protocol.md +++ b/control_protocol.md @@ -33,13 +33,14 @@ Only for acknowledging a {ref}`control_protocol.md#coordinator-sign-in`, it is p #### Naming scheme Each Component must have an individual name, given by the user, the _Component name_. -A Component name must be a series of ASCII characters, without the character ".". Component names must be unique in a {ref}`Node `, i.e. among the Components (except other Coordinators) connected to a single Coordinator. A Coordinator itself must have the Component name `COORDINATOR`. Similarly, every Node must have a name, the _Namespace_. Every Namespace must be unique in the Network. -It must be a series of ASCII characters, without the character ".". + +A Component name or a Namespace must be a series of printable ASCII characters (byte values 0x20 to 0x7E), without the character "." (byte value 0x2E). + As each Component belongs to exactly one Node, it is fully identified by the combination of Namespace and Component name, which is globally unique. This _Full name_ is the composition of Namespace, ".", and Component name. For example `N1.CA` is the Full name of the Component `CA` in the Node `N1`.