Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Network Layer (next gen) for Complete Connectivity - this represents a revolution in network connectivity! #713

Open
CMCDragonkai opened this issue May 5, 2024 · 2 comments
Labels
design Requires design enhancement New feature or request r&d:polykey:core activity 4 End to End Networking behind Consumer NAT Devices

Comments

@CMCDragonkai
Copy link
Member

CMCDragonkai commented May 5, 2024

This is a high-level issue, a large epic with multiple branches.

Requirements of this design

  1. Complete connectivity is to ensure that all Polykey nodes regardless of their network environment can connect to each other if there is a sufficient link (potential multistep) between them.
  2. The ability for 2 nodes in symmetric NAT to discover and connect to each other.
  3. The ability for all nodes to act as relays - and thus decentralized relaying without rely on the seed nodes.
  4. The ability to ensure network segregation, firstly through mainnet and testnet separation, and subsequently private networks in PKE.
  5. The ability to simulate the network - create a "chaosnet" where one can deploy autonomous Polykey agents to perform random tasks in a loop, in order to test network stability empirically.
  6. The ability to do decentralized signalling efficiently, through each hop, that is user acceptance testing and simulation testing must be done according to Decentralised NAT Signalling #365 that was closed.
  7. The ability to attempt network passthrough routers and other equipment via UPnP Network Traversal (by asking the router nicely) - UPNP #536
  8. Optimised mdns for local network discovery - to avoid conflict with existing mdns stacks, and integration into existing OS stacks.
  9. The ability to signal AND relay over IPv4 and IPv6 and change between the 2.
  10. Increased bandwidth with optimised QUIC implementation by parallelising the send and recv calls.
  11. Get past corporate firewalls that block UDP, but enabling optional HTTP based relaying.
  12. Increased debug transparency and audit transparency - local to the node and global to the dashboard whether public or private.

Additional context

Specification

Network Identity and Membership

This is needed to separate mainnet, testnet and private networks, possibly allowing PK to connect to multiple networks simultaneously but forming segregation on every command. It's probably easier to just make PK connect to only a single network.

We also need to figure out the exact role and relationship between the certificate identity chain, and the sigchain claim chain.

Decentralized Signalling and Relaying Algorithm Design

The initial design is in #365. However it has not been applied to relaying yet, nor has it been empirically tested.

Relaying ultimately has to be done through network encapsulation. We have to make use of our RPC to pass through binary blocks from one node to another. This will need to make use of our RAW mode in our RPC for efficiency reasons. We may need to test the performance of this though, because QUIC does actually have a datagram mode, I'm not sure if this is relevant here.

And DOS prevention in #556.

Network Simulation/Visualisation

Involves the usage of Orchestrator and Pulumi.

Chaosnet

It will be important to create autonomous looped Polykey agents to be able to simulate a dirty and chaotic network to see how robust our mechanisms are.

NodeGraph Garbage Collection

The NodeGraph since #537 is capable of multi-host addressing. However GC of the NodeGraph has been neglected, and I believe fast manipulation of this node graph will need to be dealt with.

Discovery Performance and Efficiency

MDNS and related algorithms suffer from some in-efficiencies.

QUIC Performance

MatrixAI/js-quic#54

IPv4 and IPv6 Conformance

#669

Sub-Issues & Sub-PRs created

  1. ...
  2. ...
  3. ...
@CMCDragonkai CMCDragonkai added enhancement New feature or request design Requires design labels May 5, 2024
Copy link

linear bot commented May 5, 2024

@CMCDragonkai CMCDragonkai self-assigned this May 5, 2024
@CMCDragonkai CMCDragonkai changed the title Network Layer for Complete Connectivity Network Layer (next gen) for Complete Connectivity - this represents a revolution in network connectivity! May 5, 2024
@CMCDragonkai
Copy link
Member Author

I expect to promote this issue to a "Project" as it something that would fit onto a roadmap. As per the discussion on the Project Graph concept - pipeline parallelism would make sense down a particular Project Path, so we can incrementally delivery "Project" rather than spreading the team across multiple "Project" each with their own paths.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
design Requires design enhancement New feature or request r&d:polykey:core activity 4 End to End Networking behind Consumer NAT Devices
Development

No branches or pull requests

1 participant