Detailed Self-Hosting Guide

[AndryHTC]

Title: Request for Detailed Self-Hosting Guide for Open Source Version

---

Hello,

I hope you're all doing well. I'm reaching out to discuss the need for a comprehensive self-hosting guide for the open-source version of Estuary.

Over the last couple of months, it has been apparent that the community, including myself, has been struggling with self-hosting Estuary. We understand that the focus has been on the managed web application, but it's equally important to cater to the needs of the open-source community.

From recent threads, it's clear that the issue is widely acknowledged, and some users have managed to piece together a makeshift guide to self-hosting (thread reference). Despite this, we believe that it would be highly beneficial to have an official, well-documented, and comprehensive guide from the team behind Estuary.

As users of open-source software, we often rely on the independence that comes with being able to host solutions on our own infrastructures. This provides us the flexibility to tailor solutions to our specific requirements and environments. It's also crucial for those who need to meet strict on-premises data storage requirements due to regulatory, security, or business reasons.

While we appreciate the updates about enabling users to run their own data-plane while using your cloud-hosted service for the control plane, we feel that a well-documented guide on self-hosting is crucial for those of us who opt for a full on-premises solution. Furthermore, clear documentation will empower more users to test, provide feedback, and even contribute to the further development of Estuary, enhancing the overall user experience and the robustness of the software.

We appreciate the work that has been done thus far, and look forward to Estuary's continued development and enhancement. It's exciting to see Estuary evolve and we hope to contribute to its growth by aiding in the debugging and expansion process.

Thank you for considering our request. We appreciate any assistance or direction you can provide regarding self-hosting.

In our case (Asset Plan) we are strictly under a VPN, other than all the security regulations, and until we have the possibility to self host Estuary Flow, for now we have to drop the idea to use it and contribute to it.

Best,
Andrea

[psFried]

Your characterization of the situation is pretty much right. We'd love for people to be able to self host, but the current state of Flow and our docs is not nearly sufficient to support that. Deploying Flow is really not trivial, and the requirements are changing rapidly as we develop more features. The "federated data-planes" feature, in particular, will make significant changes to how things are deployed. To host the whole UI and everything is also quite complex, and we're still changing some pieces of it. It's possible that there's a useful and minimal data-plane deployment that could stand on its own (not connected to our control-plane), but it's unclear whether that's useful enough to be worth it. Basically, things are just a little too up in the air for us to commit to maintaining self-hosting instructions in the short term. But we do intend to support self-hosting after the dust settles at little.

In the meantime, I'd love to hear your thoughts on what you'd want a self-hosted Flow to look like. Would you want Kuberenetes manifests? Cloud formation template? Something else? And would you want to host the full UI, control-plane, and everything, or would you prefer to have a simpler deployment that you just interact with via CLI?