Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

loadProjectFromDirectory file sync, import & export sync #159

Open
1 of 3 tasks
samuelstroschein opened this issue Aug 30, 2024 — with Linear · 8 comments
Open
1 of 3 tasks

loadProjectFromDirectory file sync, import & export sync #159

samuelstroschein opened this issue Aug 30, 2024 — with Linear · 8 comments
Assignees

Comments

Copy link
Member

samuelstroschein commented Aug 30, 2024

Context

@samuel.stroschein said in MESDK-176:

Update: saveProjectToDirectory is not needed until we need roundtrip capabilities from dev tools <-> web apps. As long as the dev tools are single player, loadProjectFromDirectory can set up file watching and auto in- and export.

cc @felix.haeberle

Task

  • add file syncing between host <-> lix
  • add watching for changes in toBeImportedFiles()
  • add auto import/export
Copy link
Member Author

@martin.lysk1 can you take this over?

progress is in branch https://github.com/opral/monorepo/tree/mesdk-200-loadprojectfromdirectory

Understanding File Syncing in Inline SDK

Copy link

sure i am on it

Copy link

Quick question about syncing from the inlang lix to the host:
Process is like this:

  1. create a project in memory
  2. import everything from the inlang folder passed in start watching - on change on
  3. on change on files in lix, copy the files to the host.

Regarding 3. some followup questions.

How shall we distinguish between files we want to sync to the host (like settings) and those we don't want to pass to the host, like the inlang.db i guess.

Option 1 - hardcode a block list and just skip the inlang.db
Option 2 - keep track of files imported via copyFilesFromDiskRecursive and sync only those that we imported to the host again.

Lets go for Option 1 @samuel.stroschein?

How shall we notice a change in a file

Option 1 - should we add a callback to the lix api - on file change, create a trigger in lix?Option 2 - should we create a trigger using the lix db object in inlang

I prefere Option 1 but 2 is faster

Both will break with https://linear.app/opral/issue/LIXDK-102/re-visit-simplifying-the-change-queue-implementation#comment-65eb3485

Copy link
Member Author

How shall we distinguish between files we want to sync to the host (like settings) and those we don't want to pass to the host, like the inlang.db i guess.

Yes, hardcode ignored files.

How shall we notice a change in a file

You can use pollQuery which only emits an event if the underlying query result changes.

(pollQuery is simulating an observable for sqlite. Granted, not real time but good enough. Lix won't need callbacks and manually setting up triggers is also not needed)

Copy link
Member Author

samuelstroschein commented Sep 9, 2024

@martin.lysk1 I completed MESDK-176. During the implementation the following observations emerged that you should consider for this issue:

  1. Don't forget to use absolutePath and withAbsolutePaths when you interact with fs stuff that plugins use. See Fix cwd bug with plugin imports by making paths absolute #174
  2. the saveProjectToDirectory tests might get flaky if loadProjectFromDirectory establishes watchers. To mitigate the problem I propose to add a watch: boolean flag to loadProjectFromDirectory that turns the sync on or off.

EDIT: yeah auto watching in loadProjectFromDirectory will likely lead to unexpected behavior see MESDK-196

Copy link
Member Author

samuelstroschein commented Sep 11, 2024

After a sync with @martin.lysk1 we concluded:

  • using lix LIXDK-126 to solve the sync issues makes sense
  • because branching is WIP, martin will work on LIXDK-147 and LIXDK-158 in the meantime

cc @felix.haeberle the sync to the fs will be delayed until LIXDK-126 exists. i deem this to be OK given that sherlock doesn't need to change when this feature ships.

Copy link

Copy link
Member Author

marking as on hold because martin is focusing on a csv hosted demo for the onsite

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants