Archiving streams, snapshots and async projections

[elexisvenator]

More of a discussion than an issue.

I've been putting together ideas for how to remodel one of our apps that uses marten events. Currently the main business entity - a versioned document - has its own stream. I see this as a problem as these documents tend to live forever and can have ongoing activity. The new design would split out parts into their own streams - most importantly each "document version" is its own stream. This along with some other changes I think is a good potential design, but it led to me thinking about archiving. So here is what I am considering:

• Event streams track changes to a document over time.
• 90% of the time, you only care about the latest shape of the document.
• Once a document is "complete", you would only care about the latest shape 99.9% of the time. The other 0.1% is auditing.
• If a document is to be archived, this would involve an event being written. There may be projections that need to consume this event.

So my questions are:

• If I want to archive a stream after this last event, how can I reliably ensure that async projections have processed this event first?
• If I want to make a "snapshot" of an archived stream, is the current projection system the right way to do this? Is there a better or future way instead?
• If I build a new projection that needs to be aware of archived streams, can it catch up these streams (suspect this is the answer)

I think the first question is the most important, as I don't like writing code that is essentially "wait until highwater mark is equal to this event sequence id".

[jeremydmiller]

I don't actually want to do #2982. I don't think that's necessarily a good idea to change the existing behavior. I'll think more on this later though, but this week is awful for me and I'm just trying to hang on til Friday:)

[jeremydmiller]

Think about this as "stream compacting". Two possible ways:

1. Retain the existing surrogate stream id, but effectively rewrite the stream as an initial event that supplies the initial state
2. "Re-stream" to a new stream altogether, but archive off the original stream so that it at least gets to the archived partition

Still need more analysis on the rest of the points that @elexisvenator brought up though