Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Foundation utilities to support the CDC outbox-pattern #8698

Open
tzolov opened this issue Aug 10, 2023 · 3 comments
Open

Foundation utilities to support the CDC outbox-pattern #8698

tzolov opened this issue Aug 10, 2023 · 3 comments
Assignees
@tzolov
Labels
in: debezium in: jpa type: enhancement
Milestone
Backlog

Comments

@tzolov
Copy link
Contributor

tzolov commented Aug 10, 2023

As part of your business logic, you often need to create/update/delete your local data store and simultaneously send messages/events to a message broker. Furthermore you need to guarantee the atomicity of both operations (e.g. the message can be sent only if the database transaction is successful). The (2PC) is not a scalable and often not supported option for most distributed systems.

The outbox pattern provides an approach for letting services execute these two operations in a safe and consistent manner.

Spring Integration can provide common utilities to facilitate the creation and management of Outbox table and entities as well as common outbox messaging abstraction.

References:
@tzolov tzolov added type: enhancement status: waiting-for-triage The issue need to be evaluated and its future decided labels Aug 10, 2023
@tzolov tzolov self-assigned this Aug 10, 2023
tzolov added a commit to tzolov/spring-integration that referenced this issue Aug 10, 2023
@tzolov
Initial support for Outbox pattern
054cb6c 
 - Provide repostiroy that represents the database that stores the business entities and message outbox.
 - Provide event abstraction and event sender service.
 - Default implementation for the event sender service that converts the events into entities and persist them into the outbox db.

 Related to spring-projects#8698
@artembilan artembilan added this to the 6.2.0-M3 milestone Aug 10, 2023
@artembilan artembilan added in: jpa in: debezium and removed status: waiting-for-triage The issue need to be evaluated and its future decided labels Aug 10, 2023
tzolov added a commit to tzolov/spring-integration that referenced this issue Aug 30, 2023
@tzolov
Initial support for Outbox pattern
ab8a5d2 
 - Provide repostiroy that represents the database that stores the business entities and message outbox.
 - Provide event abstraction and event sender service.
 - Default implementation for the event sender service that converts the events into entities and persist them into the outbox db.

 Related to spring-projects#8698
tzolov added a commit to tzolov/spring-integration that referenced this issue Sep 4, 2023
@tzolov
Initial support for Outbox pattern
4cc2eff 
 - Provide repostiroy that represents the database that stores the business entities and message outbox.
 - Provide event abstraction and event sender service.
 - Default implementation for the event sender service that converts the events into entities and persist them into the outbox db.

 Related to spring-projects#8698
@artembilan artembilan modified the milestones: 6.2.0-M3, 6.2.0-RC1 Sep 14, 2023
tzolov added a commit to tzolov/spring-integration that referenced this issue Oct 12, 2023
@tzolov
Initial support for Outbox pattern
62c74e2 
 - Provide repostiroy that represents the database that stores the business entities and message outbox.
 - Provide event abstraction and event sender service.
 - Default implementation for the event sender service that converts the events into entities and persist them into the outbox db.

 Related to spring-projects#8698
tzolov added a commit to tzolov/spring-integration that referenced this issue Oct 12, 2023
@tzolov
Initial support for Outbox pattern
e75d6a0 
 - Provide repostiroy that represents the database that stores the business entities and message outbox.
 - Provide event abstraction and event sender service.
 - Default implementation for the event sender service that converts the events into entities and persist them into the outbox db.

 Related to spring-projects#8698
tzolov added a commit to tzolov/spring-integration that referenced this issue Oct 13, 2023
@tzolov
Initial support for Outbox pattern
c4a268a 
 - Provide repostiroy that represents the database that stores the business entities and message outbox.
 - Provide event abstraction and event sender service.
 - Default implementation for the event sender service that converts the events into entities and persist them into the outbox db.

 Related to spring-projects#8698
@artembilan artembilan modified the milestones: 6.2.0-RC1, 6.3.x Oct 13, 2023
@artembilan
Copy link
Member

Pushed to the next 6.3.x since the change is too involving to make it available for upcoming 6.2.0-RC1 next Tuesday.

tzolov added a commit to tzolov/spring-integration that referenced this issue Oct 16, 2023
@tzolov
Initial support for Outbox pattern
de4bfce 
 - Provide repostiroy that represents the database that stores the business entities and message outbox.
 - Provide event abstraction and event sender service.
 - Default implementation for the event sender service that converts the events into entities and persist them into the outbox db.

 Related to spring-projects#8698
@gdupontf
Copy link

Can you illuminate me on why this can't be achieved with a JdbcChannelMessageStore-backed QueueChannel?

@artembilan
Copy link
Member

See the pattern page: https://microservices.io/patterns/data/transactional-outbox.html.

In the end there are a couple links like this:

There are two patterns for implementing the Message relay:

The [Transaction log tailing](https://microservices.io/patterns/data/transaction-log-tailing.html) pattern
The [Polling publisher](https://microservices.io/patterns/data/polling-publisher.html) pattern

So, the QueueChannel is based on a DB polling approach, while Debezium uses transaction log tailing.
Therefore fully different implementation we are pursuing with this issue unlike the mentioned solution via QueueChannel: https://github.com/artembilan/microservices-patterns-spring-integration/tree/main/outbox

@artembilan artembilan modified the milestones: 6.3.x, Backlog Jun 10, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@tzolov tzolov

Labels
in: debezium in: jpa type: enhancement
Projects
None yet
Milestone
Backlog
Development

No branches or pull requests
3 participants
@artembilan @tzolov @gdupontf