Allow Change Data Capturing on Cloud SQL Databases, using Replication slots for Logical Decoding
Companies are investing into "data lakes" supported by Kafka or Google PubSub. They typically have their data in relational databases like PostgreSQL. In order to make the change on the relational data consumable for other applications (for building caches, search engines, messaging, ...), those companies need Change Data Capturing (CDC) on their database.
Google should allow to add a Replication Slot in the managed Cloud SQL Postgres instance (https://www.postgresql.org/docs/9.6/static/logicaldecoding-explanation.html) in order to allow to publish row changes to Kafka and/or PubSub.
Siamak Haschemi
shared this idea
Cloud SQL is investigating supporting Change Data Capture (CDC) for PostgreSQL.
4 comments
-
Siamak Haschemi
commented
@LIQID User: We could not wait for the log replication, so we used a combination of
- PostgreSQL triggers to write events to a log-table
- and a self-written Change Data Capturing Software (Java)Basically implementing the Transactional Outbox Pattern (https://microservices.io/patterns/data/transactional-outbox.html)
-
Tu Nguyen
commented
Is there anyway to use logical replication with Debezium? I'm using Postgres 11
-
LIQID User
commented
@siamak Haschemi So what was your alternate to this please?
-
Siamak Haschemi
commented