Companies are investing into "data lakes" supported by Kafka or Google PubSub. They typically have their data in relational databases like PostgreSQL. In order to make the change on the relational data consumable for other applications (for building caches, search engines, messaging, ...), those companies need Change Data Capturing (CDC) on their database.
Google should allow to add a Replication Slot in the managed Cloud SQL Postgres instance (https://www.postgresql.org/docs/9.6/static/logicaldecoding-explanation.html) in order to allow to publish row changes to Kafka and/or PubSub.313 votes
Today Cloud SQL for PostgreSQL launched support for change data capture and logical decoding in Public Preview. Users can now stream data for real-time analytics, perform selective replication, set up external replicas, and more. To learn more, see our blog post:
Check out our release notes here:
Currently the only way to migrate Postgres into Cloud SQL is to dump and restore. This necessitates downtime. We need a way to migrate Postgres DB without downtime.
So in practice GCP needs to do a managed Bucardo service to transfer data into Cloud SQL, and subsequently we can switch the Replica status to Master status.10 votes
Cloud SQL and the Database Migration Service (DMS) offer managed migration support for PostgreSQL in Private Preview. To determine if DMS will work for your database migration, check out the website:
To learn more, check out our blog post:
- Don't see your idea?