by

Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
    Chris Cranford
    @Naros
    @amoswood If you have super-user access, you could try set global binlog_row_image = 'full';
    Amos Wood
    @amoswood
    @Naros , I have the most admin privileges that AWS RDS MySQL will let you have and I can't run that command.
    Chris Cranford
    @Naros
    @amoswood I had a similar discussion with another user back in February; can you check the global variables and see what its already set as? They found that while they couldn't change it, the default was FULL so no changes were necessary.
    Amos Wood
    @amoswood
    @Naros , I left it at the default and tried the connector which is what gave me that error. So, the connector code is checking the instance and saying that it isn't configured correctly. The error is The MySQL server is not configured to use a row-level binlog.
    gyh2020
    @gyh2020

    Hi,When I have used Debezium 1.1 to get data from SQL Server2014 , I found that if CDC feature is enabled on columns instead of table , Kafka connect will throw below error:

    Data row is smaller than a column index, internal schema representation is probably out of sync with real database schema\n\tat

    And I have not changed related table schema

    1 reply
    lenxia
    @lenxia
    I used debezium in Oracle standbydb, and configured by the debeizum-doc,but got a StreamException:ORA-16000 database for read-only access
    Balaji Mohandas
    @BalajiMohandas

    Hi There, I am trying to bring up my local docker instance based on the Debezium docker tutorial from GitHub. I am trying to connect to DB2 instance and I am able successfully setup DB2 server and with ASNCDC schema capturing all changes I do on the user schema. I did setup my docker instance based on the docker-compose-db2.yaml file for provided for DB2 . I did registered the connector using postman

    http://localhost:8083/connectors*
    {
    "name": "inventory-connector",
    "config": {
    "connector.class": "io.debezium.connector.db2.Db2Connector",
    "database.hostname": "192.168.0.111",
    "database.port": "50000",
    "database.user": "db2inst1",
    "database.password": "xyz78!",
    "database.dbname": "TESTDB",
    "database.server.name": "inventory",
    "table.whitelist": "DB2INST1.CUSTOMERS",
    "database.history.kafka.bootstrap.servers": "kafka:9092",
    "database.history.kafka.topic": "dbhistory.inventory"
    }
    }

    Response*
    {
    "name": "inventory-connector",
    "config": {
    "connector.class": "io.debezium.connector.db2.Db2Connector",
    "database.hostname": "192.168.0.111",
    "database.port": "50000",
    "database.user": "db2inst1",
    "database.password": "xyz78!",
    "database.dbname": "TESTDB",
    "database.server.name": "inventory",
    "table.whitelist": "DB2INST1.CUSTOMERS",
    "database.history.kafka.bootstrap.servers": "kafka:9092",
    "database.history.kafka.topic": "dbhistory.inventory",
    "name": "inventory-connector"
    },
    "tasks": [
    {
    "connector": "inventory-connector",
    "task": 0
    }
    ],
    "type": "source"
    }

    I did modified few rows in CUSTOMERS and PRODUCT_ON_HAND table, I do not see the topics generated for these table, upon update. Can someone guide me to troubleshoot the issue, especially where I look for the logs, that will help to trouble to shoot the issue.

    5 replies
    Quarantined-soul-alochna
    @alochna1_twitter
    Hello , i'm trying to set up AWS RDS Postgresql CDC to AWS MSK and hosting connect server on EC2 instance but while creating the connector i get this error that entry is missing in pg_hba.conf file. FATAL: no pg_hba.conf entry for replication connection from host \"172.z.y.x\", user \"postgres\" also looks like AWS RDS doesn't allow that property to be changed according to this documentation. https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Appendix.PostgreSQL.CommonDBATasks.html. Any pointers ? or is AWS RDS or AURORA postgres a no go place ? thank you for answers
    hanoisteve
    @hanoisteve
    Debezium for outbox pattern. I have debezium mysql connector installed and its streaming all the table changes at the moment to Kafka. For outbox pattern implementation should I have an separate instance of the connector just for monitoring outbox_event table?
    Also how do I add the outbox related config to an existing debezium connector? In general, how to do an update as opposed to a create of the config for a connector? Up to now I have been doing drop, and create to make modifications.
    Kushal Reddy
    @kushalreddy_gitlab
    Hi @jpechane I have updated the jira , can you pls take a look . Thank you .
    Vincent-Zeng
    @Vincent-Zeng
    Hi, team. Is there any way to remove source field in record and remain other fields?
    Alexander Ryzhenko
    @aryzhenko
    Hi all. What can it be ?
    [2020-05-28 12:07:01,918] ERROR WorkerSourceTask{id=mysql_connector_google_stage-0} Task threw an uncaught and unrecoverable exception (org.apache.kafka.connect.runtime.WorkerTask:179) org.apache.kafka.connect.errors.ConnectException: no viable alternative at input 'GRANT SELECT (...fields list here...' at io.debezium.connector.mysql.AbstractReader.wrap(AbstractReader.java:230) at io.debezium.connector.mysql.AbstractReader.failed(AbstractReader.java:207) at io.debezium.connector.mysql.BinlogReader.handleEvent(BinlogReader.java:536) at com.github.shyiko.mysql.binlog.BinaryLogClient.notifyEventListeners(BinaryLogClient.java:1095) at com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:943) at com.github.shyiko.mysql.binlog.BinaryLogClient.connect(BinaryLogClient.java:580) at com.github.shyiko.mysql.binlog.BinaryLogClient$7.run(BinaryLogClient.java:825) at java.base/java.lang.Thread.run(Thread.java:834) Caused by: io.debezium.text.ParsingException: no viable alternative at input 'GRANT SELECT (...fields list here...' at io.debezium.antlr.ParsingErrorListener.syntaxError(ParsingErrorListener.java:40) at org.antlr.v4.runtime.ProxyErrorListener.syntaxError(ProxyErrorListener.java:41) at org.antlr.v4.runtime.Parser.notifyErrorListeners(Parser.java:544) at org.antlr.v4.runtime.DefaultErrorStrategy.reportNoViableAlternative(DefaultErrorStrategy.java:310) at org.antlr.v4.runtime.DefaultErrorStrategy.reportError(DefaultErrorStrategy.java:136) at io.debezium.ddl.parser.mysql.generated.MySqlParser.sqlStatements(MySqlParser.java:1094) at io.debezium.ddl.parser.mysql.generated.MySqlParser.root(MySqlParser.java:828) at io.debezium.connector.mysql.antlr.MySqlAntlrDdlParser.parseTree(MySqlAntlrDdlParser.java:68) at io.debezium.connector.mysql.antlr.MySqlAntlrDdlParser.parseTree(MySqlAntlrDdlParser.java:41) at io.debezium.antlr.AntlrDdlParser.parse(AntlrDdlParser.java:80) at io.debezium.connector.mysql.MySqlSchema.applyDdl(MySqlSchema.java:306) at io.debezium.connector.mysql.BinlogReader.handleQueryEvent(BinlogReader.java:729) at io.debezium.connector.mysql.BinlogReader.handleEvent(BinlogReader.java:519) ... 5 more Caused by: org.antlr.v4.runtime.NoViableAltException at org.antlr.v4.runtime.atn.ParserATNSimulator.noViableAlt(ParserATNSimulator.java:2026) at org.antlr.v4.runtime.atn.ParserATNSimulator.execATN(ParserATNSimulator.java:467) at org.antlr.v4.runtime.atn.ParserATNSimulator.adaptivePredict(ParserATNSimulator.java:393) at io.debezium.ddl.parser.mysql.generated.MySqlParser.sqlStatements(MySqlParser.java:902) ... 12 more [2020-05-28 12:07:01,918] ERROR WorkerSourceTask{id=mysql_connector_google_stage-0} Task is being killed and will not recover until manually restarted (org.apache.kafka.connect.runtime.WorkerTask:180)
    Matt J
    @emm-jay
    this may be old news, but based on this https://github.com/debezium/debezium/blob/e4a1e5c1ada9cf6853b26981d2d175ff5d740a0d/debezium-core/src/main/java/io/debezium/config/CommonConnectorConfig.java#L162 it looks like the default poll.interval.ms is 500, however the documentation on https://debezium.io/documentation/reference/1.1/connectors/mongodb.html#mongodb-connector-properties says 1000ms. Am I looking in the right spot or is the documentation actually wrong?
    4 replies
    Jiri Pechanec
    @jpechane
    @aryzhenko Hi, this looks like a deficinecy in the DDL parser
    @aryzhenko Could you please find out which DDL is it and run it against latest Debezium 1.2 and if it is still the problem then reprot it in Jira?
    Quarantined-soul-alochna
    @alochna1_twitter

    Hello , i'm trying to set up AWS RDS Postgresql CDC to AWS MSK and hosting connect server on EC2 instance but while creating the connector i get this error that entry is missing in pg_hba.conf file. FATAL: no pg_hba.conf entry for replication connection from host \"172.z.y.x\", user \"postgres\" also looks like AWS RDS doesn't allow that property to be changed according to this documentation. https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Appendix.PostgreSQL.CommonDBATasks.html. Any pointers ? or is AWS RDS or AURORA postgres a no go place ? thank you for answers

    fixed it by enabling logical replication=1 and reboot AWS RDS. Also while creating connector pass on "plugin.name": "pgoutput"

    dctc42
    @dctc42
    Hi all, using debezium + kafka (+ a few custom workers to perform data transformations and enrichment) to replicate a couple of tables between two Postgres DBs. I'm concerned schema changes and code bugs could introduce data drift between the two databases. We feel we have decent telemetry and logging. Is this enough? What other things are recommended practices for detecting and fixing data drift? Thanks in advance for any pointers
    Vincent-Zeng
    @Vincent-Zeng

    @Naros

    @aryzhenko You can try setting snapshot.new.tables to parallel which means newly added whitelist tables when the connector is restarted will be snapshotted while existing ones will continue to be streamed.

    I've tried and it works. But after add new table to whitelist and restart connect making a snapshot goes really slof. For example: Initial tables snapshot was taken with ~50k records per second rate (I've got 600mil records in ~ 3 hours). And After add to whitelist+restart new table snapshot prepared with 500 rec/sec (INFO Step 8: - 21220000 of 21911205 rows scanned from table 'TBL_NAME' after 15:30:35.572 (io.debezium.connector.mysql.SnapshotReader:606)).
    Is there any known issues about it?

    Hi all. I encountered the same problem, any solution now?

    Jiri Pechanec
    @jpechane
    @Vincent-Zeng Hi, I am afraid it is price paid for processing both existing binlog and executing the snapshot. It might make more sense to you to start a new connector just to execute the snapshot and then resume the streamin
    6 replies
    gyh2020
    @gyh2020
    @jpechane Thanks very much for your reply
    hanoisteve
    @hanoisteve
    I just configured Debezium outbox router. I just see the outbox table has its own topic
    Is that correct or should other topics be formed as well?
    1 reply
    hanoisteve
    @hanoisteve
    How do I verify that its working? what should I see?
    Mike Kamornikov
    @mikekamornikov
    Hi, I wonder if it's reliable to depend on payload.source.snapshot to understand if snapshot is done? I'm looking at the last 2 records in my topic and see snapshot: true and snapshot: last. The thing is i want to make snapshot on database clone and only after that run binlog stage on original database. And to make transparent stitching I need to be sure snapshot is really done.
    Thanks
    Mike Kamornikov
    @mikekamornikov

    hmm, 1.1 mysql docs say:

     {
                "type": "boolean",
                "optional": true,
                "default": false,
                "field": "snapshot"
    },

    while in my case it is:

    {
                "type": "string",
                "optional": true,
                "name": "io.debezium.data.Enum",
                "version": 1,
                "parameters": {
                    "allowed": "true,last,false"
                },
                "default": "false",
                "field": "snapshot"
    },
    4 replies
    Mike Kamornikov
    @mikekamornikov
    One more thing. I make initial snapshot and get op: c. While documentation states r = read (initial snapshot only).
    Kushal Reddy
    @kushalreddy_gitlab
    @jpechane How to run ad-hoc JDBC queries using mysql connector ? We need this feature for data loss recovery . Currently we are facing data loss issue on production .
    2 replies
    hanoisteve
    @hanoisteve
    @kushalreddy_gitlab Could you use ksqldb ?
    hanoisteve
    @hanoisteve
    I have deployed Debezium outbox router. And have table. I see the updates going into outbox_event database
    How do I confirm the publishing of Kafka events? the table changes seem to be streamed like all the other tables but I am not seeing separate topics being written to or created for the aggregates.
    Also I would like to know the meaning of type, aggregateId, and aggregateType?
    Kushal Reddy
    @kushalreddy_gitlab
    @hanoisteve We are not having ksqldb on production
    hanoisteve
    @hanoisteve
    Its fully open source. I am streaming all of my topics through KSQLdb and can see and confirm everything with pull and push queries.
    Also does Debezium publish any log or event to confirm that the outbox router was successfully created etc?
    Kushal Reddy
    @kushalreddy_gitlab
    As you know every client is having some restrictions
    hanoisteve
    @hanoisteve
    Yep its relatively new too.
    So what I need to know about more is the Outbox Router. And how to confirm publish messages on a Kafka topic with just the aggregateId, type, aggregateType and payload. I am not seeing this topic created or published too.
    I feel I have configured it correctly but need to know how to troubleshoot when it does not publish events as expected.
    Instead what I see is just a topic for outbox_events table only.
    hanoisteve
    @hanoisteve
    CREATE SOURCE CONNECTOR my_outbox_connector WITH (
    'connector.class' = 'io.debezium.connector.mysql.MySqlConnector',
    'database.hostname' = '10.10.3.92’,
    'database.port' = '3306',
    'database.user' = 'root',
    'database.password' = ‘pwd12354’,
    'database.allowPublicKeyRetrieval' = 'true',
    'database.server.id' = '223344',
    'database.server.name' = ‘myDatabase’,
    'transforms' = 'outbox',
    'transforms.outbox.type' = 'io.debezium.transforms.outbox.EventRouter',
    'table.field.event.key' = 'aggregate_id',
    'table.field.event.payload.id' = 'aggregate_id',
    'database.history.kafka.bootstrap.servers' = 'localhost:9092',
    'database.history.kafka.topic' = ‘myDatase-history',
    'include.schema.changes' = 'false'
    );
    2 replies
    Screen Shot 2020-05-29 at 3.22.05 PM.png
    So why no topics being written to except the topic for outbox_events table changes? And for only see two records in topic wheras the table has 9 records in it.
    Kushal Reddy
    @kushalreddy_gitlab
    @jpechane is there any issue with Mysql debezium connector & Percona tool?
    Do you seen any issue with Debezium connector & Percona , If Percona tool is taking long time like 2 hrs or 6 hrs will face any issue with DBZ connector?
    9 replies
    @jpechane
    Jiamin WANG
    @Jiamin-WANG
    hi do you have any idea to clean the error message in tasks->trace? I ve tried to restart the task and the connect and clean the log but the error are still there
    8 replies
    Arne Huang
    @ArneHuang_twitter

    Using debezium engine, is there way to access engine internals from the processor, i.e. in handleBatch()?

    Use case: Instead of sending the DDL statement as string when schema changes I want to send the Schema object (maybe as json) since it uses debezium parser already, so I don't need to reparse the statement on the other side

    payasgupta267
    @payasgupta267
    Hi, Is there any relevant document on debezium parallelism model ?. I want to know more about ways/configurations to increase connector throughput. Can someone please help me on this ?
    saurabh8774
    @saurabh8774

    Hello, I am using S3 sink connector and want to have a custom bucket name
    topic-name : mlp-emp
    bucketname: emp

    "transforms":"dropPrefix",      
    "transforms.dropPrefix.type":"org.apache.kafka.connect.transforms.RegexRouter",  
    "transforms.dropPrefix.regex":"mlp-(.*)",  
    "transforms.dropPrefix.replacement":"$1"

    Tried using regex but it is failing with Nullpointerexception. Any ideas ?