by

Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
    若晨星
    @ruochenxing
    Hi,I use debezium at postgresql, and then I find some task disappeared for no reason in log file,but I find those task are still running where I type 'curl http://localhost:8083/connectors/myconnector/status',I don't know what happend for this,what can I do?Thanks!
    Jiri Pechanec
    @jpechane
    @pratyaksh131_twitter Hi, you either have applied change to the source table schema that schema registry is not able to handle in compatibility mode
    @pratyaksh131_twitter or there was an old schema stored in schem when you started and this is compatible with the one you are using now
    @dallariva93 Have you seen message in log Migrating schema to ...?
    Jiri Pechanec
    @jpechane
    @ashikaumanga Try setting min.row.count.to.stream.results to 0
    Ashika Umanga Umagilya
    @ashikaumanga
    @jpechane with default "min.row.count.to.stream.results=1000", it should still respect the "snapshot.fetch.size=5000" value and stream the data instead of retrieving everything ?
    PRATYAKSH SHARMA
    @pratyaksh131_twitter
    @jpechane looks like it was version issue. I was using Debezium version 0.10 which is not having event flattening support. Am I correct?
    Giovanni Dalla Riva
    @dallariva93
    @jpechane yes, after that message I delete the old capture table and at the next insert on the source table I get that error
    PRATYAKSH SHARMA
    @pratyaksh131_twitter
    @jpechane got the issue. It was related to compatibility mode only. Thank you for the help
    Chris Cranford
    @Naros
    @ashikaumanga The min.row.count.to.stream.results configuration controls how we interact with the MySQL Connector/J driver.
    If the value is 0, then a cursor will be created that allows iterating over the result-set in such a way that avoids loading the full table's contents into memory (this is done by the driver).
    If the value is > 0, then the default behavior used by the driver occurs.
    So as Jiri points out, for large result-sets you want to set min.row.count.to.stream.results=0 to keep the memory footprint reasonable.
    Chris Cranford
    @Naros
    @pratyaksh131_twitter 0.10 does support flattening, see https://debezium.io/documentation/reference/0.10/configuration/event-flattening.html
    The class names I believe changed and the old one was deprecated moving from 0.8 to 0.9 if my memory is correct.
    Ben Cohen
    @cohenbenjamin05
    has anyone used the exported snapshot.mode for postgres? The release notes (https://debezium.io/blog/2019/09/10/debezium-0-10-0-cr1-released/) suggest that no locks should be required during the snapshot mode, but in testing, I see that it is acquiring an access share during the snapshot phase. Am I doing something wrong here?
    Chris Cranford
    @Naros
    @cohenbenjamin05 Could you open an issue with all the pertinent details, including where/how you see the lock during snapshotting?
    Ben Cohen
    @cohenbenjamin05
    @Naros - will do, thanks
    Chris Cranford
    @Naros
    The goal was the tables would not be locked, allowing them to be altered and such while snapshot occurs since creating the replication slot can generate an exported snapshot reference we could use to produce the snapshot events from without locking.
    Ben Cohen
    @cohenbenjamin05
    @Naros Yeah, in our experience the old method of snapshotting was acquiring a stronger lock (I believe ACCESS EXCLUSIVE?). This still seems like a big improvement but was surprised due to the documentation. I will do a little more testing on my end and open an issue with the details. Thanks!
    xi gao
    @Kevin_gx_twitter
    Hi, I'm using MongoDB connector as the sink, how do I use the PK in original record as PK in MongoDB instead of always creating a new record?
    I have the same issue in Elasticsearch sink as well...
    ekremtoprak
    @ekremtoprak
    Hey guys, i was wondering what i need to configure in order to have multiple consumers?
    ekremtoprak
    @ekremtoprak
    how can we increase the number of partitions in our topic? We reroute all messages to a single topic named “all”.
    "transforms": "Reroute",
    "transforms.Reroute.type": "io.debezium.transforms.ByLogicalTableRouter",
    "transforms.Reroute.topic.regex": "deb_src.moto_sales.(.*)",
    "transforms.Reroute.topic.replacement": "deb_src.moto_sales.all"
    Jiri Pechanec
    @jpechane
    @ekremtoprak Hi, please use kafka-topics tool for topic repartitioning
    Lasantha Samarakoon
    @lasanthaS
    Hi guys, I am trying to connect to an Oracle 12c standalone server (No CDB/PDB) using the Debezium Oracle connector. Configured server as per the tutorial. Now when I run the client JAR I get the following stack trace. Appreciate if you can help on figuring out this.
    1763 [debezium-oracleconnector-orclserver1-change-event-source-coordinator] INFO io.debezium.relational.RelationalSnapshotChangeEventSource - Snapshot step 8 - Finalizing 
    [debezium-oracleconnector-orclserver1-change-event-source-coordinator] ERROR io.debezium.pipeline.ErrorHandler - Producer failure 
    java.lang.RuntimeException: oracle.streams.StreamsException: ORA-21560: argument last_position is null, invalid, or out of range 
        at io.debezium.connector.oracle.OracleStreamingChangeEventSource.execute(OracleStreamingChangeEventSource.java:75) 
        at io.debezium.pipeline.ChangeEventSourceCoordinator.lambda$start$0(ChangeEventSourceCoordinator.java:91) 
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) 
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) 
        at java.lang.Thread.run(Thread.java:748) 
    Caused by: oracle.streams.StreamsException: ORA-21560: argument last_position is null, invalid, or out of range 
        at oracle.streams.XStreamOut.XStreamOutAttachNative(Native Method) 
        at oracle.streams.XStreamOut.attachInternal(XStreamOut.java:373) 
        at oracle.streams.XStreamOut.attach(XStreamOut.java:343) 
        at io.debezium.connector.oracle.OracleStreamingChangeEventSource.execute(OracleStreamingChangeEventSource.java:63) 
        ... 6 more
    Lasantha Samarakoon
    @lasanthaS
    And also, is there a minimum Oracle version that supports by Debezium Oracle connector 0.10.0?
    Chris Cranford
    @Naros
    @lasanthaS Oracle 11g and 12 both should work, see https://github.com/debezium/debezium-incubator/blob/master/README.md for information specific for 11g support
    Lasantha Samarakoon
    @lasanthaS

    Hi @Naros, Thanks for the info.

    In my setup, I have Oracle 12cR1 (12.1.0.2) in a Standalone mode. So for that, the compliant instant client is 12.1 (ojdbc7/jdk8). When I run with that I got the following error just after the snapshot is done.

    Caused by: java.lang.NoSuchMethodError: oracle.streams.XStreamUtility.convertSCNToPosition(Loracle/sql/NUMBER;I)[B
        at io.debezium.connector.oracle.OracleStreamingChangeEventSource.convertScnToPosition(OracleStreamingChangeEventSource.java:131)

    When I change the instant client to version 12.2 or 19.3 (ojdbc8/jdk8) I am getting the following error instead.

    Caused by: oracle.streams.StreamsException: ORA-21560: argument last_position is null, invalid, or out of range
        at oracle.streams.XStreamOut.XStreamOutAttachNative(Native Method)

    As this is an ORA-* error, could this be due to an incompatible version? Any idea on what is wrong here?

    Chris Cranford
    @Naros
    @lasanthaS This came up before with another user on 2019/05/22; their solution was to switch to 12cR2.
    So perhaps there is some incompatibility between R1 & R2
    Lasantha Samarakoon
    @lasanthaS
    @Naros Yeah, seems like it is. BTW I couldn't find any reference for that issue on this feed. Could you please share if you have the reference? I will try to test this on R1 & R2 again to see if there is an incompatibility between these two. Thanks for the help.
    Lasantha Samarakoon
    @lasanthaS
    Thanks @Naros for the reference.
    Ben Cohen
    @cohenbenjamin05

    @Naros - Before I open up an issue, I’d like to confirm something with you. It doesn’t appear that the connector is explicitly acquiring locks. Rather, an AccessShareLock is being acquired as a result of the connector running a SELECT * FROM table query during the snapshot. Based on the documentation, I would have expected that this query would not be run given that the documentation says

    The fifth snapshot mode, exported, will perform a database snapshot based on the point in time when the replication slot was created. This mode is an excellent way to perform a snapshot in a lock-free way.
    Is this expected?

    Ashika Umanga Umagilya
    @ashikaumanga
    @Naros @jpechane We are using Mysql Connector to ingest ~20mil records.We write the messages back to HDFS using HDSF Sink. Initial snapshot works fine but then after awhile seems the CDC stopped ingesting new records.
    Our source record counts increased with like 20515736,20515744,20515986...etc But the HDFS count remained at 20515283. We confirmed that HDFS Sink works fine and there are no new data ingested (no new HDFS partitioned created). Any idea what might have gone wrong ?
    Jiri Pechanec
    @jpechane
    @ashikaumanga Hi, are the records stored in the Kafka broker or not?
    Ashika Umanga Umagilya
    @ashikaumanga
    @jpechane seems like after the initial load no new messages received to the topic
    Jiri Pechanec
    @jpechane
    @ashikaumanga Could you please try schema only snapshot to verify that streaming is/is not working? Is there any exception in the log?
    Ashika Umanga Umagilya
    @ashikaumanga
    @jpechane I flushed the Topic. Started listening to the topic using kafka-console-consumer. Finally created a new MySql Connector with 'schema_only' (with a new connector name)
    Noticed 15 new records in the source db. But no messages shown in the console-consumer.
    do i need to flush the Db History Topic as well ?
    Ashika Umanga Umagilya
    @ashikaumanga
    Then I created a new Connector with 'when_needed' and messages started flooding in the console-consumer..
    Ashika Umanga Umagilya
    @ashikaumanga
    we are using 1.0.0Beta2, Mysql version is 5.7.18
    Martin
    @mhaagens
    Hi! I was wondering if there's a way to force Debezium to re run the indexing of Postgres?
    Dawid Mazur
    @dwdmzr_twitter

    Hello everyone! I'm trying to deploy Debezium to do CDC on AWS RDS MySQL (5.7) server and today it stopped working with an exception: org.apache.kafka.connect.errors.ConnectException: Received DML 'DELETE FROM mysql.rds_sysinfo where name = 'innodb_txn_key'' for processing, binlog probably contains events generated with statement or mixed based replication format

    This is weird, because our DB is configured with correct binlog format:

    +--------------------------------------------+----------------------+
    | Variable_name                              | Value                |
    +--------------------------------------------+----------------------+
    | binlog_format                              | ROW                  |
    | binlog_row_image                           | FULL                 |
    +--------------------------------------------+----------------------+

    I'm searching around for a solution for some time now and didn't find anything useful yet. Did I do something wrong when configuring the connector, or should I create a ticket for the problem? Maybe the fact that we have replication in place could cause the DML to appear in the binlog?

    Jiri Pechanec
    @jpechane
    @dwdmzr_twitter Hi, please see https://issues.jboss.org/browse/DBZ-1492
    Dawid Mazur
    @dwdmzr_twitter
    Oh, so upgrading to 1.0 should fix the problem, thank you!
    Chris Cranford
    @Naros
    Hi @cohenbenjamin05 so after reviewing the exported snapshot functionality, the AccessShareLock is expected and here is why.
    The goal behind the exported snapshot feature was to avoid LOCK TABLE <table> IN ACCESS SHARE UPDATE EXCLUSIVE MODE (at the time this was the lock we used).