Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
    Natacha Crooks
    @nacrooks
    @jpechane: are you by any chance aware of limitations of using both avro.key.converter + schema registry, and the new trx.metadata topic in Postgres? Thank you!
    tegia
    @tegia
    @jpechane
    Hi, I tried connect version 1.1 but my grafana doesn't reflect every each the record is created at the table
    Can you tell me why ? and the solution
    Jeremy Finzel
    @jfinzel
    I am using this new feature "datatype.propagate.source.type":"numeric", but I don't see it having any affect. I am serializing numeric as string, but I would expect now to see the original scale/precision in schema registry. But this is all I see for a numeric(10, 2) field:
    {
                  "default": null,
                  "name": "numeric_reasonable",
                  "type": [
                    "null",
                    "string"
                  ]
                },
    Chuong Nguyen
    @ChuongTNguyen_twitter

    I'm trying to configure log4j for connect. My docker file is basically this without the ES part of it: https://github.com/debezium/debezium-examples/blob/master/unwrap-smt/debezium-jdbc-es/Dockerfile

    I've added my configs to the log4j.properties file and added COPY log4j.properties $KAFKA_HOME/config/log4j.properties to the end of my Dockerfile

    It seems that something is overwriting the $KAFKA_HOME/config/log4j.properties file because it still shows the default file when I look in the docker container.

    I can copy to $KAFKA_HOME/log4j.properties without issues but that's not the correct folder.

    Not sure what's going on here. Confused about the order of operations. Could someone advise? Thank you

    hydrogen
    @duycuong87vn
    @tegia , same problem with me,
    My tables have been scanned, but there was no reflection when I tried to create new record,
    @jpechane , my config as below,
    {
      "name": "inventory-connector",
      "config": {
        "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
        "tasks.max": "1",
        "database.hostname": "xxx",
        "database.port": "xxx",
        "database.user": "xxx",
        "database.password": "xxx",
        "database.server.id": "184054",
        "database.dbname": "xxx",
        "database.server.name": "dbserver1",
        "database.whitelist": "xxx",
        "database.history.kafka.bootstrap.servers": "kafka:9092",
        "database.history.kafka.topic": "schema-changes.inventory",
        "transforms": "route",
        "transforms.route.type": "org.apache.kafka.connect.transforms.RegexRouter",
        "transforms.route.regex": "([^.]+)\\.([^.]+)\\.([^.]+)",
        "transforms.route.replacement": "$3",
        "plugin.name": "pgoutput"
      }
    }
    Marwan Ghalib
    @marwanghalib

    Hello everyone,

    I have transaction metadata enabled, but I get a lot of transaction messages (BEGIN and END) where the event_count is 0.

    Any idea why that might be the case, or any way I can filter out transaction metadata with event_count = 0?

    '''
    {
    "schema": {
    "type": "struct",
    "fields": [{
    "type": "string",
    "optional": false,
    "field": "status"
    }, {
    "type": "string",
    "optional": false,
    "field": "id"
    }, {
    "type": "int64",
    "optional": true,
    "field": "event_count"
    }, {
    "type": "array",
    "items": {
    "type": "struct",
    "fields": [{
    "type": "string",
    "optional": false,
    "field": "data_collection"
    }, {
    "type": "int64",
    "optional": false,
    "field": "event_count"
    }
    ],
    "optional": false
    },
    "optional": true,
    "field": "data_collections"
    }
    ],
    "optional": false,
    "name": "io.debezium.connector.common.TransactionMetadataValue"
    },
    "payload": {
    "status": "BEGIN",
    "id": "476501",
    "event_count": null,
    "data_collections": null
    }
    }
    {
    "schema": {
    "type": "struct",
    "fields": [{
    "type": "string",
    "optional": false,
    "field": "status"
    }, {
    "type": "string",
    "optional": false,
    "field": "id"
    }, {
    "type": "int64",
    "optional": true,
    "field": "event_count"
    }, {
    "type": "array",
    "items": {
    "type": "struct",
    "fields": [{
    "type": "string",
    "optional": false,
    "field": "data_collection"
    }, {
    "type": "int64",
    "optional": false,
    "field": "event_count"
    }
    ],
    "optional": false
    },
    "optional": true,
    "field": "data_collections"
    }
    ],
    "optional": false,
    "name": "io.debezium.connector.common.TransactionMetadataValue"
    },
    "payload": {
    "status": "END",
    "id": "476501",
    "event_count": 0,
    "data_collections": []
    }
    }
    '''

    Thank you!

    Chris Cranford
    @Naros
    @marwanghalib Is this with PostgreSQL?
    Marwan Ghalib
    @marwanghalib
    @Naros yes, sir
    Chris Cranford
    @Naros
    @marwanghalib I wonder if this is because the logical replicatation stream is sending an event for a table that isn't monitored by the connector but because BEGIN/COMMIT messages aren't tied to a "table" per-se, you're seeing this.
    @marwanghalib e.g. BEGIN, INSERT (non-monitored table), COMMIT and the connector emits nothing more than BEGIN, COMMIT
    @marwanghalib What plugin are you using?
    Marwan Ghalib
    @marwanghalib
    wal2json
    Chris Cranford
    @Naros
    Ah; ok. If you had been on pgoutput, I would have been curious if setting the publication to only the be for the monitored list of tables if that would have helped but wal2json has no such feature.
    Marwan Ghalib
    @marwanghalib
    Does decoredbufs have such feature?
    Chris Cranford
    @Naros
    No, publications are a native implementation detail of PG10+ using pgoutput logical replication.
    Its a PG filtering mechanism much like the whitelist features our connectors provide but at the PG db level for the replication stream itself.
    Can you log a jira issue for this please? I'm don't have the code in-front of me atm, but trying to minimize this in the topic itself might be tricky for us depending on how we are calculating some of this detail.
    In the meantime, you could introduce a custom SMT that checks if the message is destined for the transaction metadata topic and see if it has event_count = 0 and if so, return null to avoid emitting the event.
    If none of those conditions are met, just return the original message untouched so it passes on down the pipeline.
    Marwan Ghalib
    @marwanghalib
    @Naros I will do that. Thank you. Any resource I can use as a reference to help me introduce this custom SMT?
    Chris Cranford
    @Naros
    Another option might be to use a Kstreams app where you basically read the metadata topic, filter it, and route the filtered stream to a new topic.
    That way you have both the non-filtered & filtered stream in Kafka for consumption.
    Marwan Ghalib
    @marwanghalib
    Also, @Naros , while I have you here. Wanted to confirm what is the format of the schema.whitelist identifier? I know for table.whitelist it is 'schemaName.tableName'
    Natacha Crooks
    @nacrooks

    @Chris: is there any chance, while you're here, that you could help me with my own transaction metadata question? Specifically, am I doing something wrong when specifying the KEY_CONVERTER/VALUE_CONVERTER Avro converter, I get this connect kafka error:

    connect_1 | Caused by: org.apache.avro.AvroRuntimeException: Not a record: ["null, {"type":"record","name":"ConnectDefault","namespace":"io.confluent.connect.avro","fields":[{"name":"id","type":"string

    "},{"name":"total_order","type":"long"},{"name":"data_collection_order","type":"long"}]}]

    connect_1 | at org.apache.avro.Schema.getFields(Schema.java:220)

    connect_1 | at org.apache.avro.data.RecordBuilderBase.<init>(RecordBuilderBase.java:54)

    connect_1 | at org.apache.avro.generic.GenericRecordBuilder.<init>(GenericRecordBuilder.java:38)

    connect_1 | at io.confluent.connect.avro.AvroData.fromConnectData(AvroData.java:600)

    connect_1 | at io.confluent.connect.avro.AvroData.fromConnectData(AvroData.java:606)

    connect_1 | at io.confluent.connect.avro.AvroData.fromConnectData(AvroData.java:365)

    Chris Cranford
    @Naros
    @marwanghalib For schema.whitelist its simply just the schema name, e.g. schemaA,schemaB
    @nacrooks I'm not aware of there being any limitations with avro + schema registry in conjunction with the transaction metadata topic stream, but that's not to say there might not be a bug somewhere either.
    Natacha Crooks
    @nacrooks
    (I upgraded my message with the exact error)
    Chris Cranford
    @Naros
    @nacrooks Which connector are you using?
    Karthik
    @unarmedcivilian
    hello, i was wondering if its possible to verify that no messages were missed while reading from the WAL (postgresql).
    Natacha Crooks
    @nacrooks

    @Naros, these are the options that make it go wrong
    "key.converter": "io.confluent.connect.avro.AvroConverter",

    "value.converter": "io.confluent.connect.avro.AvroConverter",

    "key.converter.schema.registry.url": "http://schema-registry:8081",

    "value.converter.schema.registry.url": "http://schema-registry:8081"

    It works without these options
    Karthik
    @unarmedcivilian
    @unarmedcivilian what i mean is if its possible to verify by using LSN or any other metadata to ensure that all changes written to WAL have been processed.
    Chris Cranford
    @Naros
    @nacrooks What version of Debezium are you presently using, 1.1.0.Final?
    Natacha Crooks
    @nacrooks
    the docker image debezium/1.1
    Chris Cranford
    @Naros
    @nacrooks Give me a few to setup a test and see. I did verify that running the SourceRecord through the AvroConverter in our transaction metadata tests passes just fine, so I'm not entirely sure why there would be this incompatibility.
    Natacha Crooks
    @nacrooks
    @naros: Thank you so much! I am using FROM confluentinc/cp-schema-registry for my schema registry (this is the only non debezium image I'm using).
    Chris Cranford
    @Naros
    @nacrooks I can reproduce it, I'll log an issue for it and attach my docker files.
    Natacha Crooks
    @nacrooks
    Thank you @naros!
    Chris Cranford
    @Naros
    @nacrooks Opened https://issues.redhat.com/browse/DBZ-1915, we'll take a look at this asap.
    @nacrooks Feel free to update that issue with any other pertinent details you believe are valuable that I might have missed.
    Jeremy Finzel
    @jfinzel
    anyone able to get this datatype.propagate.source.type to work? I tried even with a brand new table - still no good
    oh........
    I think I see the issue
    Chris Cranford
    @Naros
    @jfinzel I believe the length/scale you're after should be in the parameters section of the struct
    Jeremy Finzel
    @jfinzel
    @Naros it is not showing up anywhere in the struct.
    I also thought I misconfigured it by only putting numeric. But I also tried now dbname.pg_catalog.numeric, and that still did not work.
    Chris Cranford
    @Naros
    @jfinzel Honestly we probably should have a test for this feature in every connector, it seems DBZ-1830 only contributed a test in the MySQL connector.
    That said, the regex used in that test appears to be .+\\.FLOAT as an example, perhaps you could try .+\\.NUMERIC?
    Jeremy Finzel
    @jfinzel
    oh sheesh - I requested it originally for the postgres one :)