by

Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
Prakash
@prakashrajpoudel
how can i parse this ?
You'll notice we provide a link in the case of wkb to the format of those fields.
I tried few things to decode. using POSTGIs lib,
<dependency>
  <groupId>mil.nga</groupId>
  <artifactId>wkb</artifactId>
  <version>1.0.6</version>
</dependency>
but none of them is able to parse those
String wkb = "AQMAACDmEAAAAQAAAAoAAAAcwbik+bBRwF3UFOlRSkVAHsG4GxmxUcCRxz0ObEpFQCHBuGcYsVHAikxX941KRUAhwbjHErFRwOHhTRyoSkVAHsG4zO+wUcDdhq1Cj0pFQB3BuAfxsFHAhYqjnINKRUAewbgp7bBRwI6mgu12SkVAHMG4QvKwUcAU7JmeZkpFQBzBuELysFHAFubPYVhKRUAcwbik+bBRwF3UFOlRSkVA";
BinaryParser binaryParser = new BinaryParser();
Geometry geometry = binaryParser.parse(wkb.getBytes(StandardCharsets.US_ASCII));
Is there any test or example where I can check how to parse this
I am using java kstream to read message from kafka topic.
Prakash
@prakashrajpoudel
@Naros Thanks for link, I am looking into this document here https://mariadb.com/kb/en/well-known-binary-wkb-format/ Should not wkb starts with 0 or 1 ?
Robert B. Hanviriyapunt
@RobertHana
Hi! So I added the new postgres (and now mysql) option binary.handling.mode. Where do I "Add the new option to the Postgres connector docs"?
Shifu Xu
@shifuxu0301
Hi all, I had a quick question. For snapshotting process, by reading the debezium docs, will debezium only lock the database for structural changes? Ie, no matter how large the table it is. the locking will not matter, right?
We had a fairly large database, just want to confirm with that, so that we can make sure it will not locking the db for too long during snapshot. Thanks.
1 reply
Gunnar Morling
@gunnarmorling
hey @jpechane, good morning!
accepted your invite
but perhaps you could begin with 1.1 backporting already?
Jiri Pechanec
@jpechane
@gunnarmorling Hi, I might do it to check if the already planned issues merges cleanly or not but then I'd prefer to do it again together with the new identified issues. I always prefer to apply commits in the same order as they were created
Gunnar Morling
@gunnarmorling
ok
Kewei Shang
@keweishang
Hi @gunnarmorling it seems that mocking isn't used heavily in DBZ's tests, I guess it's a choice on purpose?
Kewei Shang
@keweishang
We're investigating on migrating to Vitess (http://vitess.io/), I was wondering if DBZ has considered supporting Vitess, probably via VStream (https://vitess.io/docs/concepts/vstream/), on the roadmap? Reading change-data event from VStream is a bit different than reading binlog event from shyiko/mysql-binlog-connector-java though.
5 replies
Mike Kamornikov
@mikekamornikov
@gunnarmorling I created https://issues.redhat.com/browse/DBZ-2128 for that "db history topic missing" error I mentioned ^^^.
1 reply
Robert B. Hanviriyapunt
@RobertHana
@gunnarmorling my PR for binary handling mode debezium/debezium#1516 is pretty much complete to what our needs are: postgres and mysql connectors utilizing the now-common binary.handling.mode option. Tests are now present in postgres and mysql ... since the option is NOT exposed in the m$sql or mongodb connectors. I just fixed formatting check issues and waiting for the build to pass, but the last build had an error with m$sql tests which i could not reproduce locally. If it happens again and i can't reproduce it, i will be coming back here to ask about it, ok?
1 reply
Gunnar Morling
@gunnarmorling
@keweishang re mocking, yes that's on purpose, we strongly prefer integration tests due to higher fidelity
1 reply
Robert B. Hanviriyapunt
@RobertHana
@gunnarmorling ok, NO m$sql test failure, but now i have a mysql test failure io.debezium.connector.mysql.MySqlConnectorTest.shouldReturnConfigurationDefinition() ...
I added binary.handling.mode to io.debezium.connector.mysql.MySqlConnectorConfig.ALL_FIELDS, was that wrong?
[ERROR]   MySqlConnectorTest.shouldReturnConfigurationDefinition:22->assertConfigDefIsValid:28->lambda$assertConfigDefIsValid$0:29 <['database.hostname', 'database.port', 'database.user', 'database.password', 'database.initial.statements', 'database.server.name', 'database.server.id', 'database.server.id.offset', 'database.ssl.mode', 'database.ssl.keystore', 'database.ssl.keystore.password', 'database.ssl.truststore', 'database.ssl.truststore.password', 'database.jdbc.driver', 'skipped.operations', 'database.history.kafka.bootstrap.servers', 'database.history.kafka.topic', 'database.history.kafka.recovery.attempts', 'database.history.kafka.recovery.poll.interval.ms', 'database.history', 'database.history.skip.unparseable.ddl', 'internal.database.history.ddl.filter', 'database.history.store.only.monitored.tables.ddl', 'include.schema.changes', 'include.query', 'table.ignore.builtin', 'database.whitelist', 'table.whitelist', 'column.blacklist', 'table.blacklist', 'database.blacklist', 'message.key.columns', 'column.mask.hash.([^.]+).with.salt.(.+)', 'column.mask.with.(d+).chars', 'column.truncate.to.(d+).chars', 'snapshot.select.statement.overrides', 'gtid.source.includes', 'gtid.source.excludes', 'gtid.source.filter.dml.events', 'gtid.new.channel.position', 'binlog.buffer.size', 'heartbeat.interval.ms', 'heartbeat.topics.prefix', 'event.deserialization.failure.handling.mode', 'event.processing.failure.handling.mode', 'inconsistent.schema.handling.mode', 'tombstones.on.delete', 'source.struct.version', 'connect.timeout.ms', 'connect.keep.alive', 'connect.keep.alive.interval.ms', 'max.queue.size', 'max.batch.size', 'poll.interval.ms', 'snapshot.mode', 'snapshot.locking.mode', 'snapshot.new.tables', 'time.precision.mode', 'decimal.handling.mode', 'bigint.unsigned.handling.mode', 'snapshot.delay.ms', 'snapshot.fetch.size', 'enable.time.adjuster']> does not contain element(s):<['binary.handling.mode']>
Gunnar Morling
@gunnarmorling
no, that was right; but you might have to adapt that assertion accordingly
Robert B. Hanviriyapunt
@RobertHana
@gunnarmorling Ah, i reproduced it locally, and fixed by adding it to io.debezium.connector.mysql.MySqlConnectorConfig.configDef()
Gunnar Morling
@gunnarmorling
k
Gunnar Morling
@gunnarmorling
Hey @Naros, one thing fyi, we had to move the 1.1.2 release to monday
meaning our quarkus version update will probably miss the platform update cutoff date
it's not ideal, but not the end of the world either
Chris Cranford
@Naros
@gunnarmorling I could prepare a Quarkus PR against 1.2.0.Beta2 if you'd like at least.
The other alternative is wait for Quarkus 1.5.1 most likely in a few weeks.
Robert B. Hanviriyapunt
@RobertHana
@gunnarmorling well, my PR (debezium/debezium#1516) didn't say that everything passed, but the details says everything passed!
Gunnar Morling
@gunnarmorling
@Naros yes, let's try that
it has the dev mode NPE fix, right?
Chris Cranford
@Naros
@gunnarmorling It does, we fixed that in 1.2.0.Alpha1
Gunnar Morling
@gunnarmorling
gotcha
Chris Cranford
@Naros
Alright I'll send them a PR based on 1.2 beta2 then
Gunnar Morling
@gunnarmorling
yeah, i remember some hesitence regarding non final releases, but give it a try, doesn't cost much
:thumbsup:
Robert B. Hanviriyapunt
@RobertHana
@gunnarmorling I have questions regarding testing of our new binary.handling.mode in our existing pipeline setup. how might we get an early build of it that gets pulled into our setup?
Robert B. Hanviriyapunt
@RobertHana
@gunnarmorling debezium/debezium#1516 PASSED! what next? how to get it into our Debezium setup? a early release Docker image? or make our own? how?
Gunnar Morling
@gunnarmorling
@RobertHana so i'll re-review the PR next week, and once it's merged, you can obtain from the maven snapshot repo
until then, you can of course just build your own branch locally
mvn clean package -pl debezium-connector-postgres -am -Passembly
this will give you the connector archive in the PG connector's target dir
re docker, see the docker images repo and the Dockerfile.local instructions
Gunnar Morling
@gunnarmorling
@jpechane hey; i forgot that tomorrow is a public holiday in Germany, so I'll be back on Tuesday; I'll leave it to you all whether you'd like to do the team meeting without me or move it to Tuesday; just let me know