Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
  • 09:50
    jpechane synchronize #1979
  • 09:26
    jpechane synchronize #1979
  • Nov 26 14:25
    jpechane converted_to_draft #1979
  • Nov 26 14:24
    jpechane opened #1979
  • Nov 26 10:25
    jpechane edited #1978
  • Nov 26 09:14
    jpechane synchronize #1978
  • Nov 26 08:52
    jpechane opened #1978
  • Nov 25 14:11
    gvaquez-ubi opened #1977
  • Nov 25 13:24
    jpechane opened #1976
  • Nov 25 12:39
    jpechane synchronize #1975
  • Nov 25 09:51
    jpechane synchronize #1975
  • Nov 25 09:11
    jpechane opened #1975
  • Nov 25 07:15
    jpechane synchronize #1974
  • Nov 25 07:07
    jpechane opened #1974
  • Nov 24 21:21
    kppullin synchronize #1973
  • Nov 24 21:15
    kppullin opened #1973
  • Nov 24 17:08
    roldanbob synchronize #1971
  • Nov 24 10:08
    gunnarmorling closed #1972
  • Nov 24 07:38
    ani-sha synchronize #1972
  • Nov 23 19:49
    roldanbob synchronize #1971
正在加载
@a6D1l1jukPRgZ8x_twitter
image.png
hello,I have some problems but i do not know how to solve them , I use debezium to synchronize Oracle data.when i create a table which like ,and then my connector hit this error:
Gunnar Morling
@gunnarmorling
hey @rk3rn3r ; are you back from PTO?
Mthulisi Nxumalo
@mnxumalo
Hey, I have an issue with MSSQL connector:
2020-11-24 08:26:47,076 WARN || Mapper for type 'uniqueidentifier' not found. [io.debezium.connector.sqlserver.SqlServerDefaultValueConverter]
connect_1 | 2020-11-24 08:26:47,643 WARN || Cannot parse column default value '(getdate())' to type 'datetime'. [io.debezium.connector.sqlserver.SqlServerDefaultValueConverter]
connect_1 | com.microsoft.sqlserver.jdbc.SQLServerException: Error converting string value 'etdate(' into data type datetime using culture ''.
connect_1 | at com.microsoft.sqlserver.jdbc.SQLServerException.makeFromDatabaseError(SQLServerException.java:262)
connect_1 | at com.microsoft.sqlserver.jdbc.SQLServerResultSet$FetchBuffer.nextRow(SQLServerResultSet.java:5427)
connect_1 | at com.microsoft.sqlserver.jdbc.SQLServerResultSet.fetchBufferNext(SQLServerResultSet.java:1758)
connect_1 | at com.microsoft.sqlserver.jdbc.SQLServerResultSet.next(SQLServerResultSet.java:1016)
connect_1 | at io.debezium.jdbc.JdbcConnection.querySingleValue(JdbcConnection.java:1299)
connect_1 | at io.debezium.connector.sqlserver.SqlServerDefaultValueConverter.lambda$createDefaultValueMappers$16(SqlServerDefaultValueConverter.java:135)
connect_1 | at io.debezium.connector.sqlserver.SqlServerDefaultValueConverter.parseDefaultValue(SqlServerDefaultValueConverter.java:82)
connect_1 | at io.debezium.connector.sqlserver.SqlServerConnection.getDefaultValue(SqlServerConnection.java:512)
connect_1 | at io.debezium.jdbc.JdbcConnection.readTableColumn(JdbcConnection.java:1181)
connect_1 | at io.debezium.jdbc.JdbcConnection.readSchema(JdbcConnection.java:1126)
connect_1 | at io.debezium.connector.sqlserver.SqlServerSnapshotChangeEventSource.readTableStructure(SqlServerSnapshotChangeEventSource.java:183)
connect_1 | at io.debezium.relational.RelationalSnapshotChangeEventSource.doExecute(RelationalSnapshotChangeEventSource.java:122)
connect_1 | at io.debezium.pipeline.source.AbstractSnapshotChangeEventSource.execute(AbstractSnapshotChangeEventSource.java:63)
connect_1 | at io.debezium.pipeline.ChangeEventSourceCoordinator.lambda$start$0(ChangeEventSourceCoordinator.java:105)
connect_1 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
connect_1 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
connect_1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
connect_1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
connect_1 | at java.base/java.lang.Thread.run(Thread.java:834)
Jiri Pechanec
@jpechane
@mnxumalo Hi, this is just a warning
Mthulisi Nxumalo
@mnxumalo
@jpechane Yes, but the topic doesn't get created in Kafka. How do I map sql server default values and the unique identifier columns. I see the Unique ID columns are excluded in the code for mappings.
Jiri Pechanec
@jpechane
@mnxumalo The daefualt value mappings is now supported only for contant values, not for functions or expressions
Mthulisi Nxumalo
@mnxumalo
@jpechane there is an exception:
com.microsoft.sqlserver.jdbc.SQLServerException: Error converting string value 'etdate(' into data type datetime using culture ''.
connect_1 | at com.microsoft.sqlserver.jdbc.SQLServerException.makeFromDatabaseError(SQLServerException.java:262)
connect_1 | at com.microsoft.sqlserver.jdbc.SQLServerResultSet$FetchBuffer.nextRow(SQLServerResultSet.java:5427)
connect_1 | at com.microsoft.sqlserver.jdbc.SQLServerResultSet.fetchBufferNext(SQLServerResultSet.java:1758)
connect_1 | at com.microsoft.sqlserver.jdbc.SQLServerResultSet.next(SQLServerResultSet.java:1016)
connect_1 | at io.debezium.jdbc.JdbcConnection.querySingleValue(JdbcConnection.java:1299)
connect_1 | at io.debezium.connector.sqlserver.SqlServerDefaultValueConverter.lambda$createDefaultValueMappers$16(SqlServerDefaultValueConverter.java:135)
connect_1 | at io.debezium.connector.sqlserver.SqlServerDefaultValueConverter.parseDefaultValue(SqlServerDefaultValueConverter.java:82)
connect_1 | at io.debezium.connector.sqlserver.SqlServerConnection.getDefaultValue(SqlServerConnection.java:512)
connect_1 | at io.debezium.jdbc.JdbcConnection.readTableColumn(JdbcConnection.java:1181)
connect_1 | at io.debezium.jdbc.JdbcConnection.readSchema(JdbcConnection.java:1126)
connect_1 | at io.debezium.connector.sqlserver.SqlServerSnapshotChangeEventSource.readTableStructure(SqlServerSnapshotChangeEventSource.java:183)
connect_1 | at io.debezium.relational.RelationalSnapshotChangeEventSource.doExecute(RelationalSnapshotChangeEventSource.java:122)
connect_1 | at io.debezium.pipeline.source.AbstractSnapshotChangeEventSource.execute(AbstractSnapshotChangeEventSource.java:63)
connect_1 | at io.debezium.pipeline.ChangeEventSourceCoordinator.lambda$start$0(ChangeEventSourceCoordinator.java:105)
connect_1 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
connect_1 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
connect_1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
connect_1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
connect_1 | at java.base/java.lang.Thread.run(Thread.java:834)
@jpechane how do I map the unique identifier data type?
@jpechane is there a way to ignore the getdate() function default value?
Jiri Pechanec
@jpechane
@mnxumalo For the latter please see https://issues.redhat.com/browse/DBZ-2642
@mnxumalo For the former, could you share the DDL we are tlkaing about?
Mthulisi Nxumalo
@mnxumalo
@jpechane
@jpechane :
CREATE TABLE [dbo].[Conversation](
[ID] [uniqueidentifier] NOT NULL,
[UserID] [uniqueidentifier] NOT NULL,
[ConversationID] [uniqueidentifier] NOT NULL,
[DateTime] [datetime] NOT NULL,
[FailedToAccept] [bit] NOT NULL,
[GroupChatID] [uniqueidentifier] NULL,
[GroupChatJoined] [datetime] NULL,
[GroupChatIsAdmin] [bit] NULL,
[GroupChatSkillID] [uniqueidentifier] NULL,
[GroupChatIsHandover] [bit] NULL,
[ConversationSessionID] [uniqueidentifier] NULL
@jpechane can't I just map a unique identifier to a string?
Jiri Pechanec
@jpechane
@mnxumalo It seems we don't support uuid for SQL Server yet. Could you please log a feature request in Jira?
Mthulisi Nxumalo
@mnxumalo
OK
Gunnar Morling
@gunnarmorling
@jpechane we can continue the grooming session
yukon71
@yukon71
Hi guys, how can I fix the problem on AWS RDS PostgreSQL connection problem?
connect_1 | Dec 19, 2017 12:30:45 AM org.postgresql.Driver connect
connect_1 | SEVERE: Connection error:
connect_1 | org.postgresql.util.PSQLException: FATAL: no pg_hba.conf entry for replication connection from host "10.73.X.X", user "**", SSL on.
i read the previous problem, now i can confirm i am using master host, username and password. but still meet this issue, any hint or suggestion i can explore?
Adishesh Kishore
@drwho1990

hi, I seem to be getting a very persistent issue while I try to use debezium with a mysql db. link to db -> https://relational.fit.cvut.cz/dataset/AdventureWorks

org.apache.kafka.connect.errors.ConnectException: Invalid SchemaBuilder call: default has already been set.
at io.debezium.connector.mysql.AbstractReader.wrap(AbstractReader.java:230)
at io.debezium.connector.mysql.AbstractReader.failed(AbstractReader.java:207)
at io.debezium.connector.mysql.SnapshotReader.execute(SnapshotReader.java:833)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.kafka.connect.errors.SchemaBuilderException: Invalid SchemaBuilder call: default has already been set.
at org.apache.kafka.connect.data.SchemaBuilder.checkCanSet(SchemaBuilder.java:437)
at org.apache.kafka.connect.data.SchemaBuilder.defaultValue(SchemaBuilder.java:126)
at io.debezium.relational.TableSchemaBuilder.addField(TableSchemaBuilder.java:380)
at io.debezium.relational.TableSchemaBuilder.lambda$create$2(TableSchemaBuilder.java:119)
at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384)
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485)
at io.debezium.relational.TableSchemaBuilder.create(TableSchemaBuilder.java:117)
at io.debezium.relational.RelationalDatabaseSchema.buildAndRegisterSchema(RelationalDatabaseSchema.java:130)
at io.debezium.connector.mysql.MySqlSchema.lambda$applyDdl$5(MySqlSchema.java:392)
at java.lang.Iterable.forEach(Iterable.java:75)
at io.debezium.connector.mysql.MySqlSchema.applyDdl(MySqlSchema.java:386)
at io.debezium.connector.mysql.SnapshotReader.lambda$readTableSchema$15(SnapshotReader.java:866)
at io.debezium.jdbc.JdbcConnection.query(JdbcConnection.java:537)
at io.debezium.jdbc.JdbcConnection.query(JdbcConnection.java:478)
at io.debezium.connector.mysql.SnapshotReader.readTableSchema(SnapshotReader.java:864)
at io.debezium.connector.mysql.SnapshotReader.execute(SnapshotReader.java:526)
... 3 more

some more logs for help,

[2020-11-25 20:21:05,097] INFO CREATE TABLE AddressType (
AddressTypeID int NOT NULL AUTO_INCREMENT COMMENT 'Primary key for AddressType records.',
Name varchar(100) NOT NULL COMMENT 'Address type description. For example, Billing, Home, or Shipping.',
rowguid varchar(64) NOT NULL COMMENT 'ROWGUIDCOL number uniquely identifying the record. Used to support a merge replication sample.',
ModifiedDate timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT 'Date and time the record was last updated.',
PRIMARY KEY (AddressTypeID),
UNIQUE KEY AK_AddressType_rowguid (rowguid),
UNIQUE KEY AK_AddressType_Name (Name),
KEY GUID_AddressType (rowguid)
) ENGINE=InnoDB AUTO_INCREMENT=7 DEFAULT CHARSET=utf8 COMMENT='Types of addresses stored in the Address table. ' (io.debezium.connector.mysql.SnapshotReader:1049)
[2020-11-25 20:21:05,098] WARN The Kafka Connect schema name 'source-mysql_111343282.adventureWorks.AddressType.Value' is not a valid Avro schema name, so replacing with 'source_mysql_111343282.adventureWorks.AddressType.Value' (io.debezium.connector.mysql.MySqlSchema:172)
[2020-11-25 20:21:05,098] WARN The Kafka Connect schema name 'source-mysql_111343282.adventureWorks.AddressType.Key' is not a valid Avro schema name, so replacing with 'source_mysql_111343282.adventureWorks.AddressType.Key' (io.debezium.connector.mysql.MySqlSchema:172)
[2020-11-25 20:21:05,099] INFO Step 7: rolling back transaction after abort (io.debezium.connector.mysql.SnapshotReader:741)
[2020-11-25 20:21:05,100] ERROR Failed due to error: Aborting snapshot due to error when last running 'SHOW CREATE TABLE adventureWorks.AddressType': Invalid SchemaBuilder call: default has already been set. (io.debezium.connector.mysql.SnapshotReader:208)

how do I fix this?
Jiri Pechanec
@jpechane
@drwho1990 Hi, which Debezium version?
H. İbrahim Güngör
@igungor
Hi everyone,
Does Debezium read from MySQL in batches, or is it a stream? If in batches, are there any Debezium configuration that changes binlog batch size read from MySQL? We have a setup where there are multiple hops required to access MySQL, which creates additional lag to Debezium (MillisecondsBehindSource). Any recommendations to reduce the lag, other than changing the current infrastructure of course :).
Gunnar Morling
@gunnarmorling
gm @jpechane
can we do grooming a bit later? say 12?
Jiri Pechanec
@jpechane
@gunnarmorling Hi
Yes, not a problem
Gunnar Morling
@gunnarmorling
i just did a session for JokerConf (75 min!) and need to recover a bit ;)
Jiri Pechanec
@jpechane
@gunnarmorling :-)
Avi Mualem
@AviMualem
hey, hopefully im addressing the right place for my question...
ive been using debezium for a while with a lot of success.
in the system im working on, we are basing our messaging payload structure upon protobuf.
The message which is being produced by debezium (mysql connector) is pure JSON, and i wanted to know if there is any option to make debezium produce a a protbuf format, or maybe even if there is any official proto message which is compatible with the produced JSON so my consumers will be able to desearlize the JSON content to protobuf message type.
Jiri Pechanec
@jpechane
@AviMualem Hi, please take a look at apicurio https://debezium.io/documentation/faq/#avro-converter it supports protobug converter too
Gunnar Morling
@gunnarmorling
woohoo, the new website is live: https://groups.google.com/g/debezium/c/Q-OJmSyn-P4
Gunnar Morling
@gunnarmorling
hey @ani-sha; could you point me to the code/configuration for the cross-repo build?
i.e. the config where a change in debezium triggers a build of debezium-incubator for instance
Jiri Pechanec
@jpechane
So it is not real cross-repo
Gunnar Morling
@gunnarmorling
ok, but still, that's what i'm looking for :)
thx
Jiri Pechanec
@jpechane
nps
Gunnar Morling
@gunnarmorling
we don't have a rebuild of the website yet after a change to path documentation, right, @jpechane?
Jiri Pechanec
@jpechane
@gunnarmorling Right
Gunnar Morling
@gunnarmorling
ok, I'll look into that then
Jiri Pechanec
@jpechane
@sha12br Hi, could you please post an example of what you get and highlight what you want to get?
Jiri Pechanec
@jpechane
@gunnarmorling I think there is missing URL rewrite/redirect. Old URLs like https://debezium.io/docs/faq/ are no longer working
@gunnarmorling It is probably docs vs documentation in URL path
Gunnar Morling
@gunnarmorling
@jpechane yes, that's already tracked, see debezium/debezium.github.io#600
issue is the trailing slash, try https://debezium.io/docs/faq
Jiri Pechanec
@jpechane
@gunnarmorling Right, thanks@