Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
  • Nov 27 18:52
    gunnarmorling synchronize #2979
  • Nov 27 18:13
    gunnarmorling synchronize #2979
  • Nov 27 17:33
    gunnarmorling synchronize #2979
  • Nov 27 17:30
    gunnarmorling synchronize #2979
  • Nov 27 15:29
    jcechace synchronize #2915
  • Nov 26 19:14
    jcechace synchronize #2915
  • Nov 26 17:15
    gunnarmorling closed #2982
  • Nov 26 16:26
    gunnarmorling synchronize #2979
  • Nov 26 16:11
    jcechace synchronize #2915
  • Nov 26 14:40
    gunnarmorling opened #2982
  • Nov 26 13:40
    jpechane synchronize #2969
  • Nov 26 13:39
    jpechane synchronize #2969
  • Nov 26 13:29
    gunnarmorling closed #2980
  • Nov 26 12:52
    msillence opened #2981
  • Nov 26 12:46
    msillence closed #2976
  • Nov 26 11:21
    gunnarmorling synchronize #2980
  • Nov 26 08:58
    gunnarmorling edited #2980
  • Nov 26 08:58
    gunnarmorling edited #2980
  • Nov 26 08:13
    gunnarmorling closed #2864
  • Nov 26 07:55
    gunnarmorling closed #2936
Chris Cranford
@Naros
It is, I'm mostly focused on the work related to event emission atm, those like CREATE PROCEDURE are things I'm going to circle back on and make sure those are filtered.
Gunnar Morling
@gunnarmorling
ok, cool
i'm a bit concerned about the huge, huge long tail of those grammar parts
Chris Cranford
@Naros
We did have a number of DDL come through that are flagged as INTERNAL DDL by LogMiner and those are now filtered.
I'm not sure how user created procedures & what-not fall into that category yet but will be tested too.
Gunnar Morling
@gunnarmorling
sweet
ha, yeah, so if we could easily limit it to table DDLs, that'd be awesome
Chris Cranford
@Naros
Ya I'm fairly certain there will be a bit of grammar extension needed unfortunately as it didn't seem to me that the Oracle DDL grammar is complete.
Gunnar Morling
@gunnarmorling
if not, well, no way around it then
Chris Cranford
@Naros
But I'm hoping that won't be too much effort in the long run.
Gunnar Morling
@gunnarmorling
ok, all good then if you got that on your roadmap, that's excellent
Chris Cranford
@Naros
But if we can restrict it at the query level up-front that helps us to only process what we need and avoids any grammar concerns too for DDL we don't really care about.
Gunnar Morling
@gunnarmorling
yeah, that'd be lovely
but perhaps not doable, we'll see
in any case, we should think about documenting the workaround in that situation
like skipping such event
anyways, step by step
Chris Cranford
@Naros
Yep, baby steps!
Gunnar Morling
@gunnarmorling
ok, dinner time
Chris Cranford
@Naros
Cheers
Gunnar Morling
@gunnarmorling
sent you an invite for tomorrw
tty then
Jiri Pechanec
@jpechane
@gunnarmorling @Naros Hi, this is because the schema change event is created using io.debezium.connector.mysql.SourceInfo.tableEvent(Set<TableId>). For rename table the MySQL DDL parser emits information about change of two tables - the old and new and this is the result. Exactly same behaviour was there before the rewrite. TBH I believe it make sense to have it this way as you are effectively touching twou tobles names. But as you've mentioned we should align on the same behviour
@Naros I am more concerned about the DDL recorded in the event. Is it really incomplete as you show in the example?
Chris Cranford
@Naros
Hi @jpechane, what do you mean incomplete?
Chris Cranford
@Naros
@jpechane If you're referring to ddl='_monitored_new' TO 'monitored', then yes that's what is emitted for RENAME TABLE monitored TO _monitored_old, _monitored_new TO monitored.

Here are the two actual records

SourceRecord{sourcePartition={server=migration}, sourceOffset={transaction_id=null, ts_sec=1617896412, file=mysql-bin.000003, pos=18470, server_id=112233}} ConnectRecord{topic='migration', kafkaPartition=0, key=Struct{databaseName=empty_1litbz2}, keySchema=Schema{io.debezium.connector.mysql.SchemaChangeKey:STRUCT}, value=Struct{source=Struct{version=1.5.0-SNAPSHOT,connector=mysql,name=migration,ts_ms=1617896412420,db=empty_1litbz2,table=monitored,_monitored_old,server_id=112233,file=mysql-bin.000003,pos=18301,row=0},databaseName=empty_1litbz2,ddl=`monitored` TO `_monitored_old`,tableChanges=[]}, valueSchema=Schema{io.debezium.connector.mysql.SchemaChangeValue:STRUCT}, timestamp=null, headers=ConnectHeaders(headers=)},

and

SourceRecord{sourcePartition={server=migration}, sourceOffset={transaction_id=null, ts_sec=1617896412, file=mysql-bin.000003, pos=18470, server_id=112233}} ConnectRecord{topic='migration', kafkaPartition=0, key=Struct{databaseName=empty_1litbz2}, keySchema=Schema{io.debezium.connector.mysql.SchemaChangeKey:STRUCT}, value=Struct{source=Struct{version=1.5.0-SNAPSHOT,connector=mysql,name=migration,ts_ms=1617896412420,db=empty_1litbz2,table=monitored,_monitored_old,server_id=112233,file=mysql-bin.000003,pos=18301,row=0},databaseName=empty_1litbz2,ddl=`_monitored_new` TO `monitored`,tableChanges=[Struct{type=ALTER,id="empty_1litbz2"."monitored",table=Struct{defaultCharsetName=latin1,primaryKeyColumnNames=[id],columns=[Struct{name=id,jdbcType=4,typeName=INT,typeExpression=INT,length=11,position=1,optional=false,autoIncremented=true,generated=true}, Struct{name=value2,jdbcType=4,typeName=INT,typeExpression=INT,length=11,position=2,optional=true,autoIncremented=false,generated=false}]}}]}, valueSchema=Schema{io.debezium.connector.mysql.SchemaChangeValue:STRUCT}, timestamp=null, headers=ConnectHeaders(headers=)},
Jiri Pechanec
@jpechane
@Naros Thanks, let me chec if this is intentional
Chris Cranford
@Naros
I should have a 1st draft ready for review on Monday for the Oracle DDL stuff btw.
Jiri Pechanec
@jpechane
@Naros Cool!
Gunnar Morling
@gunnarmorling
@jpechane hey; but so are you saying that comma-separated list is an expected value for source.table?
i don't think I agree with that
old and new table name are available from tableChanges
re ddl, showing a fragment of the original DDL statement definitely is odd
@jpechane you are you going to look at @morozov's reproducer, right?
Jiri Pechanec
@jpechane
@gunnarmorling Hi, well, that's your code, so I suppose you intended to do it exactly in that way :-)
@gunnarmorling Please check io.debezium.connector.mysql.SourceInfo.table()
@gunnarmorling And the comment specifically say this is intended
René Kerner
@rk3rn3r:matrix.org
[m]
LOL
Jiri Pechanec
@jpechane
@gunnarmorling Regarding the reproducer, yes. I am just now finishing the protobuf PR so all Anisha's work is completed
Gunnar Morling
@gunnarmorling
@jpechane ok, cool
re table(), wow, you're right
ok, in that light, let's do it like that for oracle, too then
@Naros ^^^
Gunnar Morling
@gunnarmorling
@jpechane might be late 2 min or so
Jiri Pechanec
@jpechane
@gunnarmorling nps
Gunnar Morling
@gunnarmorling
@jpechane ok, i'm in
Gunnar Morling
@gunnarmorling
@Naros could you reply here: https://issues.redhat.com/browse/DBZ-3155
Chris Cranford
@Naros
@gunnarmorling Ok, I'll attempt to manipulate Oracle's DDL parsing the sameway as MySQL then.
Gunnar Morling
@gunnarmorling
@Naros sounds good, thx!