Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
  • 07:07
    jpechane closed #2380
  • 05:39
    morozov ready_for_review #2382
  • 01:17
    morozov synchronize #2382
  • 00:35
    morozov edited #2382
  • 00:17
    morozov synchronize #2382
  • 00:03
    morozov opened #2382
  • May 13 23:44
    ccollingwood closed #2381
  • May 13 22:00
    ccollingwood opened #2381
  • May 13 17:24
    Naros synchronize #2371
  • May 13 17:16
    Naros synchronize #2347
  • May 13 17:14
    Naros opened #2380
  • May 13 17:02
    roldanbob closed #2379
  • May 13 16:49
    roldanbob opened #2379
  • May 13 15:08
    jpechane opened #226
  • May 13 14:48
    truman303 opened #2378
  • May 13 14:19
    Naros synchronize #2371
  • May 13 12:59
    novotnyJiri synchronize #2338
  • May 13 12:56
    novotnyJiri synchronize #2338
  • May 13 12:47
    Naros synchronize #2370
  • May 13 12:45
    Naros synchronize #2371
Gunnar Morling
@gunnarmorling
@avatar13 can you identify the problematic table?
avatar13
@avatar13
@gunnarmorling yes, but there is about 80 columns. java 8 not helped, i recompile debezium-mysql-connector with tokudb specific sql-tokens. tests passed, but in docker after launch on single table there is same error...
Jiri Pechanec
@jpechane
@avatar13 Could ou please share the DDL for the given table? Ideally using SHOW CREATE TABLE
avatar13
@avatar13
CREATE TABLE `conversions` (
  `id` bigint(20) unsigned NOT NULL,
  `is_test` tinyint(1) NOT NULL DEFAULT 0 COMMENT 'Транзакция является тестовой',
  `conversion_transaction_id` int(11) NOT NULL DEFAULT 0 COMMENT 'ID информации о транзакции',
  `transaction_id` varchar(32) NOT NULL DEFAULT '' COMMENT 'ID транзакции',
  `created` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00' COMMENT 'дата создания',
  `modified` timestamp NOT NULL DEFAULT current_timestamp() ON UPDATE current_timestamp() COMMENT 'время изменения конверсии на трекере',
  `management_date` timestamp NOT NULL DEFAULT current_timestamp() COMMENT 'Учётная дата конверсии',
  `account_type` enum('none','employer','advertiser') NOT NULL DEFAULT 'none' COMMENT 'Тип аккаунта который последний изменял данные по конверсии',
  `user_id` int(11) NOT NULL DEFAULT 0 COMMENT 'advertiser_user_id или employer_id',
  `period_hour` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00' COMMENT 'час в формате Y-m-d H:00:00',
  `affiliate_id` int(11) NOT NULL DEFAULT 0 COMMENT 'ID вебмастера',
  `advertiser_id` int(11) NOT NULL DEFAULT 0 COMMENT 'ID рекламодателя',
  `offer_id` int(11) NOT NULL COMMENT 'ID оффера',
  `offer_url_id` int(11) NOT NULL DEFAULT 0 COMMENT 'ID URL офера',
  `platform_id` int(11) NOT NULL DEFAULT 0 COMMENT 'ID платформы вебмастера',
  `campaign_id` int(11) DEFAULT 0 COMMENT 'ID Рекламной кампании',
  `campaign_file_id` int(11) NOT NULL DEFAULT 0 COMMENT 'Файл рекламной кампании',
  `conversion_update_id` int(11) NOT NULL DEFAULT 0 COMMENT 'Идентификатор задачи по обновлению конверсий по списку',
  `file_id` int(11) NOT NULL DEFAULT 0 COMMENT 'ID баннера',
  `goal_id` int(11) NOT NULL DEFAULT 0 COMMENT 'ID цели',
  `browser_format` enum('unknown','desktop','fully','partially','waponly') NOT NULL DEFAULT 'unknown' COMMENT 'Формат браузера: desktop, waponly и т.д.',
  `browser_device` enum('unknown','desktop','mobile','tablet') NOT NULL DEFAULT 'unknown' COMMENT 'Устройство браузера: desktop, mobile',
  `browser_app` enum('unknown','ie','firefox','chrome','safari','opera','other','wml') NOT NULL DEFAULT 'unknown' COMMENT 'Название браузера: IE, Firefox и т.д.',
  `status` enum('approved','pending','rejected') NOT NULL DEFAULT 'pending' COMMENT 'Статус конверсии',
  `status_detail` enum('approve-auto','approve-manual','approve-sync','pending-new','pending-inprogress','pending-manual','pending-wait','rejected-manual','rejected-timeout','rejected-scoring-internal','rejected-scoring-external') NOT NULL DEFAULT 'pending-new' COMMENT 'Детализированный статус. Обновление поле status автоматическое, в зависимости от этого поля',
  `status_affiliate` enum('approve-auto','approve-manual','approve-sync','pending-new','pending-inprogress','pending-manual','pending-wait','rejected-manual','rejected-timeout','rejected-scoring-internal','rejected-scoring-external') NOT NULL DEFAULT 'pending-new' COMMENT 'Внешний статус для вебмастера',
  `status_advertiser` enum('approved','pending','rejected') NOT NULL DEFAULT 'pending' COMMENT 'Статус рекламодателя (краткий без детализации)',
  `amount` decimal(10,2) DEFAULT 0.00 COMMENT 'Сумма сделки',
  `payout_type` enum('cpa_fix','cpa_percent') NOT NULL DEFAULT 'cpa_fix' COMMENT 'Тип расчета выплаты',
  `payout` decimal(10,2) DEFAULT 0.00 COMMENT 'Сумма получаемая вебмастером',
  `revenue_type` enum('cpa_fix','cpa_percent') NOT NULL DEFAULT 'cpa_fix' COMMENT 'Тип расчета дохода',
  `revenue` decimal(10,2) DEFAULT 0.00 COMMENT 'Сумма получаемая от рекламодателя',
  `protocol_type` enum('http','https','http_img','https_img','server','js_script','leadschain') NOT NULL DEFAULT 'server',
  `referrer` varchar(1024) NOT NULL DEFAULT '' COMMENT 'HTTP_REFERRER страницы перехода на посадку',
  `pixel_referrer` varchar(1024) NOT NULL DEFAULT '' COMMENT 'HTTP_REFERRER страницы спасибо',
  `user_agent` varchar(512) NOT NULL DEFAULT '' COMMENT 'HTTP_USER_AGENT',
  `country_id` int(11) NOT NULL DEFAULT 0 COMMENT 'ID страны',
`region_id` int(11) NOT NULL DEFAULT 0 COMMENT 'ID региона',
  `city_id` int(11) NOT NULL DEFAULT 0 COMMENT 'ID населенного пункта',
  `source` varchar(255) NOT NULL DEFAULT '' COMMENT 'ID населенного пункта',
  `aff_sub1` varchar(255) NOT NULL DEFAULT '' COMMENT 'SubId1 вебмастера',
  `aff_sub2` varchar(255) NOT NULL DEFAULT '' COMMENT 'SubId2 вебмастера',
  `aff_sub3` varchar(255) NOT NULL DEFAULT '' COMMENT 'SubId3 вебмастера',
  `aff_sub4` varchar(255) NOT NULL DEFAULT '' COMMENT 'SubId4 вебмастера',
  `aff_sub5` varchar(255) NOT NULL DEFAULT '' COMMENT 'SubId5 вебмастера',
  `adv_sub` varchar(255) NOT NULL DEFAULT '' COMMENT 'SubId рекламодателя',
  `ip` varchar(15) NOT NULL DEFAULT '' COMMENT 'IP адрес клиента',
  `ip_proxy` varchar(15) NOT NULL DEFAULT '' COMMENT 'IP адрес прокси сервера',
  `employee_note` varchar(255) NOT NULL DEFAULT '' COMMENT 'Заметка сотрудника',
  `advertiser_note` varchar(255) NOT NULL DEFAULT '' COMMENT 'Заметка рекламодателя',
  `advertiser_status` varchar(64) NOT NULL DEFAULT '' COMMENT 'Статус конверсии банка',
  `advertiser_info` text NOT NULL DEFAULT '' COMMENT 'сериализованные данные полученные от банка',
  `affiliate_note` varchar(255) NOT NULL DEFAULT '' COMMENT 'Заметка вебмастера',
  `affiliate_info` text NOT NULL DEFAULT '' COMMENT 'сериализованные данные полученные от вебмастера(брокера)',
  `change_balance` enum('noaction','need_approve','need_reject') NOT NULL DEFAULT 'noaction' COMMENT 'Отметка, что нужно создать изменение баланса по конверсии',
  `status_updated` timestamp NULL DEFAULT NULL COMMENT 'Дата обновления статуса конверсии',
  `points` decimal(10,2) DEFAULT 0.00 COMMENT 'Бонусные баллы',
  `os` enum('linux','windows','ios','mac','android','other') DEFAULT NULL,
  `is_scale_of_charges` tinyint(1) NOT NULL DEFAULT 0 COMMENT 'Конверсия использует тарифную сетку',
  `lead_id` bigint(20) unsigned DEFAULT NULL COMMENT 'ID анкеты по которой была создана конверсия(если есть)',
  `conversion_source` enum('lead_api','lead_form','lead_file','landing','other') NOT NULL DEFAULT 'other' COMMENT 'Источник конверсии',
  `has_personal_payout` tinyint(1) NOT NULL DEFAULT 0 COMMENT 'Коеффициент персональных условий для вебмастера',
  `sys_sub1` varchar(255) NOT NULL DEFAULT '',
  `sys_sub2` varchar(255) NOT NULL DEFAULT '',
  `sys_sub3` varchar(255) NOT NULL DEFAULT '',
  `sys_sub4` varchar(255) NOT NULL DEFAULT '',
  `sys_sub5` varchar(255) NOT NULL DEFAULT '',
  `default_payout` decimal(10,2) DEFAULT 0.00 COMMENT 'Дефолтная сумма выплат вебмастеру (без учёта персональных цен)',
  `default_revenue` decimal(10,2) DEFAULT 0.00 COMMENT 'Дефолтная сумма, получаемая от рекламодателя (без учёта персональных цен)',
  `http_host` varchar(255) DEFAULT '' COMMENT 'Хост, с которого был сделан клик, показ или конверсия',
  `hold_days` int(11) DEFAULT NULL,
  `hold_type` enum('system','offer','affiliate','group','personal') DEFAULT NULL,
  `event_http_host` varchar(255) DEFAULT '' COMMENT 'Хост события(клик,показ)',
  `preoffer_url_id` int(11) DEFAULT 0 COMMENT 'ID прелендинга',
  `is_unique_conversion` tinyint(4) DEFAULT 0,
  `contextad_dimension_id` int(11) NOT NULL DEFAULT 0,
  `valuta_rate_id` int(11) DEFAULT NULL COMMENT 'ID курса валюты выплаты',
  `rotator_id` int(11) DEFAULT 0 COMMENT 'ID ротатора',
  `virtual_affiliate_id` int(11) DEFAULT NULL COMMENT 'Виртуальный ИД вебмастера',
  `virtual_platform_id` int(11) DEFAULT NULL COMMENT 'Виртуальный ИД платформы',
  `blocked_domain_id` int(11) DEFAULT 0 COMMENT 'Реклама на бренд по домену',
  `blocked_word_id` int(11) DEFAULT 0 COMMENT 'Реклама на бренд по ключ. слову',
  `nds_rate` decimal(10,4) DEFAULT 0.0000 COMMENT 'Коэффициент от НДС',
 PRIMARY KEY (`id`),
  UNIQUE KEY `unq_adv_sub_goal_id_offer_id` (`adv_sub`,`goal_id`,`offer_id`),
  KEY `idx_conversion_transaction_id` (`conversion_transaction_id`),
  KEY `idx_transaction_id` (`transaction_id`),
  KEY `idx_is_test` (`is_test`),
  KEY `idx_is_test_change_balance` (`is_test`,`change_balance`),
  KEY `idx_status` (`status`),
  KEY `idx_modified` (`modified`),
  KEY `idx_offer_id` (`offer_id`),
  KEY `idx_affiliate_id` (`affiliate_id`),
  KEY `idx_platform_id` (`platform_id`),
  KEY `idx_adv_sub` (`adv_sub`),
  KEY `idx_change_balance` (`change_balance`),
  KEY `idx_status_updated` (`status_updated`),
  KEY `idx_status_detail` (`status_detail`),
  KEY `idx_status_affiliate` (`status_affiliate`),
  KEY `idx_adv_sub_offer_id` (`adv_sub`,`offer_id`),
  KEY `idx_transaction_id_goal_id_offer_id` (`transaction_id`,`goal_id`,`offer_id`),
  KEY `idx_transaction_id_goal_id_advertiser_id` (`transaction_id`,`goal_id`,`advertiser_id`),
  KEY `idx_lead_id` (`lead_id`),
  KEY `idx_change_balance_affiliate_id` (`change_balance`,`affiliate_id`),
  KEY `idx_advertiser_id_goal_id_adv_sub` (`advertiser_id`,`goal_id`,`adv_sub`),
  KEY `idx_created` (`created`),
  KEY `idx_hold_days` (`hold_days`)
) ENGINE=TokuDB DEFAULT CHARSET=utf8 `compression`='tokudb_zlib'
we used before 0.9 version of debezium about year+.
sry chat symbol limit
Jiri Pechanec
@jpechane
@avatar13 I was able to parse the DDL correctly. Could you please try to take our our tutorial and deploy this table to it?
avatar13
@avatar13
Снимок экрана 2021-04-16 в 12.12.57.png
@jpechane i`m trying to use compiled version debezium-mysql-connector with some changes
Снимок экрана 2021-04-16 в 12.14.37.png
for testing i use debezium docker container with replace debezium-connector-mysql jar file
in 0.9 version this method worked =)
CREATE TABLE `keyword_dimension` (
  `id` int(11) NOT NULL AUTO_INCREMENT,
  `hash` char(32) NOT NULL DEFAULT '' COMMENT 'md5(raw=true) от search_engine, keyword',
  `search_engine` enum('','yandex','google','rambler','alltheweb','aol','ask','bing','yahoo','baidu','mail.ru') NOT NULL DEFAULT '' COMMENT 'Поисковая система',
  `keyword` char(128) NOT NULL DEFAULT '' COMMENT 'Ключевое слово',
  PRIMARY KEY (`id`),
  KEY `idx_search_engine` (`search_engine`) `CLUSTERING`=YES,
  KEY `idx_keyword` (`keyword`) `CLUSTERING`=YES,
  KEY `idx_hash` (`hash`)
) ENGINE=TokuDB AUTO_INCREMENT=346233 DEFAULT CHARSET=utf8 `compression`='tokudb_zlib'
Jiri Pechanec
@jpechane
@avatar13 This exception is thrown by code that processes default values in DDL. 0.9 did not have such functionality
avatar13
@avatar13
@jpechane i need to copy other .jars too?
Jiri Pechanec
@jpechane
@avatar13 You must take the full content of the tar.gz file built by Maven
avatar13
@avatar13
@jpechane thank you! i added more jar files from build to copy into container and this helped.
René Kerner
@rk3rn3r
@gunnarmorling I think you were writing some comments while we were talking about debezium/debezium#2273 but I can't see any. Can you check if by accident you still have the tab open and submitted?
Gunnar Morling
@gunnarmorling
oh, you're right
just done
René Kerner
@rk3rn3r
:pray:
Thimxn
@thimxns_twitter

Hey guys, I'm currently setting up debezium (in kafka) for a mssql server (on another machine in the network). So far everything seemed to work fine however, when checking the status of the connector i get the following response:
{"name":"test-connector","connector":{"state":"RUNNING","worker_id":"localhost:8083"},"tasks":[{"id":0,"state":"FAILED","worker_id":"localhost:8083","trace":"org.apache.kafka.common.errors.TimeoutException: Timeout expired while fetching topic metadata\n"}],"type":"source"} while it used to be this in the first minute or so after starting it: {"name":"test-connector","connector":{"state":"RUNNING","worker_id":"localhost:8083"},"tasks":[{"id":0,"state":"RUNNING","worker_id":"localhost:8083"}],"type":"source"}.

The initial connector setup looks like the following:

curl -i -X POST -H "Accept:application/json" -H "Content-Type:application/json" localhost:8083/connectors/ -d '{ "name": "test-connector", "config": { "connector.class": "io.debezium. connector.sqlserver.SqlServerConnector", "database.hostname": "192.168.230.30",
"database.port": "1433", "database.user": "***","database.password": "***", "database.dbname": "aqotec_Daten", "database.server.name": "AQNEU", "table.whitelist": "dbo.RM360_L2R12", "database.history.kafka.bootstrap.servers": "localhost:9092",
"database.history.kafka.topic": "dbhistory.fulfillment"  } }';

Furthermore when starting the consumer using this:

sudo docker run -it --rm --name consumer --link zookeeper:zookeeper --link kafka2:kafka2 debezium/kafka:1.1 watch-topic -a AQNEU.aqotec_Daten.dbo.RM360_L2R12 --max-messages 1

I get no errors but it also doesnt track any of the changes:

WARNING: Using default BROKER_ID=1, which is valid only for non-clustered installations.
Using ZOOKEEPER_CONNECT=172.17.0.2:2181
Using KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://172.17.0.5:9092
Using KAFKA_BROKER=172.17.0.4:9092
Contents of topic AQOTECNEU.aqotec_Daten.dbo.RM360_L2R12:

If I would use the wrong watch-topic -a it would throw me an error so I guess its detecting the connector, but the connector is not working.

Since im trying to give you as much information as I can, here is my docker setup:

CONTAINER ID        IMAGE                    COMMAND                  CREATED             STATUS              PORTS
    NAMES
17ee16d7d123        debezium/kafka:1.1       "/docker-entrypoint.…"   15 minutes ago      Up 15 minutes       8778/tcp, 9092/tcp, 9779/tcp
    consumer
f41d9023971b        debezium/connect         "/docker-entrypoint.…"   8 days ago          Up 15 minutes       8778/tcp, 9092/tcp, 0.0.0.0:8083->8083/tcp, 9779/tcp
    connect
cb6da30727b3        debezium/kafka:1.1       "/docker-entrypoint.…"   8 days ago          Up 8 days           8778/tcp, 9779/tcp, 0.0.0.0:9092->9092/tcp
    kafka2
d027f3e800a7        debezium/zookeeper:1.4   "/docker-entrypoint.…"   8 days ago          Up 8 days           0.0.0.0:2181->2181/tcp, 0.0.0.0:2888->2888/tcp, 8778/tcp, 0.0.0.0:3888->3888/tcp, 9779/tcp   zookeeper

Im rather new to debezium and docker so it could be just some simple setup that I messed up.

I already tried to find solutions for this but couldn't find any that worked so I decided to ask here. Thanks already for your help.

Gunnar Morling
@gunnarmorling
@Naros do you think you have an answer for this user: https://twitter.com/karim_elna/status/1383081141465403407
perhaps an enhancement request falls out of it? I'm not sure
Chris Cranford
@Naros
@gunnarmorling It sounds to me like the SMT complains that it cannot find field aggregatetype since Hibernate creates it as "aggregate_type". Is that how you read it?
Chris Cranford
@Naros
We added a way to customize the naming at build time, I think it's something like quarkus.debezium-outbox.aggregate-type.name=<value>
That way the generated XML could match whatever naming strategy they needed in the db.
Gunnar Morling
@gunnarmorling
@Naros yes, exactly, that's how i read it
but this guy is using spring boot
so not our extension for quarkus
so probably just need to adjust the column names?
Chris Cranford
@Naros
Ah, well in that case wouldn't they just change the route.by.field in the connector's SMT configuration to use aggregate_typerather than the default aggregatetype?
Gunnar Morling
@gunnarmorling
ah, yes, right
that'd work too
Chris Cranford
@Naros
Pretty sure we made sure that all this stuff was super flexible re naming.
Gunnar Morling
@gunnarmorling
do you want to reply?
or should i
Chris Cranford
@Naros
Could you, I don't use my twitter too often and don't recall the pwd.
Gunnar Morling
@gunnarmorling
LOL, ok :)
Chris Cranford
@Naros
tbh, looking at the quarkus extension I was struggling to remember what I wrote lol
it feels like ages since I last looked at it lol
Gunnar Morling
@gunnarmorling
hehe, yeah, has been a while
Sanjeev Singh
@sanjeevhbti45_twitter

Can someone helps here i am facing below error in debezium MySQL connector while Deleting the rows.
Error : eventType=EXT_DELETE_ROWS

{"name":"connect-01","connector":{"state":"RUNNING","worker_id":"100.0.0.111:8083"},"tasks":[{"id":0,"state":"FAILED","worker_id":"00.0.0.111:8083","trace":"org.apache.kafka.connect.errors.ConnectException: com.github.shyiko.mysql.binlog.event.deserialization.EventDataDeserializationException: Failed to deserialize data of EventHeaderV4{timestamp=1618682018000, eventType=EXT_DELETE_ROWS, serverId=520243567, headerLength=19, dataLength=8137, nextPosition=3791091837, flags=0}\n\tat io.debezium.connector.mysql.AbstractReader.wrap(AbstractReader.java:230)\n\tat io.debezium.connector.mysql.AbstractReader.failed(AbstractReader.java:207)\n\tat io.debezium.connector.mysql.BinlogReader.handleEvent(BinlogReader.java:600)\n\tat com.github.shyiko.mysql.binlog.BinaryLogClient.notifyEventListeners(BinaryLogClient.java:1130)\n\tat com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:978)\n\tat com.github.shyiko.mysql.binlog.BinaryLogClient.connect(BinaryLogClient.java:581)\n\tat com.github.shyiko.mysql.binlog.BinaryLogClient$7.run(BinaryLogClient.java:860)\n\tat java.base/java.lang.Thread.run(Thread.java:834)\nCaused by: java.lang.RuntimeException: com.github.shyiko.mysql.binlog.event.deserialization.EventDataDeserializationException: Failed to deserialize data of EventHeaderV4{timestamp=1618682018000, eventType=EXT_DELETE_ROWS, serverId=520243567, headerLength=19, dataLength=8137, nextPosition=3791091837, flags=0}\n\tat io.debezium.connector.mysql.BinlogReader.handleServerIncident(BinlogReader.java:668)\n\tat io.debezium.connector.mysql.BinlogReader.handleEvent(BinlogReader.java:583)\n\t... 5 more\nCaused by: com.github.shyiko.mysql.binlog.event.deserialization.EventDataDeserializationException: Failed to deserialize data of EventHeaderV4{timestamp=1618682018000, eventType=EXT_DELETE_ROWS, serverId=520243567, headerLength=19, dataLength=8137, nextPosition=3791091837, flags=0}\n\tat com.github.shyiko.mysql.binlog.event.deserialization.EventDeserializer.deserializeEventData(EventDeserializer.java:300)\n\tat com.github.shyiko.mysql.binlog.event.deserialization.EventDeserializer.nextEvent(EventDeserializer.java:223)\n\tat io.debezium.connector.mysql.BinlogReader$1.nextEvent(BinlogReader.java:249)\n\tat com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:957)\n\t... 3 more\nCaused by: java.io.EOFException\n\tat com.github.shyiko.mysql.binlog.io.ByteArrayInputStream.read(ByteArrayInputStream.java:190)\n\tat java.base/java.io.InputStream.read(InputStream.java:271)\n\tat java.base/java.io.InputStream.skip(InputStream.java:531)\n\tat com.github.shyiko.mysql.binlog.io.ByteArrayInputStream.skipToTheEndOfTheBlock(ByteArrayInputStream.java:216)\n\tat com.github.shyiko.mysql.binlog.event.deserialization.EventDeserializer.deserializeEventData(EventDeserializer.java:296)\n\t... 6 more\n"}],"type":"source"}

Gunnar Morling
@gunnarmorling
hey @jpechane, good morning
got something for you (maybe) :)
the other day, i wanted to understand how offsets are handled for source connectors with more than one task
so to see what will change for the sql server work
in order to do so, i created a super-simple source connector for etcd: https://github.com/gunnarmorling/kcetcd