Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
  • 07:07
    jpechane closed #2380
  • 05:39
    morozov ready_for_review #2382
  • 01:17
    morozov synchronize #2382
  • 00:35
    morozov edited #2382
  • 00:17
    morozov synchronize #2382
  • 00:03
    morozov opened #2382
  • May 13 23:44
    ccollingwood closed #2381
  • May 13 22:00
    ccollingwood opened #2381
  • May 13 17:24
    Naros synchronize #2371
  • May 13 17:16
    Naros synchronize #2347
  • May 13 17:14
    Naros opened #2380
  • May 13 17:02
    roldanbob closed #2379
  • May 13 16:49
    roldanbob opened #2379
  • May 13 15:08
    jpechane opened #226
  • May 13 14:48
    truman303 opened #2378
  • May 13 14:19
    Naros synchronize #2371
  • May 13 12:59
    novotnyJiri synchronize #2338
  • May 13 12:56
    novotnyJiri synchronize #2338
  • May 13 12:47
    Naros synchronize #2370
  • May 13 12:45
    Naros synchronize #2371
Gunnar Morling
@gunnarmorling
@jpechane could you reply here: debezium/debezium#2280 ?
Gunnar Morling
@gunnarmorling
@jpechane ok, thx
(and good morning :)
people freak out about ad-hoc snapshots btw., see replies here: https://twitter.com/gunnarmorling/status/1382357364687450112
but no pressure ;)
Jiri Pechanec
@jpechane
@gunnarmorling Damn, you've raised a crowed and filled it wih unrelasitic expectations ;-)
Gunnar Morling
@gunnarmorling
yes, preparing my transition into PM ;)
Gunnar Morling
@gunnarmorling
@rk3rn3r @jpechane we're skipping grooming today, since we did it yesterday; i had a quick look, nothing special in there
mostly micro docs issues of Rob ;)
René Kerner
@rk3rn3r:matrix.org
[m]
Okay! Thx for the notification
It's so good that we have Bob
Gunnar Morling
@gunnarmorling
@jpechane shouldn't there be a debezium/connect:1.6 container image on DockerHub by now?
Jiri Pechanec
@jpechane
@gunnarmorling With the first 1.6 release
Gunnar Morling
@gunnarmorling
ah, interesting
I'd have expected something from nightly jobs
Jiri Pechanec
@jpechane
There is nightly tag
Gunnar Morling
@gunnarmorling
aaah
so debezium/connect:nightly would give me 1.6 right now?
Jiri Pechanec
@jpechane
Yes, 1.6.0-SNAPSHOT
Gunnar Morling
@gunnarmorling
ok, gotcha
thx
avatar13
@avatar13
Hello, I have problems with 1.5.0Final version after start: 2021-04-16 06:50:11,032 INFO || Snapshot - Final stage [io.debezium.pipeline.source.AbstractSnapshotChangeEventSource]
2021-04-16 06:50:11,032 ERROR || Producer failure [io.debezium.pipeline.ErrorHandler]
io.debezium.DebeziumException: java.lang.NumberFormatException: Character array is missing "e" notation exponential mark.
at io.debezium.pipeline.source.AbstractSnapshotChangeEventSource.execute(AbstractSnapshotChangeEventSource.java:82)
at io.debezium.pipeline.ChangeEventSourceCoordinator.lambda$start$0(ChangeEventSourceCoordinator.java:105)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.lang.NumberFormatException: Character array is missing "e" notation exponential mark.
at java.base/java.math.BigDecimal.<init>(BigDecimal.java:577)
at java.base/java.math.BigDecimal.<init>(BigDecimal.java:401)
at java.base/java.math.BigDecimal.<init>(BigDecimal.java:834)
at io.debezium.connector.mysql.MySqlDefaultValueConverter.convertToDecimal(MySqlDefaultValueConverter.java:194)
at io.debezium.connector.mysql.MySqlDefaultValueConverter.convert(MySqlDefaultValueConverter.java:91)
at io.debezium.connector.mysql.MySqlDefaultValueConverter.setColumnDefaultValue(MySqlDefaultValueConverter.java:403) need help with this
using TokuDb with single database and table (about 80 columns)
avatar13
@avatar13
java 9 using on build =( sry.
Gunnar Morling
@gunnarmorling
@avatar13 can you identify the problematic table?
avatar13
@avatar13
@gunnarmorling yes, but there is about 80 columns. java 8 not helped, i recompile debezium-mysql-connector with tokudb specific sql-tokens. tests passed, but in docker after launch on single table there is same error...
Jiri Pechanec
@jpechane
@avatar13 Could ou please share the DDL for the given table? Ideally using SHOW CREATE TABLE
avatar13
@avatar13
CREATE TABLE `conversions` (
  `id` bigint(20) unsigned NOT NULL,
  `is_test` tinyint(1) NOT NULL DEFAULT 0 COMMENT 'Транзакция является тестовой',
  `conversion_transaction_id` int(11) NOT NULL DEFAULT 0 COMMENT 'ID информации о транзакции',
  `transaction_id` varchar(32) NOT NULL DEFAULT '' COMMENT 'ID транзакции',
  `created` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00' COMMENT 'дата создания',
  `modified` timestamp NOT NULL DEFAULT current_timestamp() ON UPDATE current_timestamp() COMMENT 'время изменения конверсии на трекере',
  `management_date` timestamp NOT NULL DEFAULT current_timestamp() COMMENT 'Учётная дата конверсии',
  `account_type` enum('none','employer','advertiser') NOT NULL DEFAULT 'none' COMMENT 'Тип аккаунта который последний изменял данные по конверсии',
  `user_id` int(11) NOT NULL DEFAULT 0 COMMENT 'advertiser_user_id или employer_id',
  `period_hour` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00' COMMENT 'час в формате Y-m-d H:00:00',
  `affiliate_id` int(11) NOT NULL DEFAULT 0 COMMENT 'ID вебмастера',
  `advertiser_id` int(11) NOT NULL DEFAULT 0 COMMENT 'ID рекламодателя',
  `offer_id` int(11) NOT NULL COMMENT 'ID оффера',
  `offer_url_id` int(11) NOT NULL DEFAULT 0 COMMENT 'ID URL офера',
  `platform_id` int(11) NOT NULL DEFAULT 0 COMMENT 'ID платформы вебмастера',
  `campaign_id` int(11) DEFAULT 0 COMMENT 'ID Рекламной кампании',
  `campaign_file_id` int(11) NOT NULL DEFAULT 0 COMMENT 'Файл рекламной кампании',
  `conversion_update_id` int(11) NOT NULL DEFAULT 0 COMMENT 'Идентификатор задачи по обновлению конверсий по списку',
  `file_id` int(11) NOT NULL DEFAULT 0 COMMENT 'ID баннера',
  `goal_id` int(11) NOT NULL DEFAULT 0 COMMENT 'ID цели',
  `browser_format` enum('unknown','desktop','fully','partially','waponly') NOT NULL DEFAULT 'unknown' COMMENT 'Формат браузера: desktop, waponly и т.д.',
  `browser_device` enum('unknown','desktop','mobile','tablet') NOT NULL DEFAULT 'unknown' COMMENT 'Устройство браузера: desktop, mobile',
  `browser_app` enum('unknown','ie','firefox','chrome','safari','opera','other','wml') NOT NULL DEFAULT 'unknown' COMMENT 'Название браузера: IE, Firefox и т.д.',
  `status` enum('approved','pending','rejected') NOT NULL DEFAULT 'pending' COMMENT 'Статус конверсии',
  `status_detail` enum('approve-auto','approve-manual','approve-sync','pending-new','pending-inprogress','pending-manual','pending-wait','rejected-manual','rejected-timeout','rejected-scoring-internal','rejected-scoring-external') NOT NULL DEFAULT 'pending-new' COMMENT 'Детализированный статус. Обновление поле status автоматическое, в зависимости от этого поля',
  `status_affiliate` enum('approve-auto','approve-manual','approve-sync','pending-new','pending-inprogress','pending-manual','pending-wait','rejected-manual','rejected-timeout','rejected-scoring-internal','rejected-scoring-external') NOT NULL DEFAULT 'pending-new' COMMENT 'Внешний статус для вебмастера',
  `status_advertiser` enum('approved','pending','rejected') NOT NULL DEFAULT 'pending' COMMENT 'Статус рекламодателя (краткий без детализации)',
  `amount` decimal(10,2) DEFAULT 0.00 COMMENT 'Сумма сделки',
  `payout_type` enum('cpa_fix','cpa_percent') NOT NULL DEFAULT 'cpa_fix' COMMENT 'Тип расчета выплаты',
  `payout` decimal(10,2) DEFAULT 0.00 COMMENT 'Сумма получаемая вебмастером',
  `revenue_type` enum('cpa_fix','cpa_percent') NOT NULL DEFAULT 'cpa_fix' COMMENT 'Тип расчета дохода',
  `revenue` decimal(10,2) DEFAULT 0.00 COMMENT 'Сумма получаемая от рекламодателя',
  `protocol_type` enum('http','https','http_img','https_img','server','js_script','leadschain') NOT NULL DEFAULT 'server',
  `referrer` varchar(1024) NOT NULL DEFAULT '' COMMENT 'HTTP_REFERRER страницы перехода на посадку',
  `pixel_referrer` varchar(1024) NOT NULL DEFAULT '' COMMENT 'HTTP_REFERRER страницы спасибо',
  `user_agent` varchar(512) NOT NULL DEFAULT '' COMMENT 'HTTP_USER_AGENT',
  `country_id` int(11) NOT NULL DEFAULT 0 COMMENT 'ID страны',
`region_id` int(11) NOT NULL DEFAULT 0 COMMENT 'ID региона',
  `city_id` int(11) NOT NULL DEFAULT 0 COMMENT 'ID населенного пункта',
  `source` varchar(255) NOT NULL DEFAULT '' COMMENT 'ID населенного пункта',
  `aff_sub1` varchar(255) NOT NULL DEFAULT '' COMMENT 'SubId1 вебмастера',
  `aff_sub2` varchar(255) NOT NULL DEFAULT '' COMMENT 'SubId2 вебмастера',
  `aff_sub3` varchar(255) NOT NULL DEFAULT '' COMMENT 'SubId3 вебмастера',
  `aff_sub4` varchar(255) NOT NULL DEFAULT '' COMMENT 'SubId4 вебмастера',
  `aff_sub5` varchar(255) NOT NULL DEFAULT '' COMMENT 'SubId5 вебмастера',
  `adv_sub` varchar(255) NOT NULL DEFAULT '' COMMENT 'SubId рекламодателя',
  `ip` varchar(15) NOT NULL DEFAULT '' COMMENT 'IP адрес клиента',
  `ip_proxy` varchar(15) NOT NULL DEFAULT '' COMMENT 'IP адрес прокси сервера',
  `employee_note` varchar(255) NOT NULL DEFAULT '' COMMENT 'Заметка сотрудника',
  `advertiser_note` varchar(255) NOT NULL DEFAULT '' COMMENT 'Заметка рекламодателя',
  `advertiser_status` varchar(64) NOT NULL DEFAULT '' COMMENT 'Статус конверсии банка',
  `advertiser_info` text NOT NULL DEFAULT '' COMMENT 'сериализованные данные полученные от банка',
  `affiliate_note` varchar(255) NOT NULL DEFAULT '' COMMENT 'Заметка вебмастера',
  `affiliate_info` text NOT NULL DEFAULT '' COMMENT 'сериализованные данные полученные от вебмастера(брокера)',
  `change_balance` enum('noaction','need_approve','need_reject') NOT NULL DEFAULT 'noaction' COMMENT 'Отметка, что нужно создать изменение баланса по конверсии',
  `status_updated` timestamp NULL DEFAULT NULL COMMENT 'Дата обновления статуса конверсии',
  `points` decimal(10,2) DEFAULT 0.00 COMMENT 'Бонусные баллы',
  `os` enum('linux','windows','ios','mac','android','other') DEFAULT NULL,
  `is_scale_of_charges` tinyint(1) NOT NULL DEFAULT 0 COMMENT 'Конверсия использует тарифную сетку',
  `lead_id` bigint(20) unsigned DEFAULT NULL COMMENT 'ID анкеты по которой была создана конверсия(если есть)',
  `conversion_source` enum('lead_api','lead_form','lead_file','landing','other') NOT NULL DEFAULT 'other' COMMENT 'Источник конверсии',
  `has_personal_payout` tinyint(1) NOT NULL DEFAULT 0 COMMENT 'Коеффициент персональных условий для вебмастера',
  `sys_sub1` varchar(255) NOT NULL DEFAULT '',
  `sys_sub2` varchar(255) NOT NULL DEFAULT '',
  `sys_sub3` varchar(255) NOT NULL DEFAULT '',
  `sys_sub4` varchar(255) NOT NULL DEFAULT '',
  `sys_sub5` varchar(255) NOT NULL DEFAULT '',
  `default_payout` decimal(10,2) DEFAULT 0.00 COMMENT 'Дефолтная сумма выплат вебмастеру (без учёта персональных цен)',
  `default_revenue` decimal(10,2) DEFAULT 0.00 COMMENT 'Дефолтная сумма, получаемая от рекламодателя (без учёта персональных цен)',
  `http_host` varchar(255) DEFAULT '' COMMENT 'Хост, с которого был сделан клик, показ или конверсия',
  `hold_days` int(11) DEFAULT NULL,
  `hold_type` enum('system','offer','affiliate','group','personal') DEFAULT NULL,
  `event_http_host` varchar(255) DEFAULT '' COMMENT 'Хост события(клик,показ)',
  `preoffer_url_id` int(11) DEFAULT 0 COMMENT 'ID прелендинга',
  `is_unique_conversion` tinyint(4) DEFAULT 0,
  `contextad_dimension_id` int(11) NOT NULL DEFAULT 0,
  `valuta_rate_id` int(11) DEFAULT NULL COMMENT 'ID курса валюты выплаты',
  `rotator_id` int(11) DEFAULT 0 COMMENT 'ID ротатора',
  `virtual_affiliate_id` int(11) DEFAULT NULL COMMENT 'Виртуальный ИД вебмастера',
  `virtual_platform_id` int(11) DEFAULT NULL COMMENT 'Виртуальный ИД платформы',
  `blocked_domain_id` int(11) DEFAULT 0 COMMENT 'Реклама на бренд по домену',
  `blocked_word_id` int(11) DEFAULT 0 COMMENT 'Реклама на бренд по ключ. слову',
  `nds_rate` decimal(10,4) DEFAULT 0.0000 COMMENT 'Коэффициент от НДС',
 PRIMARY KEY (`id`),
  UNIQUE KEY `unq_adv_sub_goal_id_offer_id` (`adv_sub`,`goal_id`,`offer_id`),
  KEY `idx_conversion_transaction_id` (`conversion_transaction_id`),
  KEY `idx_transaction_id` (`transaction_id`),
  KEY `idx_is_test` (`is_test`),
  KEY `idx_is_test_change_balance` (`is_test`,`change_balance`),
  KEY `idx_status` (`status`),
  KEY `idx_modified` (`modified`),
  KEY `idx_offer_id` (`offer_id`),
  KEY `idx_affiliate_id` (`affiliate_id`),
  KEY `idx_platform_id` (`platform_id`),
  KEY `idx_adv_sub` (`adv_sub`),
  KEY `idx_change_balance` (`change_balance`),
  KEY `idx_status_updated` (`status_updated`),
  KEY `idx_status_detail` (`status_detail`),
  KEY `idx_status_affiliate` (`status_affiliate`),
  KEY `idx_adv_sub_offer_id` (`adv_sub`,`offer_id`),
  KEY `idx_transaction_id_goal_id_offer_id` (`transaction_id`,`goal_id`,`offer_id`),
  KEY `idx_transaction_id_goal_id_advertiser_id` (`transaction_id`,`goal_id`,`advertiser_id`),
  KEY `idx_lead_id` (`lead_id`),
  KEY `idx_change_balance_affiliate_id` (`change_balance`,`affiliate_id`),
  KEY `idx_advertiser_id_goal_id_adv_sub` (`advertiser_id`,`goal_id`,`adv_sub`),
  KEY `idx_created` (`created`),
  KEY `idx_hold_days` (`hold_days`)
) ENGINE=TokuDB DEFAULT CHARSET=utf8 `compression`='tokudb_zlib'
we used before 0.9 version of debezium about year+.
sry chat symbol limit
Jiri Pechanec
@jpechane
@avatar13 I was able to parse the DDL correctly. Could you please try to take our our tutorial and deploy this table to it?
avatar13
@avatar13
Снимок экрана 2021-04-16 в 12.12.57.png
@jpechane i`m trying to use compiled version debezium-mysql-connector with some changes
Снимок экрана 2021-04-16 в 12.14.37.png
for testing i use debezium docker container with replace debezium-connector-mysql jar file
in 0.9 version this method worked =)
CREATE TABLE `keyword_dimension` (
  `id` int(11) NOT NULL AUTO_INCREMENT,
  `hash` char(32) NOT NULL DEFAULT '' COMMENT 'md5(raw=true) от search_engine, keyword',
  `search_engine` enum('','yandex','google','rambler','alltheweb','aol','ask','bing','yahoo','baidu','mail.ru') NOT NULL DEFAULT '' COMMENT 'Поисковая система',
  `keyword` char(128) NOT NULL DEFAULT '' COMMENT 'Ключевое слово',
  PRIMARY KEY (`id`),
  KEY `idx_search_engine` (`search_engine`) `CLUSTERING`=YES,
  KEY `idx_keyword` (`keyword`) `CLUSTERING`=YES,
  KEY `idx_hash` (`hash`)
) ENGINE=TokuDB AUTO_INCREMENT=346233 DEFAULT CHARSET=utf8 `compression`='tokudb_zlib'
Jiri Pechanec
@jpechane
@avatar13 This exception is thrown by code that processes default values in DDL. 0.9 did not have such functionality
avatar13
@avatar13
@jpechane i need to copy other .jars too?
Jiri Pechanec
@jpechane
@avatar13 You must take the full content of the tar.gz file built by Maven
avatar13
@avatar13
@jpechane thank you! i added more jar files from build to copy into container and this helped.
René Kerner
@rk3rn3r
@gunnarmorling I think you were writing some comments while we were talking about debezium/debezium#2273 but I can't see any. Can you check if by accident you still have the tab open and submitted?
Gunnar Morling
@gunnarmorling
oh, you're right
just done
René Kerner
@rk3rn3r
:pray:
Thimxn
@thimxns_twitter

Hey guys, I'm currently setting up debezium (in kafka) for a mssql server (on another machine in the network). So far everything seemed to work fine however, when checking the status of the connector i get the following response:
{"name":"test-connector","connector":{"state":"RUNNING","worker_id":"localhost:8083"},"tasks":[{"id":0,"state":"FAILED","worker_id":"localhost:8083","trace":"org.apache.kafka.common.errors.TimeoutException: Timeout expired while fetching topic metadata\n"}],"type":"source"} while it used to be this in the first minute or so after starting it: {"name":"test-connector","connector":{"state":"RUNNING","worker_id":"localhost:8083"},"tasks":[{"id":0,"state":"RUNNING","worker_id":"localhost:8083"}],"type":"source"}.

The initial connector setup looks like the following:

curl -i -X POST -H "Accept:application/json" -H "Content-Type:application/json" localhost:8083/connectors/ -d '{ "name": "test-connector", "config": { "connector.class": "io.debezium. connector.sqlserver.SqlServerConnector", "database.hostname": "192.168.230.30",
"database.port": "1433", "database.user": "***","database.password": "***", "database.dbname": "aqotec_Daten", "database.server.name": "AQNEU", "table.whitelist": "dbo.RM360_L2R12", "database.history.kafka.bootstrap.servers": "localhost:9092",
"database.history.kafka.topic": "dbhistory.fulfillment"  } }';

Furthermore when starting the consumer using this:

sudo docker run -it --rm --name consumer --link zookeeper:zookeeper --link kafka2:kafka2 debezium/kafka:1.1 watch-topic -a AQNEU.aqotec_Daten.dbo.RM360_L2R12 --max-messages 1

I get no errors but it also doesnt track any of the changes:

WARNING: Using default BROKER_ID=1, which is valid only for non-clustered installations.
Using ZOOKEEPER_CONNECT=172.17.0.2:2181
Using KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://172.17.0.5:9092
Using KAFKA_BROKER=172.17.0.4:9092
Contents of topic AQOTECNEU.aqotec_Daten.dbo.RM360_L2R12:

If I would use the wrong watch-topic -a it would throw me an error so I guess its detecting the connector, but the connector is not working.

Since im trying to give you as much information as I can, here is my docker setup:

CONTAINER ID        IMAGE                    COMMAND                  CREATED             STATUS              PORTS
    NAMES
17ee16d7d123        debezium/kafka:1.1       "/docker-entrypoint.…"   15 minutes ago      Up 15 minutes       8778/tcp, 9092/tcp, 9779/tcp
    consumer
f41d9023971b        debezium/connect         "/docker-entrypoint.…"   8 days ago          Up 15 minutes       8778/tcp, 9092/tcp, 0.0.0.0:8083->8083/tcp, 9779/tcp
    connect
cb6da30727b3        debezium/kafka:1.1       "/docker-entrypoint.…"   8 days ago          Up 8 days           8778/tcp, 9779/tcp, 0.0.0.0:9092->9092/tcp
    kafka2
d027f3e800a7        debezium/zookeeper:1.4   "/docker-entrypoint.…"   8 days ago          Up 8 days           0.0.0.0:2181->2181/tcp, 0.0.0.0:2888->2888/tcp, 8778/tcp, 0.0.0.0:3888->3888/tcp, 9779/tcp   zookeeper

Im rather new to debezium and docker so it could be just some simple setup that I messed up.

I already tried to find solutions for this but couldn't find any that worked so I decided to ask here. Thanks already for your help.

Gunnar Morling
@gunnarmorling
@Naros do you think you have an answer for this user: https://twitter.com/karim_elna/status/1383081141465403407