SymmetricDS/Lobby

jmckgenerali

@jmckgenerali:matrix.org

I've created a bug report regarding the jdbc issue. It includes logs with stack traces.

joshahicks

@joshahicks

@jmckgenerali:matrix.org thank you for submitting the issue we are trying to release 3.14 so that has been the focus most recently but we will look at it soon

joshahicks

@joshahicks

@adamitsch it looks like an issue with Android or a class loading issue with apache commons jar. Once we get this 3.14 released we will try to take a look. Please let us know if you find anything on your side that fixes it

@jyotid1815_gitlab do you have any nodes offline? Nodes that are registered but not actually running or connecting? This would backlog the data. Also make sure routing and purge jobs are not throwing any errors in the logs. Do you have any batches in error

Jyoti D

@jyotid1815_gitlab

Nodes are not offline. There is no error in the logs. The outgoing batches showing the status NE.

joshahicks

@joshahicks

@jyotid1815_gitlab is the target set up to pull these batches ?

adamitsch

@adamitsch

@joshahicks Any idea how to debug this? I have tried different with different release versions but just encountered other errors.

Jyoti D

@jyotid1815_gitlab

Yes, target set to pull batches. right now out of 150000 only 4000 data is synced in the table. But error log still showing one entry.

veripolis-ms

@veripolis-ms

milliseconds are not synced in datetime(6) fields.

dikum

@dikum

@joshahicks Please when I enter a new record, it syncs to the target node successfully, also works when I update such a record. But when an older record is updated, it does not sync. Is there a possible cause for this?

mitpjones

@mitpjones

Hi, I am upgrading from 3.8 to 3.13 and noticed that the entries in the sym_outgoing_batch table for channel_id = 'heartbeat' are being inserted but when the target node is offline that the sym_outgoing_batch.status of the previous entries are no longer updated. Hence when the node comes back online it can have many 'outdated superfluous' heartbeat entries that must be synched rather than just the most recent. I noticed that in revision 0003883 5/03/19 that the PushHeartbeatListener.heartbeat() method was changed from

        log.debug("Updating my node info");
        engine.getOutgoingBatchService().markHeartbeatAsSent();   <---- this has been removed
        engine.getNodeService().updateNodeHostForCurrentNode();
        log.debug("Done updating my node info");

        log.debug("Updating my node info");
        if (engine.getOutgoingBatchService().countOutgoingBatchesUnsentHeartbeat() == 0) {
            engine.getNodeService().updateNodeHostForCurrentNode();
        }
        log.debug("Done updating my node info");

Is there some other functionality that replaced the updating of the 'outdated' sym_outgoing_batch heartbeat entries that I need to configure for 3.13 or is it now intended for these to be synched?

Jyoti D

@jyotid1815_gitlab

SymmetricDS is taking a lot of time to sync the data. I'm able to see in logs the data is in process to sync . In log it is showing data of last 3 to 4 days is still pending to sync. Please help me out.Please give inputs for resolving this issue.

dikum

@dikum

The incoming purge process is about to run Getting range for incoming batch : Creating dump About to purge incoming batch Done purging 62 of data rows Getting range for outgoing batch Getting first batch_id for outstanding batches About to purge data_event_range using range 26324 thro>2022-07-15 00:00:00.201 INFO 2584 --- [server-000-Done purging 0 of data_event_range rows About to purge outgoing_batch_range using range 26324 >2022-07-15 00:00:00.201 INFO 2584 --- [server-000-Done purging 0 of outgoing_batch_range rows About to purge data_event using range 26061 through 26>2022-07-15 00:00:00.225 INFO 2584 --- [server-000-Done purging 47 incoming batch rows Purging incoming error rows Purged 0 incoming error rows Purging registration requests that are older than Sun>2022-07-15 00:00:00.227 INFO 2584 --- [server-000-job-12] Purging monitor events that are older than Thu Jul 14>2022-07-15 00:00:00.250 INFO 2584 --- [server-000-job-The incoming purge process has completed Done purging 62 of data_event rows About to purge outgoing_batch using range 26061 throug>2022-07-15 00:00:00.426 INFO 2584 --- [server-000-Done purging 49 of outgoing_batch rows Looking for lingering batches before batch ID 26389 Found 14 lingering batches to purge Done purging 14 lingering batches and 32 rows Getting range for stranded data events About to purge stranded_data_event using range 0 throu>2022-07-15 00:00:00.561 INFO 2584 --- [server-000-Done purging 0 of stranded_data_event rows Getting range for stranded data

I have an issue where old records that are updated don't get synchronized. Please here is my log from the target node.
Please can anyone provide any help?

veripolis-ms

@veripolis-ms

@dikum I'm pretty new myself, but I suggest you check whether there is an update trigger on the source table if its updates don't get synchronized out.

I have an issue too. I am synchronizing four databases all ways, and it looks correct short-term, but if I wait and check later, deleted rows are back.

veripolis-ms

@veripolis-ms

How is the correct way to alter the structure of a synchronized table?

veripolis-ms

@veripolis-ms

It seems that earlier rows, which are deleted, are stored in files somewhere, and not only in the symmetricds database? Where are they stored?

I have re-created all databases and still old rows are showing up

dikum

@dikum

@veripolis-ms Thanks. The thing is, if I enter a new record it synchronizes. The update made on that new record also synchronizes. The problem is with the existing records in the source db. It doesn't get inserted nor updated in the target DB.

dikum

@dikum

I think I figured it out. It was the initial_load_select column on the sym_trigger_router table that was restricting the number of records being loaded. I set this field to null and it seems to be fine.

dikum

@dikum

Hi everyone,
Is it normal for a sync of 14,000 records to take hours to sync?
Here's my current configuration on a source node:

job.routing.period.time.ms=5000
job.push.period.time.ms=3000
job.pull.period.time.ms=3000
dataloader.max.rows.before.commit=100
auto.resolve.foreign.key.violation.reverse=true
dataloader.use.primary.keys.from.source=true

joshahicks

@joshahicks

@dikum if you look at sym_outgoing_batch for batches with a status of OK on channel default (or whatever channel you are using), there are stats in there regarding where the slowness might be coming from. Look at extract_millis, network_millis, and load_millis to start to see if any of them are much larger than the others

adamitsch

@adamitsch

I would like to add ORM layer over SQLite in Android. Which solution/library works best together with SymmetricDS?

joshahicks

@joshahicks

SymmetricDS works directly with the database so any applications on top should not matter

adamitsch

@adamitsch

I mean some create its own database and you need to manually copy from existing database etc. I just tried DBFlow and somehow can't specify database name following the documentation... And just wanna know what ORM people use the most.

Jyoti D

@jyotid1815_gitlab

extract_millis=169 showing in sym_outgoing_batch and status is NE . Is this the reason for slow syncing?

joshahicks

@joshahicks

@jyotid1815_gitlab this means it is extracting very fast and sitting in new status waiting for the target node to pull the change. So the source seems to look good but the target is probably where you want to look next because it is not pulling the change.

Did you find any batches with OK status? They would have all the stats populated

Jyoti D

@jyotid1815_gitlab

very few batches are in OK status and most of the batches are in NE and LD status. What changes need to do at target to make PULL fast? Because it is taking 4 to 5 days to sync 100000 record in a table.

joshahicks

@joshahicks

@jyotid1815_gitlab without any changes SymmetricDS will sync every 60s and will send up to 100 batches at a time with a max size of 10,000. A lot of other factors come into play regarding performance though such as the network speed, the size of the data, and the machines it is running on. You may need to check some of these areas to determine where the slowness is occurring before adjustments can be made. You could start by changing the parameters arournd the frequency of push and pulls : job.push.period.time.ms & job.push.period.time.ms

You could also adjust the channel max batch to send from 100 down to 5 to see if it is having issues sending that many batches at once

Jyoti D

@jyotid1815_gitlab

checked performance of both source and target machine. It is quite normal. The source node is having symmetricDS 3.13.5 version and target which is low in pulling data is SymmetricDS 3.9.11 .

This version difference can be the issue for syymetricDS performance at target side

joshahicks

@joshahicks

@jyotid1815_gitlab we have customers using those versions without issues so Im not sure its version related but could be if you are syncing a lot of data.

philipmarzullo64

@philipmarzullo64

@veripolis-ms The best way to alter the structure of a synchronized table is to set enabled=0 on the sym_trigger_router record for the trigger that is referencing the table to change, allow SymmetricDS to drop the triggers on the table to change, make the alterations to the table, then set enabled=1 on the trigger router record for the trigger that is referencing the table to change.

adamitsch

@adamitsch

I want to sync between android clients (store) via main instance (corp) on mssql. Android clients have "localhost" as address and they can't push to each other. Is turning sync_on_incoming_batch the only way to do It? And how common is this update loop mentioned in documentation?
How to solve the problem when two clients insert row with same PK (and also edit dependent rows from other tables)?

joshahicks

@joshahicks

Yes you need a set of triggers for the corp that has sync on incoming on. You can setup another set of client trigges that do not have it on as it is not needed at that level.
Conflict resolution is built in and last to update will win the conflict by default but you can setup other variations

adamitsch

@adamitsch

But what to do in case when I want to keep data from both clients (and only change primary key of newest insert)? In my case there won't be many users but they could be offline a lot of the time. So I was thinking about creating a new table(s) and insert only after it gets synchronized.

joshahicks

@joshahicks

@adamitsch you may want to consider a composite primary key then at central that includes the client id and the current PK. Are these auto increment PKs at the clients? That is how most of our replication customers solve this issue. You can use a SymmetricDS transform on the way to central to put in the client id value

adamitsch

@adamitsch

This primary key is some kind of order number and it is varchar so it is manually incremented and I wouldn't like to have two with the same. Is it possible to do something like that with beanshell (to increment this to next value with transformation)?

joshahicks

@joshahicks

@adamitsch the issue is if you increment at the target you also need to go back and change at the source or you are out of sync. That is why most common solution we see for this use case is to allow each source to have its own unique value and then when arriving at the target the combination of source id and the id itself creates the uniqueness. This also allows replication and tracing back to the original value at the source

Alfeighty

@Alfeighty

@joshahicks Good Day, Can anyone please help with the format required for the "source_catalog_name" and "source_schema_name" in the 'sym_trigger table'? I tried a few combinations, currently //IPAddress/DatabaseName:1433, but it gives a connect error (Symmetric is an owner for this MSSQL database). I am trying to keep the sym tables in a separate database. Thank you in advance.

joshahicks

@joshahicks

db.url property in the engine file is what is used to connect to the database and it will create the sym tables in the databased used at setup in this property

Alfeighty

@Alfeighty

Thank you Joshua, if I understand correctly, this db.url will be where the "sym" tables will be created. I want to edit this databases sym_trigger table, to point or create the triggers in an external database. (So that the external database only has the triggers installed, and the sym tables are hosted in the db.url pointed database.

joshahicks

@joshahicks

Yes db.url is where SymmetricDS will install its SYM_ runtime tables. You can then configure change capture through symtrigger table as you mentioned. Here you can provide other catalogs and schemas that the sym user account (used to connect to the database) for the tables you wish to capture changes on. This does depend on the database you are using though as well. For instance MSSQL will allow you to connect and setup SYM tables in for example the “SymmetricDS” database and then setup triggers to capture changes in “DB1” and “DB2” …. However in Postgres you can not have a trigger fire and write to a table in another database so this would not work.

Ogie Borja

@justinejules_twitter

Hi! I'm still new to SymmetricDS and just want to ask if it is okay to install and run SymmetricDS replication with existing data on the target DB? Do I still need to configure the initial load? TIA

Alfeighty

@Alfeighty

Thank you @joshahicks, will give this a try, but it makes sense. (Will give symmetric user access to all databases involved.)

joshahicks

@joshahicks

@justinejules_twitter No you do not have to do an intial load. It is usually needed to get in sync initially but if you have that covered you are set. It will perform inserts though that would fall back to updates (if you have proper primary keys) and do not harm

Ogie Borja

@justinejules_twitter

Thank you @joshahicks! How about if source db and target db data are not in sync? For example, target db's last data is 2 days older than the source db.

joshahicks

@joshahicks

@justinejules_twitter then you may need to send a load from target to source to get the 2 days of data back to the source. Inserts will fallback to updates so as long as you have PKs in place it should not cause any issues.

Ogie Borja

@justinejules_twitter

Thanks again @joshahicks! :)

Where communities thrive

People

Repo info

Activity