Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
  • 06:39

    the4thdoctor on ver2.0

    fix link for release badge fix link for releases second try (compare)

  • 06:37

    the4thdoctor on master

    fix link for releases second try (compare)

  • 06:36

    the4thdoctor on master

    fix link for release badge (compare)

  • 06:33

    the4thdoctor on master

    fix the location for the packag… stamp 2.0.12 (compare)

  • Dec 11 16:13

    the4thdoctor on v2.0.12

    (compare)

  • Dec 11 16:11

    the4thdoctor on ver2.0

    stamp 2.0.12 (compare)

  • Dec 11 15:53

    the4thdoctor on ver2.0

    fix the location for the packag… (compare)

  • Dec 11 15:41

    the4thdoctor on setup_fix

    fix the location for the packag… (compare)

  • Dec 10 17:41
  • Dec 07 09:50
    the4thdoctor closed #92
  • Dec 07 09:50
    the4thdoctor commented #92
  • Dec 07 09:50
    the4thdoctor unlabeled #92
  • Dec 07 09:50
    the4thdoctor unlabeled #92
  • Dec 07 09:50
    the4thdoctor labeled #92
  • Dec 05 15:10
    clifff closed #91
  • Dec 05 15:10
    clifff commented #91
  • Dec 01 20:16
    the4thdoctor closed #89
  • Dec 01 20:16
    the4thdoctor commented #89
  • Nov 29 13:32
    the4thdoctor commented #92
  • Nov 28 16:16
    the4thdoctor closed #96
Esa Halsti
@ekohalsti_gitlab
Thank you! That worked like a charm!
Federico Campoli
@the4thdoctor
cool :)
Achraf Elafrit
@afritfr
Hello, is it possible to reduce rights for replication user in mysql ? I mean Does it need write privileges ? I am asking this because i want to use pg_chameleon in production but my team is kinda not confortable with the user granted all in mysql (our database is really critical and a minor error can get us all fired :p ). if i delete entries, tables, even the schema from the replica does it affect the source database ?
Thank in advance for your response.
Federico Campoli
@the4thdoctor
hi, I remember that a user hit some issues with the privileges. I agree that the write privileges shouldn't be necessary.
Achraf Elafrit
@afritfr
so does the change in the replica affect the source ?
Federico Campoli
@the4thdoctor
nope
the replica is mysql to postgres
I remember I got these settings from the python mysql replica library
Achraf Elafrit
@afritfr
Thank you.
Federico Campoli
@the4thdoctor
:thumbsup:
Bryan Pedini
@bryanpedini
Hello everyone.
Does anybody know if I can use Chameleon to migrate Gitea backend DB from MySQL to PgSQL?
Federico Campoli
@the4thdoctor
no idea. is gitea another git provider?
Bryan Pedini
@bryanpedini
yep, it's another gitlab-like web based git interface
but much more stable, easy upgradable, maintenance-free and painless (not to mention, lighter a f)
and as mentioned with Gitea, postgres (opposite of my current MySQL solution) is lighter, faster, easier (and generally better)
Federico Campoli
@the4thdoctor
cool. I don't see why shouldn't be possible to migrate. unless gitea use specific design hacks diverging from the postgresql logic. for example back in the days I wrote a script to migrate explicitly phpbb forums from mysql to postgresql that required a particular naming convention for the objects
Bryan Pedini
@bryanpedini
I could try to migrate the database, change the connection configuration on Gitea files, try to run it and see if everything works fine...
Federico Campoli
@the4thdoctor
:thumbsup:
Bryan Pedini
@bryanpedini
@the4thdoctor have you included examples (or better off, sqldump to sqldump conversions) in the documentation?
I wouldn't like to install PIP nor virtualenv nor a python module on the production server, and don't have wish enough to waste at least one hour to setup two (mysql&pgsql) docker containers to make the migration...
I would like to migrate from dump file to dump file if possible, then re-import the "converted" dump into postgres..
Federico Campoli
@the4thdoctor
you can setup a third machine for running pgchameleon only
and this is an example on how to migrate https://pgdba.org/post/migration/
no need for sql dump
running a migration from dump file is not matter of pgchameleon. there is a wiki page on postgresql dedicated to the several programs available https://wiki.postgresql.org/wiki/Converting_from_other_Databases_to_PostgreSQL#MySQL
Bryan Pedini
@bryanpedini
I see, pg_chameleon does live migrations only...
Federico Campoli
@the4thdoctor
same as pgloader. the dump manipulation have a lot of drawbacks and is not suitable if your db is huge
Bryan Pedini
@bryanpedini
but it allows you to work "offline" (from your production server)
and it's quicker to download and re-upload a migrated dump than to setup a third (virtual) machine
(btw dumps for git interfaces usually contain only repo informations and users, not the git object themselves, so not to much of a big deal in terms of space..)
I will try to setup a VM to perform the migration (also because I trust more an open source python code written by a compatriot of mine, than some random foreign dotnet dudad technology provided AS-IS with no source code :P)
Federico Campoli
@the4thdoctor
thanks. anyway if you can afford offline work I suggest you to use pgloader. the tool is migration oriented and the author, dimitri is a good friend and a highly respected person in the postgresql community
Bryan Pedini
@bryanpedini
will try. Thanks for your help and suggestions Federico.
Federico Campoli
@the4thdoctor
you are welcome :)
jovankricka
@jovankricka

Hi all,
I am trying to setup replication with pg_chameleon but keep getting this error:

2019-12-12 14:30:56 MainProcess CRITICAL mysql_lib.py (1400): init replica for source mysql failed
Traceback (most recent call last):
File "/Users/jovankricka/evb/venv/lib/python3.7/site-packages/pg_chameleon/lib/pg_lib.py", line 2743, in get_data_type
type_override = self.type_override[column["column_type"]]
KeyError: 'point'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/Users/jovankricka/evb/venv/bin/chameleon", line 5, in <module>
exec(compile(open(file).read(), file, 'exec'))
File "/Users/jovankricka/evb/venv/bin/chameleon.py", line 58, in <module>
getattr(replica, args.command)()
File "/Users/jovankricka/evb/venv/lib/python3.7/site-packages/pg_chameleon/lib/global_lib.py", line 328, in init_replica
self.init_mysql_replica()
File "/Users/jovankricka/evb/venv/lib/python3.7/site-packages/pg_chameleon/lib/global_lib.py", line 338, in
init_mysql_replica
self.mysql_source.init_replica()
File "/Users/jovankricka/evb/venv/lib/python3.7/site-packages/pg_chameleon/lib/mysql_lib.py", line 1379, in init_replica
self.create_destination_tables()
File "/Users/jovankricka/evb/venv/lib/python3.7/site-packages/pg_chameleon/lib/mysql_lib.py", line 394, in create_destination_tables
self.pg_engine.create_table(table_metadata, table, schema, 'mysql')
File "/Users/jovankricka/evb/venv/lib/python3.7/site-packages/pg_chameleon/lib/pg_lib.py", line 2972, in create_table
table_ddl = self.build_create_table_mysql( table_metadata,table_name, schema)
File "/Users/jovankricka/evb/venv/lib/python3.7/site-packages/pg_chameleon/lib/pg_lib.py", line 2382, in
build_create_table_mysql
column_type = self.get_data_type(column, schema, table_name)
File "/Users/jovankricka/evb/venv/lib/python3.7/site-packages/pg_chameleon/lib/pg_lib.py", line 2751, in get_data_type
column_type = self.type_dictionary[column["data_type"]]
KeyError: 'point'

Anyone stumbled upon this one before?
Federico Campoli
@the4thdoctor
mmmm may be a point data type not handled? can you post please the table definition?
jovankricka
@jovankricka
Found one table with POINT column. Could be the culprit, has this column:
COORDINATES point not null,
So MySQL POINT data type is not handled by pg_chameleon?
Is there a plan to support POINT?
jovankricka
@jovankricka
@the4thdoctor ?

Ok this fixed it:

type_override:
"point":
override_to: point
override_tables:

  - "*"
Now I got this error: "AttributeError: 'NoneType' object has no attribute 'settimeout'"
jovankricka
@jovankricka
Full error:
2019-12-12 16:01:54 MainProcess CRITICAL mysql_lib.py (1400): init replica for source mysql failed
Traceback (most recent call last):
File "/Users/jovankricka/evb/venv/bin/chameleon", line 5, in <module>
exec(compile(open(file).read(), file, 'exec'))
File "/Users/jovankricka/evb/venv/bin/chameleon.py", line 58, in <module>
getattr(replica, args.command)()
File "/Users/jovankricka/evb/venv/lib/python3.7/site-packages/pg_chameleon/lib/global_lib.py", line 328, in init_replica
self.init_mysql_replica()
File "/Users/jovankricka/evb/venv/lib/python3.7/site-packages/pg_chameleon/lib/global_lib.py", line 338, in
init_mysql_replica
self.mysql_source.init_replica()
File "/Users/jovankricka/evb/venv/lib/python3.7/site-packages/pg_chameleon/lib/mysql_lib.py", line 1381, in init_replica
self.copy_tables()
File "/Users/jovankricka/evb/venv/lib/python3.7/site-packages/pg_chameleon/lib/mysql_lib.py", line 699, in
copy_tables
master_status = self.copy_data(schema, table)
File "/Users/jovankricka/evb/venv/lib/python3.7/site-packages/pg_chameleon/lib/mysql_lib.py", line 552, in copy_data
csv_results = self.cursor_unbuffered.fetchmany(copy_limit)
File "/Users/jovankricka/evb/venv/lib/python3.7/site-packages/pymysql/cursors.py", line 503, in fetchmany
row = self.read_next()
File "/Users/jovankricka/evb/venv/lib/python3.7/site-packages/pymysql/cursors.py", line 464, in read_next
return self._conv_row(self._result._read_rowdata_packet_unbuffered())
File "/Users/jovankricka/evb/venv/lib/python3.7/site-packages/pymysql/connections.py", line 1167, in _read_rowdata_packet_unbuffered
row = self._read_row_from_packet(packet)
File "/Users/jovankricka/evb/venv/lib/python3.7/site-packages/pymysql/connections.py", line 1206, in _read_row_from_packet
data = data.decode(encoding)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe6 in position 207: invalid continuation byte
Exception ignored in: <function SSCursor.close at 0x105d2a7b8>
Traceback (most recent call last):
File "/Users/jovankricka/evb/venv/lib/python3.7/site-packages/pymysql/cursors.py", line 441, in close
File "/Users/jovankricka/evb/venv/lib/python3.7/site-packages/pymysql/connections.py", line 1177, in _finish_unbuffered_query
File "/Users/jovankricka/evb/venv/lib/python3.7/site-packages/pymysql/connections.py", line 657, in _read_packet
File "/Users/jovankricka/evb/venv/lib/python3.7/site-packages/pymysql/connections.py", line 688, in _read_bytes
AttributeError: 'NoneType' object has no attribute 'settimeout'
Exception ignored in: <function MySQLResult.__del__ at 0x106119f28>
Traceback (most recent call last):
File "/Users/jovankricka/evb/venv/lib/python3.7/site-packages/pymysql/connections.py", line 1071, in del
File "/Users/jovankricka/evb/venv/lib/python3.7/site-packages/pymysql/connections.py", line 1177, in _finish_unbuffered_query
File "/Users/jovankricka/evb/venv/lib/python3.7/site-packages/pymysql/connections.py", line 657, in _read_packet
File "/Users/jovankricka/evb/venv/lib/python3.7/site-packages/pymysql/connections.py", line 688, in _read_bytes
AttributeError: 'NoneType' object has no attribute 'settimeout'
jovankricka
@jovankricka
Changed charset on my MySQL source to latin1 and it worked. Although charset set on my MySQL instance is utf8... strange..
jovankricka
@jovankricka

Now I initialized replica successfuly, this is the status after initialization:
Source id Source name Type Status Consistent Read lag Last read Replay lag Last replay


      1  mysql          mysql   initialised  No            N/A                      N/A

== Schema mappings ==
Origin schema Destination schema


database database

== Replica status ==


Tables not replicated 0
Tables replicated 41
All tables 41
Last maintenance N/A
Next maintenance N/A
Replayed rows
Replayed DDL
Skipped rows


, but when I start replica and show status after that I get this:

Source id Source name Type Status Consistent Read lag Last read Replay lag Last replay


      1  mysql          mysql   error     No            N/A                      N/A

== Schema mappings ==
Origin schema Destination schema


database database

== Replica status ==


Tables not replicated 0
Tables replicated 41
All tables 41
Last maintenance N/A
Next maintenance N/A
Replayed rows
Replayed DDL
Skipped rows


chameleon show_errors gives just "There are no errors in the log"
How to debug from this point?
jovankricka
@jovankricka
As I can see all tables were copied, but when I insert record in MySQL it does not come up in Postgres so replication does not work at this point. But not sure how to debug why is it not working
jovankricka
@jovankricka
In my MySQL logs I see:
"Aborted connection to db: '...' user: '...' host: '...' (Got an error reading communication packets)"