Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
    Dimitri Fontaine
    @dimitri
    thanks, my pleasure!
    rocas82
    @rocas82
    Hello Dimitri,
    During data migration from mysql to postgresql database. I am had the following error message:
    Database error 22001: value too long for type character varying(5000)
    2 CONTEXT: COPY mass_treat_logs, line 2860, column logtext: "Opration UP1TR023-60707-1397462564755 : Quick Feature : lment 1613 : termin en Erreur : Des ..."
    3 Database error 22001: value too long for type character varying(5000)
    4 CONTEXT: COPY mass_treat_logs, line 769, column logtext: "Opration UP1TR023-60707-1397462564755 : Quick Feature : lment 2208 : termin en Erreur : Des ..."
    5 Database error 22001: value too long for type character varying(5000)
    6 CONTEXT: COPY mass_treat_logs, line 586, column logtext: "Opration UP1TR023-60707-1397462564755 : Quick Feature : lment 2534 : termin en Erreur : Des ..."
    7 Database error 22001: value too long for type character varying(5000)
    8 CONTEXT: COPY mass_treat_logs, line 3023, column logtext: "Opration UP1TR023-60707-1397462564755 : Quick Feature : lment 5123 : termin en Erreur : Des ..."
    9 Database error 22001: value too long for type character varying(5000)
    10 CONTEXT: COPY mass_treat_logs, line 13, column logtext: "Opration UP1TR023-60707-1397462564755 : Quick Feature : lment 5136 : termin en Erreur : Des ..."
    11 Database error 22001: value too long for type character varying(5000)
    12 CONTEXT: COPY mass_treat_logs, line 90, column logtext: "Opration UP1TR023-60707-1397462564755 : Quick Feature : lment 5226 : termin en Erreur : Des ..."
    13 Database error 22001: value too long for type character varying(5000)
    14 CONTEXT: COPY mass_treat_logs, line 272, column logtext: "Opration UP1TR023-60707-1397462564755 : Quick Feature : lment 5498 : termin en Erreur : Des ..."
    15 Database error 22001: value too long for type character varying(5000)
    16 CONTEXT: COPY mass_treat_logs, line 50, column logtext: "Opration UP1TR023-60707-1397462564755 : Quick Feature : lment 5548 : termin en Erreur : Des ..."
    17 Database error 22001: value too long for type character v 16:25
    in one of my tables there is a column " logText varchar(5000)". Is there any restriction on size of varchar with pgloader ? Thanks for your feedback.
    Dimitri Fontaine
    @dimitri
    please open an issue, the problem seems to be with PostgreSQL, so I need to know if the table definition has been ported over by pgloader of if it's filling-up a pre-existing target?
    rocas82
    @rocas82
    I opened the issue #469 : dimitri/pgloader#469
    Dimitri Fontaine
    @dimitri
    answered on the issue and closed it, thanks!
    rocas82
    @rocas82
    Hello Dimitri
    regarding the issue #469, I posted some additional questions in the ticket. Could you have a look please ?
    Tengfei Niu
    @spartucus

    @dimitri Hello Dimitri, from the site

    Building for the Windows™ Operating System is easy enough and the platform is fully supported.

    and I have no knowledges of Lisp, how do i start build pgloader on Windows?
    Andrew Graham-Yooll
    @andrewgy8
    The readme says "Build error". I see the .spec file was updated 8 days ago. Is this representative of the project or is the project fine and this warning is erroneous for now?
    Dimitri Fontaine
    @dimitri
    The build error is Travis integration, the .travis.yaml file endured bitrot when Travis changed things in their hosting solution months ago and I never had time to properly handle that situation. Help welcome here!
    Johannes Visintini
    @joker234
    hey there. I want to move a ruby on rails production database from sqlite to postgres. I think your tool might be good for this. My problem is, that rails have to create the database schema to work properly with the new database. Can I use pgloader to get only the data
    so in SQL i would only want to have just the INSERT INTO statements
    I used “WITH data only”, but that also changed my tables;
    the indexes moved away after using pgloader
    Johannes Visintini
    @joker234
    Ah. Is it correct, that my problem should be solved by dimitri/pgloader@1023577 resp. dimitri/pgloader#504
    Andrew Graham-Yooll
    @andrewgy8
    @dimitri Sorryfor the late reply. I would help with that issue, although Im not sure what the fix would be. Perhaps a rewriting of the yml file?
    I could try something out this weekend...
    Andrew Graham-Yooll
    @andrewgy8

    Anyways, I have a question. Im sucessfully porting over numerous sqlite dbs to postgres and pgloader is doing a great job! However, Im running into a couple of errors. One is that that some of my sqlite tables dont have a unique id which messes up the addition of foreign keys into the postgres db. The error:
    Database error 42830: there is no unique constraint matching given keys for referenced table "shop"

    So I ran this in my load file:

    AFTER LOAD DO 
        $$ ALTER TABLE shop$ ADD UNIQUE (id);$$
    However, it didnt seem to fire that query and add the unique to the id column.
    Any thoughts?
    Dimitri Fontaine
    @dimitri
    can you add the same thing in the SQLite database so that it's converted from sources in the right shape?
    the after load do clause will only run after everything that pgloader does internally, there's no other hooks to run at intermediary stages yet (well, apart from the alter schema and alter table pgloader clauses that impact the catalogs used to migrate)
    Andrew Graham-Yooll
    @andrewgy8
    Yeah, thats what I was going to go for if those hooks didnt fire until after the pgloader processes.
    Another issue Im running into is using the type int with extra auto_increment to serial. It does not seem to add that modification to the postgres tables from the sqlite3 tables that have autoincrement.
    Andrew Graham-Yooll
    @andrewgy8
    This is my current load file which doesnt seem to convert auto-incremnt to serial:
    
    LOAD DATABASE
         from ../../data/databases/main_db.sqlite
         into postgresql://postgres:postgres@127.0.0.1:5432/main_db
    
    WITH include drop, 
        create tables, 
        create indexes, 
        reset sequences
    
    CAST type datetime to timestamp drop default drop not null using zero-dates-to-null, 
    type int with extra auto_increment to serial keep typemod keep default keep not null 
    
    SET work_mem to '16MB', 
        maintenance_work_mem to '512 MB';
    Dimitri Fontaine
    @dimitri
    @andrewgy8 please see dimitri/pgloader#556, should be fixed now
    Andrew Graham-Yooll
    @andrewgy8
    Fantastic @dimitri !
    Ill look into the timezone issue. Time is such a strange thing...
    Marcel Hoppe
    @hobbypunk90

    Hello,
    i habe a little problem, i'm using the 3.1 version an a debian machine with this little script:

    LOAD DATABASE
         FROM    mysql://<user>:<password>@localhost/gildamesh_test_db
         INTO postgres:///gildamesh_test
     WITH data only, truncate;

    and i get this very strange error:
    Error on Github

    Dimitri Fontaine
    @dimitri
    3.1 is old now, please upgrade to latest Debian provided package (3.3.2, see https://packages.debian.org/search?keywords=pgloader) or build from current sources ; then try again.
    Patrik Lermon
    @kler
    Is it possible to handle empty dates when converting from dbase files?
    I get the following error: Database error 22007: invalid input syntax for type date: " - - "
    and it seems like casting is not compatible using dbase as source.
    Dimitri Fontaine
    @dimitri
    Is it possible for you to send me the dbase file so that I can debug pgloader with that input?
    Patrik Lermon
    @kler
    Sorry, I can't do that. But basically the date is empty, which I believe is interpreted as the empty string "", while I believe it should be interpreted as null.
    Dimitri Fontaine
    @dimitri
    Can you open an issue on github I'm going to have a look at the situation, I might be able to blind-code a patch for you there, but it would be much easier with a test case
    Dimitri Fontaine
    @dimitri
    ok as I was looking here, I tried something: dimitri/pgloader#589
    Patrik Lermon
    @kler
    I think it's as easy to reproduce to have a dbf database containing dates which are empty (""). I saw you opened and also closed the issue.
    Dimitri Fontaine
    @dimitri
    It sounds easy to reproduce, but how exactly do you generate a dbf file? My guess is that an empty date will not show up the same depending on the tooling used to produce the file, as is often the case... anyway, let me know if the defensive approach I tried solve your issue!
    Patrik Lermon
    @kler
    Hmm... Problem is that I don't program lisp, so I can't really tell.
    And also thx for the effort!
    Dimitri Fontaine
    @dimitri
    Well all you need to do now is to build pgloader from sources, try again, and report success or failure...
    Patrik Lermon
    @kler
    I tested it now, and it seems to have fixed the issue, but I cannot fully test it, because I ran into the next one... :)
    Same error, but for type numeric:
    2017-07-03T07:17:33.101000Z ERROR Database error 22P02: invalid input syntax for type numeric: " "
    CONTEXT: COPY per, line 1, column timezone: " "
    Dimitri Fontaine
    @dimitri
    Can you reopen issue #589 on GitHub with the details and if possible a test case? The dbf numeric field is supposed to host either integers or floats, but I don't know what float look like in this format... Also why do you have a single blank in a Numeric column? I can again blind code something but would really appreciate a test case (that I don't know how to produce).
    Patrik Lermon
    @kler
    I would love to help, but I can't share the database and I don't know how it's created :/
    So unfortunately I see only two options, either you push another fix "blindly" and I'll test it or else I believe we need to leave the issue.
    I solved my problem by writing my own convert script, but if we could improve pgloader that would be great.
    Dimitri Fontaine
    @dimitri
    Ok I see, will push a blind patch
    Dimitri Fontaine
    @dimitri
    pushed now
    Patrik Lermon
    @kler
    Moved discussion over to bug - seems more legit :)