by

Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
    Patrik Lermon
    @kler
    Hmm... Problem is that I don't program lisp, so I can't really tell.
    And also thx for the effort!
    Dimitri Fontaine
    @dimitri
    Well all you need to do now is to build pgloader from sources, try again, and report success or failure...
    Patrik Lermon
    @kler
    I tested it now, and it seems to have fixed the issue, but I cannot fully test it, because I ran into the next one... :)
    Same error, but for type numeric:
    2017-07-03T07:17:33.101000Z ERROR Database error 22P02: invalid input syntax for type numeric: " "
    CONTEXT: COPY per, line 1, column timezone: " "
    Dimitri Fontaine
    @dimitri
    Can you reopen issue #589 on GitHub with the details and if possible a test case? The dbf numeric field is supposed to host either integers or floats, but I don't know what float look like in this format... Also why do you have a single blank in a Numeric column? I can again blind code something but would really appreciate a test case (that I don't know how to produce).
    Patrik Lermon
    @kler
    I would love to help, but I can't share the database and I don't know how it's created :/
    So unfortunately I see only two options, either you push another fix "blindly" and I'll test it or else I believe we need to leave the issue.
    I solved my problem by writing my own convert script, but if we could improve pgloader that would be great.
    Dimitri Fontaine
    @dimitri
    Ok I see, will push a blind patch
    Dimitri Fontaine
    @dimitri
    pushed now
    Patrik Lermon
    @kler
    Moved discussion over to bug - seems more legit :)
    (and your code is tested btw)
    Dimitri Fontaine
    @dimitri
    Thanks! That way other users withe the same kind of problems will easily find what we did to solve it...
    Irio Musskopf
    @Irio
    Hello, been struggling for a few minutes trying to use the app for migrating a large MySQL database into PostgreSQL. Maybe could get other eyes to help me? 😬
    Tried a few variations of the command, but the output is always the same. I assume I'm missing something obvious...
    Dimitri Fontaine
    @dimitri
    Hi, yes it's pretty obvious. It's missing some error messages tho, which version are you using?
    Your problem is that the PostgreSQL db URI is malformed, it has postgresql:///, and three slashes mean you didn't provide username, password, host, port. The part after the third slash is expected to be the database name. Just as in the MySQL connection string, actually.
    Brian Jensen
    @bjensen
    I am still getting the pgloader failed to find schema in target catalog with the newest build from head..wasnt it suppose to be fixed? in a migration from mysql to 9.6.3 and WITH data only, truncate
    @dimitri :-)
    Brian Jensen
    @bjensen
    nm
    Luke Snape
    @lsnape

    Hello, I'm trying to find a way of casting a mysqlbinary(36)to a postgres uuid type, or, failing that, a varchar(36). I'm frequently getting the error message:

    2017-08-04T16:17:33.884000Z ERROR The value
      #(53 55 101 50 55 49 99 51 45 55 48 98 56 45 52 55 55 52 45 57 97 98 50 45 55
        52 50 50 55 98 57 101 55 56 100 97)
    is not of type
      STRING

    Does anyone know a way of casting the above using the options provided with pgloader, or do I have to write my own using lisp function?

    Luke Snape
    @lsnape

    OK, I got it working with the varbinary-to-string function, but I get lots of these errors for nullable fields:

    2017-08-05T08:15:45.427000Z ERROR NIL fell through ETYPECASE expression. Wanted one of (STRING VECTOR).

    Should varbinary-to-string handle nulls gracefully?

    Dimitri Fontaine
    @dimitri
    @lsnape yeah it can be argued it should, please open an issue at github and I'll fix it!
    @bjensen yes it's supposed to be fixed, do you have a test case for me to reproduce the bug?
    @lsnape or even see about making a Pull Request, it's a very good introduction topic for hacking on pgloader -- see https://github.com/dimitri/pgloader/blob/master/src/utils/transforms.lisp with plenty of examples in there
    Luke Snape
    @lsnape
    Thanks @dimitri! I'll raise a PR.
    Tim Mendez
    @timendez
    Hey gang, if I have a Postgres database, and a similar looking database in MySQL, can I use this tool to plop the contents of the MySQL database in the Postgres one?
    Dimitri Fontaine
    @dimitri
    Sure, try the --with "data only" parameter.
    louiz’
    @louiz
    dimitri/pgloader#556 I’m running into this issue. It’s weird though, because I have pgloader version "3.4.1", wich is supposed to contain the fix…
    My sqlite3 column is “id INTEGER PRIMARY KEY AUTOINCREMENT,” and it’s converted into “id | bigint | not null”, and not serial
    (with or without the --cast 'type int with extra auto_increment to serial' option)
    Dimitri Fontaine
    @dimitri
    Hi @louiz , it seems like your auto_increment property is misspelled...
    louiz’
    @louiz
    With or without the underscore, you mean?
    Joonas Koivunen
    @koivunej

    just tried to use pgloader from mssql:// into postgresql:// by compiling in debian-testing (hit issue #330) and with latest docker image the output is:

    CORRUPTION WARNING in SBCL pid 1(tid 139666838079232):
    Memory fault at 0 (pc=0x7f06b75d6c3a, sp=0x7f06b7106e58)
    The integrity of this image is possibly compromised.
    Continuing with fingers crossed.
    KABOOM!
    FATAL error: Failed to connect to mssql at "192.168.10.200" (port 1433) as user "XXXX": Unhandled memory fault at #x0.

    the part related to #330 is the alien sslv3 api usage.

    Dimitri Fontaine
    @dimitri
    did you compile from GitHub sources, freshly cloned?
    Joonas Koivunen
    @koivunej
    @dimitri yes, both the master and latest tag v3.4.1
    @dimitri in latest debian testing there are both libssl1.0.2 (a lot of deps) and libssl1.1 (python3) installed so that might be the source of confusion (alien sslv3 api) ... I remember finding some discussion on ironclad (or it's debian package) about the alien sslv3 api and it was a fixed issue. however I can't see how to get the build to use system installed cl-ironclad (and I know nothing of the build chain)
    Joonas Koivunen
    @koivunej
    found the actual output for "alien sslv3 api":
    FATAL error: Undefined alien: "SSLv3_client_method"
    An unhandled error condition has been signalled:
       Undefined alien: "SSLv3_client_method"
    Dimitri Fontaine
    @dimitri
    I wish I knew how to handle those kind of errors better, or at all really, but the linking in between the Lisp host system (SBCL here) and the C-world .so libraries is still something I don't know how to properly handle. My understanding is that the problem occurs because the lisp image is compiled with symbols gravelled from one version of the lib, and then at runtime the linker provides another version of the lib. Do you compile and run on the same system?
    Joonas Koivunen
    @koivunej
    @dimitri yes, on this same pristine debian testing virtual machine i created yesterday
    Luke Snape
    @lsnape

    Some of our daily pgloader jobs have started failing recently. Rerunning them again with the debug flag set to true, I get the following output before the job hangs:

    debugger invoked on a SB-FORMAT:FORMAT-ERROR in thread
    #<THREAD "lparallel" RUNNING {1009158173}>:
      error in FORMAT: no more arguments
      copy-batch[~a] ~a ~d row~:p [~a] in ~6$s~@[ [oversized]~]
                      ^
    while processing indirect format string:
      ~{~}
       ^
    pgloader version "3.4.1"
    compiled with SBCL 1.2.9-1.el7.centos

    If I rerun the error reappears at a different time, but always early on in the job.

    We run 5 pgloader jobs each day, copying data from 5 RDS databases into postgres. It's been running for months and been quite reliable up until now. The only thing that's changed is the load size increase.
    Dimitri Fontaine
    @dimitri
    @lsnape consider building pgloader from current sources, there has been quite some improvements recently (stability, bug fixes, speed) ; and also tell your boss to help sustain the project financially thanks to the Moral License / Patron program, available at https://pgloader.io/moral-licence/
    @lsnape the bug you mention in particular has been fixed, I believe. It's a format string that doesn't get all its argument at run time, which is somewhat strange, but should no longer be the case
    Luke Snape
    @lsnape
    Thanks @dimitri! I will try that. And yes, I will speak to my boss about supporting the work that you're doing. I doubt we'd be able to become a Patron as we're quite a small team, but we may be able to make some sort of contribution. Is there a way for us to make a one-off donation to you?
    Dimitri Fontaine
    @dimitri
    Sure! either sign for a monthly payment then cancel your subscription, or see https://gumroad.com/l/pgloader for a one-off payment covering a Moral Licence (you can buy as many as you want/need of course, and account for it as a classic/usual software licence cost)
    Tim Pierson
    @o1lo01ol1o
    @dimitri Is there a way to pass the text of a loadfile as a command-line argument? I need to use the docker image to migrate some databases on windows but I'm having trouble dealing with the docker cli indirection.
    Tim Pierson
    @o1lo01ol1o
    nvm, passed that hurdle. now getting
    FATAL error: Undefined alien: "SSLv3_client_method"
    An unhandled error condition has been signalled:
       Undefined alien: "SSLv3_client_method"