by

Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
    Luke Snape
    @lsnape
    Thanks @dimitri! I'll raise a PR.
    Tim Mendez
    @timendez
    Hey gang, if I have a Postgres database, and a similar looking database in MySQL, can I use this tool to plop the contents of the MySQL database in the Postgres one?
    Dimitri Fontaine
    @dimitri
    Sure, try the --with "data only" parameter.
    louiz’
    @louiz
    dimitri/pgloader#556 I’m running into this issue. It’s weird though, because I have pgloader version "3.4.1", wich is supposed to contain the fix…
    My sqlite3 column is “id INTEGER PRIMARY KEY AUTOINCREMENT,” and it’s converted into “id | bigint | not null”, and not serial
    (with or without the --cast 'type int with extra auto_increment to serial' option)
    Dimitri Fontaine
    @dimitri
    Hi @louiz , it seems like your auto_increment property is misspelled...
    louiz’
    @louiz
    With or without the underscore, you mean?
    Joonas Koivunen
    @koivunej

    just tried to use pgloader from mssql:// into postgresql:// by compiling in debian-testing (hit issue #330) and with latest docker image the output is:

    CORRUPTION WARNING in SBCL pid 1(tid 139666838079232):
    Memory fault at 0 (pc=0x7f06b75d6c3a, sp=0x7f06b7106e58)
    The integrity of this image is possibly compromised.
    Continuing with fingers crossed.
    KABOOM!
    FATAL error: Failed to connect to mssql at "192.168.10.200" (port 1433) as user "XXXX": Unhandled memory fault at #x0.

    the part related to #330 is the alien sslv3 api usage.

    Dimitri Fontaine
    @dimitri
    did you compile from GitHub sources, freshly cloned?
    Joonas Koivunen
    @koivunej
    @dimitri yes, both the master and latest tag v3.4.1
    @dimitri in latest debian testing there are both libssl1.0.2 (a lot of deps) and libssl1.1 (python3) installed so that might be the source of confusion (alien sslv3 api) ... I remember finding some discussion on ironclad (or it's debian package) about the alien sslv3 api and it was a fixed issue. however I can't see how to get the build to use system installed cl-ironclad (and I know nothing of the build chain)
    Joonas Koivunen
    @koivunej
    found the actual output for "alien sslv3 api":
    FATAL error: Undefined alien: "SSLv3_client_method"
    An unhandled error condition has been signalled:
       Undefined alien: "SSLv3_client_method"
    Dimitri Fontaine
    @dimitri
    I wish I knew how to handle those kind of errors better, or at all really, but the linking in between the Lisp host system (SBCL here) and the C-world .so libraries is still something I don't know how to properly handle. My understanding is that the problem occurs because the lisp image is compiled with symbols gravelled from one version of the lib, and then at runtime the linker provides another version of the lib. Do you compile and run on the same system?
    Joonas Koivunen
    @koivunej
    @dimitri yes, on this same pristine debian testing virtual machine i created yesterday
    Luke Snape
    @lsnape

    Some of our daily pgloader jobs have started failing recently. Rerunning them again with the debug flag set to true, I get the following output before the job hangs:

    debugger invoked on a SB-FORMAT:FORMAT-ERROR in thread
    #<THREAD "lparallel" RUNNING {1009158173}>:
      error in FORMAT: no more arguments
      copy-batch[~a] ~a ~d row~:p [~a] in ~6$s~@[ [oversized]~]
                      ^
    while processing indirect format string:
      ~{~}
       ^
    pgloader version "3.4.1"
    compiled with SBCL 1.2.9-1.el7.centos

    If I rerun the error reappears at a different time, but always early on in the job.

    We run 5 pgloader jobs each day, copying data from 5 RDS databases into postgres. It's been running for months and been quite reliable up until now. The only thing that's changed is the load size increase.
    Dimitri Fontaine
    @dimitri
    @lsnape consider building pgloader from current sources, there has been quite some improvements recently (stability, bug fixes, speed) ; and also tell your boss to help sustain the project financially thanks to the Moral License / Patron program, available at https://pgloader.io/moral-licence/
    @lsnape the bug you mention in particular has been fixed, I believe. It's a format string that doesn't get all its argument at run time, which is somewhat strange, but should no longer be the case
    Luke Snape
    @lsnape
    Thanks @dimitri! I will try that. And yes, I will speak to my boss about supporting the work that you're doing. I doubt we'd be able to become a Patron as we're quite a small team, but we may be able to make some sort of contribution. Is there a way for us to make a one-off donation to you?
    Dimitri Fontaine
    @dimitri
    Sure! either sign for a monthly payment then cancel your subscription, or see https://gumroad.com/l/pgloader for a one-off payment covering a Moral Licence (you can buy as many as you want/need of course, and account for it as a classic/usual software licence cost)
    Tim Pierson
    @o1lo01ol1o
    @dimitri Is there a way to pass the text of a loadfile as a command-line argument? I need to use the docker image to migrate some databases on windows but I'm having trouble dealing with the docker cli indirection.
    Tim Pierson
    @o1lo01ol1o
    nvm, passed that hurdle. now getting
    FATAL error: Undefined alien: "SSLv3_client_method"
    An unhandled error condition has been signalled:
       Undefined alien: "SSLv3_client_method"
    Dimitri Fontaine
    @dimitri
    hi @o1lo01ol1o ! good that you're on to the next error. This SSLv3 problem should be avoided in the docker image, it's usually a discrepancy in between build time and run time selection of the SSL library by the OS, so that's strange. It should be possible to build a native pgloader.exe on windows, mind you.
    Petr Osetrov
    @ParadoxFilm
    guys, please help me! #765
    Dimitri Fontaine
    @dimitri
    I think you need fields escaped by '\'' in your case, can you test with that? backslash-quote is actually \".
    Petr Osetrov
    @ParadoxFilm
    It seems like it helped, thank you very much!
    Benjamin David
    @benmodd_twitter
    Has anybody ever had the error The value NIL is not of type PGLOADER.CATALOG:COLUMN? I'm trying to migrate from an SQLite database to a postgres and consistently run into this error. I tried with both the latest development version and the latest stable version and get the same error. The SQLite file doesnt seem corrupt and I rebuilt it and the problem remained. Any ideas?
    Dimitri Fontaine
    @dimitri
    Hi @benmodd_twitter ; please share a --debug stack trace, and if you can, even better, send me the SQLite database file at my email address (see my Github profile)!
    Julien Bouquillon
    @revolunet
    hello :hand:
    Dimitri Fontaine
    @dimitri
    Hi @revolunet !
    Anton Kulaga
    @antonkulaga
    Hello. I have a question: how to setup heap for pgloader?
    I get:
    Heap exhausted during garbage collection: 80 bytes available, 448 requested.
     Gen StaPg UbSta LaSta LUbSt Boxed Unboxed LB   LUB  !move  Alloc  Waste   Trig    WP  GCs Mem-age
       0: 99868     0 121100     0  6780  2023     0     0     7 280578176 7878528 108952008    0   1  0,0000
       1: 129702 131071     0     0   296  7315    16     0     7 242074016 7847520 236408680  180   1  0,7992
       2: 68068 127740     0     0   932 27071    31     0    27 882164496 36453616 554860888  934   1  0,5803
       3: 20274 107904     0     0  1558 47997    28     0    25 1564283776 60451968 42949672 1555   0  0,5538
       4: 124721 124715     0     0  1068 31141   188     0    84 1007615136 53969760  2000000 1149   0  0,0000
       5:     0     0     0     0     0     0     0     0     0        0     0  2000000    0   0  0,0000
       6:     0     0     0     0  3437  1191     0     0     0 151650304     0  2000000 3303   0  0,0000
       Total bytes allocated    = 4128365904
       Dynamic-space-size bytes = 4294967296
    GC control variables:
       *GC-INHIBIT* = true
       *GC-PENDING* = true
       *STOP-FOR-GC-PENDING* = false
    fatal error encountered in SBCL pid 17889(tid 140737022129920):
    Heap exhausted, game over.
    Dimitri Fontaine
    @dimitri
    it's a compile time option, usually the best move here is reducing the batch size parameters, see http://pgloader.readthedocs.io/en/latest/pgloader.html#with and “prefetch rows” in particular.
    Anton Kulaga
    @antonkulaga
    I have 32GB RAM and according to top command it uses very small portion of it
    Anton Kulaga
    @antonkulaga
    The database I am trying to convert is http://starbuck1.s3.amazonaws.com/sradb/GEOmetadb.sqlite.gz (around 8GB when ungziped)
    Anton Kulaga
    @antonkulaga
    Looks like changing batch parameters helped. Even though even when it crashed with heap error it never used more than 10% of RAM, according to top
    I also wonder what is HEAP for list programs, is it avaliable memory, memory allocated with enviroment variables (like stupid Java's -Xmx) or something else?
    brendan susens-jackson
    @bsusensjackson
    Hello @dimitri, thanks for this great tool
    I left a comment on dimitri/pgloader#812, I am trying to do something very similar to the OP
    please let me know if you need additional information. Thanks!
    Hendi Santika
    @hendisantika
    Very Sorry...
    I am using RHEL 7.
    I am still having issue to start with pgloader. How do I know my pgloader installation is success and how to start loading txt file using pgloader?
    I can't find any guidance for basic user like me.
    Any one please help me.
    Dimitri Fontaine
    @dimitri
    Hi @hendisantika ; what have your tried? things like pgloader --version and pgloader --help might help you get started.
    Hendi Santika
    @hendisantika

    Hi @dimitri , I was installing pgloader using bootstrap-centos7.sh.
    But after installation completed, I was trying pgloader --version my RHEL said pgloader : command not found.

    What should I do then?

    Thanks

    shashank
    @shashankn91
    I want to make a data ware house using postgres . I want to refresh data regularly (12hrs or daily) . If I use pgloader to do so will it be incremental copy or it does delete older data and refresh it with new data ? I think it falls under CI/CD @dimitri white paper.
    Mario Delgado
    @peachepe
    Hi! What's the recommended environment right now for compiling and running pgloader? I'm unable to run it on macos. Compiled from source and get the Control stack exhausted (no more space for function call frames). that is reported in the issues.
    Also I just compiled from 3.5.1 source and I get the same error :/
    Mario Delgado
    @peachepe
    My database has 701 tables, are there any settings I have to use for a DB that big?