Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
    Tim Pierson
    @o1lo01ol1o
    @dimitri Is there a way to pass the text of a loadfile as a command-line argument? I need to use the docker image to migrate some databases on windows but I'm having trouble dealing with the docker cli indirection.
    Tim Pierson
    @o1lo01ol1o
    nvm, passed that hurdle. now getting
    FATAL error: Undefined alien: "SSLv3_client_method"
    An unhandled error condition has been signalled:
       Undefined alien: "SSLv3_client_method"
    Dimitri Fontaine
    @dimitri
    hi @o1lo01ol1o ! good that you're on to the next error. This SSLv3 problem should be avoided in the docker image, it's usually a discrepancy in between build time and run time selection of the SSL library by the OS, so that's strange. It should be possible to build a native pgloader.exe on windows, mind you.
    Petr Osetrov
    @ParadoxFilm
    guys, please help me! #765
    Dimitri Fontaine
    @dimitri
    I think you need fields escaped by '\'' in your case, can you test with that? backslash-quote is actually \".
    Petr Osetrov
    @ParadoxFilm
    It seems like it helped, thank you very much!
    Benjamin David
    @benmodd_twitter
    Has anybody ever had the error The value NIL is not of type PGLOADER.CATALOG:COLUMN? I'm trying to migrate from an SQLite database to a postgres and consistently run into this error. I tried with both the latest development version and the latest stable version and get the same error. The SQLite file doesnt seem corrupt and I rebuilt it and the problem remained. Any ideas?
    Dimitri Fontaine
    @dimitri
    Hi @benmodd_twitter ; please share a --debug stack trace, and if you can, even better, send me the SQLite database file at my email address (see my Github profile)!
    Julien Bouquillon
    @revolunet
    hello :hand:
    Dimitri Fontaine
    @dimitri
    Hi @revolunet !
    Anton Kulaga
    @antonkulaga
    Hello. I have a question: how to setup heap for pgloader?
    I get:
    Heap exhausted during garbage collection: 80 bytes available, 448 requested.
     Gen StaPg UbSta LaSta LUbSt Boxed Unboxed LB   LUB  !move  Alloc  Waste   Trig    WP  GCs Mem-age
       0: 99868     0 121100     0  6780  2023     0     0     7 280578176 7878528 108952008    0   1  0,0000
       1: 129702 131071     0     0   296  7315    16     0     7 242074016 7847520 236408680  180   1  0,7992
       2: 68068 127740     0     0   932 27071    31     0    27 882164496 36453616 554860888  934   1  0,5803
       3: 20274 107904     0     0  1558 47997    28     0    25 1564283776 60451968 42949672 1555   0  0,5538
       4: 124721 124715     0     0  1068 31141   188     0    84 1007615136 53969760  2000000 1149   0  0,0000
       5:     0     0     0     0     0     0     0     0     0        0     0  2000000    0   0  0,0000
       6:     0     0     0     0  3437  1191     0     0     0 151650304     0  2000000 3303   0  0,0000
       Total bytes allocated    = 4128365904
       Dynamic-space-size bytes = 4294967296
    GC control variables:
       *GC-INHIBIT* = true
       *GC-PENDING* = true
       *STOP-FOR-GC-PENDING* = false
    fatal error encountered in SBCL pid 17889(tid 140737022129920):
    Heap exhausted, game over.
    Dimitri Fontaine
    @dimitri
    it's a compile time option, usually the best move here is reducing the batch size parameters, see http://pgloader.readthedocs.io/en/latest/pgloader.html#with and “prefetch rows” in particular.
    Anton Kulaga
    @antonkulaga
    I have 32GB RAM and according to top command it uses very small portion of it
    Anton Kulaga
    @antonkulaga
    The database I am trying to convert is http://starbuck1.s3.amazonaws.com/sradb/GEOmetadb.sqlite.gz (around 8GB when ungziped)
    Anton Kulaga
    @antonkulaga
    Looks like changing batch parameters helped. Even though even when it crashed with heap error it never used more than 10% of RAM, according to top
    I also wonder what is HEAP for list programs, is it avaliable memory, memory allocated with enviroment variables (like stupid Java's -Xmx) or something else?
    brendan susens-jackson
    @bsusensjackson
    Hello @dimitri, thanks for this great tool
    I left a comment on dimitri/pgloader#812, I am trying to do something very similar to the OP
    please let me know if you need additional information. Thanks!
    Hendi Santika
    @hendisantika
    Very Sorry...
    I am using RHEL 7.
    I am still having issue to start with pgloader. How do I know my pgloader installation is success and how to start loading txt file using pgloader?
    I can't find any guidance for basic user like me.
    Any one please help me.
    Dimitri Fontaine
    @dimitri
    Hi @hendisantika ; what have your tried? things like pgloader --version and pgloader --help might help you get started.
    Hendi Santika
    @hendisantika

    Hi @dimitri , I was installing pgloader using bootstrap-centos7.sh.
    But after installation completed, I was trying pgloader --version my RHEL said pgloader : command not found.

    What should I do then?

    Thanks

    shashank
    @shashankn91
    I want to make a data ware house using postgres . I want to refresh data regularly (12hrs or daily) . If I use pgloader to do so will it be incremental copy or it does delete older data and refresh it with new data ? I think it falls under CI/CD @dimitri white paper.
    Mario Delgado
    @peachepe
    Hi! What's the recommended environment right now for compiling and running pgloader? I'm unable to run it on macos. Compiled from source and get the Control stack exhausted (no more space for function call frames). that is reported in the issues.
    Also I just compiled from 3.5.1 source and I get the same error :/
    Mario Delgado
    @peachepe
    My database has 701 tables, are there any settings I have to use for a DB that big?
    Dimitri Fontaine
    @dimitri
    @shashankn91 you're right that in falls into the CI/CD idea, but refreshing data is not implemented in pgloader as of yet. Have a look at PostgreSQL logical decoding if that applies to your situation?
    @peachepe I think I've fixed the “Control stack exhausted” problem recently in dimitri/pgloader@8b1acba ; please try from fresh sources and make clean before building again?
    about 701 tables, the defaults should be good enough to let you get started, then you might want to review the parallelism and concurrency settings. Also run in VERBOSE mode so that you can follow along what's happening when trying it out.
    Supriya singh
    @supriya.sk_gitlab
    can i install pgloader in windows?
    Dimitri Fontaine
    @dimitri
    @supriya.sk_gitlab yes pgloader should work on windows, although it seldom get tested and might require slight adjustments
    Supriya singh
    @supriya.sk_gitlab
    how can i install pgloader in windows,can u send me any reference or the steps to install and connect with database
    Dimitri Fontaine
    @dimitri
    please review https://github.com/dimitri/pgloader/issues?utf8=✓&q=label%3A%22Windows+support%22+ ; some of them have extended install steps
    Alex V Gonchar
    @dezconnect
    Which version of pgloader correctly works with SQL Server 2012 ? I trying to use 3.5.2 version but got error when pgloader tried get info about columns from SQL Server
    Christoph Berg
    @df7cb
    @dezconnect It might help if you show the error message
    Alex V Gonchar
    @dezconnect
    it is looks like this: https://pastebin.com/ZgfWhbzb
    Alex V Gonchar
    @dezconnect
    KABOOM!
    FATAL error: %dbsqlexec fail
    Date/time: 2019-01-24-15:49An unhandled error condition has been signalled:
    %dbsqlexec fail
    at the end before CL stacktrace
    Dimitri Fontaine
    @dimitri
    well version 3.5.2 is a little old now, so please try 3.6.1 which contains specific improvements for MS SQL support ; also review your FreeTDS configuration, which is often the problem at hand (in particular, the protocol version setup in there is important) (see http://pgloader.org for a quick intro on the topic)
    Alex V Gonchar
    @dezconnect
    Ok thanks
    Reinhardt van Rooyen
    @rvanrooy

    Hi all. I'm trying to migrate a MSSQL db and I seem to be stuck on the datetime2 columns:

    KABOOM!
    FATAL error: not supported type SYB-DATETIME2

    error

    I'm using freetds-RC1.1 and pgloader from git master 3.6.213ebde.

    can anyone point me in the right direction?

    Dimitri Fontaine
    @dimitri
    @rvanrooy please hack https://github.com/dimitri/pgloader/blob/master/src/monkey/mssql.lisp#L98 and add :syb-msdatetime2 in the list there, there's a slight chance it will just work; please then report back
    if that's all it takes, you could then open a PR that I'd be happy to accept
    Reinhardt van Rooyen
    @rvanrooy
    thanks, I'll give it a go
    Dimitri Fontaine
    @dimitri
    make sure to double check the values migrated (if any) should that edit avoid the errors you're having now, it might be that we would be handling the data in a way that was not intended, I have no idea if the syb-msdatetime2 type works like the others or not, after all
    Reinhardt van Rooyen
    @rvanrooy
    @dimitri the process definitely gets further along, it correctly scans the target rows, but gives an error status: "invalid input syntax for type timestamp with time zone: "Aug 7 2014 12:00:00:0000000" I found a github issue #431 related to msdatetime4 and and will try to add those fixes in src/sources/mssql/mssql-schema.lisp
    Ok, that seemed to work, dates looks correct at a quick glance, I'll do some more checks.
    Reinhardt van Rooyen
    @rvanrooy
    Is there a way to limit the amount of rows copied over and do incremental batches? I'm trying to copy over a huge (several hundred million rows) table, but from the debug logs, it seems that all rows are pre-fetched even though I have specified the batch rows to be fairly small. Is there a best practice to move over a large database?
    Dimitri Fontaine
    @dimitri
    see https://pgloader.readthedocs.io/en/latest/pgloader.html#with and the parameters batch rows, batch size, and prefetch rows; maybe the last one specifically here
    oh and yeah the convert()call to be used in the SELECT statement needs some attention too, in mssql-schema.lisp, good point! I guess with that you have a PR ready for improving pgloader?