Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
    Dimitri Fontaine
    @dimitri
    see https://github.com/dimitri/pgloader/blob/master/test/csv-districts.load for an example of using the “batch rows” parameter (I think you want batch rows rather than batch size, which is intended as memory size limitation rather than number of rows per batch)
    Scott Hillman
    @hillmanov
    Perfect. Thanks!
    Justin
    @xxdesmus
    can I ask a likely dumb question?
    I'm bugging Dimitri directly on Twitter.
    sorry Dimitri :-/
    Justin
    @xxdesmus
    all set. needed to include WITH data only
    Dimitri Fontaine
    @dimitri
    oh you tried here first... well twitter is good too I guess
    Murilo Capanema
    @mcapanema
    Hello guys, we are intending to use the pgloader as a helping tool for our Snow Plow implementation. The problem is that as far as I can see, pgloader still does not support JSON load, is that correct ?
    sspr28
    @sspr28
    Hello Guys, Good day..I am facing a issues using pgloader configuration file...We are using external file to load data to postgresql database using pgloader..After insert , some of the data's are not inserted as expected null (source has null or blank space)..can you help me how to mention null if blanks on configuration file or any possible way to fix
    Dimitri Fontaine
    @dimitri
    @mcapanema : can you detail your JSON use case a bit please?
    sspr28
    @sspr28
    Thank you Dimitri...er are tryong to load from Text file.
    sorry for typo...We are trying to load from a text file
    sspr28
    @sspr28
    Hi All,
    I want to skip the 1st line in pgloader.configuration file..I have checked using skip head_lines=1 in my configuration but am getting the error global name 'nb_lines' is not defined..
    Could you help on this .
    rocas82
    @rocas82
    Hi All,
    I would like to build a pgloader binary on my centOS 6.7 from source code:
    rocas82
    @rocas82
    first I used bootstrap-centos.sh to create the environment, then run make and I have the following error message: debugger invoked on a LOAD-SYSTEM-DEFINITION-ERROR in thread #<THREAD "main thread" RUNNING {1002A8BA13}>: Error while trying to load definition for system bordeaux-threads from pathname /home/pgloader-3.2.2/build/quicklisp/dists/quicklisp/software/bordeaux-threads-v0.8.5/bordeaux-threads.asd: READ error during COMPILE-FILE: You need ASDF >= 3.1 to load this system correctly.(in form starting at line: 1, column: 0, file-position: 0)
    could you help on that please ?
    rocas82
    @rocas82
    Could you please help me to compile pgloader-3.2.2 on CentOS 6.7 to obtain a binary format ?
    is there any procedure that I can follow to compile pgloader-3.2.2 on Centos 6.7 ?
    Thank sfor your help
    Dimitri Fontaine
    @dimitri
    The ASDF being too old often is because the compiler itself (SBCL) is too old, which version do you have?
    rocas82
    @rocas82
    I had got the issue with sbcl-1.1.14 , then I installed SBCL 1.2.6 and it seems better,
    I loaded pgloader "3.2.02ecd88" binary format as it is available on your github and I am trying to use it
    Do I need to install SBCL before runing this pgloader binary format ?
    Dimitri Fontaine
    @dimitri
    If you download a pgloader binary, it's self contained (it includes SBCL) so you don't need SBCL at run-time. However it's a good idea to fetch the current git master's version and build from sources, and for that you need SBCL installed.
    rocas82
    @rocas82
    Hello Dimitri

    ./pgloader -V

    pgloader version "3.3.0.50"
    compiled with SBCL 1.3.5
    When I run pgloader, I have got the following error:
    mkdir /tmp/pgloader
    [root@VM_kurmi_Perso pgloaderAndTools-6.2.0]# ./pgloader -v commands
    2016-06-16T12:23:53.269000+02:00 NOTICE Starting pgloader, log system is ready.
    2016-06-16T12:23:53.569000+02:00 LOG Main logs in '/tmp/pgloader/pgloader.log'
    2016-06-16T12:23:53.624000+02:00 LOG Data errors in '/tmp/pgloader/'
    2016-06-16T12:23:53.624000+02:00 LOG Parsing commands from file #P"/root/pgloaderAndTools-6.2.0/commands"
    An unhandled error condition has been signalled:
    PostgreSQL protocol error: Unexpected message received: Z
    rocas82
    @rocas82
    When I run pgaloader in debug mode as following:
    ./pgloader -v -d commands
    the load of data startsand run for about 2 minutes and stop with this output:
    ./pgloader -v -d commands
    2016-06-16T13:56:21.730000+02:00 DEBUG MySQL: sending query:
    select column_name, data_type
    from information_schema.columns
    where table_schema = 'natesdb_master' and table_name = 'department'
    order by ordinal_position
    2016-06-16T13:56:21.730000+02:00 DEBUG MySQL: sending query: SELECT idDEPARTMENT, name, description, id_parent_department, path, location, eqtAvailableDefaultValue, idORGANIZATION FROM department;
    2016-06-16T13:56:21.730000+02:00 DATA < #("1" "OBS" "" NIL "/1/" "/OBS/" "1" "1")
    2016-06-16T13:56:21.730000+02:00 DATA < #("2" "Equant" "" "1" "/1/2/" "/OBS/Equant/" "1" "1")
    2016-06-16T13:56:21.730000+02:00 DATA < #("3" "FT-SCE" "" "1" "/1/3/" "/OBS/FT-SCE/" "1" "1")
    2016-06-16T13:56:21.730000+02:00 DATA < #("4" "tests" "" "3" "/1/3/4/" "/OBS/FT-SCE/tests/" "1" "1")
    2016-06-16T13:56:21.731000+02:00 DATA < #("5" "testsIt" "" "4" "/1/3/4/5/" "/OBS/FT-SCE/tests/testsIt/" "1" "1")
    2016-06-16T13:56:21.731000+02:00 DATA < #("6" "testsIng" "" "4" "/1/3/4/6/" "/OBS/FT-SCE/tests/testsIng/" "1" "1")
    2016-06-16T13:56:21.731000+02:00 DATA < #("7" "THATEST00" "" NIL "/7/" "/THATEST00/" "1" "1")
    2016-06-16T13:56:21.732000+02:00 DATA < #("8" "THATEST10" "" "7" "/7/8/" "/THATEST00/THATEST10/" "1" "1")
    2016-06-16T13:56:21.732000+02:00 DATA < #("9" "THATEST11" "" "8" "/7/8/9/" "/THATEST00/THATEST10/THATEST11/" "1" "1")
    2016-06-16T13:56:21.732000+02:00 DEBUG finish-batch[3] 9 rows in 0.018000s
    2016-06-16T13:56:21.732000+02:00 DEBUG Reader for department is done in 0.018000s
    2016-06-16T13:56:21.732000+02:00 DEBUG Finished processing READER for "department" 0.018000s
    2016-06-16T13:56:21.732000+02:00 DEBUG Transformer 3 in action for department_restriction!
    I don't know what is going wrong. the out of the execution is freezed without any error message
    Then when I used CTR+C to exit, I have:

    2016-06-16T13:56:21.732000+02:00 DEBUG Finished processing READER for "department" 0.018000s
    2016-06-16T13:56:21.732000+02:00 DEBUG Transformer 3 in action for department_restriction!
    ^C2016-06-16T14:16:27.047000+02:00 FATAL We have a situation here.
    An unhandled error condition has been signalled:
    Interactive interrupt at #x322AAE5197.

    Date/time: 2016-06-16-14:16An unhandled error condition has been signalled:
    Interactive interrupt at #x322AAE5197.

    Backtrace for: #<SB-THREAD:THREAD "main thread" RUNNING {10071F67B3}>
    0: ("bogus stack frame")
    1: ((FLET #:WITHOUT-INTERRUPTS-BODY-646 :IN SB-THREAD::%CONDITION-WAIT))
    2: (SB-THREAD:CONDITION-WAIT #<SB-THREAD:WAITQUEUE Anonymous condition variable {10140174E3}> #<SB-THREAD:MUTEX "Anonymous lock" (free)> :TIMEOUT NIL)
    3: (LPARALLEL.CONS-QUEUE:POP-CONS-QUEUE/NO-LOCK #<unavailable lambda li

    rocas82
    @rocas82
    I can share pgloadr.log file
    rocas82
    @rocas82
    ERROR found in pgloader.log
    2016-06-16T13:56:21.727000+02:00 ERROR Database error 42P01: relation "database_version_dev_history" does not exist
    2016-06-16T13:56:21.728000+02:00 ERROR Database error 42P01: relation "database_version_dev_history" does not exist
    2016-06-16T13:56:21.728000+02:00 ERROR A thread failed with error: Database error 42P01: relation "database_version_dev_history" does not exist
    2016-06-16T13:56:21.728000+02:00 INFO Transformer[3] for database_version_dev_history is done in 0.058000s
    When a thread failed with error, does pgloader execution stop ?
    rocas82
    @rocas82
    Hello Dimitri, I solved out this issue. Thanks
    database_version_dev_history table doesn't exist in my postgresql db
    Dimitri Fontaine
    @dimitri
    Hi ! Did you mention this table explicitly in your load command? Can you open an issue on github so that I can reproduce and fix the issue? As you're saying errors like that should end in a clean FATAL message
    Chico Venancio
    @chicocvenancio
    Hello, I'm trying to build from sources in Windows and I'm getting problems with the freetds dll (sybdb.dll)
    I tried building it, or using different versions to no avail.
    Chico Venancio
    @chicocvenancio
    since I won't be using mssql I also tried the patch suggested at #159 to remove it, but I still get the problem
    rocas82
    @rocas82
    Hello Dimitri, Sorry for the delay. This is my answer to your last question. The problem was that the "database_version_dev_history" table exists in my MySQL database but it doesn't exist in my PostgreSQL database. So I explicitly excluded this table from load command with "EXCLUDING TABLE NAMES MATCHING 'database_version_dev_history'" and it works fine. So for me there was no bug. Many thanks