Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
    Dimitri Fontaine
    @dimitri
    fixing that to be "2015-10-10 10:10:00" instead (defaulting non-provided parts from the format string to 00 in the time parts, so that you can ignore seconds here)
    see dimitri/pgloader@da05782 for your problem case and please recompile from a fresh git clone, you should be good then
    Karri Niemelä
    @kakoni
    Aah, ok. thanks!
    Scott Hillman
    @hillmanov
    Hi guys - I am looking to modify the batch options in my config file, but I am having a hard time figuring out where I need to put that, and what the syntax is. From what I gathered from looking at the source it should simply be a matter of putting "batch size = 10000" somewhere.
    Dimitri Fontaine
    @dimitri
    yeah, in the WITH clause
    see https://github.com/dimitri/pgloader/blob/master/test/csv-districts.load for an example of using the “batch rows” parameter (I think you want batch rows rather than batch size, which is intended as memory size limitation rather than number of rows per batch)
    Scott Hillman
    @hillmanov
    Perfect. Thanks!
    Justin
    @xxdesmus
    can I ask a likely dumb question?
    I'm bugging Dimitri directly on Twitter.
    sorry Dimitri :-/
    Justin
    @xxdesmus
    all set. needed to include WITH data only
    Dimitri Fontaine
    @dimitri
    oh you tried here first... well twitter is good too I guess
    Murilo Capanema
    @mcapanema
    Hello guys, we are intending to use the pgloader as a helping tool for our Snow Plow implementation. The problem is that as far as I can see, pgloader still does not support JSON load, is that correct ?
    sspr28
    @sspr28
    Hello Guys, Good day..I am facing a issues using pgloader configuration file...We are using external file to load data to postgresql database using pgloader..After insert , some of the data's are not inserted as expected null (source has null or blank space)..can you help me how to mention null if blanks on configuration file or any possible way to fix
    Dimitri Fontaine
    @dimitri
    @mcapanema : can you detail your JSON use case a bit please?
    sspr28
    @sspr28
    Thank you Dimitri...er are tryong to load from Text file.
    sorry for typo...We are trying to load from a text file
    sspr28
    @sspr28
    Hi All,
    I want to skip the 1st line in pgloader.configuration file..I have checked using skip head_lines=1 in my configuration but am getting the error global name 'nb_lines' is not defined..
    Could you help on this .
    rocas82
    @rocas82
    Hi All,
    I would like to build a pgloader binary on my centOS 6.7 from source code:
    rocas82
    @rocas82
    first I used bootstrap-centos.sh to create the environment, then run make and I have the following error message: debugger invoked on a LOAD-SYSTEM-DEFINITION-ERROR in thread #<THREAD "main thread" RUNNING {1002A8BA13}>: Error while trying to load definition for system bordeaux-threads from pathname /home/pgloader-3.2.2/build/quicklisp/dists/quicklisp/software/bordeaux-threads-v0.8.5/bordeaux-threads.asd: READ error during COMPILE-FILE: You need ASDF >= 3.1 to load this system correctly.(in form starting at line: 1, column: 0, file-position: 0)
    could you help on that please ?
    rocas82
    @rocas82
    Could you please help me to compile pgloader-3.2.2 on CentOS 6.7 to obtain a binary format ?
    is there any procedure that I can follow to compile pgloader-3.2.2 on Centos 6.7 ?
    Thank sfor your help
    Dimitri Fontaine
    @dimitri
    The ASDF being too old often is because the compiler itself (SBCL) is too old, which version do you have?
    rocas82
    @rocas82
    I had got the issue with sbcl-1.1.14 , then I installed SBCL 1.2.6 and it seems better,
    I loaded pgloader "3.2.02ecd88" binary format as it is available on your github and I am trying to use it
    Do I need to install SBCL before runing this pgloader binary format ?
    Dimitri Fontaine
    @dimitri
    If you download a pgloader binary, it's self contained (it includes SBCL) so you don't need SBCL at run-time. However it's a good idea to fetch the current git master's version and build from sources, and for that you need SBCL installed.
    rocas82
    @rocas82
    Hello Dimitri

    ./pgloader -V

    pgloader version "3.3.0.50"
    compiled with SBCL 1.3.5
    When I run pgloader, I have got the following error:
    mkdir /tmp/pgloader
    [root@VM_kurmi_Perso pgloaderAndTools-6.2.0]# ./pgloader -v commands
    2016-06-16T12:23:53.269000+02:00 NOTICE Starting pgloader, log system is ready.
    2016-06-16T12:23:53.569000+02:00 LOG Main logs in '/tmp/pgloader/pgloader.log'
    2016-06-16T12:23:53.624000+02:00 LOG Data errors in '/tmp/pgloader/'
    2016-06-16T12:23:53.624000+02:00 LOG Parsing commands from file #P"/root/pgloaderAndTools-6.2.0/commands"
    An unhandled error condition has been signalled:
    PostgreSQL protocol error: Unexpected message received: Z
    rocas82
    @rocas82
    When I run pgaloader in debug mode as following:
    ./pgloader -v -d commands
    the load of data startsand run for about 2 minutes and stop with this output:
    ./pgloader -v -d commands
    2016-06-16T13:56:21.730000+02:00 DEBUG MySQL: sending query:
    select column_name, data_type
    from information_schema.columns
    where table_schema = 'natesdb_master' and table_name = 'department'
    order by ordinal_position
    2016-06-16T13:56:21.730000+02:00 DEBUG MySQL: sending query: SELECT idDEPARTMENT, name, description, id_parent_department, path, location, eqtAvailableDefaultValue, idORGANIZATION FROM department;
    2016-06-16T13:56:21.730000+02:00 DATA < #("1" "OBS" "" NIL "/1/" "/OBS/" "1" "1")
    2016-06-16T13:56:21.730000+02:00 DATA < #("2" "Equant" "" "1" "/1/2/" "/OBS/Equant/" "1" "1")
    2016-06-16T13:56:21.730000+02:00 DATA < #("3" "FT-SCE" "" "1" "/1/3/" "/OBS/FT-SCE/" "1" "1")
    2016-06-16T13:56:21.730000+02:00 DATA < #("4" "tests" "" "3" "/1/3/4/" "/OBS/FT-SCE/tests/" "1" "1")
    2016-06-16T13:56:21.731000+02:00 DATA < #("5" "testsIt" "" "4" "/1/3/4/5/" "/OBS/FT-SCE/tests/testsIt/" "1" "1")
    2016-06-16T13:56:21.731000+02:00 DATA < #("6" "testsIng" "" "4" "/1/3/4/6/" "/OBS/FT-SCE/tests/testsIng/" "1" "1")
    2016-06-16T13:56:21.731000+02:00 DATA < #("7" "THATEST00" "" NIL "/7/" "/THATEST00/" "1" "1")
    2016-06-16T13:56:21.732000+02:00 DATA < #("8" "THATEST10" "" "7" "/7/8/" "/THATEST00/THATEST10/" "1" "1")
    2016-06-16T13:56:21.732000+02:00 DATA < #("9" "THATEST11" "" "8" "/7/8/9/" "/THATEST00/THATEST10/THATEST11/" "1" "1")
    2016-06-16T13:56:21.732000+02:00 DEBUG finish-batch[3] 9 rows in 0.018000s
    2016-06-16T13:56:21.732000+02:00 DEBUG Reader for department is done in 0.018000s
    2016-06-16T13:56:21.732000+02:00 DEBUG Finished processing READER for "department" 0.018000s
    2016-06-16T13:56:21.732000+02:00 DEBUG Transformer 3 in action for department_restriction!
    I don't know what is going wrong. the out of the execution is freezed without any error message
    Then when I used CTR+C to exit, I have:

    2016-06-16T13:56:21.732000+02:00 DEBUG Finished processing READER for "department" 0.018000s
    2016-06-16T13:56:21.732000+02:00 DEBUG Transformer 3 in action for department_restriction!
    ^C2016-06-16T14:16:27.047000+02:00 FATAL We have a situation here.
    An unhandled error condition has been signalled:
    Interactive interrupt at #x322AAE5197.

    Date/time: 2016-06-16-14:16An unhandled error condition has been signalled:
    Interactive interrupt at #x322AAE5197.

    Backtrace for: #<SB-THREAD:THREAD "main thread" RUNNING {10071F67B3}>
    0: ("bogus stack frame")
    1: ((FLET #:WITHOUT-INTERRUPTS-BODY-646 :IN SB-THREAD::%CONDITION-WAIT))
    2: (SB-THREAD:CONDITION-WAIT #<SB-THREAD:WAITQUEUE Anonymous condition variable {10140174E3}> #<SB-THREAD:MUTEX "Anonymous lock" (free)> :TIMEOUT NIL)
    3: (LPARALLEL.CONS-QUEUE:POP-CONS-QUEUE/NO-LOCK #<unavailable lambda li

    rocas82
    @rocas82
    I can share pgloadr.log file
    rocas82
    @rocas82
    ERROR found in pgloader.log
    2016-06-16T13:56:21.727000+02:00 ERROR Database error 42P01: relation "database_version_dev_history" does not exist
    2016-06-16T13:56:21.728000+02:00 ERROR Database error 42P01: relation "database_version_dev_history" does not exist
    2016-06-16T13:56:21.728000+02:00 ERROR A thread failed with error: Database error 42P01: relation "database_version_dev_history" does not exist
    2016-06-16T13:56:21.728000+02:00 INFO Transformer[3] for database_version_dev_history is done in 0.058000s
    When a thread failed with error, does pgloader execution stop ?
    rocas82
    @rocas82
    Hello Dimitri, I solved out this issue. Thanks
    database_version_dev_history table doesn't exist in my postgresql db