Where communities thrive

  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
Repo info
    I am converting legacy database (dBASE III) with encoding IBM (german) using :

    FROM 'daten.dbf' WITH ENCODING CP850
    INTO postgresql:///staging
    WITH truncate, create table

    and it works well with only a small glitch. Beside the table 'public.daten', I also got a schema 'daten' with no dependencies at all.

    has the --config option been deprecated?
    I am trying to use a .load but every time i ran pgloader csv-writer.load I get an error saying: No such file or directory
    I am using pgloader version "3.6.2"
    we are having an issue where we have some images stored in a database that we're moving from mssql into pg and the fields storing the image data seems to be truncating at 4096 - any ideas if there is an option we need to set to allow larger chunks of data to pass through?
    1 reply
    Hello everyone and hello @dimitri
    I have a question about pgloader. I want to import a table from MSSQL database to a PGSQL.
    I use pgloader version 3.6.3~devel from docker image dimitri/pgloader:latest with a modified freetds conf explained in pgloader readthedoc
    I launch pgloader like this
    According to debug log, it create the table, read the source data but imported nothing
    In the log i found
    Finished processing WRITER for "dbo.csection" 31.060000s
    writers-counts[dbo.csection] = 0
    The summary said it had read x data but 0 imported
    It may be difficult for you to help me with those details but if you have any clue it will be great !
    It also may be a classical issue that i don't found any solution
    Thanks in advance
    Hi, is it possible to limit the number of copied rows?

    Hello everyone,
    I have a question about an issue am facing on. Am very new to pgloader, so any inputs will really help me.

    1) Getting an error as " the octet sequence #(195) cannot be decoded." It is because of the string getting in the source file "Women’s, Girls’, and Infants’ Cut and Sew Apparel Manufacturing ". Is there a way to handle this in load file and load data, without replacing the character in source file ?

    2) Also, the file has 1000 rows and this error happened on 425th row.. when the error happened it just stopped processing and did not continue. Is it possible to force ignore bad record & continue to next row ?

    My Load file below

    FROM INLINE with encoding 'ascii'
    Name, Value
    INTO postgresql://user:pwd@server:port/db?schema.table
    Name, Value

    WITH skip header = 1,
          batch rows = 200,
          prefetch rows = 200,
          batch size = 1024 kB,
          batch concurrency = 3,
          fields escaped by '\',
          fields terminated by '|'
    SET client_encoding to 'utf8',
        standard_conforming_strings to 'on'

    droptableifexistsschema.table; drop table if exists schema.table; ,
    dropINDEXifexistsschema.tableidx; drop INDEX if exists schema.table_idx; ,
    createtableifnotexistsschema.table(Namevarchar(80),Valuevarchar(255)); create table if not exists schema.table ( Name varchar(80), Value varchar(255) );

    029327952|"Women’s, Girls’, and Infants’ Cut and Sew Apparel Manufacturing "

    Rifhan Akram

    Hello everyone,

    I'm trying a migration from MSSQL to Postgresql. I'm getting an error as in this issue dimitri/pgloader#1018. I noticed the fix is in v3.6.2 but when I do apt-get install on Debian I get v3.6.1.

    am I missing something ? could someone please help me with this ? should I build from source? :)

    Rifhan Akram

    Hello everyone,

    I'm trying a migration from MSSQL to Postgresql. I'm getting an error as in this issue dimitri/pgloader#1018. I noticed the fix is in v3.6.2 but when I do apt-get install on Debian I get v3.6.1.

    am I missing something ? could someone please help me with this ? should I build from source? :)

    Update -:
    I was able to build version 3.6.2 from source and that resolved the issue. I'm not sure if 3.6.2 should be available in apt-get repository or if its intended to not have it ?

    mase - meat popsicle
    Does anyone know if in a command file, I am able to choose which src database columns are included ?

    @rifhanakram I'm in the process of trying to do the same. :)

    The new versions of software won't typically appear in APT until the maintainer pulls in the change

    mase - meat popsicle
    Hey all, using the latest docker image of pgloader drops me in to ldb
    I am not really sure how to proceed from here
    can anyone give any guideance ?
    mase - meat popsicle
    @mcgri Yes, I'm currently battling another issue, but I got past that stage
    is it possible for me to choose which mssql (src) columns I want to include ?
    mase - meat popsicle
    anyone.... ?
    Dario Frongillo
    HI All
    @dimitri i love your postgres book :D
    hey guys .. i m reading a field of a csv that have this format 999999999D99 and it use as thousand separator the char , and for decimal the point
    is there a way to transform it in number during an import csv ?
    Dario Frongillo
    hi all, someone can help me in this issue dimitri/pgloader#1192 ?
    Keerthikan Ratnarajah
    is this active?
    Benjamin David
    Hey! I'm having this error "there is no unique constraint matching given keys for referenced table" when trying to load form SQLite to Postgres
    but there is the proper constraint in sqlite
    so I dont understand
    Benjamin David
    It seems that the primary keys are not migrated
    hence causing the issues
    or the Unique constraint of the primary key is not passed
    is there any wya to force that?
    I looked over the whole doc and cant find it
    Benjamin David
    here's my empty DB
    just migrating the schema fails
    whereas its pretty straightforward
    Hi I am trying to load data from MS SQL server to Postgres, both password have special characters, for postgres I am using PGPASSWORD environment variable , what is the environment variable for MSSQL server password it tried PWD, PASSWORD it does not work
    Manoj Jadhav

    HI All,

    I am facing this issue?

    31: ("foreign function: call_into_lisp")
    can any one guide me here?
    ariel yosef
    hello everyone
    so i'v tried use pgloader and i have problem that mysql wont let me pass the password Visible
    any suggestion on how to pass the password without be Visible?
    is there a way to load mysql dump to postgresql with pgloader?
    Wagner Caixeta
    Hi, I have a messed table, some rows were inserted using latin1 and others were windows-1252. How can i apply my pgsql function for each varchar/text columns on import?
    CREATE OR REPLACE FUNCTION fix_text_encoding(text) returns text AS 
      RETURN convert_from(convert_to($1, 'latin1'), 'utf8');
        WHEN sqlstate '22P05' THEN
      RETURN convert_from(convert_to($1, 'windows-1252'), 'utf8');
    language plpgsql immutable;
    Is possible to use this fix_text_encoding? Or do I need to convert it to LISP? I know nothing about LISP =)
    Hey, when trying to execute pgloader to migrate a mariadb to posgres I receive following error: "ERROR Database error 42703: column "adsrc" does not exist" I'm using pgloader v3.6.1 - PostgreSQL 13.2
    Tatzlwurm Eads
    Hi, I have lots of columns that are like this "is_nonconcurrent character varying(1),", they are boolean but can't seem to get pgloader syntax right. I can workaround by altering columns post migration like : alter table acq.qrtz2_job_details alter is_durable type boolean using is_durable::boolean;
    Tatzlwurm Eads
    Sorry trying to migrate mysql -> postgresql.
    Tatzlwurm Eads
    nm figured out:CAST type varchar when (= precision 1) to boolean drop typemod
    I was trying to build pgloader-3.6.2 on RHEL 7 and got following errors

    mkdir -p build
    curl -o build/quicklisp.lisp http://beta.quicklisp.org/quicklisp.lisp
    % Total % Received % Xferd Average Speed Time Time Time Current
    Dload Upload Total Spent Left Speed
    100 3628 100 3628 0 0 521k 0 --:--:-- --:--:-- --:--:-- 708k
    sbcl --noinform --no-sysinit --no-userinit --load build/quicklisp.lisp \
    --load src/getenv.lisp \
    --eval '(quicklisp-quickstart:install :path "build/quicklisp" :proxy (getenv "http_proxy"))' \
    --eval '(quit)'
    While evaluating the form starting at line 1, column 0
    of #P"/var/opt/mwang/pgloader/pgloader-3.6.2/build/quicklisp.lisp":

    debugger invoked on a UNBOUND-VARIABLE in thread

    <THREAD "main thread" RUNNING {9461061}>:

    The variable <!DOCTYPE is unbound.

    Type HELP for debugger help, or (SB-EXT:EXIT) to exit from SBCL.

    restarts (invokable by number or by possibly-abbreviated name):
    0: [RETRY ] Retry EVAL of current toplevel form.
    1: [CONTINUE] Ignore error and continue loading file "/var/opt/mwang/pgloader/pgloader-3.6.2/build/quicklisp.lisp".
    2: [ABORT ] Abort loading file "/var/opt/mwang/pgloader/pgloader-3.6.2/build/quicklisp.lisp".
    3: Ignore runtime option --load "build/quicklisp.lisp".
    4: Skip rest of --eval and --load options.
    5: Skip to toplevel READ/EVAL/PRINT loop.
    6: [EXIT ] Exit SBCL (calling #'EXIT, killing the process).