Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
    JP Phillips
    @jipperinbham
    my guess is we don't provide all that's needed for the update to work as you'd expect
    can you file an issue with some details and we can investigate further there?
    SamBartrum
    @SamBartrum
    ok will do!
    John Jung
    @johnjjung
    @SamBartrum are you saying that when you're tailing mongo and you're deleting a record it doesn't delete in ES? Also just so I cover all basis, this isn't softdeletes (updates) right?
    SamBartrum
    @SamBartrum
    Not deleting a mongo document, but deleting a field on the mongo document. The field is not removed in the elasticsearch record.
    John Jung
    @johnjjung
    @jipperinbham I think I encountered a bug where my logs after using the parent-child throws a lot of routing errors
    I sent in a pull request where I identified the problem, and I tested it with my own data set
    This might have something to do with updates and delete issues. For me, deletes aren’t a big deal because we use soft deletes then batch delete them in ES, but I’ll patch that up as well. It has something to do with elasticsearch routing so that parent-childs are on the same shards
    time="2017-09-29T19:57:28Z" level=error msg="_bulk failed list [10]" error="&elastic.ErrorDetails{Type:"routing_missing_exception", Reason:"routing is required for [db]/[collection]/[id]", ResourceType:"", ResourceId:"", Index:"june", Phase:"", Grouped:false, CausedBy:map[string]interface {}(nil), RootCause:[]*elastic.ErrorDetails(nil), FailedShards:[]map[string]interface {}(nil)}" executionID=8 id={...id...} index=db type=collection version=5 writer=elasticsearch
    JP Phillips
    @jipperinbham
    @johnjjung thanks for the PR! how hard would it be to add a test to cover this?
    John Jung
    @johnjjung
    i was just about to ping you before getting pulled away
    I have to update the tests
    it doesn’t look too bad I’m already halfway there
    also question, can you share the dockerfile? I’m curious as to how you’re building your docker containers
    I’m trying to use docker-compose to make testing easier but I had trouble building it
    JP Phillips
    @jipperinbham
    you mean for running tests locally?
    John Jung
    @johnjjung
    yes
    I’m trying to do something like this:
    version: "3"
    
    services:
      test:
        build:
          context: .
          dockerfile: ./docker/Dockerfile
        volumes:
          - ./:/go/src/github.com/compose/transporter
        environment:
          - ES_V5_URL=http://es_v1:9205
          - ES_V2_URL=http://es_v2:9202
          - ES_V1_URL=http://es_v5:9201
        command: "go test -v ./adaptor/elasticsearch/..."
        depends_on:
          - es_v1
          - es_v2
          - es_v5
        links:
          - es_v1
          - es_v2
          - es_v5
      mongo:
        image: mongo:3.4.4
        ports:
          - "27000:27017"
      es_v1:
        image: elasticsearch:1.7.6
        ports:
          - "9201:9200"
      es_v2:
        image: elasticsearch:2.4.4
        ports:
          - "9202:9200"
      es_v5:
        image: elasticsearch:5.1.2
        ports:
          - "9205:9200”
    Also that dockerfile would help me with our kubernetes deployment as well
    I think all our tests would be easier and reproducable this way. the vagrant file was giving me trouble
    JP Phillips
    @jipperinbham
    nice, yea, we need to just remove that vagrant setup at this point
    the Dockerfile I use is really just for publishing releases, do you just need an example?
    John Jung
    @johnjjung
    sure anything would help, I tried a few times, with the native golang:latest but I’m having trouble building it
    I think because it has something to do with how runs go-wrapper it always errors out
    RUN go-wrapper download   # "go get -d -v ./..."
    RUN go-wrapper install    # "go install -v ./…"
    JP Phillips
    @jipperinbham
    gotcha, I've never used go-wrapper, I'll paste something in here later
    John Jung
    @johnjjung
    thanks
    having trouble with updates, still working through it
    debug
    John Jung
    @johnjjung
    @jipperinbham updated with tests
    John Jung
    @johnjjung
    @jipperinbham does x-log work with mongodb to elasticsearch?
    JP Phillips
    @jipperinbham
    @johnjjung what is x-log?
    John Jung
    @johnjjung
    t.Config({"xlog_dir":"/data/transporter"}) resuming
    @jipperinbham
    JP Phillips
    @jipperinbham
    FROM golang:1.8 as builder
    
    # Setting up working directory
    ADD . /go/src/github.com/compose/transporter/
    WORKDIR /go/src/github.com/compose/transporter/
    
    RUN CGO_ENABLED=0 GOOS=linux go build -a -installsuffix cgo -o transporter ./cmd/transporter/...
    
    FROM alpine:latest 
    
    RUN apk --no-cache add ca-certificates
    
    COPY --from=builder /go/src/github.com/compose/transporter/transporter /usr/local/bin/
    
    # Alpine Linux doesn't use pam, which means that there is no /etc/nsswitch.conf,
    # but Golang relies on /etc/nsswitch.conf to check the order of DNS resolving
    # (see https://github.com/golang/go/commit/9dee7771f561cf6aee081c0af6658cc81fac3918)
    # To fix this we just create /etc/nsswitch.conf and add the following line:
    RUN echo 'hosts: files mdns4_minimal [NOTFOUND=return] dns mdns4' >> /etc/nsswitch.conf
    
    CMD ["/usr/local/bin/transporter"]
    @johnjjung try that Dockerfile out ^^^ I haven't tested it but pretty sure it's right
    and yes, MongoDB -> elasticsearch supports the new commit log feature
    John Jung
    @johnjjung
    @jipperinbham thanks I’ll try this out
    John Jung
    @johnjjung
    Thanks! this docker file worked beautifully! Perfect!
    JP Phillips
    @jipperinbham
    good deal
    Jonathan Sheely
    @jsheely
    Trying to build the project on windows and I'm getting a package github.com/compose/transporter/log: cannot find package "github.com/compose/transporter/log" in any of: C:\Go\src\github.com\compose\transporter\log (from $GOROOT) Seems strange it would be looking at the $GORoot for Compose. shouldn't that be in the $GoPath ?
    Nevermind. Looks like a git-lfs issue
    Ilia Lobsanov
    @nurey
    i'm using rethinkdb as source and postgresql as sink. the data gets across successfully but I can't get the tail functionality to work.
    Kautilya Tripathi
    @knrt10
    Hey all. Is this project still maintained? I was looking to contribute to it.
    Black-Gold
    @Black-Gold
    does anyone know how to sync all database just one time?
    Black-Gold
    @Black-Gold
    maybe this project is dead,so bad!this is great!
    gun powder
    @gunpowder1_gitlab
    hello world
    srilekhaMarktine
    @srilekhaMarktine
    Hi Team, can I use this project to run in realtime production