Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
  • Jan 06 2016 17:57
    test2
  • Jan 06 2016 17:56
    test
Charles Dixon
@chvck
Which is management, views, query, search services
If you have those ports blocked and only care about kv then you can tell WaitUntilReady to only test connectivity to the kv service (although I’m pretty sure we’d not recommend running the other services with the ports blocked)
WaitUntilReady(5*time.Second, &gocb.WaitUntilReadyOptions{ ServiceTypes: []gocb.ServiceType{gocb.ServiceTypeKeyValue}, })
Odell Joseph
@dragonpiper

I mean no ports should be blocked all i did was update to the sdk.

So i did try that but in the sdk code for wait on ready it does

for _, srv := range opts.ServiceTypes { if srv == MemdService { return nil, wrapError(errInvalidArgument, "memd service is not valid for use with clusterAgent") } else if srv == CapiService { return nil, wrapError(errInvalidArgument, "capi service is not valid for use with clusterAgent") } }

@chvck So it will throw an error when using gocb.ServiceType(gocbcore.MemdService),
Charles Dixon
@chvck
Oh, so you’re using cluster.WaitUntilReady? Maybe try bucket.WaitUntilReady instead - that will allow keyvalue
You can also not use WaitUntilReady and if you’re only doing kv then it looks like it’ll work but I expect that any HTTP based ops would fail
creddym
@creddym
Hi All,
I am noticing delays with golang couchbase sdk 2.2.3. The profiling shows the below trace. I am not suing WaitUntilReady for cluster level though I it used at bucket level. I can share code that is used to connect to couchbase. what is going wrong here? the sam application works with gocb 1.6. I could see most the time is spent in cccpConfigController/Doloop.

(pprof) top -cum
Showing nodes accounting for 20ms, 3.64% of 550ms total
Showing top 10 nodes out of 123
flat flat% sum% cum cum%
0 0% 0% 210ms 38.18% github.com/couchbase/gocbcore/v9.(*cccpConfigController).DoLoop
0 0% 0% 210ms 38.18% github.com/couchbase/gocbcore/v9.(*pollerController).Start
0 0% 0% 200ms 36.36% github.com/couchbase/gocbcore/v9.(*memdClient).SaslAuth.func1
0 0% 0% 200ms 36.36% github.com/couchbase/gocbcore/v9.(*memdClient).doBootstrapRequest.func1
0 0% 0% 200ms 36.36% github.com/couchbase/gocbcore/v9.(*memdClient).resolveRequest
0 0% 0% 200ms 36.36% github.com/couchbase/gocbcore/v9.(*memdClient).run.func2
0 0% 0% 200ms 36.36% github.com/couchbase/gocbcore/v9.(*memdQRequest).tryCallback
0 0% 0% 200ms 36.36% github.com/couchbase/gocbcore/v9.saslAuthScram.func1
0 0% 0% 200ms 36.36% github.com/couchbase/gocbcore/v9/scram.(*Client).Step
20ms 3.64% 3.64% 200ms 36.36% github.com/couchbase/gocbcore/v9/scram.(*Client).saltPassword

Code snippet for connecting to DB.
// Connect to DB
lCBConn, err := ConnectToCBv2(couchDbCfg.Url(), couchDbCfg.BucketName(), couchDbCfg.UserName(), couchDbCfg.Password())
clusterOptions := gocb.ClusterOptions{
Authenticator: gocb.PasswordAuthenticator{
Username: userName,
Password: passWd,
},
}
lCluster, err := gocb.Connect(couchUrl, clusterOptions)
// error handling if err!=nil or lCluster==nil
lBucket := lCluster.Bucket(bucketName)
lCollection := lBucket.DefaultCollection()

default collection is being used for KV operations.
I can quickly share any information that helps to debug this issue. can anyone help me on this?
Charles Dixon
@chvck
@creddym what do you mean by delays?
creddym
@creddym
@chvck KV operations taking more time compared to gocb v1.6.
Charles Dixon
@chvck
@creddym as in time to first kv operation or just kv operations in general? The trace shows a bunch of bootstrapping and config fetching, which shouldn’t be affecting how long kv operations take.
creddym
@creddym

@chvck I have set ConnectTimeout, KVTimeout, and run the tests. It is not the first few kv operations. In general, kv operations (upsert/patch/get) taking time (> 500 ms)
resolveRequest and ReadPacket are in top 20 functions that take more time cumulatively.
(pprof) top -cum -20
Showing nodes accounting for 5.14s, 74.49% of 6.90s total
Dropped 318 nodes (cum <= 0.03s)
flat flat% sum% cum cum%
0 0% 0% 2.18s 31.59% golang.org/x/net/http2.(serverConn).runHandler
0.01s 0.14% 0.14% 2.15s 31.16% github.com/gorilla/mux.(
Router).ServeHTTP
0 0% 0.14% 2s 28.99% net/http.HandlerFunc.ServeHTTP
0.02s 0.29% 0.43% 2s 28.99% udr/provision/subscription/api.ProvisionSubsAmDataV2
0.01s 0.14% 0.58% 1.58s 22.90% github.com/couchbase/gocbcore/v9.(memdClient).run.func2
0 0% 0.58% 1.43s 20.72% runtime.mcall
0 0% 0.58% 1.42s 20.58% runtime.park_m
0.02s 0.29% 0.87% 1.42s 20.58% runtime.schedule
0.17s 2.46% 3.33% 1.21s 17.54% runtime.findrunnable
1.16s 16.81% 20.14% 1.16s 16.81% runtime.futex
0.03s 0.43% 20.58% 1.07s 15.51% internal/poll.ignoringEINTR
0.74s 10.72% 31.30% 1.01s 14.64% syscall.Syscall
0 0% 31.30% 0.93s 13.48% udr/provision/subscription/api.CreateAmSubsData
0 0% 31.30% 0.85s 12.32% runtime.futexwakeup
0 0% 31.30% 0.83s 12.03% runtime.notewakeup
0.02s 0.29% 31.59% 0.81s 11.74% github.com/couchbase/gocbcore/v9.(
memdClient).resolveRequest
0.01s 0.14% 31.74% 0.80s 11.59% runtime.systemstack
0.11s 1.59% 33.33% 0.73s 10.58% runtime.mallocgc
0.01s 0.14% 33.48% 0.72s 10.43% github.com/couchbase/gocbcore/v9.(memdConnWrap).ReadPacket
0.02s 0.29% 33.77% 0.72s 10.43% io.ReadFull (inline)
0.01s 0.14% 33.91% 0.71s 10.29% github.com/couchbase/gocbcore/v9/memd.(
Conn).ReadPacket
0.03s 0.43% 34.35% 0.70s 10.14% io.ReadAtLeast
0.02s 0.29% 34.64% 0.67s 9.71% bufio.(Reader).Read
0.01s 0.14% 34.78% 0.65s 9.42% runtime.wakep
0 0% 34.78% 0.64s 9.28% net.(
conn).Read
0.01s 0.14% 34.93% 0.64s 9.28% net.(*netFD).Read

FYI,
Change is below.
clusterOptions := gocb.ClusterOptions{
Authenticator: gocb.PasswordAuthenticator{
Username: userName,
Password: passWd,
},
TimeoutsConfig: gocb.TimeoutsConfig{
ConnectTimeout: 60000 time.Millisecond,
KVTimeout: 2500
time.Millisecond,
},
}

I want to understand the reason for below functions taking some significant time.
0.01s 0.14% 0.58% 1.58s 22.90% github.com/couchbase/gocbcore/v9.(memdClient).run.func2
0.02s 0.29% 31.59% 0.81s 11.74% github.com/couchbase/gocbcore/v9.(memdClient).resolveRequest
0.01s 0.14% 33.48% 0.72s 10.43% github.com/couchbase/gocbcore/v9.(memdConnWrap).ReadPacket
0.01s 0.14% 33.91% 0.71s 10.29% github.com/couchbase/gocbcore/v9/memd.(Conn).ReadPacket
Charles Dixon
@chvck
Do you have any logs that you can share? Which operations are you running?
creddym
@creddym
@chvck - sorry for the delay. I have extracted the required code and made an independent application. it is just 4-5 go files. how to share go code here?
    startTime := time.Now()
    mutateResult, err := collection.Upsert(key, value, &upsertOptions)
    logElapsedTime(startTime, "UpsertDocV2()", cc.UPSERT, key)
2021/05/25 08:26:47 Exceeded db elapsed time for UpsertDocV2() key= TEST::405050000000483 DB elapsed time:266.365033ms
creddym
@creddym
@chvck please find the source code here. it is easy to compile and run by just changing a DB config file.
https://github.com/creddym/gocbTest.
Charles Dixon
@chvck
Hi @creddym how are you testing once you have the gocbTest running?
Running it seems to just start up the http server and establish connections but do no work
Charles Dixon
@chvck
I’ve tried the commands in the cmd.txt.txt in the test dir and I’ve modified the dbApi to always log elapsed time and I see
2021/05/25 08:47:06 Exceeded db elapsed time for UpsertDocV2() key= TEST::key1 DB elapsed time:537.478µs
2021/05/25 08:47:10 Exceeded db elapsed time for GetDocV2() key= TEST::key1 DB elapsed time:469.239µs
creddym
@creddym
@chvck Good. It is working for you. I am seeing the issue when I am running 100 requests per second. I can help you to provide a tool if you want to generate traffic.
what is the problem with my setup? what details do you need to debug the issue?
I am providing more details.
Client - gocb 2.2.3.
Server:
DB - VM based - 3 node cluster. DB type is couchbase (persistent)
Enterprise Edition 6.5.0 build 4960 - © 2019 Couchbase, Inc
Charles Dixon
@chvck
@creddym it’s hard to say what the issue would be. SDK log output would possibly help to identify where the issue is.
creddym
@creddym
gocb.SetLogger(gocb.VerboseStdioLogger()) -> this generates huge logging. can you please suggest to filter required info?
Charles Dixon
@chvck
Reducing the number of connections you create to 1 connection would help with that, and make the logs much easier to follow too.
creddym
@creddym
sure. will do that and share the logs. thanks for the suggestion.
one quick question -
https://docs.couchbase.com/go-sdk/current/ref/client-settings.html
under Constrained Network Environments
Config Poll Interval to 10s -> how to set this to a higher value? any impact by doing this?
I could see huge CCCPOLL messages b/w client and server.
Charles Dixon
@chvck
It means that in the case of a node failover or adding a new node etc… it’d take longer for the SDK to become aware of the cluster topology change.
vibhanshu sharma
@vibhanshu2014_twitter
I am debating between using joins through N1QL or creating data frames in spark and join there, so far N1QL join have better performance, any suggestions on pros and cons.
creddym
@creddym
@chvck I have captured logs for the latency issue in my setup. Can you please check the logs and help me to understand what is going wrong with my code?
https://github.com/creddym/gocbTest/blob/main/test/startup.log
https://github.com/creddym/gocbTest/blob/main/test/full.log
Charles Dixon
@chvck
@creddym just looking at this log output it looks like it’s time in network/on server. If you look at the first instance of the exceeded db in the log then it’s at around 19:24:04.055 and the elapsed time is 161.869194ms. Working backward from that timestamp and that elapsed time gives us 19:24:03.894 and at that time we can see GOCB 19:24:03.894157 memdpipelineclient.go:164: Writing request. 10.10.205.55:33784 to 10.10.220.190:11210 OP=0x1. Opaque=40 which tallies with GOCB 19:24:04.055836 ???:0: Resolving response OP=0x1. Opaque=40. This seems to be the case for the exceeded outputs that I looked at. If I worked back from the timestamp with the elapsed time there’s a matching opaque. The opaque is unique per request on a per connection basis (so in the case of a connection per node, a per node basis)
You could try a tool like Wireshark (packet capturing) to confirm/deny this too.
creddym
@creddym
ok Thank you @chvck. I had captured tcpump/pcap and saw the same. I will capture one more and share it with you. we have recently migrated from gocb 1.6 to gocb 2.2.x and observing this.
creddym
@creddym
@chvck i have collected pcap and logs for the delay issue in my setup.
https://github.com/creddym/gocbTest
delayLog - > filtered log
full.txt -> full log
Delay_Pcap_1.pcap -> full cap
Delay_Pcap_1.JPG -> request and response pkts.
My observation is upsert db request took around 400 ms!!! I have captured pcap at the server (couchbase) and looks me DB took around 7.5 ms. can you cross-check it once?
I will add another flag to select gocb 1.6 and run the command with both sdks and I will shared the details.
FYI
GOCB 20:05:20.430262 memdpipelineclient.go:164: Writing request. 10.10.205.55:43746 to 10.10.220.190:11210 OP=0x1. Opaque=11
GOCB 20:05:20.908758 ???:0: Resolving response OP=0x1. Opaque=11
GOCB 20:05:20.908790 asm_amd64.s:1374: Handling response data. OP=0x1. Opaque=11. Status:0
GOCB 20:05:20.908823 asm_amd64.s:1374: Dispatching response callback. OP=0x1. Opaque=11
2021/05/26 20:05:20 Exceeded db elapsed time for UpsertDocV2() key= TEST::405050000000000 DB elapsed time:478.874854ms
as per timestamp elapsed time is correct.
Charles Dixon
@chvck
Hi @creddym I’ve looked into this and discussed it with a colleague and it looks like there’s some sort of weird network issue that you’re seeing. If you look at your PCAP and filter it with couchbase && ip.addr==10.10.205.55 && couchbase.opaque==0x0b000000 (assuming you use Wireshark) then you can see there are 2 extra requests after the initial request response cycle for opaque 11. These are marked TCP Spurious Transmission which suggests that the client didn’t get a TCP ack when it sent the request initially. If you look at when the TCP Spurious requests are then they tally to the 400ms that you see.
vibhanshu sharma
@vibhanshu2014_twitter
Hi All, I am trying to connect couchbase from scala sdk within a spark session, do you think it’s a good idea I want to understand the internals if there is any downside of this approach
creddym
@creddym
@chvck I want to confirm that the performance issue was not due to sdk v2.
i have update test app with a flag to select sdk v1/v2 etc... and tested with two instances one with v1 and another with v2. the issue is due to istio configuration that is being used as part of my pod(db app)... thanks for your support.
FYI,
this test app may help to compare different sdks when upgrading to new sdk.
https://github.com/creddym/gocbTest
Noteworthy
@LordNoteworthy
Hi Folks, is there any naming conventions for buckets ? My bucket contain different types of documents: users, comments, ... Can't seem to find the right name :) amy suggestion ?
vibhanshu sharma
@vibhanshu2014_twitter
@LordNoteworthy, best to give name based on data or the downstream going to consume that data .
Noteworthy
@LordNoteworthy
Thanks @vibhanshu2014_twitter
Matt Ingenthron
@ingenthr
nice, thanks @creddym
Noteworthy
@LordNoteworthy
Can anyone thinks why a username could be a bad thing to use as a key in a bucket ?
Michael Reiche
@mikereiche
@LordNoteworthy it would limit the amount of data to one document for that username. Also. If a username changed, the data would need to be changed. Li
Brant Burnett
@brantburnett
@LordNoteworthy I'd probably combine with a doc type, like "user-<username>" for added flexibility. Also, I wouldn't do it unless you have a length limit on the user name, doc keys have a length limit (around 250 I think) and practically more like 50-ish is advisable.
Noteworthy
@LordNoteworthy
I see thanks a lot @mikereiche and @brantburnett . I do have a limitation of the username (20 max char)
If I sell a software that depends on couchbase enterprise, my customers would need also to buy an enterprise license ?
Brant Burnett
@brantburnett
@LordNoteworthy depends on the features you use and the scale they want to run at, Community may be sufficient
Noteworthy
@LordNoteworthy
I am using the couchbase-operator, that's part of the enterprise right ?
Pat
@patrick-stephens
Yes, the operator requires EE.