Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
  • Jan 06 2016 17:57
    test2
  • Jan 06 2016 17:56
    test
Leonid Boykov
@leonidboykov
To be more specific, I have 2 buckets with 95K documents and with 125M records, with the same update code I'm getting these issues only with a larger bucket. This is really confusing that I have no issues with updating my data, only with replication.
Also, I've set the durability level to Majority.
Michael Reiche
@mikereiche_gitlab
Seeing that the issue only occurs with replication - it's possible that a later update/Cas is replicated before an earlier update/Cas. In that case the earlier update/Cas is skipped. Which is ok.
As a side note - cb upsert isn't supposed to accept a Cas - if you have a Cas, then the document already exists, use replace (although the gocb sdk or your code may already do that).
Leonid Boykov
@leonidboykov
I have a different document count after a couple of days in both DC, it does not look like ok. I will try ReplaceOp instead, thanks.
Michael Reiche
@mikereiche_gitlab
What's the if m.IsTombstone() code for? Isn't a cb tombstone for an already removed document? Or is that your application's tombstone?
Leonid Boykov
@leonidboykov
This means that message must be removed from Couchbase, I'm transferring data from Kafka. This is not related to Couchbase.
Michael Reiche
@mikereiche_gitlab
Are all these kv operations? Or are any of them done with query?
Michael Reiche
@mikereiche_gitlab
Are you using transactions? Bi-directional XDCR?
Leonid Boykov
@leonidboykov
Yes, I'm using only these operations, no N1QL, no transactions.
Only one DC replicates to another. No bi-directional replication, no multi-master or anything fancy at all.
Michael Reiche
@mikereiche_gitlab
Are there consistently more (or less) documents on the primary DC or the replicated DC? I'm trying to figure out if it is inserts or deletes that are causing the difference.
Also - what is your server version?
Leonid Boykov
@leonidboykov

I've flushed both buckets (master DC and replicated) and reuploading 120M of records from scratch right now. I have no issues with replication at the moment, but every time issues come a few days after when I'm pulling updates from Kafka and pushing to Couchbase.

Primary DC has more records than replicated.

Couchbase Server Community Edition 6.6.0 build 7909
Michael Reiche
@mikereiche_gitlab

thinking out loud here - more documents in the primary DC would indicate that it is inserts that are not being replicated (at least more inserts than removes). [ the document count discrepancy being due to more removes being processed on the replicated DC than on the primary is not a possibility]. Some of the "skipped mutations" may also be replaces. Theoretically, a remove cannot be a "skipped mutation" because once a document is removed, there cannot be any subsequent operations on it to replicate. Even when there are "skipped mutations" detected, it doesn't mean that there are "lost permutations" - a "skipped permutation" just means that a later permutation arrived before an earlier one arrived/was processed - when the replication DC "caught up" they would be consistent. I can't really think of anything that would cause your situation.

From the XDCR Mutations Skipped graph - did something happen around 8pm when the mutations skipped started to climb? A rebalance, maybe? Or was that just when the kafka intake started?
Is there a possibility that the difference is document counts is temporary, if the replicated DC was allowed to "catch up" they would be the same? I can't think of anything else.

Leonid Boykov
@leonidboykov
The last rebalance was several days ago (and a few months for replica DC). I've noticed that the changelog for Couchbase 6.6.1 and 6.6.2 contains some fixes for race conditions, I'll try to update my Couchbase version and see if these fixes help. Thanks for the help, @mikereiche_gitlab!
Nick Sherron
@nicksherron

I’ve just pushed https://github.com/couchbase/gocb-opentelemetry which currently uses HEAD of gocb, it will be updated to 2.3.0 when released in July

Thanks for the helpful answers! @chvck

Pat
@patrick-stephens
What versions are you using of operator and server image? Is it complaining but running ok?
The server container is a multi process one that does some interesting things from a container perspective, mostly to keep things aligned between the container and native stacks.
I'm not in the office until tomorrow unfortunately but we do use GKE for the CICD and the docs are normally pretty good for setting it up so apologies if we've missed something. I can have a look for you once in but be good to have more details on your config, ideally a cbopinfo to see the logs and all the configuration in a support request.
Pat
@patrick-stephens
There are things like the security context and fs config in the CRD that need to be correct for more secure setups. I'll see if I can dig out the info or feel free to do a forum post as I'll see that once back in tomorrow.
Pat
@patrick-stephens
I've found an existing issue: https://issues.couchbase.com/browse/K8S-1512
This doesn't prevent the container running, it's a spurious error message due to the server launch script trying to chown
Pat
@patrick-stephens
Are you running with Istio?
There's also a known certificate issue you have to deal with on older versions so let me dig that up.
Pat
@patrick-stephens
It looks like the certificate is not right so I'd look at the TLS tutorial to see the specifics you need for it.
Pat
@patrick-stephens
Ah yeah, that's the one I was looking for I think. There should be a call out in the documentation as to what needs doing. Are you using helm to deploy and upgrading then?
It should be resolved in latest Helm charts but depends if you're upgrading, helm has a few issues with that: it can't do CRD updates for example either so these just be done manually.
Pat
@patrick-stephens
I'd say if you're still struggling, use the cbopinfo tool to grab the logs and a snapshot of your configuration for a support request. It's hard to debug things without knowing the specifics and that will capture your config plus anything else relevant that might be affecting it.
Leonid Boykov
@leonidboykov

Hello,

I've spotted strange behavior. When I've tried to override custom-compaction settings, I've found all metrics in grafana are absent. Can you confirm this issue?

image.png
Here, I've just enabled custom auto-compaction (haven't change anything else) and disabled after a short time. Metrics for all buckets have returned.
Justin Ashworth
@malscent
This is a known issue tracked in https://issues.couchbase.com/browse/PE-51
ngaugler
@ngaugler
Using the couchbase go driver (gocb) and I noticed that my application initiates just 2 connections to each couchbase server. I am worried that this is causing a bottleneck. Is this intended? Does anyone have scalability issues with this? Is there anyway or reason to increase this?
Brant Burnett
@brantburnett
@ngaugler I can't speak specifically to the go driver, I'm much more familiar with C#. However, I can speak to the architecture of the connections with some confidence (not an expert though)
The modern SDKs support multiplexing over the connections, meaning multiple requests can be in flight at once on a single connection. Modern versions of Couchbase Server also support returning the results out of order for even more efficiency.
This means that, in theory, a single connection can cover a lot of load. The exception tends to be if you have a mix of large and small documents. In that case, sending a large doc over the network can plug up that connection for a bit and delay the small documents. This is part of the reason for having more than one connection, reduces the risk of this.
typically the size of this connection pool is configurable, though I'm not familiar with where those settings are in Go
Brant Burnett
@brantburnett
In C#, I even helped implement an autoscaling system for the connection pool that sizes it up and down based on needs, not sure if other SDKs have a system like this or not
Shraman Das
@Shrman007658
Hey Team :D Our DBA says that they have disabled querying in a couchbase bucket. I want to extract a specific document from the bucket, a random one would work too. without using N1QL or the doucment ID(because I wouldnt know the document ID). Is it possible to do that using the JAVA SDK? 2.7
Denis Rosa
@deniswsrosa_twitter
@Shrman007658 Without knowing the Id of the document and with N1QL disabled I'm afraid that this would be difficult. I would assume that your DBA potentially blocked all the other ways as well (N1QL rest api, web console, etc)
Brant Burnett
@brantburnett
Only other way I know of is to use the APIs used by the web console on port 8091, http://localhost:8091/pools/default/buckets/bucketName/docs?skip=0&include_docs=false&limit=100, but it's probably not the most performant option if you're hitting it often. My understanding is the 8091 HTTP server isn't designed for high load.
Christian Boulanger
@cboulanger

Hi, I am getting a segmentation fault with the NodeJS client (v 3.1.0) on MacOS 11.4:

PID 26285 received SIGSEGV for address: 0x0
0   segfault-handler.node               0x0000000104bbbf90 _ZL16segfault_handleriP9__siginfoPv + 304
1   libsystem_platform.dylib            0x00007fff20684d7d _sigtramp + 29
2   ???                                 0x000000010a1ab6c0 0x0 + 4464490176
3   couchbase_impl.node                 0x00000001070281a3 _ZN3lcb4http7Request6finishE10lcb_STATUS + 275
4   couchbase_impl.node                 0x000000010702c71b _ZL12on_connectedP12lcbio_SOCKETPv10lcb_STATUSi + 123
5   couchbase_impl.node                 0x0000000107036bdb _ZN3lcb2io11PoolRequest6invokeEv + 251
6   couchbase_impl.node                 0x0000000107038013 timer_callback + 595
7   node                                0x0000000100a000a7 uv__run_timers + 103
8   node                                0x0000000100a048dd uv_run + 205
9   node                                0x00000001000f2075 _ZN4node16NodeMainInstance3RunEv + 309
10  node                                0x00000001000851b6 _ZN4node5StartEiPPc + 294
11  libdyld.dylib                       0x00007fff2065af5d start + 1
sh: line 1: 26285 Segmentation fault: 11  ts-node test.ts

Does that sound familiar to anyone? I followed all of Google's suggestions like rm -rf node_modules && npm install, ulimit unlimited and npm rebuildbut to no avail...

ramesh-cell
@ramesh-cell
image.png
image.png
image.png
hi all,
very good afternoon every one.
can you please help me i have query about couchbase server.
i need to get highlighted "Query History" through REST API.
The Exported "Output"Json file need to get using REST API or java sdk.
can you please help me, is it possible in couchbase.
I have gone through all REST API endpoints but i did not found any where in docs.
is any one please help me.
Brant Burnett
@brantburnett
@ramesh-cell I may be wrong, but I don't think that query workbench history exists on the server. That's stored in your browser's local storage. So there would be no server API to get it.
2021-08-30_9-32-06.png
ML
@manideeplanka_twitter
Is it safe to do updates on large number of documents during rebalancing?