Find us on slack with invite: http://bit.ly/m3slack - Next community meeting Tuesday July 23, 2019 (11am PST, West Coast US): https://github.com/m3db/m3#community-meetings
Seed Node: First and foremost, these hosts are storage nodes themselves. In addition to that responsibility, they run an embedded ETCD server.
so does that mean I can run just seed nods without storage nodes?
so storage nodes are a thing only if I have a separate ETCD cluster, correct?
but this is confusing because
An M3DB storage instance (m3dbnode) for timeseries storage. This includes an embedded tag-based metrics index, as well as as an embedded etcd server for storing the above mentioned cluster topology and runtime configuration.
so which is it, is the seed node that runs embedded etcd, or is it the storage node
these two docs are contradictory
btw, can there be multiple m3 coordinators?
and I wonder if I want both masters writing to a separate m3 coordinator host, or maybe run 2 m3 coordinators on the same master prometheus hosts
would that make sense?
based on this presentation I need only a coordinator and db nodes:
https://www.youtube.com/watch?v=EFutyuIpFXQ
https://www.youtube.com/watch?v=EFutyuIpFXQ
Another question: I can see settings for pretty much all ports described here:
https://github.com/m3db/m3/blob/master/src/dbnode/config/m3dbnode-all-config.yml
https://github.com/m3db/m3/blob/master/src/dbnode/config/m3dbnode-all-config.yml
but I can't find where the ETCD(2379) and cluster peer port(2380) are set
where are those settings?
is that correct?
{"error":"invalid retention time: time: unknown unit d in duration 1825d"}
i aslo tried with 5y
i am configuring m3db with carbon injestion i also added one namespace with retention as 4840h and added some aggragte function in that .. but coordinator is not starting getting this error
Apr 1 22:09:12 mtc01-tej m3coordinator[15902]: {"level":"fatal","ts":1585739352.1716218,"msg":"cannot enable carbon ingestion without a corresponding aggregated M3DB namespace","resolution":"10s","retention":"4h0m0s"}
so does it mean that i need to create different name space for each aggregation which i mention in config
Hi, there
If I use this[https://github.com/m3db/m3#write-a-datapoint] API to write the metrics to m3db and then use prometheus to read the metrics remotely, how do I specify the data type of the metrics, such as Counter in prometheus
If I use this[https://github.com/m3db/m3#write-a-datapoint] API to write the metrics to m3db and then use prometheus to read the metrics remotely, how do I specify the data type of the metrics, such as Counter in prometheus
you need a different namespace for each combination of retention and resolution. So if you are aggregating data at 10s for 4 hours, you will need an aggregated namespace that is created for 10s data for 4hrs of retention
@unknown-admin metrics don't have a type in M3DB by default, that's a prometheus construct. You can just give the metric a name and some tags and as long as the values are running values (increasing over time), you can query it as a counter using Prometheus
this error could also indicate a back-up of metrics inside your Prom remote write queue
hello, I am using m3db as prometheus remote storage! I have 3 nodes( Memory:512GB, CpuCore: 56, 3T SSD), I have deployed cluster, And the resource usage is very Low(But Network IO is very High).
Right now , I only used m3db as storage, not connected to grafana.
CPU < 20%
Memory about 100G
NetworkIO(node_network_receive_bytes_total) > 100MB/s
How can I make full usage of my Servers resource?
Does m3db save data use less memory and cpus?
db:
metrics:
prometheus:
handlerPath: /metrics
listenAddress: 0.0.0.0:7203
sanitization: prometheus
samplingRate: 1.0
extended: detailed
hostID:
resolver: hostname
config:
service:
env: default_env
zone: embedded
service: m3db
cacheDir: /media/ssd1/m3db/cache
etcdClusters:
- zone: embedded
endpoints:
- node1:2379
- node2:2379
- node:2379
listenAddress: 0.0.0.0:9000
clusterListenAddress: 0.0.0.0:9001
httpNodeListenAddress: 0.0.0.0:9002
httpClusterListenAddress: 0.0.0.0:9003
debugListenAddress: 0.0.0.0:9004
client:
writeConsistencyLevel: majority
readConsistencyLevel: unstrict_majority
gcPercentage: 100
writeNewSeriesAsync: true
writeNewSeriesLimitPerSecond: 10485760
writeNewSeriesBackoffDuration: 2ms
bootstrap:
bootstrappers:
- filesystem
- commitlog
- peers
- uninitialized_topology
commitlog:
returnUnfulfilledForCorruptCommitLogFiles: false
cache:
series:
policy: lru
postingsList:
size: 2621440
commitlog:
flushMaxBytes: 52428800
flushEvery: 5s
queue:
calculationType: fixed
size: 2097152000
fs:
filePathPrefix: /media/ssd1/m3db/datacurl -X POST http://localhost:7201/api/v1/namespace -d '{
"name": "metrics",
"options": {
"bootstrapEnabled": true,
"flushEnabled": true,
"writesToCommitLog": true,
"cleanupEnabled": true,
"snapshotEnabled": true,
"repairEnabled": false,
"retentionOptions": {
"retentionPeriodDuration": "30d",
"blockSizeDuration": "6h",
"bufferFutureDuration": "1h",
"bufferPastDuration": "1h",
"blockDataExpiry": true,
"blockDataExpiryAfterNotAccessPeriodDuration": "5m"
},
"indexOptions": {
"enabled": true,
"blockSizeDuration": "6h"
}
}
}'curl -X POST localhost:7201/api/v1/services/m3db/placement/init -d '{
"num_shards": 512,
"replication_factor": 3,
"instances": [
{
"id": "node1",
"isolation_group": "node1",
"zone": "embedded",
"weight": 100,
"endpoint": "node1:9000",
"hostname": "node1",
"port": 9000
} , {
"id": "node2",
"isolation_group": "node2",
"zone": "embedded",
"weight": 100,
"endpoint": "node2:9000",
"hostname": "node2",
"port": 9000
} , {
"id": "node3",
"isolation_group": "node3",
"zone": "embedded",
"weight": 100,
"endpoint": "node3:9000",
"hostname": "node3",
"port": 9000
} ]
}'
Prometheus config
remote_write:
- url: "http://node4:7201/api/v1/prom/remote/write"
remote_timeout: 30s
queue_config:
capacity: 1000
max_shards: 1000
min_shards: 1
max_samples_per_send: 100
batch_send_deadline: 20s
min_backoff: 30ms
max_backoff: 100ms
coordinator config
listenAddress:
type: "config"
value: "0.0.0.0:7201"
metrics:
scope:
prefix: "coordinator"
prometheus:
handlerPath: /metrics
listenAddress: 0.0.0.0:7203 # until https://github.com/m3db/m3/issues/682 is resolved
sanitization: prometheus
samplingRate: 1.0
extended: none
tagOptions:
idScheme: quoted
clusters:
# Fill-out the following and un-comment before using, and
# make sure indent by two spaces is applied.
- namespaces:
- namespace: metrics
retention: 30d
type: unaggregated
client:
config:
service:
env: default_env
zone: embedded
service: m3db
cacheDir: /media/ssd1/m3db/cache
etcdClusters:
- zone: embedded
endpoints:
- node1:2379
- node2:2379
- node3:2379
writeConsistencyLevel: majority
readConsistencyLevel: unstrict_majority
writeTimeout: 10s
fetchTimeout: 15s
connectTimeout: 20s
writeRetry:
initialBackoff: 500ms
backoffFactor: 3
maxRetries: 2
jitter: true
fetchRetry:
initialBackoff: 500ms
backoffFactor: 2
maxRetries: 3
jitter: true
backgroundHealthCheckFailLimit: 4
backgroundHealthCheckFailThrottleFactor: 0.5
which param can i adjust to make full usage of my server ?
thanks very much