Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
    Jakub
    @jakubgs
    I was also wondering about the distinction between the nodes
    Seed Node: First and foremost, these hosts are storage nodes themselves. In addition to that responsibility, they run an embedded ETCD server.
    so does that mean I can run just seed nods without storage nodes?
    so storage nodes are a thing only if I have a separate ETCD cluster, correct?
    but this is confusing because
    An M3DB storage instance (m3dbnode) for timeseries storage. This includes an embedded tag-based metrics index, as well as as an embedded etcd server for storing the above mentioned cluster topology and runtime configuration.
    so which is it, is the seed node that runs embedded etcd, or is it the storage node
    these two docs are contradictory
    martin-mao
    @martin-mao
    good question, the single_node is for a single instance with everything (etcd, coordinator) embedded all in one
    martin-mao
    @martin-mao
    the cluster the hard way recommends a separate cluster for etcd and a cluster for M3DB storage and also separate the read and write coordinators as well
    Jakub
    @jakubgs
    that seems like it's quite an involved setup could I not run just the coordinator and some storage nodes?
    btw, can there be multiple m3 coordinators?
    Jakub
    @jakubgs
    because I'm running 2 master prometheus instances that federate from slave instances in different DCs
    and I wonder if I want both masters writing to a separate m3 coordinator host, or maybe run 2 m3 coordinators on the same master prometheus hosts
    would that make sense?
    Jakub
    @jakubgs
    based on this presentation I need only a coordinator and db nodes:
    https://www.youtube.com/watch?v=EFutyuIpFXQ
    Jakub
    @jakubgs
    Another question: I can see settings for pretty much all ports described here:
    https://github.com/m3db/m3/blob/master/src/dbnode/config/m3dbnode-all-config.yml
    but I can't find where the ETCD(2379) and cluster peer port(2380) are set
    where are those settings?
    is that correct?
    martin-mao
    @martin-mao
    I'll 1:1 you
    tejasbhosale009
    @tejasbhosale009
    can anyone please help how to create namespace with 5 year as retention time
    {"error":"invalid retention time: time: unknown unit d in duration 1825d"}
    i aslo tried with 5y
    tejasbhosale009
    @tejasbhosale009
    hi all
    i am configuring m3db with carbon injestion i also added one namespace with retention as 4840h and added some aggragte function in that .. but coordinator is not starting getting this error
    Apr 1 22:09:12 mtc01-tej m3coordinator[15902]: {"level":"fatal","ts":1585739352.1716218,"msg":"cannot enable carbon ingestion without a corresponding aggregated M3DB namespace","resolution":"10s","retention":"4h0m0s"}
    so does it mean that i need to create different name space for each aggregation which i mention in config
    unknown
    @unknown-admin
    Hi, there
    If I use this[https://github.com/m3db/m3#write-a-datapoint] API to write the metrics to m3db and then use prometheus to read the metrics remotely, how do I specify the data type of the metrics, such as Counter in prometheus
    martin-mao
    @martin-mao
    @tejasbhosale009 can you try in hours? 43800h?
    you need a different namespace for each combination of retention and resolution. So if you are aggregating data at 10s for 4 hours, you will need an aggregated namespace that is created for 10s data for 4hrs of retention
    @unknown-admin metrics don't have a type in M3DB by default, that's a prometheus construct. You can just give the metric a name and some tags and as long as the values are running values (increasing over time), you can query it as a counter using Prometheus
    tejasbhosale009
    @tejasbhosale009
    Ok thanks @martin-mao i done that but coordinator is not working giving error like datapoint too far in past
    martin-mao
    @martin-mao
    @tejasbhosale009 There is a buffer past period which can be set on the coordinators. You need to ensure that that is larger than the delays you expect on your time stamps
    this error could also indicate a back-up of metrics inside your Prom remote write queue
    tejasbhosale009
    @tejasbhosale009
    no i am using graphite ingestion not promethus exporter
    martin-mao
    @martin-mao
    oh interesting, well same thing there, if the timestamps on Carbon are a behind, yuo will need to set higher bufferPast periods to account for it
    Ween
    @weenxin

    hello, I am using m3db as prometheus remote storage! I have 3 nodes( Memory:512GB, CpuCore: 56, 3T SSD), I have deployed cluster, And the resource usage is very Low(But Network IO is very High).

    Right now , I only used m3db as storage, not connected to grafana.

    CPU < 20%
    Memory about 100G
    NetworkIO(node_network_receive_bytes_total) > 100MB/s

    How can I make full usage of my Servers resource?
    Does m3db save data use less memory and cpus?

    Ween
    @weenxin
    db:
      metrics:
        prometheus:
          handlerPath: /metrics
          listenAddress: 0.0.0.0:7203
        sanitization: prometheus
        samplingRate: 1.0
        extended: detailed
    
      hostID:
        resolver: hostname
    
      config:
        service:
          env: default_env
          zone: embedded
          service: m3db
          cacheDir: /media/ssd1/m3db/cache
          etcdClusters:
            - zone: embedded
              endpoints:
                - node1:2379
                - node2:2379
                - node:2379
    
      listenAddress: 0.0.0.0:9000
      clusterListenAddress: 0.0.0.0:9001
      httpNodeListenAddress: 0.0.0.0:9002
      httpClusterListenAddress: 0.0.0.0:9003
      debugListenAddress: 0.0.0.0:9004
    
      client:
        writeConsistencyLevel: majority
        readConsistencyLevel: unstrict_majority
    
      gcPercentage: 100
    
      writeNewSeriesAsync: true
      writeNewSeriesLimitPerSecond: 10485760
      writeNewSeriesBackoffDuration: 2ms
    
      bootstrap:
        bootstrappers:
            - filesystem
            - commitlog
            - peers
            - uninitialized_topology
        commitlog:
          returnUnfulfilledForCorruptCommitLogFiles: false
    
      cache:
        series:
          policy: lru
        postingsList:
          size: 2621440
    
      commitlog:
        flushMaxBytes: 52428800
        flushEvery: 5s
        queue:
          calculationType: fixed
          size: 2097152000
    
      fs:
        filePathPrefix: /media/ssd1/m3db/data
    curl -X POST http://localhost:7201/api/v1/namespace -d '{
      "name": "metrics",
      "options": {
        "bootstrapEnabled": true,
        "flushEnabled": true,
        "writesToCommitLog": true,
        "cleanupEnabled": true,
        "snapshotEnabled": true,
        "repairEnabled": false,
        "retentionOptions": {
          "retentionPeriodDuration": "30d",
          "blockSizeDuration": "6h",
          "bufferFutureDuration": "1h",
          "bufferPastDuration": "1h",
          "blockDataExpiry": true,
          "blockDataExpiryAfterNotAccessPeriodDuration": "5m"
        },
        "indexOptions": {
          "enabled": true,
          "blockSizeDuration": "6h"
        }
      }
    }'
    curl -X POST localhost:7201/api/v1/services/m3db/placement/init -d '{
        "num_shards": 512,
        "replication_factor": 3,
        "instances": [
            {
                "id": "node1",
                "isolation_group": "node1",
                "zone": "embedded",
                "weight": 100,
                "endpoint": "node1:9000",
                "hostname": "node1",
                "port": 9000
            } ,        {
                "id": "node2",
                "isolation_group": "node2",
                "zone": "embedded",
                "weight": 100,
                "endpoint": "node2:9000",
                "hostname": "node2",
                "port": 9000
            } ,        {
                "id": "node3",
                "isolation_group": "node3",
                "zone": "embedded",
                "weight": 100,
                "endpoint": "node3:9000",
                "hostname": "node3",
                "port": 9000
            }     ]
    }'
    Prometheus config
    remote_write:
    - url: "http://node4:7201/api/v1/prom/remote/write"
      remote_timeout: 30s
      queue_config:
        capacity: 1000
        max_shards: 1000
        min_shards: 1
        max_samples_per_send: 100
        batch_send_deadline: 20s
        min_backoff: 30ms
        max_backoff: 100ms
    Ween
    @weenxin
    coordinator config
    listenAddress:
      type: "config"
      value: "0.0.0.0:7201"
    
    metrics:
      scope:
        prefix: "coordinator"
      prometheus:
        handlerPath: /metrics
        listenAddress: 0.0.0.0:7203 # until https://github.com/m3db/m3/issues/682 is resolved
      sanitization: prometheus
      samplingRate: 1.0
      extended: none
    
    tagOptions:
      idScheme: quoted
    
    clusters:
    # Fill-out the following and un-comment before using, and
    # make sure indent by two spaces is applied.
       - namespaces:
           - namespace: metrics
             retention: 30d
             type: unaggregated
         client:
           config:
             service:
               env: default_env
               zone: embedded
               service: m3db
               cacheDir: /media/ssd1/m3db/cache
               etcdClusters:
                 - zone: embedded
                   endpoints:
                     - node1:2379
                     - node2:2379
                     - node3:2379
           writeConsistencyLevel: majority
           readConsistencyLevel: unstrict_majority
           writeTimeout: 10s
           fetchTimeout: 15s
           connectTimeout: 20s
           writeRetry:
             initialBackoff: 500ms
             backoffFactor: 3
             maxRetries: 2
             jitter: true
           fetchRetry:
             initialBackoff: 500ms
             backoffFactor: 2
             maxRetries: 3
             jitter: true
           backgroundHealthCheckFailLimit: 4
           backgroundHealthCheckFailThrottleFactor: 0.5
    image.png
    Ween
    @weenxin
    this is my resource grafana
    which param can i adjust to make full usage of my server ?
    thanks very much