Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
    René Ribaud
    @uggla
    Currently we need to have access to the power repl driver (/sys/devices/virual/powercap) to calculate consumption. On a vm you need to do the procedure you highlighted above in order to know the consumption on a per vm basis.
    Unfortunately, in order to do the procedure you need access to the hypervisor and this is not the case on cloud providers.
    We have an issue opened hubblo-org/scaphandre#25 to estimate power consumption as a fall back plan when we don't have access to the sensors. But this is not a simple problem so far from being done.
    bpetit
    @bpetit

    Hi there !

    I felt like we had many topics on going and one could be a bit lost about what's going on and what PR should be merged first etc.

    I wrote a quick "monthly report" here

    I encourage you to discuss on that thread for everything related. It seems to me this is much more readable than the chat on the long term.

    please also tell me if this adds value or not, and why. I'd like to make the management of the project more transparent.

    bpetit
    @bpetit
    :point_up: MONTHLY REPORT 06/2021 :point_up:
    Samuel Langlois
    @slanglois
    Hi there! Newbie here… I love the project, and would love to help. However, I can't do much on the technical side, that's way beyond me. But if that's okay, I'll have a look at proofreading the documentation. Also, I think I can make the Readme more "snappy"… See you in a pull request soon ;-)
    bpetit
    @bpetit
    Hi @slanglois ! That's great to hear ! See you in a pull request then ! :)
    bpetit
    @bpetit

    Hi !

    Working on #109 I've changed a bit the way MetricGenerator can be used. It now owns the Topology struct (it held a reference previously). It makes easier having the MetricGenerator struct instantiated outside of the loop of most exporters and thus optimize performances (holding counters for instance to create some cache-like mecanisms). My case was with prometheus exporter, when using --containers, to not query the docker socket to often.

    Please have a look if your are interested. A company interested in the feature (extra labels for processes metrics when in a docker container) will test it before I merge. Feel free to tell me your thoughts about this (and the feature) in the mean time.

    cc @uggla
    olivier de Meringo
    @demeringo

    Hi !
    Do you use scaphandre in CI ?
    I would like to measure the energy consumed by a test suite to get a kind of power score for different iterations of an application code.

    My naive approach is the following:

    • use scaphandre container customized (with the application under test + test suite)
    • start scaph (json export) for a fixed time
    • run the test
    ... and retrieve the json restults as CI artefacts...
    But there surely must be a better way, how do you do this ?
    olivier de Meringo
    @demeringo
    This is a micro project where I use gitlabci + scaph (and simulate a test run with stress-ng).
    The gitlab runner is a custom runner on a Linux laptop.
    bpetit
    @bpetit

    Hi @demeringo !

    I'd say you could store metrics in a TSDB and export graphs for a given time window through grafana API for example.
    I don't have a detailed procedure available but it seems completely possible.

    Depends on what you want to do with the data afterwards
    René Ribaud
    @uggla
    @demeringo , interesting use case. I think Riemann is a good candidate for this ! Riemann is working with a push model. So you can imagine the following pipeline: (ci image with scaphandre) --> (Riemann) --> (Influx DB or ES)
    Then you can see live data in the Riemann dashboard and TS data (history) in influx with Grafana or ES with (Grafana ou Kibana)
    René Ribaud
    @uggla
    By the way I could slightly modify Scaphandre to add metadata (keys/values) to Riemann, so your scaph client could send these metadata (build id, etc...) alongside with power/process values.
    olivier de Meringo
    @demeringo
    @uggla @bpetit great idea ! I did not think about using Riemann, the push model is interesting in this case. I will try it.
    bpetit
    @bpetit
    You can also use warp10, or prometheus with a pushgateway, if that's the push mode you are looking for. Just to mention :)
    René Ribaud
    @uggla
    @demeringo , yes I think Riemann is underestimated. @bpetit I did not know about the pushgateway, do you speak about this project https://github.com/prometheus/pushgateway ?
    René Ribaud
    @uggla
    https://prometheus.io/docs/instrumenting/pushing/ seems more complex to use as it will required to send scaphandre metrics to the prom gateway with another program or script. Unless if I'm wrong or use Riemann https://riemann.io/api/riemann.prometheus.html. :) (we can not beat Riemann ! :D )
    bpetit
    @bpetit
    yes you're right, we did not develop the feature to send metrics to the push gateway :)
    René Ribaud
    @uggla
    Anyway warp10 is the simplest solution. As it requires only to setup warp10.
    To note that vector (https://vector.dev/) can be interesting too. By the way it seems datadog acquired it.
    bpetit
    @bpetit

    Hi there !

    Short but kind of urgent message in french :

    Le magazine https://www.programmez.com/ nous demande si on peut écrire un article (en français) pour présenter scaphandre. L'idée serait de décrire un use case standard de surveillance de la conso d'une application en développement, de version en version. Un dashboard grafana, un scénario gatling ou équivalent et quelques modifs dans la base de code d'une app pour donner un exemple de réduction de conso lors d'un test de charge, dans l'idée.

    La mauvaise nouvelle c'est que ce serait pour le 5 août (semaine prochaine). C'est ma faute j'ai laissé trainé et maintenant c'est le rush ^^

    Si quelqu'un a un use case en stock sur son laptop et veut bien en parler dans un article, qu'il/elle me fasse signe !

    Merci !

    (pas de rémunération prévue, mais toute la gratiture de la part des contributeurs de scaphandre et j'offre un verre sans probleme si on est dans le même coin ^^)
    (PS: vous pouvez en profiter pour parler au nom de votre boite, ou de vous meme si vous etes freelance et vous faire un peu de pub, du moment que vous donnez les infos sur le projet)

    René Ribaud
    @uggla
    @bpetit , arf chaud en terme de temps dispo. Surtout que je viens juste de voir le msg. J'ai peut être qq chose qui pourrait être utile. J'ai en préparation pour une conf du code qui calcule des nombres premiers et cela me permet de debunker certaines idées reçus. Bref on pourrait imaginer de lancer le truc avec un algo naif qui devrait donc consommer un max de power (car beaucoup de cpu). Puis ensuite de lancer la version avec un meilleur algo (Crible d'Ératosthène) et voir que cela consomme moins. Le use case est pas super réaliste et fou, mais l'idée serait de montrer la corrélation efficacité algo et power / usage de la batterie. Et que monitorer la conso permet de trouver se genre de chose.... Par contre ça me semble un peu chaud pour fournir ça d'ici à Jeudi avec la semaine qui arrive. Après il y a une chance, c'est que Rose est chez ses grand parents donc j'ai un peu plus de temps libre cette semaine. Dis moi vite si ça le ferait ou si tu as des propositions pour améliorer le sujet, et si on peut prévoir une session pour avancer/relire/améliorer le truc si ça te semble une bonne idée.
    4 replies
    Ross Fairbanks
    @rossf7
    Hi @bpetit just to say I'm back after my break. So if I can help with the kubernetes integration just let me know. Cheers Ross
    3 replies
    bpetit
    @bpetit
    Hi ! I'm back after a short break. PRs should be moving on soon ! Thanks a lot for the work you have done so far on those PRs ! :)
    1 reply
    bpetit
    @bpetit

    Hi !

    @rossf7 did you see this ticket ? https://github.com/hubblo-org/scaphandre/issues/122#issuecomment-914061716

    It seems to me the only way to install scaphandre with helm for now is to use the local helm folder as the chart right ?

    3 replies
    bpetit
    @bpetit
    Solved, don't mind my last message :)
    bpetit
    @bpetit

    Hi there ! I have a Rust question for anyone able to answer it :)

    Here is my code :

    let top_consumers = consumers
                .iter()
                .filter_map(|(process, _value)|{
                    if let Some(metric) = metrics.iter().find(|x| {
                        x.name == "scaph_process_power_consumption_microwatts" 
                        && process.pid == x.attributes.get("pid").unwrap().parse::<i32>().unwrap()
                    }) {
                        Some(
                            Consumer {
                                exe: PathBuf::from(metric.attributes.get("exe").unwrap()),
                                pid: process.pid,
                                consumption: format!("{}", metric.metric_value).parse::<f32>().unwrap(),
                                timestamp: metric.timestamp.as_secs_f64(),
                            }
                        )
                    } else { None }
                })
                .collect::<Vec<_>>();

    I want to to isolate Process items (from the consumers vec) with their Metric cousin (from the metrics vec) and finally have a vec containing Consumer items with data from the Process and the Metric.

    This code compiles, but clippy gives me this warning :

    warning: manual implementation of `Option::map`
       --> src/exporters/json.rs:196:17
        |
    196 | /                 if let Some(metric) = metrics.iter().find(|x| {
    197 | |                     x.name == "scaph_process_power_consumption_microwatts" 
    198 | |                     && process.pid == x.attributes.get("pid").unwrap().parse::<i32>().unwrap()
    199 | |                 }) {
    ...   |
    207 | |                     )
    208 | |                 } else { None }
        | |_______________________________^
        |
        = note: `#[warn(clippy::manual_map)]` on by default
        = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#manual_map
    help: try this
        |
    196 |                 metrics.iter().find(|x| {
    197 |                     x.name == "scaph_process_power_consumption_microwatts" 
    198 |                     && process.pid == x.attributes.get("pid").unwrap().parse::<i32>().unwrap()
    199 |                 }).map(|metric| Consumer {
    200 |                             exe: PathBuf::from(metric.attributes.get("exe").unwrap()),
    201 |                             pid: process.pid,
      ...
    
    warning: 1 warning emitted

    I'd like to make him happy but I struggle to find a similar way to do that and avoid that warning :)

    I need to return either Some or None for filter_map, so using map as proposed by clippy doesn't work...

    any advice @uggla @PierreRust or someone else ? :)

    (its for PR #113 by the way :P)
    bpetit
    @bpetit
    It seems I was just overthinking the requirements of filter_map, the answer is there: https://users.rust-lang.org/t/trapped-in-clippys-warning-manual-implementation-of-option-map/65118/4 :)
    René Ribaud
    @uggla
    Du coup je comprends que c'est ok ? Tu confirmes ?