Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
    Markus Bukowski
    @mrbuk
    Hey there it seems that the current version of snotel is using an old dropsonde protocole definition
    is there a plan to fix that?
    (i see that there are many changes required, as i tried to do it myself)
    abeguerra
    @abeguerra
    I am trying to use snotel to gather data on cloud foundry usage. I'd like to store a subset of the firehose data in DB for querying/analysis.
            firehose.open().filter( envelope -> {
                if((envelope.eventType.toString().equals("HttpStartStop") && null != envelope.httpStartStop.applicationId)
                        || envelope.eventType.toString().equals("ContainerMetric")){
                    return Boolean.TRUE;
                }else{
                    return Boolean.FALSE;
                }
            })
            .forEach( envelope -> {
                switch(envelope.eventType.toString()) {
                    case "HttpStartStop":
                        httpReqDocs.add(createEventDoc(envelope));
                        if(httpReqDocs.size() == 250) {
                            List<DocumentOperationResult> errorDocs = dbHttpReqs.executeBulk(httpReqDocs);
                            // FIXME - Use logging
                            if(errorDocs.size() > 0) System.err.println("There was a problem saving stream data");
                            httpReqDocs.clear();
                        }
                        break;
                    case "ContainerMetric":
                        containerMetricDocs.add(createEventDoc(envelope));
                        if(containerMetricDocs.size() == 250) {
                            List<DocumentOperationResult> errorDocs = dbContainerMetric.executeBulk(containerMetricDocs);
                            // FIXME - Use logging
                            if(errorDocs.size() > 0) System.err.println("There was a problem saving stream data");
                            containerMetricDocs.clear();
                        }
                        break;
                }
            });
    abeguerra
    @abeguerra
    seems to be moving along and then it just stops loading events into my db. Is there something blatantly wrong with this approach? I'm not sure where/how to begin debugging. I extended the NettyFirehoseOnSubscribe so that I can print the message being processed by the channelRead methods at time of failure. The message at the time of failure is CloseWebSocketFrame and shortly above it I often see a PingWebSocketFrame. It makes me think my java app is not processing the stream fast enough. Any help/advice/pointers would be greatly appreciated.