These are chat archives for cloudfoundry-community/snotel

15th
Aug 2017
abeguerra
@abeguerra
Aug 15 2017 19:34
I am trying to use snotel to gather data on cloud foundry usage. I'd like to store a subset of the firehose data in DB for querying/analysis.
        firehose.open().filter( envelope -> {
            if((envelope.eventType.toString().equals("HttpStartStop") && null != envelope.httpStartStop.applicationId)
                    || envelope.eventType.toString().equals("ContainerMetric")){
                return Boolean.TRUE;
            }else{
                return Boolean.FALSE;
            }
        })
        .forEach( envelope -> {
            switch(envelope.eventType.toString()) {
                case "HttpStartStop":
                    httpReqDocs.add(createEventDoc(envelope));
                    if(httpReqDocs.size() == 250) {
                        List<DocumentOperationResult> errorDocs = dbHttpReqs.executeBulk(httpReqDocs);
                        // FIXME - Use logging
                        if(errorDocs.size() > 0) System.err.println("There was a problem saving stream data");
                        httpReqDocs.clear();
                    }
                    break;
                case "ContainerMetric":
                    containerMetricDocs.add(createEventDoc(envelope));
                    if(containerMetricDocs.size() == 250) {
                        List<DocumentOperationResult> errorDocs = dbContainerMetric.executeBulk(containerMetricDocs);
                        // FIXME - Use logging
                        if(errorDocs.size() > 0) System.err.println("There was a problem saving stream data");
                        containerMetricDocs.clear();
                    }
                    break;
            }
        });
abeguerra
@abeguerra
Aug 15 2017 19:42
seems to be moving along and then it just stops loading events into my db. Is there something blatantly wrong with this approach? I'm not sure where/how to begin debugging. I extended the NettyFirehoseOnSubscribe so that I can print the message being processed by the channelRead methods at time of failure. The message at the time of failure is CloseWebSocketFrame and shortly above it I often see a PingWebSocketFrame. It makes me think my java app is not processing the stream fast enough. Any help/advice/pointers would be greatly appreciated.