These are chat archives for OpenCosmics/fits-evaluation

5th
Jun 2015
Anirudha Bose
@onyb
Jun 05 2015 07:49
So what are we working on right now?
Achintya Rao
@RaoOfPhysics
Jun 05 2015 07:51
I’ll let @berghaus answer. :P
Frank Berghaus
@berghaus
Jun 05 2015 07:57
So we now have nicely formatted the event and station data
we want to pull together the location and text description of the station data with the event description
Achintya Rao
@RaoOfPhysics
Jun 05 2015 07:57
See #9 as well.
Frank Berghaus
@berghaus
Jun 05 2015 07:57
and then output these in the formats discussed yesterday
Anirudha Bose
@onyb
Jun 05 2015 08:01
@berghaus: So we should add geolocation data in our event dictionary (JSON)?
Frank Berghaus
@berghaus
Jun 05 2015 08:02
Yeah, it would be sipping together the location information with the event data
There should be GPS coords, an altitude parameter if there is one
and a free text field for a description of the detector (maybe we can put the name of the institution in there for now
Anirudha Bose
@onyb
Jun 05 2015 08:04
Okay. I am looking into these now. Thanks for the input.
Frank Berghaus
@berghaus
Jun 05 2015 08:07
OpenCosmic.png
I made a mind map during out discussion of data yesterday, I'll post it here for now. Will need to update it wrt the data we have just gathered.
Anirudha Bose
@onyb
Jun 05 2015 08:09
What does the 'direction' field contain?
Anirudha Bose
@onyb
Jun 05 2015 08:20
@berghaus: Under what field do we keep the event data? Note that the event summary is specific to the time interval we provide. So I guess the interval must also be a part of the format.
Hugo Day
@Tontonis
Jun 05 2015 08:34
Hi all, yes station data object should have enough keywords latitude longitude and altitude already from the get_station_data func
(will be all text today, mostly sat around in an airport)
Anirudha Bose
@onyb
Jun 05 2015 09:02
@Tontonis Does this make sense? https://gist.github.com/onyb/110c2338fd3ad1a17633
Frank Berghaus
@berghaus
Jun 05 2015 09:28
had to run off for a while, back now
would it be sensible to combine the date, time and nanoseconds fields?
There is a loy of station information there, we probably don't need all of it
we need the date and time from the event. The date and time ranges from the station we could throw away
let's take a step back fro a moment, we need the fields for the event to be pretty much free to be different key:value pairs that can be defined by the experiments
it would be really good to have one of the guys actually involved in teh experiments connected :)
Anirudha Bose
@onyb
Jun 05 2015 09:35
@berghaus: Should the JSON have only one event, or a list of events within a given range?
Sahal Yacoob
@SirHalie
Jun 05 2015 09:39
As long as the list CAN be only one event long, I guess a list would be better
Frank Berghaus
@berghaus
Jun 05 2015 09:45
Just one event
Anirudha Bose
@onyb
Jun 05 2015 09:45
Single event in one JSON document makes sense. This way we can do away with the date-time ranges. What do others think?
Okay, got it.
Frank Berghaus
@berghaus
Jun 05 2015 09:46
yes onyb that is what i was thinking as well
then we can query for lists of events later right?
So an entry should be an event with fields for: date, time, location, a text description, plus the event information.
Anirudha Bose
@onyb
Jun 05 2015 09:51
Sounds okay to me. Let me get a patch ready for this.
Achintya Rao
@RaoOfPhysics
Jun 05 2015 10:07
@berghaus and @SirHalie currently arguing about what the datasets should include. :P More soon, with pictures.
Anirudha Bose
@onyb
Jun 05 2015 10:08
Waiting for the results. :smile:
Anirudha Bose
@onyb
Jun 05 2015 10:28
@berghaus: We can use RFC 3339 standard to combine date and time with precision upto nanoseconds.
Anirudha Bose
@onyb
Jun 05 2015 10:29
ISO 8601 also solves the same purpose.
We can just ignore the UNIX timestamp. The combibation of the 'date', 'time', and 'nanoseconds' fields can be used to construct the time according to RFC 3339 or ISO 8601.
Achintya Rao
@RaoOfPhysics
Jun 05 2015 10:34
Thanks, noted. Now we’re haggling over what kind of time information is needed depending on detector type. More soon.
Achintya Rao
@RaoOfPhysics
Jun 05 2015 10:54
@onyb: We’re breaking for lunch. We’ll send you our discussion feedback soon. Sadly, I think it will be quite late in the day for you. :(
If I hear from ERGO and/or CERN@School, I’ll point you to their data so you can figure out how to extract it.
Anirudha Bose
@onyb
Jun 05 2015 10:55
No problem. If I am not in Gitter, ping me on my email when you guys are back.
Achintya Rao
@RaoOfPhysics
Jun 05 2015 12:17
Ok, back from lunch but not much more progress on agreeing on the data needed to be recorded. Soon, I guess.
Anirudha Bose
@onyb
Jun 05 2015 12:25
Any inputs on what discussions have been done till now?
Frank Berghaus
@berghaus
Jun 05 2015 12:27
we had a long discussion about how to structure our output, mostly thinking about the use cases for different detectors
let me see if i can come up with a sensible graphic to post, give me a few minutes
Anirudha Bose
@onyb
Jun 05 2015 12:28
@berghaus: The timestamp doesn't contain any information about the timezone. This is going to vary across different stations. Have you thought about this issue?
Achintya Rao
@RaoOfPhysics
Jun 05 2015 12:30
@onyb: Is this from the HiSPARC data?
Anirudha Bose
@onyb
Jun 05 2015 12:32
@RaoOfPhysThecs: Yes. I UNIX timestamps as well as the 'time' and 'nanoseconds' fields are local times.
Achintya Rao
@RaoOfPhysics
Jun 05 2015 12:32
Can this be off-set based on the GPS coördinates?
Anyway, comes down to use-cases.
If you want to measure how the flux varies with time of day, local times are needed.
But I’m not the right person to address this.
Anirudha Bose
@onyb
Jun 05 2015 12:36
@RaoOfPhysics: Can be done based on the coordinates. Your point is good. We can keep local time and fetch offset information from GPS coordinates.
RFC 3339 time looks like this: 2015-06-05 17:11:15.006895000+05:30
So we will need offset information any way.
Achintya Rao
@RaoOfPhysics
Jun 05 2015 12:48
OT: IST FTW. Sigh.
Yeah, if we’re following those specifications, then we can include the offset, which can be converted to UTC as needed.
Frank Berghaus
@berghaus
Jun 05 2015 12:51
OpenCosmic.png
idea after discussion
the "i" icon means these values should allow a value for uncertainty
SO next steps are to get this into JSON or FITS
Anirudha Bose
@onyb
Jun 05 2015 12:53
@berghaus: Why a start and end rage for time? Do you plan to have multiple events?
Achintya Rao
@RaoOfPhysics
Jun 05 2015 12:54
@onyb: Can you investigate whether FITS can be mapped to a JSON schema? (Does that make sense?)
Sahal Yacoob
@SirHalie
Jun 05 2015 12:55
@onyb: We're trying to account for detectors which will not report every count as an event, but where an 'event' is a time window. Time start is required, time end is for detectors what reports counts in a time window.
Anirudha Bose
@onyb
Jun 05 2015 12:58
So in case of HiSPARC data, we will have more than one events in the "Measurement" field?
Patricia Herterich
@pherterich
Jun 05 2015 12:58
Looks like there are APIs that have FITS and JSON as output options, they have to have an internal mapping then http://api.sdss3.org/api_spectrum.html
Frank Berghaus
@berghaus
Jun 05 2015 13:10
No, single event
just the integral, number_of_mips, and pulse_heights fields I think think
The end-time can be left out for this,
Anirudha Bose
@onyb
Jun 05 2015 13:11
Got it.
Frank Berghaus
@berghaus
Jun 05 2015 13:12
@SirHalie pointed out that there are detector designs that only report an accumulation after a time. The idea for this field is to support those detectors
but we should ignore it until we come across such a detector
Anirudha Bose
@onyb
Jun 05 2015 13:14
What about "livetime"?
Achintya Rao
@RaoOfPhysics
Jun 05 2015 13:15
@berghaus’s mindmap migrated to online viewer: https://www.mindmup.com/#m:a1ac5cdf30edb20132dfcf1222860cd564
Frank Berghaus
@berghaus
Jun 05 2015 13:17
The goal for this would be to capture the time at which the detector was switched on
Anirudha Bose
@onyb
Jun 05 2015 13:18
How is it different from the Time -> Start field?
Frank Berghaus
@berghaus
Jun 05 2015 13:18
you are making a good obervation here
this is not well named
the time->start should be the time the event was recoded
the naming here is ambiguous
we should come up with something better
OK I've updated the map that @RaoOfPhysics linked
so we just have an event time
and for those special detectors they can fill the window field
otherwise (our case) we just set it to 0
Achintya Rao
@RaoOfPhysics
Jun 05 2015 13:26
@berghaus: Did you save the map? It doesn’t auto-save. Button on top RHS.
Frank Berghaus
@berghaus
Jun 05 2015 13:31
RAO said he could not see it before
Achintya Rao
@RaoOfPhysics
Jun 05 2015 13:32
Yup, the URL changes everytime you hit “Save”. For future reference, in case anyone else is editing the map.
Anirudha Bose
@onyb
Jun 05 2015 13:37
@berghaus : How can I get the value of Livetime in the HiSPARC data set?
Should I set it to 0 (or None) ?
Frank Berghaus
@berghaus
Jun 05 2015 13:40
Yeah set it to 0
actually None is better
for both window and livetime
Anirudha Bose
@onyb
Jun 05 2015 13:42
Python will automatically convert NoneType to appropriate type while dumping to other formats. In case of JSON, it will be coverted to null during a JSON dump.
Achintya Rao
@RaoOfPhysics
Jun 05 2015 13:42
Maybe @153957 can comment on whether there’s any livetime info.
Anirudha Bose
@onyb
Jun 05 2015 13:45
Created a pull request #11 .
Anirudha Bose
@onyb
Jun 05 2015 13:52
An excerpt from the JSON output: https://gist.github.com/onyb/67eab81bd10588956a05
Frank Berghaus
@berghaus
Jun 05 2015 13:53
looks fantastic
Anirudha Bose
@onyb
Jun 05 2015 13:53
:smile:
Achintya Rao
@RaoOfPhysics
Jun 05 2015 13:55
@onyb: Just wanted to confirm if the UTC offset will be included under time.
Anirudha Bose
@onyb
Jun 05 2015 13:56
@RaoOfPhysics: Working on that now. Looking for an API which can resolve geolocation into the correct offset.
Achintya Rao
@RaoOfPhysics
Jun 05 2015 13:56
@onyb: Cool. :D
Anirudha Bose
@onyb
Jun 05 2015 13:58
@berghaus: Don't merge it. I am pushing one more commit.
Frank Berghaus
@berghaus
Jun 05 2015 13:58
OK I'll wait
Arne de Laat
@153957
Jun 05 2015 13:59
livetime? do you mean when a station was active (online/measuring)?
Frank Berghaus
@berghaus
Jun 05 2015 13:59
correct
the time the station "turned on" if you will
Arne de Laat
@153957
Jun 05 2015 14:01
That is not available. A good estimate can be made by looking at the number of events the station has been detecting.
Oh, Do you mean the first date the station turned on? (even if it may have been down in between)?
Achintya Rao
@RaoOfPhysics
Jun 05 2015 14:03
@153957: For each event, the last time the detector was turned on, I believe.
Arne de Laat
@153957
Jun 05 2015 14:04
Ok, nope.
Achintya Rao
@RaoOfPhysics
Jun 05 2015 14:04
@153957: Thanks for confirming.
Frank Berghaus
@berghaus
Jun 05 2015 14:13
Well we could extract it from the station data
for the station, the last time it was turned on
clarification: just wondering if this information is available in a different context
Arne de Laat
@153957
Jun 05 2015 14:22
The best way (?) would possibly by looking at the number of events and see if there way a moment when the station was not measuring any event.
Anirudha Bose
@onyb
Jun 05 2015 14:22
@berghaus: Done.
Arne de Laat
@153957
Jun 05 2015 14:23
On average a 4-detector station detects ~60 000 events per day. So you can look for dips.
However, please do not query our server for the number of events per day for extended periods many times.. it probably doesnt like that..
Achintya Rao
@RaoOfPhysics
Jun 05 2015 14:30
@153957: We’ll keep that in mind. :)
Anirudha Bose
@onyb
Jun 05 2015 14:49
@RaoOfPhysics: Does BST translate to UTC +2:00 if daylight savings is taken into account?
Frank Berghaus
@berghaus
Jun 05 2015 15:17
the brits use UTC (GMT) when it's not summertime there
James Devine
@pingud98
Jun 05 2015 15:18
hello world
Frank Berghaus
@berghaus
Jun 05 2015 15:19
hi there
Sahal Yacoob
@SirHalie
Jun 05 2015 15:22
Hola!
Achintya Rao
@RaoOfPhysics
Jun 05 2015 15:23
@onyb: Ok, I need to think about this a bit more.
Anyway, it’s late in the night in India and you should sign off from the sprint! :P
Anirudha Bose
@onyb
Jun 05 2015 15:47
@RaoOfPhysics: I am having a good time collaborating with you guys. :smile: