atruskie on nsw-dpi-recognizers
atruskie on master
Stub out initial recogniser fil… Updated all recognisers with ne… Bumped CSV version and fixed To… and 17 more (compare)
atruskie on nsw-dpi-recognizers
Fix bugs introduced in feedback… Fixed broken recognisers @tows… (compare)
atruskie on nsw-dpi-recognizers
Stub out initial recogniser fil… Updated all recognisers with ne… Bumped CSV version and fixed To… and 15 more (compare)
atruskie on Issue466-FixWritingOfVoidList
atruskie on master
Fix bug in EventFilters.cs See… Merge pull request #475 from Qu… (compare)
hi @panchorivasf ,
1) We call that configuration value
IndexCalculationDuration. See here https://github.com/QutEcoacoustics/audio-analysis/blob/master/src/AnalysisConfigFiles/Towsey.Acoustic.yml#L22-L27
2) Just because the audiomoths default to 1-minute chunks doesn't mean that's a good size audio file. Record in segments that suit your study. The overhead in metadata and filesystem load for thousands of small files can be significant.
To answer your question, the basic process is to:
concatenateindexfilesto stitch the results together
Here is an example script https://github.com/QutEcoacoustics/audio-analysis/blob/master/scripts/indices_and_concat.ps1 , and some guidance from our documentation: https://ap.qut.ecoacoustics.info/technical/commands/concatenate_index_files.html https://ap.qut.ecoacoustics.info/basics/scripting.html
This process can be extremely inefficient for lots of small files. It's something we're looking to improve upon. See QutEcoacoustics/audio-analysis#378
3) Michael (@towsey) did plot day phases. The middle FCS in slide 13 has day phases attached in https://research.ecosounds.org/research/eadm-towsey/long-duration-audio-recordings-of-the-environment#slide-13-cross-site-comparison-varying-latitudes.
As for temps, I don't think so
@atruskie Awesome! I'll navigate through those commands and see what happens. Did Michael attach the day phases manually (e. g. using some graphics software)?
He did it in AP - but we removed that code, it only worked in one area and wasn't very good too
You can attach any image you want simply by placing it underneath the image. For a IndexCaculationDuration of 60, each column of pixel is 1 minute long. So to produce a graph that is 24 hours long, make an image 1440px wide
Ah, that's a holdover from past technicalities. It used to be we couldn't do too much at one time or else we'd run out of memory. Capping the rendering resolution to 60 kept the indices generation fast.
Anyway, generate indices first with
audio2csv then use
DrawLongDurationSpectrograms to make the images
there is distinction betwen arguments and options.
arguments: yes, order is important. Arguments are values for which you don't have to put their name in front. Example:
AnalysisPrograms.exe audio2csv C:\Audio Towsey.Acoustic.yml C:\output <-- there are three arguments there, each separated by a space
options: order is not important. Options are always prefixed with their name (like
--input-data-directory) and then the value. Example:
--help should be available on every command. Even though I've written most of AP.exe, I still regularly forget things.
--help is the first thing I do everytime.
Some commands will take arguments, some options, and some both. Techincally an option is a special type of argument (hence my confusing statement)
Towsey.Acousticconfig file with
audio2csvto calculate acoustic indices for short (1 minute) files. It works well; however, it generates a bunch of other analyses that I am not using and I now want to unleash it on ~10 TB of 1-minute files. Essentially I am interested in only creating this file for each sound file:
*__Towsey.Acoustic.Indices.csv. Is this possible? can you point me in the right direction? Thanks for a great tool!
@abfleishman the short answer is don't do this.
AP.exe is not made for processing 1-minute files - it will perform slowly and will be quite inefficient. 1-minute files are a bad choice for storing and collecting data, especially for consecutive minutes. Unfortunately, people keep doing this, so we intend to add support for it (QutEcoacoustics/audio-analysis#378); however, it will be a while before that is implemented.
With the warning out of the way: it's possible. We've done it before, on datasets larger than 10TB.
, it generates a bunch of other analyses
Slightly incorrect. It generates other sorts of indices, which we call spectral indices. They are very much acoustic indices though.
Currently, we do not have the options to calculate just a subset of the indices. It's been suggested a few times and when I went searching for the relevant feature request, I found your name mentioned!
All I can say is sorry. We've been under a lot of pressure. For our typical use case (2, or 6-hour long files) the default settings make sense and aren't that annoying, even when we don't need a subset of the files which makes it a low priority.
I'd recommend just deleting the files you don't need after each loop. It'd be worthwhile keeping some of the metadata (like the index generation data json).
@abfleishman yeah, I get there are plenty of valid reasons to sample data. The issue I keep running into is people doing it by default, especially for consecutive minutes. Small files create huge strain on filesystems... and I'm over here trying to write software that works for not just their project, but every project we get all at once.
FWIW, we've been collecting data on continuous schedules by default. With our new sensors service time is every 6 or 12 months.
And yeah, AP.exe has no obvious issue working with short files, but audio2csv in particular expects it. It does a bunch of stuff like duration checking, chunking, minute alignment, and outputting data files in shapes that really don't make sense for short files. It's --parallel option is also unused.
Anyway /rant :-)
My contract is changing soon and I'll be back to working on AP.exe more regularly. Hopefully I'll be able to provide some good news soon!
@fcorivasf_gitlab no there isn't. Sounds like a good feature request? Do you want to add it the feature request to the issue tracker?
Historically, we intended the images to sit side by side for quick and dirty alignments without any padding.
We're encountering cases now we're people are looking for prettier images and aesthetic options like this are becoming more important
@mitesser no, not currently. The best resource is Towsey's whitepaper on the topic - see https://eprints.qut.edu.au/110634/
The documentation is a much newer (part-time labour of love) than the indices and it is nowhere near finished. We aim to merge that white paper into our documentation.
The index you're looking for is PMN (power minus noise). The true power could be obtained by doing
PMN + BGN if you need it (I'll double check that and get back to you)
The tutorial mentions “We recommend you do not aggregate data files together to create a composite view of a day of data.” Our data are however in 10 minute recordings - is there a way to identify a folder of data instead of one long recording and if not, is there really a problem when compiling multiple 10 minute files into a single way file?
I get an error shows up when processing dataset, which I think relates to 24 bit format of audio files. An output is still derived but just in case, is this “unrecognized media” warning important? And can you specify that your data is 24-bit data as opposed to 16 bit?
For the joining of audio files: Every audio file produced by a typical sensor includes metadata in the file's header. If you join files together, that metadata will not be preserved. Also, most recorders leave a small gap between recordings. So, joining files together introduces a small amount of time error. There are exceptions to both of these rules, but it is generally easier to not mutate source data. We recommend writing a script to analyse multiple files. Here is an example that might do what you want: https://github.com/QutEcoacoustics/audio-analysis/blob/master/scripts/indices_and_concat.ps1
We should support 24-bit audio files. If there is an error, please file a bug report.
Our indices analysis be default resamples files to 22050Hz. You can change the value in the
Towsey.Acoustic.yml config file to keep your original sample rate.
The frequency bin width is essentially
sample_rate / window_size = bin_width. Thus the default is
22050 / 512 = 43.06640625 Hz. That will change if you change either the
FrameLength settings in the config file