## Where communities thrive

• Join over 1.5M+ people
• Join over 100K+ communities
• Free without limits
##### Activity
• Mar 05 05:35

atruskie on nsw-dpi-recognizers

Fix merge conflicts and update … (compare)

• Mar 05 05:30

atruskie on nsw-dpi-recognizers

Fixed bug in when removing encl… resolved merge conflict (compare)

• Mar 05 05:09

atruskie on nsw-dpi-recognizers

Added validation to config file… Work on recognizers Glossy-bla… (compare)

• Mar 05 00:03

atruskie on FixBugWithRemoveEnclosedEvents

• Mar 05 00:03

atruskie on master

Fixed bug in when removing encl… (compare)

• Mar 05 00:03
atruskie closed #462
• Mar 05 00:03
atruskie opened #462
• Mar 04 23:43

towsey on FixBugWithRemoveEnclosedEvents

Fixed bug in when removing encl… (compare)

• Mar 04 03:30
atruskie edited #461
• Mar 04 02:31
atruskie edited #461
• Mar 04 00:56
atruskie edited #461
• Mar 04 00:50
atruskie assigned #461
• Mar 04 00:50
atruskie labeled #461
• Mar 04 00:50
atruskie opened #461
• Mar 04 00:38
atruskie closed #451
• Mar 03 23:45

atruskie on nsw-dpi-recognizers

Fix merge commit, add noise red… First draft of glossy-black coc… (compare)

• Mar 03 05:02

atruskie on nsw-dpi-recognizers

• Mar 03 04:48

atruskie on Issue451_DIYRecognizerFeedback

• Mar 03 04:48
atruskie closed #459
• Mar 03 04:48

atruskie on master

panchorivasf
@panchorivasf
Hi everyone! Like many of you, I'm a newcomer to AP so I come with few silly questions for the team:
1) How can I script AP to change the time window at which it calculates the AIs? I have a bunch of full-day recordings with a non-continuous recording schedule: 1min-on / 9min-off. Obviously, when I run AP for a full day of recordings, it shows a 2.5 long spectrogram (or around that), which is not very useful. So I wonder if I can set the AI calculation to 1 sec per pixel.
(above I meant 2.5 hour-long spectrogram)
panchorivasf
@panchorivasf
2) What happens if I want to load a full day of continuous recordings that are already in 1 min chunks? This is the default for Audiomoths and I generally prefer to use this schedule because it allows me to fast scanning with Adobe Audition spectrogram view. By the moment, I've been putting all recordings together in a single file before loading into AP, but it seems redundant that the program will cut it into 1 min chunks again.
(sorry if that was too silly)
3) Have someone out there plotted temperature (and/or humidity, light metrics, etc.) over a false-color spectrogram? That would be amazing!
Anthony Truskinger
@atruskie

hi @panchorivasf ,

1) We call that configuration value IndexCalculationDuration. See here https://github.com/QutEcoacoustics/audio-analysis/blob/master/src/AnalysisConfigFiles/Towsey.Acoustic.yml#L22-L27

2) Just because the audiomoths default to 1-minute chunks doesn't mean that's a good size audio file. Record in segments that suit your study. The overhead in metadata and filesystem load for thousands of small files can be significant.

1. loop through each file, calculate indices for each file
2. use concatenateindexfiles to stitch the results together

Here is an example script https://github.com/QutEcoacoustics/audio-analysis/blob/master/scripts/indices_and_concat.ps1 , and some guidance from our documentation: https://ap.qut.ecoacoustics.info/technical/commands/concatenate_index_files.html https://ap.qut.ecoacoustics.info/basics/scripting.html

This process can be extremely inefficient for lots of small files. It's something we're looking to improve upon. See QutEcoacoustics/audio-analysis#378

3) Michael (@towsey) did plot day phases. The middle FCS in slide 13 has day phases attached in https://research.ecosounds.org/research/eadm-towsey/long-duration-audio-recordings-of-the-environment#slide-13-cross-site-comparison-varying-latitudes.

As for temps, I don't think so

panchorivasf
@panchorivasf
@atruskie I was reading the ConfigFiles parameters, and this warning puzzled me: # IMPORTANT NOTE: FC spectrograms are produced only IF the above IndexCalculationDuration := 60.0. So if I reduce the ICD, there will be no FC spectrogram output? or is that a warning for when you want to change FCS's color parameters only?
Anthony Truskinger
@atruskie

@panchorivasf :

@atruskie Awesome! I'll navigate through those commands and see what happens. Did Michael attach the day phases manually (e. g. using some graphics software)?

He did it in AP - but we removed that code, it only worked in one area and wasn't very good too

You can attach any image you want simply by placing it underneath the image. For a IndexCaculationDuration of 60, each column of pixel is 1 minute long. So to produce a graph that is 24 hours long, make an image 1440px wide

Anthony Truskinger
@atruskie

Ah, that's a holdover from past technicalities. It used to be we couldn't do too much at one time or else we'd run out of memory. Capping the rendering resolution to 60 kept the indices generation fast.

Anyway, generate indices first with audio2csv then use DrawLongDurationSpectrograms to make the images

panchorivasf
@panchorivasf
Awesome, thank you @atruskie !
Francisco Rivas Fuenzalida
@fcorivasf_gitlab
Hi again! I don't know how does the workflow goes to make the LDSs afterwards. I'm typing: C:\AP\AnalysisPrograms.exe DrawLongDurationSpectrograms
(...) and then I guess I should give the path to the csv file right?
Anthony Truskinger
@atruskie

The first thing to do is use the --help option to see what arguments the command takes

AnalysisPrograms.exe DrawLongDurationSpectrograms --help

Then see what happens

Francisco Rivas Fuenzalida
@fcorivasf_gitlab
Thanks I did. The thing I don't get yet is how many arguments are required to make the spectrogram.
Anthony Truskinger
@atruskie

It's been a while - so I don't know for sure!
But the simple answer is as many as needed 😂

Looking at the help docs, probably at least -i, -o, -ip, -fcs

Francisco Rivas Fuenzalida
@fcorivasf_gitlab
ok I'll keep trying, thanks!
when I typed the path for -fcs it gives me an error message saying it was waiting for a .yml instead of a .csv
Anthony Truskinger
@atruskie
mmm bad documentation - i'll fix that
But it is looking for a config file, you can infer that from the rest of the description -fcs|--false-colour-spectrogram-config <FALSE_COLOUR_SPECTROGRAM_CONFIG>
it'll be the ConfigFiles/SpectrogramFalseColourConfig.yml one
Updated, you'll see it inlcuded in Monday's automated release of AP.exe QutEcoacoustics/audio-analysis@f78c09a
Francisco Rivas Fuenzalida
@fcorivasf_gitlab
Let me see if I understand. The actual file from which AP will make the LDS must be given after "-i"?
Anthony Truskinger
@atruskie

well the directory, yeah. The output of indices generation is the input to image drawing

generate indices ➡ draw false colour spectrogram

Francisco Rivas Fuenzalida
@fcorivasf_gitlab
ok, I'll see what happens. Thank you!
is the order of the arguments important?
Anthony Truskinger
@atruskie

there is distinction betwen arguments and options.

arguments: yes, order is important. Arguments are values for which you don't have to put their name in front. Example: AnalysisPrograms.exe audio2csv C:\Audio Towsey.Acoustic.yml C:\output <-- there are three arguments there, each separated by a space

options: order is not important. Options are always prefixed with their name (like -i or --input-data-directory) and then the value. Example: --input-data-directory C:\indices\

Francisco Rivas Fuenzalida
@fcorivasf_gitlab
gotcha! I was confused when you said I should enter --help to see what arguments the command takes... thanks for explaining!
Anthony Truskinger
@atruskie

yeah --help should be available on every command. Even though I've written most of AP.exe, I still regularly forget things. --help is the first thing I do everytime.

Some commands will take arguments, some options, and some both. Techincally an option is a special type of argument (hence my confusing statement)

Francisco Rivas Fuenzalida
@fcorivasf_gitlab
gotcha. Other question: when I enter -i, should I include a file name at the end of the directory?
Anthony Truskinger
@atruskie
Definitely a directory. I think also it needs Towsey.Acoustic appended. E.g. C:\Indices\IndicesForFileA\Towsey.Acoustic
Francisco Rivas Fuenzalida
@fcorivasf_gitlab
within -i?
or that is -ip?
or -fcs?
Francisco Rivas Fuenzalida
@fcorivasf_gitlab
I'm here:
C:\AP\AnalysisPrograms.exe DrawLongDurationSpectrograms -i D:\PhD\TANZANIA\IndicesOutput\Towsey.Acoustic\S4A03863 -O D:\PhD\TANZANIA\IndicesOutput\Towsey.Acoustic\S4A03863 -ip C:\AP\ConfigFiles\IndexPropertiesConfig.yml
Anthony Truskinger
@atruskie

Your directory structure looks a bit weird. In D:\PhD\TANZANIA\IndicesOutput\Towsey.Acoustic\S4A03863 is there a Towsey.Acoustic folder?

Does that command work?

Other than that, it's the right format

Francisco Rivas Fuenzalida
@fcorivasf_gitlab
I've run that without the Towsey.Acoustic and it gives an error message: FATAL - Unhandled exception ->
13 replies
There is the Towsey.Acoustic folder created by AP to store the AIs
I've made the \S4A03863 to separate the data from different sensors
So within the Towsey.Acoustic folder there is no file but other three folders
Abram Fleishman
@abfleishman
I have been using the Towsey.Acoustic config file with audio2csv to calculate acoustic indices for short (1 minute) files. It works well; however, it generates a bunch of other analyses that I am not using and I now want to unleash it on ~10 TB of 1-minute files. Essentially I am interested in only creating this file for each sound file: *__Towsey.Acoustic.Indices.csv. Is this possible? can you point me in the right direction? Thanks for a great tool!
Anthony Truskinger
@atruskie

@abfleishman the short answer is don't do this.

AP.exe is not made for processing 1-minute files - it will perform slowly and will be quite inefficient. 1-minute files are a bad choice for storing and collecting data, especially for consecutive minutes. Unfortunately, people keep doing this, so we intend to add support for it (QutEcoacoustics/audio-analysis#378); however, it will be a while before that is implemented.

With the warning out of the way: it's possible. We've done it before, on datasets larger than 10TB.

, it generates a bunch of other analyses

Slightly incorrect. It generates other sorts of indices, which we call spectral indices. They are very much acoustic indices though.

Currently, we do not have the options to calculate just a subset of the indices. It's been suggested a few times and when I went searching for the relevant feature request, I found your name mentioned!

QutEcoacoustics/audio-analysis#240

All I can say is sorry. We've been under a lot of pressure. For our typical use case (2, or 6-hour long files) the default settings make sense and aren't that annoying, even when we don't need a subset of the files which makes it a low priority.

I'd recommend just deleting the files you don't need after each loop. It'd be worthwhile keeping some of the metadata (like the index generation data json).

Abram Fleishman
@abfleishman
@atruskie Thanks for your reply. I thought I had asked a similar question a few years back but searched for me github handle and didn't find anything. looks like it came from an email?! I understand your caution away from short files, however, we find that consecutive minutes are often highly autocorrelated and thus do not provide much additional biological information. Also, we often can visit sites infrequently and thus need to use a duty cycle of 1 min every 5 or 10 to extend deployment periods (both battery and memory). I have used your tools on 1 minute files successfully in the past
Anthony Truskinger
@atruskie

@abfleishman yeah, I get there are plenty of valid reasons to sample data. The issue I keep running into is people doing it by default, especially for consecutive minutes. Small files create huge strain on filesystems... and I'm over here trying to write software that works for not just their project, but every project we get all at once.

FWIW, we've been collecting data on continuous schedules by default. With our new sensors service time is every 6 or 12 months.

And yeah, AP.exe has no obvious issue working with short files, but audio2csv in particular expects it. It does a bunch of stuff like duration checking, chunking, minute alignment, and outputting data files in shapes that really don't make sense for short files. It's --parallel option is also unused.

Anyway /rant :-)

My contract is changing soon and I'll be back to working on AP.exe more regularly. Hopefully I'll be able to provide some good news soon!

Abram Fleishman
@abfleishman
Thanks for the info, I will key my eye on the repo and look forward to seeing it continue to evolve
Francisco Rivas Fuenzalida
@fcorivasf_gitlab
Hi @atruskie ! I wonder if there is an option to show the frequency values outside the FCS. They mask the midnight events and they don't look very well aesthetically. Thank you!
Anthony Truskinger
@atruskie

@fcorivasf_gitlab no there isn't. Sounds like a good feature request? Do you want to add it the feature request to the issue tracker?

Historically, we intended the images to sit side by side for quick and dirty alignments without any padding.

We're encountering cases now we're people are looking for prettier images and aesthetic options like this are becoming more important

mitesser
@mitesser
Dear @atruskie, seems that I missed it in the documentation: Is there a list of descriptions for all the index abbreviations used in the csv spectral output files *.Acoustic.ACI, BGN, CVR, DIF, ENT, EVN, ...., SPT, SUM? My specific aim is to find one that represents (or could be used as starting point for calculating) simple db mean for each frequeny bin (per minute).
Anthony Truskinger
@atruskie

@mitesser no, not currently. The best resource is Towsey's whitepaper on the topic - see https://eprints.qut.edu.au/110634/

The documentation is a much newer (part-time labour of love) than the indices and it is nowhere near finished. We aim to merge that white paper into our documentation.

The index you're looking for is PMN (power minus noise). The true power could be obtained by doing PMN + BGN if you need it (I'll double check that and get back to you)