This is a channel focused on ScanCode support and not as noisy as the main discuss channel
Ah, so as far as the ispan is concerned all the words are matching, it is not concerned about the extra words in there, that'd be for the qspan. So what is still missing from the key_phrase_filter is checking if key_phrase_span is uninterrupted in the qspan.
If I would use the zip(qspan, ispan) to create a query_key_phrase_span = Span(qpos, qpos + len(key_phrase_span)) if ipos in key_phrase_span to create a Span offset by where it matches. And then check if that is in query_key_phrase_span in qspan. Would that reasoning be in the right direction?
for qpos, ipos in zip(match.qspan, match.ispan):
if ipos in key_phrase_span:
query_key_phrase_span = Span(qpos, qpos + len(key_phrase_span))
if query_key_phrase_span not in match.qspan:
has_key_phrases = False
break
unknown wordsDO NOT exist anywhere in any RULE or LICENSE. They can be seen only in the Query.unknowns_by_pos where we only track how many unknown words exist after a known word position. They are not present in the ispan nor the qspanstopwordsexist in RULEs and LICENSEs are short, too common words to be useful. They are skipped both on the index and query side. They are not present in the ispan nor the qspan. They can be seen only in the Query.stopwords_by_pos where we only track how many stopwords exist after a known word position.
by construction, key_phrase_span should be :
- containing no stopwords
- containing no unknowns
Therefore, it should be possible to do key_phrase_span in match.qspan.
I reckon the code snippet above is for the next step.
Yes, that's what I was trying to communicate. So that under the {{Creative Commons Attribution 4.0 International License}} (the "License"); won't match under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.
The key_phrase_span is a single key phrase.
Yes, that's what I was trying to communicate. So that under the {{Creative Commons Attribution 4.0 International License}} (the "License"); won't match under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.
yes, remember my comment on your PR... if you have a only one rule in your test index, the univers of unknown words is very large :D
That was just a bad attempt to make my example easier to run, the same problem persists when running as a datadriven test (they use the full index right?). Hence why I am taking another look.
cc-by-nc-sa-4.0 match IMHO
Your new filtering code ... does filter! and it exposes that we are missing some rules there
I added a few comments in https://github.com/nexB/scancode-toolkit/issues/2767#issuecomment-987735258
--max-in-memory to use disk-caching...
1 reply
sorry if this is not the right place to ask this but I'm having some toubles with scancode while excluding some files. I've tried both --ignore-author and --ignore-copyright-holder like in this example:
https://scancode-toolkit.readthedocs.io/en/latest/cli-reference/output-filters-and-control.html?highlight=ignore
but none of the above worked for me.
How can I skip one folder in my repository, is there a way to do that with adding the parameter while executing the command, or should I use separated config file to specify particular directories I want to scan? I'll just add that it's not about skipping the unwanted files in post scan activities, I need to skip the unwanted folder from skanning as it will save 10h(!) of my time.
Thanks in advance, any help will be appreciated!