Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
  • Dec 06 22:11
    kumekay starred nexB/scancode-toolkit
  • Dec 06 09:08
    silverskyvicto starred nexB/scancode-toolkit
  • Dec 06 06:53
    sky-kokubu-h starred nexB/scancode-toolkit
  • Dec 05 22:00
    Gator8 commented #3156
  • Dec 05 16:44
    pombredanne commented #3165
  • Dec 05 16:44
    pombredanne commented #3165
  • Dec 05 16:41
    pombredanne commented #3156
  • Dec 05 14:29
    pombredanne commented #3168
  • Dec 05 13:50
    Gator8 commented #3156
  • Dec 05 13:12
    pombredanne commented #3102
  • Dec 05 13:11
    pombredanne commented #3139
  • Dec 05 09:31
    MarcelWorschech opened #3168
  • Dec 05 09:31
    MarcelWorschech labeled #3168
  • Dec 04 22:49
    SupremeHedgehog starred nexB/scancode-toolkit
  • Dec 04 14:29
    webtech-tom starred nexB/scancode-toolkit
  • Dec 03 20:52
    pedroraft starred nexB/scancode-toolkit
  • Dec 03 19:51
    TimothyGillespie starred nexB/scancode-toolkit
  • Dec 01 17:26
    DennisClark labeled #3167
  • Dec 01 17:26
    DennisClark assigned #3167
  • Dec 01 17:26
    DennisClark opened #3167
Akhil Raj
@lf32
img
remove url for directories, and in case if visited the path of the directory then the blue info will be displayed
Philippe Ombredanne
@pombredanne
@lf32 that's looking to me, but that's just me... do you min putting the suggestion in the issue proper?
Akhil Raj
@lf32
proper?
Philippe Ombredanne
@pombredanne
@lf32 I meant to say that your issue would benefit of more details... I know understand it based on our discussion in today's call... But I am sure you can make it bettr :)
Akhil Raj
@lf32
Yes I have updated the information about the issue
Philippe Ombredanne
@pombredanne
@lf32 Thanks!
@tdruez at some point we should discuss improved UX design... such as what @lf32 suggested above an in an issue
Akhil Raj
@lf32
I'll join the regular meet as I have less questions
Philippe Ombredanne
@pombredanne
@lf32 this works too :)
but please ask the question ahead ... the meeting is there to bring up issues, not to resolve all of them (otherwise it would be too long ;) )
Akhil Raj
@lf32
Ok
Akhil Raj
@lf32
for nexB/scancode-toolkit#2951 in which section of CHANGELOG.rst the update should be added?
Akhil Raj
@lf32
the update is scancode-toolkit now does support usage of shortcut flags like -A, -q, -v, -V
Philippe Ombredanne
@pombredanne
@lf32 in the miscellaneous section
Akhil Raj
@lf32
Ok, Thanks
Philippe Ombredanne
@pombredanne
ScanCode.io on K8s ... woot! https://github.com/xerrni/scancode-kube
Steven Esser
@steven-esser
:+1:
very cool
Philippe Ombredanne
@pombredanne
:)
@steven-esser Omkar may need a bit of attention on his SCWB PR if you have a few minutes
Akhil Raj
@lf32
what is your opinion on this https://github.com/django-ace/django-ace
Philippe Ombredanne
@pombredanne

@lf32 I do not know about it so I have no opinion... we are using ace I think already though https://github.com/nexB/scancode.io/blob/main/scanpipe/templates/scanpipe/resource_detail.html#L78

@lf32 returning the question, what do you thing of it?

Akhil Raj
@lf32
I think its better to let it stay the way it is, this has less features
Philippe Ombredanne
@pombredanne
is there something better than ace?
Akhil Raj
@lf32
I think ace is good
Akhil Raj
@lf32
@mjherzog @tdruez do you think this option looks good for scancode.io
tdruez
@tdruez
@lf32 see my comments in the PR
lf32
@lf32:matrix.org
[m]
Yes Im looking at them
🤯
Philippe Ombredanne
@pombredanne
;)
@tdruez
tdruez
@tdruez
@lf32 ?
lf32
@lf32:matrix.org
[m]
yes @tdruez
tdruez
@tdruez

lf32 @lf32 Jun 22 14:03 @tdruez

You needed something?

lf32
@lf32:matrix.org
[m]
I thought we had a meet
tdruez
@tdruez
There's no recurrent weekly meeting, last week was exceptional to get you started.
Sorry if that was not clear.
lf32
@lf32:matrix.org
[m]
Ok then
Ben Johnson
@bjohnson4_gitlab
For using scancode to scan for FOSS licenses, is there a way to make the scan faster by using heuristics about file names and extensions that commonly contain LICENSE information to reduce the number of files needed to scan, or would doing that cause potential false negatives in that it would not scan files that do contain license information?
Philippe Ombredanne
@pombredanne
@bjohnson4_gitlab I guess you could use heuristics, but IMHO these heuristics are what make detection poor and brittle in tools that only rely on these (such as GitHub licensee and several others)
that said, it would be a nice addition to have a "quick and dirty" scan mode that only looks at these "key" files
note that the "--classify" CLI option does extensive classification already in https://github.com/nexB/scancode-toolkit/blob/develop/src/summarycode/classify.py reporting the "is_key_files"
I could see how we could have an option that would basically use is_key_file and only scan these for licenses
@bjohnson4_gitlab I would be interested to know what's the volume you deal with and what's the speed hold up
4 replies
Ben Johnson
@bjohnson4_gitlab
I'm attempting to use scancode.io, and I've run into an issue with running a scan_codebase pipeline. It seems it fails, but doesn't provide enough info on why. Here's the information it provides:
2022-09-09 21:52:44.04 Pipeline [scan_codebase] starting
2022-09-09 21:52:44.15 Step [copy_inputs_to_codebase_directory] starting
2022-09-09 21:52:44.24 Step [copy_inputs_to_codebase_directory] completed in 0.09 seconds
2022-09-09 21:52:44.25 Step [extract_archives] starting
2022-09-09 21:52:45.00 Step [extract_archives] completed in 0.76 seconds
2022-09-09 21:52:45.01 Step [collect_and_create_codebase_resources] starting
2022-09-09 21:52:49.75 Step [collect_and_create_codebase_resources] completed in 4.74 seconds
2022-09-09 21:52:49.75 Step [tag_empty_files] starting
2022-09-09 21:52:49.81 Step [tag_empty_files] completed in 0.06 seconds
2022-09-09 21:52:49.81 Step [scan_for_application_packages] starting
2022-09-09 21:53:17.12 Step [scan_for_application_packages] completed in 27.30 seconds
2022-09-09 21:53:17.13 Step [scan_for_files] starting
2022-09-09 21:53:20.83 Pipeline failed
Task output
A process in the process pool was terminated abruptly while the future was running or pending.

Traceback:
  File "/app/scanpipe/pipelines/__init__.py", line 115, in execute
    step(self)
  File "/app/scanpipe/pipelines/scan_codebase.py", line 99, in scan_for_files
    scancode.scan_for_files(self.project)
  File "/app/scanpipe/pipes/scancode.py", line 310, in scan_for_files
    _scan_and_save(resource_qs, scan_file, save_scan_file_results)
  File "/app/scanpipe/pipes/scancode.py", line 297, in _scan_and_save
    scan_results, scan_errors = future.result()
  File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 439, in result
    return self.__get_result()
  File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 391, in __get_result
    raise self._exception
7 replies
Ben Johnson
@bjohnson4_gitlab
Sorry, another question that I had about how scancode works. I'm working with packages from a few different languages (go/php/js), and noticed that with go (and maybe others) that scancode doesn't seem to be able to read the go.mod file (that contains all the dependencies) and interpret dependencies automatically. For example, with go projects, I can run go mod vendor to populate the ./vendor directory, and then run scancode, pointing it at the vendor directory, and it scans all the files. Thus to scan source code deps, I can't just point it at a cloned version of the repo, but I'm required to first prep the repo by pulling in all the source files of all the deps, using the appropriate package manage per each. Is my understanding right? I was looking at using https://github.com/pivotal/LicenseFinder, and see that it works with the package managers for you to do this prep step automatically. Is this something that scancode supports and I'm just not seeing it? Does scancode.io work differently than the scancode-toolkit in this case?
Philippe Ombredanne
@pombredanne

scancode doesn't seem to be able to read the go.mod file (that contains all the dependencies) and interpret dependencies automatically.

It reads the mod file, with the --package option but will not actually run any package management tool

4 replies

Thus to scan source code deps, I can't just point it at a cloned version of the repo, but I'm required to first prep the repo by pulling in all the source files of all the deps, using the appropriate package manage per each. Is my understanding right?

correct... though we are eventually building a series of dependency resolvers, the first being https://github.com/nexB/python-inspector and https://github.com/nexB/nuget-inspector ... and more to come for all the main package ecosystems

I was looking at using https://github.com/pivotal/LicenseFinder, and see that it works with the package managers for you to do this prep step automatically. Is this something that scancode supports and I'm just not seeing it? Does scancode.io work differently than the scancode-toolkit in this case?

neither the toolkit nor scancode.io do this running of package management tools. Instead you need to run your build first for now, but as mentioned above that's definitely on the roadmap