Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
    Jörg Stucke
    @jstucke
    You could search for the files with results for the plugin "users_and_passwords" and iterate over the entries and look for ones that include "password-hash". This script should accomplish that if you run it from the src directory:
    import json
    
    from pymongo import MongoClient
    
    # start MongoDB without auth before running (i.e.: mongod --config config/mongod.conf)
    client = MongoClient("mongodb://127.0.0.1:27018", connect=False)
    fo_collection = client['fact_main']['file_objects']
    query = {"processed_analysis.users_and_passwords.summary": {"$not": {"$size": 0}}}
    
    results = {}
    for entry in fo_collection.find(query, {"processed_analysis.users_and_passwords": 1}):
        try:
            for pw_result in entry["processed_analysis"]["users_and_passwords"].values():
                if isinstance(pw_result, dict) and "password-hash" in pw_result:
                    results.setdefault(entry["_id"], []).append(pw_result["password-hash"])
        except KeyError:
            pass
    print(json.dumps(results, indent=2))
    but be aware that there might be false positives among the results
    IoT-junkrat
    @IoT-junkrat
    Thank you for that code snippet :-) It worked great.
    hairlessbear
    @hairlessbear

    What you're talking about (identical files with different paths) is exactly the scenario I'm facing. I'll work on producing an archive that can reproduce the problem by mimicking the layout of the firmware. Once I figure out an archive that triggers it, I'll throw it your way!

    I haven't had the chance to try to produce an example archive yet, but I have a theory on what's causing this. In my samples, all of the instances of this bug occur when the same file is present in different "levels" of archives. Here's an example of what I mean:

    archive.zip
    |--file_1
    |--file_2
    |--nested_archive.zip
       |--file_1
       |--file_3
    |--file_4

    In the above example, file_1 is present at the top level of the archive, as well as present in another archive within the root archive. If my theory is right, that's what causes the display bug and file_1 will show up in the wrong place.

    My FACT instance is currently occupied analyzing a big batch of firmware, so I can't test this at the moment. I'll try to do it within the next few days, but I wanted to let y'all know now on the off chance someone else feels like testing this :)
    hairlessbear
    @hairlessbear
    Notably, when identical files are all only present at the same "level" of archive, everything displays properly (at least in all of my samples)
    Jörg Stucke
    @jstucke

    I'm not sure I understand the problem exactly but I was definitely able to find a bug in the file tree:
    A test file uploaded as

    ├── test1.zip
        └── test_file_1.txt
    └── test_dir
        └── test_file_1.txt

    will display in the file tree as

    ├── test1.zip
        ├── test_file_1.txt
        └── test_dir
            └── test_file_1.txt
    ├── test_file_1.txt
    └── test_dir
        └── test_file_1.txt

    I will take a look and try to find the bug that causes this

    Jörg Stucke
    @jstucke
    fkie-cad/FACT_core#541 should (hopefully) fix the problem
    hairlessbear
    @hairlessbear
    Thanks! I'll take a look!
    One other thing, while I'm bringing up display bugs 😅 In some of my firmware samples, the same files exist between multiple different firmwares. When viewing one of these files, FACT properly shows that this file exists in multiple firmwares (in the "parent firmware" section). But in the "complete file paths in container" section, it only shows the paths from a single firmware, not from all of the parent firmwares.
    hairlessbear
    @hairlessbear

    fkie-cad/FACT_core#541 should (hopefully) fix the problem

    Initial test looks great, thank you!

    Jörg Stucke
    @jstucke

    it only shows the paths from a single firmware, not from all of the parent firmwares

    That is actually intended: It should only show the paths of the firmware you are currently looking at. We could display all paths when no "root uid" is selected (the 2nd endpoint parameter after "ro") -- currently the paths of a random firmware are displayed in that case.

    hairlessbear
    @hairlessbear
    That's not the behavior I'm seeing, unfortunately. While the path under the "Download" button is updated based on which firmware you're looking at, the "complete file paths in container" section always displays the paths from the same firmware sample, even if that's not the one I'm looking at.
    Possibly relevant, the sample whose paths are always shown is the first sample I uploaded that had this file in it.
    Jörg Stucke
    @jstucke
    @hairlessbear I finally found the time to look at this problem. Turns out there was a bug in the revised general information section which led to the "root_uid" (the id of the parent firmware) always being None. fkie-cad/FACT_core#544 should fix the issue and also display all paths in the case no "root_uid" is provided
    hairlessbear
    @hairlessbear
    Sweet, thanks! I'll try to take a look later today.
    IoT-junkrat
    @IoT-junkrat

    Hey guys 🙂
    I want to get all the software_components from all my ~800 firmware images.

    If I use the normal curl on the REST interface "/rest/FW_UID", I get only very few software_component summaries which are not empty. But the retrieval is fast.

    If I use the normal curl on the REST interface with the summary option "/rest/FW_UID&summary=true" I get the software_component summaries. But retrieving the results of a single firmware takes like 2 hours. Don't want to waste that much time.

    If I use the curl command on "/rest/firmware/?recursive=true" and the mongo query "'processed_analysis.software_components':{'exists':'true'}}" I get a fast result but with the file UIDs. So I would retrieve the parent FW UID of every file. Then I could merge all those files grouped per FW UID, right? (no clue how via the command line/script)

    Maybe you guys implemented something similar already...
    Can you imagine any other solution which is "fast" and outputs me all the software_component summaries? 😅

    Jörg Stucke
    @jstucke
    This should be possible with a MongoDB aggregation:
    import json
    from pathlib import Path
    
    from pymongo import MongoClient
    
    # start MongoDB without auth before running (i.e.: mongod --config config/mongod.conf)
    client = MongoClient("mongodb://127.0.0.1:27018", connect=False)
    fo_collection = client['fact_main']['file_objects']
    query = {"processed_analysis.users_and_passwords.summary": {"$not": {"$size": 0}}}
    
    
    aggregation_pipeline = [
        {"$match": {
            "processed_analysis.software_components.summary": {
                "$exists": True, "$not": {"$size": 0}  # only match entries that are not empty
            }
        }},
        {"$unwind": "$parent_firmware_uids"},  # unravel entries (so that we have one element per parent uid)
        {"$group": {
            '_id': '$parent_firmware_uids',  # group entries by parent uid
            "uids": {"$push": "$_id"},
            'software_components': {
                 '$push': '$processed_analysis.software_components.summary'
            }
        }}
    ]
    
    result = list(fo_collection.aggregate(aggregation_pipeline))
    
    for entry in result:
        print(f"Results for firmware {entry['_id']}")
        for uid, software_list in zip(entry["uids"], entry["software_components"]):
            print(f"\tfile {uid} matched software {software_list}")
    
    Path("software_components.json").write_text(json.dumps(result))
    print("saved as software_components.json")
    IoT-junkrat
    @IoT-junkrat
    Wow thank you so much! This was the solution I was looking for. :heart_eyes: You made my day :grinning:
    Nikita Bublikov
    @firmadyne:matrix.org
    [m]

    Hi! successfully install FACT with instruction https://github.com/fkie-cad/FACT_core/blob/master/INSTALL.md

    start $ ./start_all_installed_fact_components

    successfully get start screen FACT, but when try to upload file, get this error, why?

    Error log

    Exception: No available plug-ins found. FACT backend might be down!
    [2021-03-30 03:04:28][app][ERROR]: Exception on /upload [GET]
    Traceback (most recent call last):
    File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 2447, in wsgi_app
    response = self.full_dispatch_request()
    File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1952, in full_dispatch_request
    rv = self.handle_user_exception(e)
    File "/usr/local/lib/python3.8/dist-packages/flask_restful/init.py", line 272, in error_router
    return original_handler(e)
    File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1821, in handle_user_exception
    reraise(exc_type, exc_value, tb)
    File "/usr/local/lib/python3.8/dist-packages/flask/_compat.py", line 39, in reraise
    raise value
    File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1950, in full_dispatch_request
    rv = self.dispatch_request()
    File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1936, in dispatch_request
    return self.view_functionsrule.endpoint
    File "./web_interface/security/decorator.py", line 11, in decorated_view
    return fn(args, *kwargs)
    File "./web_interface/components/io_routes.py", line 50, in _app_upload
    analysis_plugins = sc.get_available_analysis_plugins()
    File "./intercom/front_end_binding.py", line 36, in get_available_analysis_plugins
    raise Exception("No available plug-ins found. FACT backend might be down!")
    Exception: No available plug-ins found. FACT backend might be down!
    [pid: 49220|app: 0|req: 15/70] 127.0.0.1 () {36 vars in 609 bytes} [Tue Mar 30 03:04:28 2021] GET /upload => generated 290 bytes in 61 msecs (HTTP/1.1 500) 2 headers in 84 bytes (1 switches on core 1)

    Jörg Stucke
    @jstucke
    This seems to be quite similar to fkie-cad/FACT_core#559.
    What version are you running? Master or stable?
    Is there any error in the backend log (might be easier to observe when starting each component independently instead of using start_all_installed_fact_components) prior to this?
    Nikita Bublikov
    @firmadyne:matrix.org
    [m]

    i think i use master, how i can start each component independently?
    i should run this commands separately?

    ./start_fact_db
    ./start_fact_frontend

    Jörg Stucke
    @jstucke

    i should run this commands separately?

    Yes, but you should start with db before running start_fact_frontend and start_fact_backend

    Nikita Bublikov
    @firmadyne:matrix.org
    [m]

    try it and when i start backend, got this error

    sudo ./start_fact_backend.py
    [sudo] password for eagle:
    [2021-03-30 04:50:18][plugin][WARNING]: Plugin has no code directory: /home/eagle/FACT_core/src/plugins/analysis/pycache
    [2021-03-30 04:50:18][YaraPluginBase][ERROR]: Signature file /home/eagle/FACT_core/src/analysis/signatures/crypto_hints.yc not found. Did you run "compile_yara_signatures.py"?
    [2021-03-30 04:50:18][start_fact_backend][CRITICAL]: Error during initialization of plugin crypto_hints. Shutting down FACT backend
    [2021-03-30 04:50:18][process][CRITICAL]: SHUTTING DOWN SYSTEM
    Killed

    Jörg Stucke
    @jstucke
    Did you try running compile_yara_signatures.py? It seems the plugin misses the compiled signatures
    Nikita Bublikov
    @firmadyne:matrix.org
    [m]

    just run it, previous error fixed, and now this error, when click on upload in UI

    sudo ./start_fact_backend.py
    [2021-03-30 04:57:14][plugin][WARNING]: Plugin has no code directory: /home/eagle/FACT_core/src/plugins/analysis/pycache
    [2021-03-30 04:58:19][cwe_checker][INFO]: Version is cwe_checker 0.5.0-dev

    [2021-03-30 04:58:20][input_vectors][INFO]: Up and running.
    Traceback (most recent call last):
    File "./start_fact_backend.py", line 54, in <module>
    analysis_service = AnalysisScheduler(config=config)
    File "/home/eagle/FACT_core/src/scheduler/Analysis.py", line 37, in init
    self.load_plugins()
    File "/home/eagle/FACT_core/src/scheduler/Analysis.py", line 212, in load_plugins
    plugin = source.load_plugin(plugin_name)
    File "/usr/local/lib/python3.8/dist-packages/pluginbase.py", line 301, in load_plugin
    return import(self.base.package + '.' + name,
    File "/usr/local/lib/python3.8/dist-packages/pluginbase.py", line 438, in plugin_import
    return self._system_import(import_name, globals, locals,
    File "/home/eagle/FACT_core/src/plugins/analysis/ip_and_uri_finder/code/ip_and_uri_finder.py", line 7, in <module>
    import geoip2.database
    File "/usr/local/lib/python3.8/dist-packages/pluginbase.py", line 438, in plugin_import
    return self._system_import(import_name, globals, locals,
    ModuleNotFoundError: No module named 'geoip2'

    Jörg Stucke
    @jstucke
    Did the installation of FACT run completely and successfully? This suggests that the installation of the plugin did not complete or run at all. You could try rerunning src/install.py or just the installation of the plugin src/plugins/analysis/ip_and_uri_finder/install.sh
    Nikita Bublikov
    @firmadyne:matrix.org
    [m]
    Ok, now I will try the first and then the second option.
    Jörg Stucke
    @jstucke
    So did it work?