Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
    Jörg Stucke
    @jstucke

    It should be possible to do this with MongoDB operations more efficiently, e.g.

    db.getCollection('firmwares').aggregate([
        { "$group": { "_id": "$device_class"}},
        { "$count": "device_class_count"}
    ])

    should give you the number of device classes. We could add this to the stats without too much of a hassle.

    IoT-junkrat
    @IoT-junkrat
    That looks definitively more handy jstucke, thanks. The code of dorpvom did it as well, which is ok for my use-case. So thanks for offering, but you don't need to implement it just for me!
    Jörg Stucke
    @jstucke
    I opened a PR fkie-cad/FACT_core#480 for those new stats as "general stats" on the stats page. It seems like there is no way to get them using REST currently, though 😕 (maybe we can add that at a later point).
    Fabian Steiner
    @fabiansteiner
    Hi guys! I wand to extract the Flattened Device Tree automatically with fact_core (if one exists) and then access it via the REST API. binwalk has the ability to show me where the device tree is the file and how long it is. Now my questions: How do i write a plugin for that specific case? Is it better to add it as an analysis plugin (maybe to the existing binwalk plugin?) or as an extraction plugin? In your documentation it says that there is an 1:n relationship from input file to extraction plugin, but when i try the extractor, it always chooses only one plugin for the extraction (according to the meta.json file) Would be grateful for an answer.
    Jörg Stucke
    @jstucke

    Do you want to unpack the Flattened Device Tree as a file from the firmware (firmware archive to binary blob) or do you intend to unpack the Flattened Device Tree itself (binary blob to source)? For the latter, an unpacking plugin would make sense. If you want to unpack it from the firmware and then run an analysis on it, an analysis plugin would be better. Adding to the binwalk plugin could then be the easiest way to go.

    it says that there is an 1:n relationship from input file to extraction plugin, but when i try the extractor, it always chooses only one plugin for the extraction

    The "fact_extractor" uses the MIME type definitions from https://github.com/fkie-cad/fact_helper_file/tree/master/fact_helper_file/mime and then selects the plugin with the fitting MIME_PATTERNS (usually only one)

    Fabian Steiner
    @fabiansteiner
    The goal is to extract it from the firmware (firmware archive to binary blob(.dtb)) and save it as every other extracted file with an identifier and name in the mongodb, to make it accessible over the REST API later. So no analysis, just unpacking and saving. From my understanding, device trees can be attached pretty much on every file type so i need something that always scans for a device tree with binwalk and if it exists it should extract it, independent of the mime type of the archive and without interfering the other extractor plugins, they should remain working as always.
    image.png
    fact obviously knows about device trees already, but does not extract them yet.
    Johannes vom Dorp
    @dorpvom
    The binwalk analysis plugin blacklists ['audio', 'image', 'video']. So it runs on all non-media files. Does that suffice for your purpose? If so you can simply add the dtb extraction to that plugin and then access the results via the REST API. You should then also add an identifier to the summary to avoid looking in the wrong places.
    Jörg Stucke
    @jstucke

    I was able to find two (apparent) .dtb files in kernel images with this YARA signature:

    rule flattened_device_tree {
        strings:
            $a = { D0 0D FE ED 00 00 ?? ?? 00 00 00 ?? 00 00 }
        condition:
            $a
    }

    As you said, the files do not get unpacked, though. In this case, binwalk is used to unpack the image. It detects the "Flattened device tree" but does not unpack it. We are currently discussing extending the capabilities of our file carver, which currently only uses binwalk. Unpacking the files should not be hard as you can find the offset with the signature and the file size is in the header (the ?? ?? in the signature).

    Fabian Steiner
    @fabiansteiner
    @dorpvom Yes it does suffice for my purpose. So the analysis plugins also extract files? thats a little confusing, intuitively i would have tried to write a plugin for the extractor, good thing that i wrote you guys first! So if want to actually see the Flattened Device Tree within the FACT website and also be Accessible over REST, i have to extract the file in the binwalk plugin and add a database entry referencing that file with an existing: processed_analysis.binwalk.summary for Example: "Flattened Device Tree" + parent_firmware_uids to the firmware. Is that all i have to do??
    @jstucke It would be !awesome! if you could extend it to do that, that would probably save me alot of time. Please let me know if you decide to do/not to do it.
    Jörg Stucke
    @jstucke
    @fabiansteiner we will probably do it but it will not be a short-term thing so it will probably be better for you to find a solution for your specific problem. You could also write a Yara-Plugin that searches for flattened device trees with a signature and saves it as analysis result. Then you could also access it using the REST interface
    Johannes vom Dorp
    @dorpvom
    @fabiansteiner So generally you are right. Analysis Plug Ins don't usually extract files. The sound method would be to extract the file using an extractor plugin. That said, as the device tree files seem to typically be somewhere inside larger container files where we already extract stuff from, the way is to enhance the extraction to include the device tree files.
    As this looks to be a larger adaptation though, you can cheat your way towards the results quicker by adding to the analysis plugin. My proposition would be to store the binary contents of the extracted file as a field in the analysis results. The binwalk plugin already does this for the entropy graph, which is stored on the fs, read from there and then stored as base64 string.
    lines 64 + 65 in the binwalk.py:
            pic_path = os.path.join(dir_path, '{}.png'.format(os.path.basename(file_object.file_path)))
            result['entropy_analysis_graph'] = get_binary_from_file(pic_path)
    Fabian Steiner
    @fabiansteiner
    @dorpvom I did it like this and it works, thanks for your advice, also to @jstucke !
    Johannes vom Dorp
    @dorpvom
    Great to hear! Would you say this resulted in something we should add to FACT permanently?
    Fabian Steiner
    @fabiansteiner
    maybe, i mean it works....just have a look: https://github.com/fabiansteiner/dtb_finder
    acarr91
    @acarr91
    Hello! I have two questions to ask you. Is it possible to schedule analyzes at a certain time or through an order? Is it possible to make advanced queries for example to find which firmware in the db have a specific version of a software inside them? Thanks!
    Johannes vom Dorp
    @dorpvom
    Hi, I can quickly answer the second question. You can use the Advanced Search from the Web UI to create complex search queries. You can effectively do any valid MongoDB query that does not involve some form of SQL-like JOIN operation (aggregate in the Mongo world)
    There are some examples in the UI that demonstrate the different query mechanisms, along with a link to the Mongo query documentation
    To construct a final query you can either consult the structure information given on the UI page or dig through the code when you have to go deeper into the structure of a specific analysis result
    Johannes vom Dorp
    @dorpvom
    BTW the same feature is also available through the REST API
    Johannes vom Dorp
    @dorpvom
    The software-in-specific-version query would look like {"processed_analysis.software_components.BusyBox.meta.version": {"$in": ["1.22.1"]}} for the example of busybox in version 1.22.1
    Regarding the first question: We don't offer such a service yet. If I wanted to do it I'd use the REST API and handle the scheduling with some small script.
    acarr91
    @acarr91
    @dorpvom Thanks for all the information. I tried that query it lists in the gui the busybox files (chosen version) of the various firmware loaded on FACT. Is there a chance to have the affected firmware list, not the busybox file list? Thanks
    Jörg Stucke
    @jstucke
    @acarr91 yes, there is a small box under the search input, labeled show parent firmware instead of matching file which does exactly that
    acarr91
    @acarr91
    @jstucke Great! I did not notice that :)
    acarr91
    @acarr91
    Hi! After a git pull done a few days ago I noticed some strange behavior in the users_and_password plugin (0.4.5). I don't know whether to open a new thread in the issues, maybe it's not needed. Basically the plugin detects (I think some false positives) in the case of username: $ 1 $ 'string' $ 'string' that the password is '1'. It detects passwords equal to '1' even in html files or different formats for strings that apparently have nothing to do with passwords. This also happens with update analysis where there were not these results previously.
    Johannes vom Dorp
    @dorpvom
    I'll let @jstucke answer this. But this seems to be unwanted behavior originating from a recent change to the plugin.
    Jörg Stucke
    @jstucke
    We recently added another password format to the plugin. Those results could be false positives of this new format. I tried to reproduce the bug, but failed. Could you provide a simple example that generates a wrong match?
    acarr91
    @acarr91
    @jstucke Before git pull the analysis of a firmware had cracked a string in etc/shadow as root: root, after the git pull and an update analysis on this firmware, at the top left of the firwmare page appear both root:root and root:unix:1 and going to check in the users_and_passwords plugin on the etc/shadow file I only see the entry with the cracked password '1'. Another example would be an html file where the analysis found AuthType: unix:1 identifying as the password hash $ ("# modal_field").val(), Type
    Jörg Stucke
    @jstucke
    The password type was added to the key in the analysis results in order to support different password types without breaking backwards compatibility. It was not planned to display it in the firmware tags, though. This bug should be fixed soon.
    acarr91
    @acarr91
    ok! But in addition to the type of password displayed in the tags, it is as if the plugin crack many passwords providing the password in clear text as '1'.
    Jörg Stucke
    @jstucke
    You may need to run the installation of this plugin again because the latest version uses a different version of John the Ripper for password cracking. You can simply run the install.sh in the plugin folder (src/plugins/analysis/users_and_passwords/install.sh).
    acarr91
    @acarr91
    @jstucke ok! in a new analysis after ran install.sh, it would seem that there are no more clear text passwords like '1' . I will be able to provide more information by doing more analyzes.
    acarr91
    @acarr91
    Now I have a similar problem like issue #504 with the source_code_analysis plugin. The last analysis seems to have stopped with this plugin in timeout. There are some degub errors (file .h, .map that are not a supported script) near the timeout message. After these messages, there is an OSError input/output error and than some "child process failed, exited with error number 100". How could this be solver? I apologize for the messages, but i hope they can also be usefuls as a test in order to improve this very useful tool.
    Jörg Stucke
    @jstucke
    This plugin also received an update recently -- sorry, it slipped my mind to mention that. Could you try running the install script of that plugin?
    acarr91
    @acarr91
    @jstucke I have no good news. After the install.sh, I rerun the analysis on the same firmware, but from the log I noticed an ERROR message with timeout of the source_code_analysis and after that an Exception in Analysis process that produced an OSError Input/Output. After this message it seems that the analysis continues until it reaches a [source code analysis] METADATA is not a supported script with a finished source_code_analysis message of worker0 and then only 'throttle down unpacking to reduce memory consumption' messages. In the GUI there were still 35 files in the queue for the source_code_analysis plugin and the firmware was about 3/4 in the progress bar.
    acarr91
    @acarr91
    I add that although the analysis was not successful, in the source code analysis tab on the gui there are some results, so I believe that the plugin works, but in a limited way it causes errors that does not allow to complete the firmware analysis.
    Jörg Stucke
    @jstucke
    @acarr91 I tried to reproduce the errors by re-analyzing a larger firmware but didn't get any exceptions. Did you get any stack traces in your logs that might be helpful? Was there something the files for which the analysis failed had in common (e.g. being very large or a certain file type)?
    acarr91
    @acarr91
    @jstucke Thanks for the update! Being the logs in /tmp and having restarted the machine I lost the log; I made some changes on the number of cores and as soon as possible I will launch the firmware analysis adding source_code_analysis and I will try to provide more information. Anyway, as a recommended configuration, is it better to start the analysis with all the plugins, or select only some and add the others later with an update analysis? From the logs I did not understand if the update with the addition of plugins involves an analysis from scratch or the analyzes already carried out are skipped.
    Jörg Stucke
    @jstucke

    From the logs I did not understand if the update with the addition of plugins involves an analysis from scratch or the analyzes already carried out are skipped.

    Analyses will be skipped if there is already an analysis result for this combination of object and analysis plugin in the database and the plugin version hasn't changed. The unpacking process is done again when using "update" or "redo analysis", though.

    is it better to start the analysis with all the plugins, or select only some and add the others later with an update analysis?

    If you're running out of memory or the analysis is very slow and you need faster results, it might be advantageous to only run certain plugins and update later but it should not affect the individual plugin results

    hairlessbear
    @hairlessbear
    Hi! Are there any known issues with files over a certain size? I'm currently trying to analyze a 32 GB image and FACT appears "stuck". I uploaded it in a 7zipped container that was ~6 GB in size. FACT properly extracted the 32 GB image from this file, but doesn't appear to be doing anything since then. That is, the logs just show [Unpacking][INFO]: Queue Length (Analysis/Unpack): 1 / 1 over and over and in htop there doesn't appear to be any analysis or unpacking running (i.e. very low CPU utilization by all processes). The logging is at the DEBUG level, but I don't see any errors. The mongo logs also don't have any errors, although when the unpacking of the 7zipped container finished, the mongo logs did contain [ftdc] serverStatus was very slow, which might be relevant?
    If there aren't any known issues with large files, any pointers on where in the FACT core codebase I should look to debug this would be appreciated.
    Oh, forgot to mention, it's been in this stuck state for about an hour and a half.
    I also used the REST API to grab status, which showed that all of the analyzer plugins have empty queues. Here's backend.analysis (file name removed):
    "analysis": {
            "analysis_main_scheduler": 1,
            "current_analyses": {
              "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx_6687791786": {
                "analyzed_count": 0,
                "start_time": 1609877170.701255,
                "total_count": 2,
                "unpacked_count": 1
              }
            }
    hairlessbear
    @hairlessbear
    And this is running on Ubuntu 20.04 with python 3.8.5, single system setup, 16 cores, 64 GB RAM