Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
  • Sep 02 14:10

    nuta on move-cursor

    Update Update Update and 1 more (compare)

  • Sep 02 10:03

    nuta on move-cursor

    Update (compare)

  • Sep 02 09:54

    nuta on move-cursor

    Update (compare)

  • Sep 01 13:23

    nuta on move-cursor

    Update Update Update (compare)

  • Sep 01 10:01

    nuta on move-cursor

    Update (compare)

  • Aug 28 13:18
    nuta edited #41
  • Aug 28 13:18
    nuta edited #41
  • Aug 26 15:10

    nuta on move-cursor

    Update Update Update (compare)

  • Aug 26 09:48
    nuta edited #41
  • Aug 26 09:47
    nuta edited #41
  • Aug 26 09:47
    nuta edited #41
  • Aug 26 09:47
    nuta edited #41
  • Aug 21 07:13

    nuta on move-cursor

    wip (compare)

  • Aug 21 01:28

    nuta on move-cursor

    Update (compare)

  • Aug 20 16:21

    nuta on move-cursor

    Update (compare)

  • Aug 20 15:14

    nuta on move-cursor

    Update (compare)

  • Aug 20 15:10

    nuta on move-cursor

    Update Update Update (compare)

  • Aug 20 14:05

    nuta on move-cursor

    Update (compare)

  • Aug 19 12:51

    nuta on move-cursor

    Update Update (compare)

  • Aug 18 17:23

    nuta on move-cursor

    Update Update (compare)

Seiya Nuta
@nuta
I think you don't have to. The original ASan uses single bit state for each memory address to reduce the memory footprint. However, I think we don't need to do that since we don't have large scale apps.
Yashraj Kakkad
@yashrajkakkad
I can surely do that, but then we'll not be able to mark different regions with different states right? It'll be either addressable or unaddressable. Wouldn't the scheme mentioned in the paper you shared be better?
Seiya Nuta
@nuta

we'll not be able to mark different regions with different states right?

Can you provide a concrete example? It seems I misunderstood something.

Yashraj Kakkad
@yashrajkakkad

Relevant section from the paper -

We use the following encoding for each shadow byte:
0 means that all 8 bytes of the corresponding application
memory region are addressable; k (1 ≤ k ≤ 7) means that
the first k bytes are addressible; any negative value indi-
cates that the entire 8-byte word is unaddressable. We
use different negative values to distinguish between dif-
ferent kinds of unaddressable memory (heap redzones,
stack redzones, global redzones, freed memory).

Seiya Nuta
@nuta
Thanks for the sharing! It makes sense. I misunderstood how ASan records the shadow memory. Give me a time to skim through the paper...
Yashraj Kakkad
@yashrajkakkad
Thank you! I'll be waiting for your nod :)
Seiya Nuta
@nuta
@yashrajkakkad As I commented in #29, please continue using >> 3.
Prayag Savsani
@PrayagS
So I noticed that virtq_allocate_buffers() allocates vq->num_descs number of buffers of both the same size and r/w access (either all are device-writeable for reading or all are device-readable for transmission).
In the case of virtio-blk, as I had mentioned before, the buffer containing the header needs to be device-readable. The payload buffer's r/w access depends on the operation. And the status byte needs to be device-writeable.
What I did was create three packets for writing and then three for reading. I chained them using the VIRTIO_DESC_F_NEXT flag. Most of what I did is according to the examples given in the spec (2.7.21.3 and 2.7.22).
The missing headers error doesn't come up now but the device isn't responding at the read request for some reason. You can take a look at my code here.
I think it's either the way I'm handling the buffer IDs (the spec doesn't go in much detail but looking at their example, it seemed like it is important) or it is the buffer allocation process since I'm using virtq_allocate_buffers() to allocate buffers of the same size regardless of their type.
Any help would be appreciated. Especially if you can elaborate on the id part since all the elements of a block request are referred to as elements of a buffer whose ID is stored in the last descriptor of the chain.
Yashraj Kakkad
@yashrajkakkad
@nuta I was thinking I can take up one more task in parallel with KASAN, now that my exams are over and I have some time (and curiosity xD). Are we considering a slab allocation scheme in malloc?
Seiya Nuta
@nuta
@PrayagS Good point. Currently, I'm working on the virtio-gpu driver and plan to improve the virtio library in couple of days. In the improvement, I'll try addressing the issue you described. FWIW, you can investigate why your virtio driver doesn't work by inserting some printf functions into QEMU (hw/virtio/virtio.c).

@nuta I was thinking I can take up one more task in parallel with KASAN, now that my exams are over and I have some time (and curiosity xD). Are we considering a slab allocation scheme in malloc?

A slab allocator sounds good idea :)

Yashraj Kakkad
@yashrajkakkad

A slab allocator sounds good idea :)

Great! Can you please create an issue for the same? I will start working on it!

Seiya Nuta
@nuta
@yashrajkakkad Sure! I've created an issue for the slab allocator. By the way, if you have feature requests for proposal, please feel free to create an issue by yourself :) nuta/resea#39
Yashraj Kakkad
@yashrajkakkad
Sure, thank you very much @nuta ! :D
Yashraj Kakkad
@yashrajkakkad

Hi @nuta, our 6 weeks are about to get over. We have our project presentation scheduled this Sunday evening. where we have to explain what we did and show our work (Fingers crossed haha :D). But that is just for our University and we'd love to continue working on the features we're implementing. Meantime, we'd be really grateful if you can share what you feel about our work, which we can use as a testimony in the concluding part of our presentation.

It's the first time we worked on an open-source at this scale and you've been a true guide to us in our journey! Thanks a ton :heart:

Prayag Savsani
@PrayagS
@nuta I totally agree with @yashrajkakkad and would love to continue contributing to this project. It has been a great learning experience working with you. I would really appreciate it if you can share a few words regarding the work we done. And thanks again for being such a great mentor to us.
Seiya Nuta
@nuta
@yashrajkakkad @PrayagS I'm glad to hear that! Especially because, I originally started writing Resea for educational purposes so that university students can understand and develop microkernel-based operating system without pains. I'm impressed that you folks have successfully submitted good pull requests (99% of my review comments were trivial nitpicks) within your deadline despite Resea lacks documentation. You should be proud of your software engineering and discussion skills. I'm looking forward you contributing famous open-source projects :)

@PrayagS Good point. Currently, I'm working on the virtio-gpu driver and plan to improve the virtio library in couple of days. In the improvement, I'll try addressing the issue you described. FWIW, you can investigate why your virtio driver doesn't work by inserting some printf functions into QEMU (hw/virtio/virtio.c).

I've pushed improvements on virtio library to allow constructing a descriptor chain. Can you try it out?

Prayag Savsani
@PrayagS
@nuta Thanks for your kind words. I completely agree with the "without pains" part as the way you've designed the OS did make it quite comfortable for us to work on it. Kudos to you for that!
And I'll be sure to take a look at the changes you made in the virtio library.
Yashraj Kakkad
@yashrajkakkad
Thanks a lot for your kind words, @nuta. We are honoured indeed! We really look forward to interacting with you and contributing more to Resea :smiley:
Yashraj Kakkad
@yashrajkakkad
Hi @nuta. I am stuck with that page fault thing in KASAN for quite a while now. Can you please suggest something?
Seiya Nuta
@nuta
@yashrajkakkad Sure. Can you share your git branch for KASAN?
Yashraj Kakkad
@yashrajkakkad

@yashrajkakkad Sure. Can you share your git branch for KASAN?

https://github.com/yashrajkakkad/resea/tree/asan

Seiya Nuta
@nuta

@yashrajkakkad I tried your code and it failed with general protection fault.

[kernel] WARN: Exception #13

[kernel] RIP = 0000000001005cc5 CS  = 000000000000002b  RFL = 0000000000000246
[kernel] SS  = 0000000000000023 RSP = 000000000300ad70  RBP = 000000000300adb0
[kernel] RAX = 0a110ced0a110ced RBX = 0000000000000004  RCX = 0000000000000001
[kernel] RDX = 0000000004124120 RSI = 00000000000000aa  RDI = f8f8f8f8f8f8f910
[kernel] R8  = 0000000000000003 R9  = 0000000000000000  R10 = 000000000300abd0
[kernel] R11 = 0000000000000246 R12 = f8f8f8f8f8f8f910  R13 = 0000000000000010
[kernel] R14 = 0000000000000020 R15 = f8f8f8f8f8f8f8f8  ERR = 0000000000000000
[vm] WARN: test: exception occurred, killing the task...
[kernel] destroying test...
QEMU 4.2.1 monitor - type 'help' for more information
(qemu) xp /i 0x1005cc5
0x01005cc5:  00 00                    addb     %al, (%rax)

I think this is caused by accessing a non-canonical address in RAX by ADD instruction.

I'm not sure but I guess some part of the memory (e.g. return address in the stack) is accidentally overwritten because RIP = 0x1005cc5 seems to be incorrect:

$ objdump -d build/vm.debug.elf|egrep '1005cc[0-9a-z]' 
 1005930:       0f 85 8d 03 00 00       jne    1005cc3 <vaddr2paddr+0x3e3>
 1005cc3:       48 bf c0 19 00 03 00    movabs $0x30019c0,%rdi
 1005cca:       00 00 00        <-- instruction at 1005cc5 does not exist
 1005ccd:       48 89 de                mov    %rbx,%rsi

Sadly, fixing your bug might take a time...

Yashraj Kakkad
@yashrajkakkad
@nuta Is the problem caused by something in my code? I was seeing such a problem even when my check_address function had nothing but a DBG statement.
Seiya Nuta
@nuta
I'm not sure but enabling KASan itself might cause the problem (unintended infinite recursion in check_address, etc.).
Yashraj Kakkad
@yashrajkakkad
@nuta I think that's the case as putting anything at all in check_address creates a problem. Any way I can help fix this?
Seiya Nuta
@nuta

@nuta I think that's the case as putting anything at all in check_address creates a problem. Any way I can help fix this?

Sure! I'll try fixing it in this weekend.

Seiya Nuta
@nuta
@yashrajkakkad OK I figured out the bug. It seems __shadow is too small and shadow_free accidentally destroyed other memory area. Thus it reproduces even if KASAN is disabled. Try the following patch:
diff --git a/libs/resea/malloc.c b/libs/resea/malloc.c
index e8040b25..5f78190c 100644
--- a/libs/resea/malloc.c
+++ b/libs/resea/malloc.c
@@ -345,6 +345,14 @@ void shadow_free(struct malloc_chunk *chunk)
     reladdr = ptr_cur - (paddr_t)__heap;
     reladdr >>= 3;
     DBG("%u", reladdr);
+
+    size_t len = chunk->capacity + MALLOC_REDZONE_LEN;
+    if ((vaddr_t) &__shadow[reladdr] + len >= (vaddr_t) __shadow_end) {
+        WARN_DBG("__shadow is too small: addr=%p, len=%d, reladdr=%p",
+                 &__shadow[reladdr], len, reladdr);
+        return;
+    }
+
     for(size_t i = 0; i < (chunk->capacity + MALLOC_REDZONE_LEN); i++)
     {
         __shadow[reladdr++] = SHADOW_FREED;
Yashraj Kakkad
@yashrajkakkad
Hi @nuta. Thanks for your time. I will work further on it this weekend
Manav-Kumar
@Manav-Kumar
Hello All, I'm new to resea and wanted to actively contribute in it. I'm a final year CSE btech student enthusiast about the OS concepts, tools etc. Can somebody give me a quick head start for the current active projects and whats the goal briefly. I'll be glad.
Seiya Nuta
@nuta

Hi, @Manav-Kumar. We have already implemented basic features like the microkernel, FAT file system, TCP/IP, and some device drivers. We're working on some improvements and new features:

Since I'm working on another OS project in Rust in very recent weeks, the progress is slow. That said, if you'd like to contribute to Resea, I can support you :)

Aside from those projects, we welcome your ideas and would like you to work on a project whatever you want. For example, integrating Linux Kernel Library would be an interesting topic.

Manav-Kumar
@Manav-Kumar
Yeah, it would be fun to implement it. Can you please share some material around it and how do we plan for it.
Does resea supports multitasking features in current version. Can we plan to improve it in any way if it does.
Manav-Kumar
@Manav-Kumar
Memory Management, vm, paging, interruptions, scheduling.. this might already be implemented in basic kernel features, are there any scopes of improvement.
Seiya Nuta
@nuta

Does resea supports multitasking features in current version. Can we plan to improve it in any way if it does.

Yes it supports. In a nutshell, Resea kernel implements a multi-tasking concept called task (so-called process). Each task has its own virtual memory space and can communicate with other tasks over message passing IPC. Threads are not supported intentionally because all applications in Resea are implemented in a single-threaded, event-driven way to make it easier to debug (we can still implement sort of threads in the userland by mapping same memory pages through vm_map syscalls btw).

Memory Management, vm, paging, interruptions, scheduling.. this might already be implemented in basic kernel features, are there any scopes of improvement.

We only have rudimental implementation so I think there's room for further improvements. Among the mentioned areas, user-level scheduling (nuta/resea#16) might be interesting to work on. A MINIX wiki page is informative on this topic.

I've implemented task priority in the kernel (task_schedule syscall) but the feature is unused at all.
Seiya Nuta
@nuta
Regarding Linux Kernel Library, this recent paper and its references might be helpful. Porting LKL into Resea would be an advanced topic, but it should be fun.
Manav-Kumar
@Manav-Kumar
Hello Sorry for the late response, got busy with my full time job. I'm planning to start working upon it immediately. Can you still please provide support for it. Really interested to implement linux kernel library or if possible something around adding more tricks for resource management like Copy On Write, Zero filled on demand, demand paging etc if it's not implemented yet.
Seiya Nuta
@nuta
You don't have to be sorry! Let's enjoy hacking only when you have time :) Linux Kernel Library should be fun but I'm not sure it's difficulty. For the first step, It might be good to work on smaller improvement on VM server, the core of memory management such as physical page allocation and page fault handling for normal apps. I guess you're interested in improving the server.
Seiya Nuta
@nuta
Here's my some comments on improvements you mentioned:
  • copy-on-write: Not implemented. Mapping zero-filled pages (e.g .bss section) using CoW might improve the startup time of a userland application.
  • zero filled on demand: I don't know what this feature does. BTW, preparing zero-filled pages in advance during an idle state would be good for performance.
  • demand paging: Implemented. In fact, every page is mapped by the pager task on the first access.
Manav-Kumar
@Manav-Kumar
Thanks for the pointers, i'll start working upon improving small things. Yes, you are right about the interest and particularly about core of memory management.