Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
  • Sep 02 14:10

    nuta on move-cursor

    Update Update Update and 1 more (compare)

  • Sep 02 10:03

    nuta on move-cursor

    Update (compare)

  • Sep 02 09:54

    nuta on move-cursor

    Update (compare)

  • Sep 01 13:23

    nuta on move-cursor

    Update Update Update (compare)

  • Sep 01 10:01

    nuta on move-cursor

    Update (compare)

  • Aug 28 13:18
    nuta edited #41
  • Aug 28 13:18
    nuta edited #41
  • Aug 26 15:10

    nuta on move-cursor

    Update Update Update (compare)

  • Aug 26 09:48
    nuta edited #41
  • Aug 26 09:47
    nuta edited #41
  • Aug 26 09:47
    nuta edited #41
  • Aug 26 09:47
    nuta edited #41
  • Aug 21 07:13

    nuta on move-cursor

    wip (compare)

  • Aug 21 01:28

    nuta on move-cursor

    Update (compare)

  • Aug 20 16:21

    nuta on move-cursor

    Update (compare)

  • Aug 20 15:14

    nuta on move-cursor

    Update (compare)

  • Aug 20 15:10

    nuta on move-cursor

    Update Update Update (compare)

  • Aug 20 14:05

    nuta on move-cursor

    Update (compare)

  • Aug 19 12:51

    nuta on move-cursor

    Update Update (compare)

  • Aug 18 17:23

    nuta on move-cursor

    Update Update (compare)

Seiya Nuta
@nuta

@nuta Won't the test for nuta/resea#28 be similar to usage of shm in linux.

I think so. Specifically, we need at least two tasks (apps/test and a new app like app/shm_test). app/test creates a shared memory and tell it app/shm_test. It maps the shm and writes something to the shm. Lastly, apps/test checks if the written data is visible.

Also, I believe shm.c to be a part of resea/include/ or will it be in the kernel/ as well ?

Do you mean API wrappers like libs/resea/include/async.h? If so, I think it should be resea/include/experimental/shm.h since its not yet matured feature.

Prayag Savsani
@PrayagS
Congratulations on the new release @nuta! I'm glad that our team could be part of it.
Seiya Nuta
@nuta
Thank you a lot for your contribution @PrayagS! :D
Yashraj Kakkad
@yashrajkakkad
Hi @nuta I apologise for being inactive. I am not able to contribute due to my upcoming examinations. I will be up soon.
Seiya Nuta
@nuta
@yashrajkakkad That's absolutely no problem and don't have to be sorry! You're totally right and preparing for examinations is more and more important. Please work on only when you have time to spare :)
malbx
@malbx
@nuta I have started working on the random number server. Can you take a look and let me know if this approach is what you had in mind? Also, do you have any thoughts on how you would like to handle the seed?
malbx/resea@803d14e
Seiya Nuta
@nuta

@malbx Your design looks good to me! Could you create a PR? Let's improve it incrementally.

Also, do you have any thoughts on how you would like to handle the seed?

Towards a cryptographic-secure RNG, we might need to add kernel support and a system call to get the seed from the hardware. I'm not familiar with random number generators but AFAIK Linux extracts randomness from interrupt timings and RDRAND instruction on x64.

Yashraj Kakkad
@yashrajkakkad
Thank you @nuta! See you soon hopefully :D
Seiya Nuta
@nuta
good luck 👍
Prayag Savsani
@PrayagS
@nuta Our presentations are due soon and we've around a week until that. I was thinking of another project to work on and saw virtio-blk and Userspace debugging using LLDB on the board. Which one do you think is more suitable given the short timeframe? Does the LLDB one involve that much work? Please let me know your thoughts.
Or if you have any ideas on something completely else that fits the timeframe, please feel free to share those as well.
Prayag Savsani
@PrayagS

@nuta So I started some work on the virtio-blk driver and went through the spec and some existing implementations (1, 2).

I noticed how they're sending three different buffers (one for header, one for data and one for device writeable status byte) and chaining them using the next field. resea's virtio_modern implementation doesn't have that but does mention descriptor chaining in the code. So can you please explain how that would work in resea?

I think I should be able to make it work like those existing ones using the legacy interface but I'm confused regarding how chaining works in the modern implementation.

Seiya Nuta
@nuta

@PrayagS I think implementing virtio-blk driver (for legacy virtio device) is more realistic.

You may already know, Resea's virtio_modern implementation uses Packed Virtqueue. See 2.7.6 Next Flag: Descriptor Chaining in the spec (the specification describes both legacy and modern devices in the single page and it makes a bit confusing...).

I haven't tried yet but according to the QEMU's implemenetation (in virtqueue_packed_read_next_desc()), we don't use next field in the packed virtqueue. Instead, we need to use continuous descriptors as a chain and set VIRTIO_DESC_F_NEXT to the descriptor flags (except the last one).

The current virtio-net driver does not use that simply because it's not needed. Thus, you may also need to improve also the Resea's virtio_modern to handle the descriptor chaining.

Prayag Savsani
@PrayagS
@nuta Thanks for clearing this out. So if I'm getting it correctly, since we're using packed virtqueues, we don't need to set the next field and just set the flags accordingly. Please let me know if I wrongly interpreted this. I'll still implement it for the legacy device first and then proceed with the modern implementation.
Seiya Nuta
@nuta

since we're using packed virtqueues, we don't need to set the next field and just set the flags accordingly

i'm not 100% sure but I think so.

I'll still implement it for the legacy device first and then proceed with the modern implementation.

Ah I got it wrong. It looks that Resea's virtio library abstracts modern/legacy interfaces so you won't have to care the differences.

Yashraj Kakkad
@yashrajkakkad
@nuta Should I continue with '>>3' in shadow? The paper describes a way of assigning "states" to our 8 bits.
Seiya Nuta
@nuta
I think you don't have to. The original ASan uses single bit state for each memory address to reduce the memory footprint. However, I think we don't need to do that since we don't have large scale apps.
Yashraj Kakkad
@yashrajkakkad
I can surely do that, but then we'll not be able to mark different regions with different states right? It'll be either addressable or unaddressable. Wouldn't the scheme mentioned in the paper you shared be better?
Seiya Nuta
@nuta

we'll not be able to mark different regions with different states right?

Can you provide a concrete example? It seems I misunderstood something.

Yashraj Kakkad
@yashrajkakkad

Relevant section from the paper -

We use the following encoding for each shadow byte:
0 means that all 8 bytes of the corresponding application
memory region are addressable; k (1 ≤ k ≤ 7) means that
the first k bytes are addressible; any negative value indi-
cates that the entire 8-byte word is unaddressable. We
use different negative values to distinguish between dif-
ferent kinds of unaddressable memory (heap redzones,
stack redzones, global redzones, freed memory).

Seiya Nuta
@nuta
Thanks for the sharing! It makes sense. I misunderstood how ASan records the shadow memory. Give me a time to skim through the paper...
Yashraj Kakkad
@yashrajkakkad
Thank you! I'll be waiting for your nod :)
Seiya Nuta
@nuta
@yashrajkakkad As I commented in #29, please continue using >> 3.
Prayag Savsani
@PrayagS
So I noticed that virtq_allocate_buffers() allocates vq->num_descs number of buffers of both the same size and r/w access (either all are device-writeable for reading or all are device-readable for transmission).
In the case of virtio-blk, as I had mentioned before, the buffer containing the header needs to be device-readable. The payload buffer's r/w access depends on the operation. And the status byte needs to be device-writeable.
What I did was create three packets for writing and then three for reading. I chained them using the VIRTIO_DESC_F_NEXT flag. Most of what I did is according to the examples given in the spec (2.7.21.3 and 2.7.22).
The missing headers error doesn't come up now but the device isn't responding at the read request for some reason. You can take a look at my code here.
I think it's either the way I'm handling the buffer IDs (the spec doesn't go in much detail but looking at their example, it seemed like it is important) or it is the buffer allocation process since I'm using virtq_allocate_buffers() to allocate buffers of the same size regardless of their type.
Any help would be appreciated. Especially if you can elaborate on the id part since all the elements of a block request are referred to as elements of a buffer whose ID is stored in the last descriptor of the chain.
Yashraj Kakkad
@yashrajkakkad
@nuta I was thinking I can take up one more task in parallel with KASAN, now that my exams are over and I have some time (and curiosity xD). Are we considering a slab allocation scheme in malloc?
Seiya Nuta
@nuta
@PrayagS Good point. Currently, I'm working on the virtio-gpu driver and plan to improve the virtio library in couple of days. In the improvement, I'll try addressing the issue you described. FWIW, you can investigate why your virtio driver doesn't work by inserting some printf functions into QEMU (hw/virtio/virtio.c).

@nuta I was thinking I can take up one more task in parallel with KASAN, now that my exams are over and I have some time (and curiosity xD). Are we considering a slab allocation scheme in malloc?

A slab allocator sounds good idea :)

Yashraj Kakkad
@yashrajkakkad

A slab allocator sounds good idea :)

Great! Can you please create an issue for the same? I will start working on it!

Seiya Nuta
@nuta
@yashrajkakkad Sure! I've created an issue for the slab allocator. By the way, if you have feature requests for proposal, please feel free to create an issue by yourself :) nuta/resea#39
Yashraj Kakkad
@yashrajkakkad
Sure, thank you very much @nuta ! :D
Yashraj Kakkad
@yashrajkakkad

Hi @nuta, our 6 weeks are about to get over. We have our project presentation scheduled this Sunday evening. where we have to explain what we did and show our work (Fingers crossed haha :D). But that is just for our University and we'd love to continue working on the features we're implementing. Meantime, we'd be really grateful if you can share what you feel about our work, which we can use as a testimony in the concluding part of our presentation.

It's the first time we worked on an open-source at this scale and you've been a true guide to us in our journey! Thanks a ton :heart:

Prayag Savsani
@PrayagS
@nuta I totally agree with @yashrajkakkad and would love to continue contributing to this project. It has been a great learning experience working with you. I would really appreciate it if you can share a few words regarding the work we done. And thanks again for being such a great mentor to us.
Seiya Nuta
@nuta
@yashrajkakkad @PrayagS I'm glad to hear that! Especially because, I originally started writing Resea for educational purposes so that university students can understand and develop microkernel-based operating system without pains. I'm impressed that you folks have successfully submitted good pull requests (99% of my review comments were trivial nitpicks) within your deadline despite Resea lacks documentation. You should be proud of your software engineering and discussion skills. I'm looking forward you contributing famous open-source projects :)

@PrayagS Good point. Currently, I'm working on the virtio-gpu driver and plan to improve the virtio library in couple of days. In the improvement, I'll try addressing the issue you described. FWIW, you can investigate why your virtio driver doesn't work by inserting some printf functions into QEMU (hw/virtio/virtio.c).

I've pushed improvements on virtio library to allow constructing a descriptor chain. Can you try it out?

Prayag Savsani
@PrayagS
@nuta Thanks for your kind words. I completely agree with the "without pains" part as the way you've designed the OS did make it quite comfortable for us to work on it. Kudos to you for that!
And I'll be sure to take a look at the changes you made in the virtio library.
Yashraj Kakkad
@yashrajkakkad
Thanks a lot for your kind words, @nuta. We are honoured indeed! We really look forward to interacting with you and contributing more to Resea :smiley:
Yashraj Kakkad
@yashrajkakkad
Hi @nuta. I am stuck with that page fault thing in KASAN for quite a while now. Can you please suggest something?
Seiya Nuta
@nuta
@yashrajkakkad Sure. Can you share your git branch for KASAN?
Yashraj Kakkad
@yashrajkakkad

@yashrajkakkad Sure. Can you share your git branch for KASAN?

https://github.com/yashrajkakkad/resea/tree/asan

Seiya Nuta
@nuta

@yashrajkakkad I tried your code and it failed with general protection fault.

[kernel] WARN: Exception #13

[kernel] RIP = 0000000001005cc5 CS  = 000000000000002b  RFL = 0000000000000246
[kernel] SS  = 0000000000000023 RSP = 000000000300ad70  RBP = 000000000300adb0
[kernel] RAX = 0a110ced0a110ced RBX = 0000000000000004  RCX = 0000000000000001
[kernel] RDX = 0000000004124120 RSI = 00000000000000aa  RDI = f8f8f8f8f8f8f910
[kernel] R8  = 0000000000000003 R9  = 0000000000000000  R10 = 000000000300abd0
[kernel] R11 = 0000000000000246 R12 = f8f8f8f8f8f8f910  R13 = 0000000000000010
[kernel] R14 = 0000000000000020 R15 = f8f8f8f8f8f8f8f8  ERR = 0000000000000000
[vm] WARN: test: exception occurred, killing the task...
[kernel] destroying test...
QEMU 4.2.1 monitor - type 'help' for more information
(qemu) xp /i 0x1005cc5
0x01005cc5:  00 00                    addb     %al, (%rax)

I think this is caused by accessing a non-canonical address in RAX by ADD instruction.

I'm not sure but I guess some part of the memory (e.g. return address in the stack) is accidentally overwritten because RIP = 0x1005cc5 seems to be incorrect:

$ objdump -d build/vm.debug.elf|egrep '1005cc[0-9a-z]' 
 1005930:       0f 85 8d 03 00 00       jne    1005cc3 <vaddr2paddr+0x3e3>
 1005cc3:       48 bf c0 19 00 03 00    movabs $0x30019c0,%rdi
 1005cca:       00 00 00        <-- instruction at 1005cc5 does not exist
 1005ccd:       48 89 de                mov    %rbx,%rsi

Sadly, fixing your bug might take a time...

Yashraj Kakkad
@yashrajkakkad
@nuta Is the problem caused by something in my code? I was seeing such a problem even when my check_address function had nothing but a DBG statement.
Seiya Nuta
@nuta
I'm not sure but enabling KASan itself might cause the problem (unintended infinite recursion in check_address, etc.).
Yashraj Kakkad
@yashrajkakkad
@nuta I think that's the case as putting anything at all in check_address creates a problem. Any way I can help fix this?
Seiya Nuta
@nuta

@nuta I think that's the case as putting anything at all in check_address creates a problem. Any way I can help fix this?

Sure! I'll try fixing it in this weekend.

Seiya Nuta
@nuta
@yashrajkakkad OK I figured out the bug. It seems __shadow is too small and shadow_free accidentally destroyed other memory area. Thus it reproduces even if KASAN is disabled. Try the following patch:
diff --git a/libs/resea/malloc.c b/libs/resea/malloc.c
index e8040b25..5f78190c 100644
--- a/libs/resea/malloc.c
+++ b/libs/resea/malloc.c
@@ -345,6 +345,14 @@ void shadow_free(struct malloc_chunk *chunk)
     reladdr = ptr_cur - (paddr_t)__heap;
     reladdr >>= 3;
     DBG("%u", reladdr);
+
+    size_t len = chunk->capacity + MALLOC_REDZONE_LEN;
+    if ((vaddr_t) &__shadow[reladdr] + len >= (vaddr_t) __shadow_end) {
+        WARN_DBG("__shadow is too small: addr=%p, len=%d, reladdr=%p",
+                 &__shadow[reladdr], len, reladdr);
+        return;
+    }
+
     for(size_t i = 0; i < (chunk->capacity + MALLOC_REDZONE_LEN); i++)
     {
         __shadow[reladdr++] = SHADOW_FREED;
Yashraj Kakkad
@yashrajkakkad
Hi @nuta. Thanks for your time. I will work further on it this weekend
Manav-Kumar
@Manav-Kumar
Hello All, I'm new to resea and wanted to actively contribute in it. I'm a final year CSE btech student enthusiast about the OS concepts, tools etc. Can somebody give me a quick head start for the current active projects and whats the goal briefly. I'll be glad.
Seiya Nuta
@nuta

Hi, @Manav-Kumar. We have already implemented basic features like the microkernel, FAT file system, TCP/IP, and some device drivers. We're working on some improvements and new features:

Since I'm working on another OS project in Rust in very recent weeks, the progress is slow. That said, if you'd like to contribute to Resea, I can support you :)

Aside from those projects, we welcome your ideas and would like you to work on a project whatever you want. For example, integrating Linux Kernel Library would be an interesting topic.

Manav-Kumar
@Manav-Kumar
Yeah, it would be fun to implement it. Can you please share some material around it and how do we plan for it.
Does resea supports multitasking features in current version. Can we plan to improve it in any way if it does.