This will supersede
@data + @variant + @embed combo we’ve had before
I’ve started experimenting with this in https://github.com/densh/scala-offheap/commits/topic/scala-native
Migration to scala-native-style api will take a few days
I have run benchmarks and it seems that there are cases where the offheap version is slower than specialized on-heap implementation, so for now I'd like to figure out why that happens.
System’s
malloc is really slow compared to jvm’s gc and our own region allocator.
Here are recent results: https://gist.github.com/adamwy/3db10a9b93751d5b4555b35a15667ee4
It looks like it's slower with operations that are not dependent on used allocator like sequential read/update.
This message was deleted
hi denys, when i use an "@data" class that is defined in a different file in the same package i get a "class [name of @data class] must be defined before it's used" compilation error. defining it in the same file is fine as is using it from a different package when it is imported. any idea what might cause this?
Can you please provide minimal reproduction and file it as an issue?
Hi @densh . i think i managed to create a reproducer: https://github.com/jeroentervoorde/offheap-error-reproducer . Odd thing is that when moving the code to a different file or package it compiles, when moving the code back to the original file the bug also disappeared. I hope the bug is triggered on your machine. I did try it using 2 laptops.
Hey, pls help me understand, why does this snippet needs at least
-Xmx3400mto not throw OOM exception ? https://gist.github.com/l15k4/89d79c0acbf98f86904c190d35283f1f
in theory, there is just one object
scala.array on heap and it should be Int.MaxValue big, so smth. like -Xmx2300m should suffice, right ?
It’s similar to the behaviour of direct byte buffers in nio.
it just requires more heap memory than it should
as besides from the
scala.Array everything should be off-heap
You should either do a manual free or use a region allocator inside the loop.
object Play extends App {
import scala.offheap
implicit val alloc = offheap.malloc
val arr = new scala.Array[offheap.Array[Int]](Int.MaxValue/4)
var counter = 0
while (counter < Int.MaxValue/4) Region { implicit alloc =>
arr(counter) == offheap.Array.fromArray("foobar".getBytes)
counter+=1
}
}
This should be better.
it's about the Region Pool using different way of allocation
Region allocator frees everything automatically as soon as region is over.
the only think unclear to me now is that after your snippet ends, you cannot use that
arr anymore, because its elements have been deallocated right away
so it was just for an explanatory purpose, to show that without the allocation overhead it would need less
-Xmx memory, right?
I don’t really have time to move it forward, if anyone is interested in stepping in please let me now.