by

Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • Nov 30 2019 21:47
    performantdata commented #113
  • Nov 30 2019 04:34
    performantdata commented #113
  • Nov 30 2019 03:47
    performantdata commented #113
  • Apr 08 2019 17:53

    densh on master

    Scala 2.12 support and updated … Merge pull request #115 from lo… (compare)

  • Apr 08 2019 17:53
    densh closed #115
  • Apr 08 2019 17:52
    densh commented #115
  • Mar 28 2019 20:03
    lolgab commented #113
  • Mar 28 2019 16:02
    lolgab opened #115
  • Feb 11 2019 14:24
    xeno-by removed as member
  • Apr 06 2018 15:30
    satabin commented #113
  • Apr 06 2018 15:22
    satabin commented #113
  • Sep 15 2017 09:01
    daron666 commented #113
  • Jan 24 2017 22:31
    l15k4 commented #47
  • Jan 24 2017 22:18
    l15k4 commented #114
  • Jan 24 2017 22:18
    l15k4 commented #114
  • Jan 24 2017 22:10
    l15k4 closed #114
  • Jan 24 2017 22:10
    l15k4 commented #114
  • Jan 24 2017 22:07
    l15k4 commented #114
  • Jan 24 2017 21:52
    l15k4 commented #114
  • Jan 24 2017 19:49
    DarkDimius commented #114
Denys Shabalin
@densh
After giving it some more thought, I’ve decided that we should just merge Scala Native and Offheap APIs
There is going to be a Ptr[T] type implemented on top of sun.misc.Unsafe
And it will have the same semantics as Ptr[T] on Scala Native
This will supersede @data + @variant + @embed combo we’ve had before
Denys Shabalin
@densh
I’ve started experimenting with this in https://github.com/densh/scala-offheap/commits/topic/scala-native
Denys Shabalin
@densh
@adamwy What have you been up to lately?
Migration to scala-native-style api will take a few days
Adam Wyłuda
@adamwy
@densh So far I have implemented the offheap Seq/Set/Map using data classes and I'm just generating code directly to tests and benchmark modules.
I have run benchmarks and it seems that there are cases where the offheap version is slower than specialized on-heap implementation, so for now I'd like to figure out why that happens.
Denys Shabalin
@densh
What allocator do you use for benchmarks?
System’s malloc is really slow compared to jvm’s gc and our own region allocator.
Adam Wyłuda
@adamwy
Where applicable I'm using both malloc and region in benchmarks.
It looks like it's slower with operations that are not dependent on used allocator like sequential read/update.
Denys Shabalin
@densh
Very interesting results
Denys Shabalin
@densh
This message was deleted
This message was deleted
Philip Stutz
@pstutz
hi denys, when i use an "@data" class that is defined in a different file in the same package i get a "class [name of @data class] must be defined before it's used" compilation error. defining it in the same file is fine as is using it from a different package when it is imported. any idea what might cause this?
jeroentervoorde
@jeroentervoorde
Hi @densh, any ideas for the issue above?
Denys Shabalin
@densh
Hi there, seems to be some unfortunate interaction between separate compilation and our macros.
Can you please provide minimal reproduction and file it as an issue?
jeroentervoorde
@jeroentervoorde
Thanks, i will try to find some time this week.
jeroentervoorde
@jeroentervoorde
Hi @densh . i think i managed to create a reproducer: https://github.com/jeroentervoorde/offheap-error-reproducer . Odd thing is that when moving the code to a different file or package it compiles, when moving the code back to the original file the bug also disappeared. I hope the bug is triggered on your machine. I did try it using 2 laptops.
Denys Shabalin
@densh
@jeroentervoorde Thanks for the reproduction!
Jakub Liska
@l15k4
Hey, pls help me understand, why does this snippet needs at least -Xmx3400mto not throw OOM exception ? https://gist.github.com/l15k4/89d79c0acbf98f86904c190d35283f1f
in theory, there is just one object scala.array on heap and it should be Int.MaxValue big, so smth. like -Xmx2300m should suffice, right ?
Denys Shabalin
@densh
Offheap allocations are not GC managed but they still count toward hotspot memory limit as we use sun.misc.Unsafe.allocateMemory under the hood.
It’s similar to the behaviour of direct byte buffers in nio.
Jakub Liska
@l15k4
I'm talking abotu on-heap here
it just requires more heap memory than it should
as besides from the scala.Array everything should be off-heap
Denys Shabalin
@densh
Every time you do offheap.Array.fromArray("foobar".getBytes) you allocate without freeing.
You should either do a manual free or use a region allocator inside the loop.
object Play extends App {

  import scala.offheap
  implicit val alloc = offheap.malloc

  val arr = new scala.Array[offheap.Array[Int]](Int.MaxValue/4)

  var counter = 0
  while (counter < Int.MaxValue/4) Region { implicit alloc =>
    arr(counter) == offheap.Array.fromArray("foobar".getBytes)
    counter+=1
  }

}
This should be better.
Jakub Liska
@l15k4
hmm, interesting riddle, just to be clear, it is not about pace of Garbage Collection of "foobar".getBytes arrays, right ?
it's about the Region Pool using different way of allocation
Denys Shabalin
@densh
Offheap allocations are not garbage-collected. If you use malloc allocator it requires you to free manually.
Region allocator frees everything automatically as soon as region is over.
Matthew de Detrich
@mdedetrich
@l15k4 If yo think about it, its the equivalent of using malloc/free in C, unless you use a region allocator (which will track this for you)
Jakub Liska
@l15k4
I think I get it, I think that you meant that JVM still tracks sun.misc.Unsafe.allocateMemory allocations and that has some memory overhead
the only think unclear to me now is that after your snippet ends, you cannot use that arr anymore, because its elements have been deallocated right away
so it was just for an explanatory purpose, to show that without the allocation overhead it would need less -Xmx memory, right?
Jakub Liska
@l15k4
hm, I still don't understand, why sun.misc.Unsafe.allocateMemory would yield any other on-heap memory overhead besides 8 bytes for the address, which is stored in the array value classes ... from what I can see, every offheap.Array allocation takes 8+8 for some reason
Jakub Liska
@l15k4
it is all clear now densh/scala-offheap#114
Jakub Liska
@l15k4
I just didn't know that Value Classes get boxed when stored to Array
Kaspar Fischer
@hbf
Is this project still active? Or has development moved to another project?
Denys Shabalin
@densh
@hbf The project has been “on hold” for quite some time now.
I don’t really have time to move it forward, if anyone is interested in stepping in please let me now.
Kaspar Fischer
@hbf
@densh Thanks for the clarifications, much appreciated!