Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
    Gil Tene
    @giltene
    Language changes are for Java 10 or later (probably later).
    Volker Simonis
    @simonis
    I don't want to be overly stubborn, but I still think that using "getCallerClass()" and the "CallerSensitive" annotation don't contradict with your "zero required language or JVM changes" and "works on/with unmodified JDKs" requirements. "getCallerClass()" and the "CallerSensitive" ARE used in the current JDK classes. The fact that we currently have to put the ObjectLayout classes in the boot class path with my proposal is just until we get a JEP and an repository. Once we're there, we put the ObjectLayout classes into the standard libraries anyway.
    That said, I'm not against a solution with method handles if that works well. I just want to concentrate on trying some of the intrinsification magic.
    Gil Tene
    @giltene
    Don't worry about the "stubborn" part. It's good to keep arguing this until we feel comfortable, or at least feel like we have to accept whatever the compromise is.
    Gil Tene
    @giltene
    Looking for input: What if we take the opposite track on constructor accessibility: We don't set accessibility ourselves (anywhere). And if a caller (creating a StructuredArray, primitive array, or intrinsic objects) wants to use a non-public constructor, they would need to pass it via a CtorAndArgs (or constructor parameter where appropriate). With this scheme, it becomes the caller's responsibility to set accessibility on the constructor before passing it in.
    This would have the downside of making "spelling" more cumbersome for non-public classes.
    But it has the upside of keeping things clean. And has no extra burdens for public classes.
    Volker Simonis
    @simonis

    Hi,
    I've just added a first attempt to intrinsify some parts of ObjectLayout in OpenJDK 9.
    Please have a look at https://github.com/simonis/ObjectLayout/tree/hotspot_intrinsification

    There's some documentation in https://github.com/simonis/ObjectLayout/blob/hotspot_intrinsification/hotspot/README

    Wishfully awaiting your comments (I especially like the DeadReckoning example :)

    PS: ..happy new year:)

    This message was deleted
    Gil Tene
    @giltene
    Hey everyone. Long time no chat here... I just pushed a set of changes to ObjectLayout (on github) that make use of a lookup (MethodHandles.Lookup) to test legitimate accessibility of constructors for StructiredArray and PrimitiveArray variants, and to test for legitimate accessibility to the private final fields in IntrinsicObjects.constructWithin(). In general, (with one exception right now, I believe) setAccessible is now only used when accessibility by the caller is verified.
    The exception (right now) is used for StructuredArray.shallowCopy(), which I intend to clean up (by either removing shallowCopy() altogether, or by making it only work for accessible fields, or by adding a lookup variant for that too).
    In the new scheme, lookup objects can be passed whenever constructors or fields need to be looked up internally by the call.
    In IntrinsicObjects.constructWithin() they are mandatory (constructWithin only works on private final fields, which by definition will require a lookup object to gain valid access).
    In StructuredArray, primitive arrays, the array builders, and in CtorAndArgs, using a lookup object is optional whenever a constructor will be looked up internally by the call.
    Gil Tene
    @giltene
    When constructors are passed in (e.g. a a constructor, or via CtorAndArgs, or CtorAndArgsProvider objects), it is the responsibility of the caller to make the constructor it passes is properly accessible. This is most easily done by using the CtorAndArgs(lookup, instanceClass, constructorArgTypes, args...) variant rather than passing a constructor (when using the variant, there is no need to do your own setAccessible(), and no need for static try/catch blocks that would result).
    You can see examples of using the lookup objects in BPlusTree (for constructing non-public StructuredArray subclass, using a non-public element Class, and a non-public ReferenceArray subclass : just search for "lookup").
    Gil Tene
    @giltene
    Overall, this is a fairly minor thing from an API point of view for StructuredArray and primitive arrays (it is optional, and only needed when dealing with non-public subclasses or elements). [Minor from an API point of view doesn't mean minor from a code point of view; this took a while to figure out...].
    For IntrinsicObjects, this is more fundamental, because it is mandatory. As such it affects the "spelling" of intrinsic object initialization (adds a lookup parameter to all calls). While it was possible to have this be "optional" (needed only for non-public fields), I felt that it would be un-natural to have intrinsic objects in anything other than private final fields.
    Ivan Krylov
    @ivankrylov
    Sounds good. Now that on the OL implementation side we have reached some milestone (0 blockers), are you planning the summarize the state in some writeup "the state of object layout"?
    Michael Barker
    @mikeb01
    I still think there is an issue with subclassing the PrimitiveArray and StructuredArray to implement 'struct with array at the end' pattern. In that it exposes methods that can mutate the content of the array. I still think these methods should be protected and then the extending class to choose to expose the behaviour that it wants.
    It would be even better if there was a way to use annotation to express the above pattern in a similar manner to struct with a struct, but I don't know if that is possible.
    Gil Tene
    @giltene
    Hmm... This problem really comes from the fact that the way we represent "struct with array at the end" in OL is within "is a (array)" rather than "has a (array)" relationship. I.e. Via subclassing...
    And while that is certainly a leaky abstraction, I think that it is unfortunately inherent (right now) to Java semantics.
    Gil Tene
    @giltene
    As to whether or not to expose the get() methods on StructuredArray as public, or the get(0, set(), and asArray() methods in primitive arrays as public, or default to protected to for both: that's a tough choice. While protecting them and leaving their public exposure up to the subclass will help with encapsulation, doing so will also make it much harder to make direct use of them as arrays. I.e. a subclass will actually have to expose the various calls (with int and long variants of each, this is annoyingly cumbersome). Then you have similar choices when it comes to visibility when no mutation is possible. E.g. should getLength(), getElementClass(), etc. be protected as well?
    I think that this fundamentally comes down to wanting to provide a "has a (array) relationship" rather than a "is a (array)" relationship. And subclassing is not the best fit for doing that...
    I've been trying to think about the viability of providing "has a (array)" capabilities via @Intrinsic. IntrinsicObject.constrcutWithin() already supports this for constant-lenth arrays (specifies in the @Intrinsic(length = X) form). But what we all want here is the ability to specify a variable length array.
    Gil Tene
    @giltene
    The issue there is that (if we want the best performance here), a class can have at most one such variable length array. And this includes both it's own fields and any fields of it's superclasses. Some of which it may not know about, and some of which may appear as private implementation detail after the subclass was already implemented. This brings up all sorts of messy rule related questions.
    E.g. I'd really want to be able to re-implement String with an @Intrinsic PrimitiveCharArray member. And I can see supporting that. But that would obviously mean that no subclass of String can make use of a variable length @Intrinsic string. What if I chose to do this cool under-the-hood implementation in a year, but someone had already implemented a String subclass that validly used an @Intrinsic variable length string (because at the time, no super class was using it)?....
    We can handle this in a few ways: (A) One simple way is to refuse to instantiate a class with multiple @Intrinsic variable size members. This would create a backwards compatibility issue for base classes, so they'll need to carefully choose what they use for internal implementation over time, but will probably be easiest to work with from a performance-understanding point of view.
    Gil Tene
    @giltene
    The we can (B) allow multiple variable size @Intrinsic members, but only actually "flatten" one of them (the others will end up being implemented with a regular de-referencing step). This has the benefit of interoperability and generality, but makes it harder to predict speed implications of implementations (especially since variable sized members will be private to different levels of the class hierarchy). Some policy would be needed for picking the right member to "flatten", and perhaps for "hinting" at which one it should be. An interesting thing this leaves open is that with enough smartness the JVM can actually make this decision based on profiling information. Either by re-laying-out a class at runtime (hard but possible once Unsafe is "fixed"), or by using cross-run recorded profile guided optimization information in some ways to drive it's "hints".
    Interestingly, allowing @Intrinsic on variable length arrays at this point is actually very easy. Both for the support-multiple-and-pick-one-internally and for the enforce-only-one-in-the-hierarchy forms. Since the vanilla implementation doesn't actually flatten anything, they will all just work. This is why I'm very tempted to support it...
    The real issue is that one we support it, there is no going back. So we need to figure out which form makes the most sense once intrinsic implementations start optimizing this stuff for real.
    Gil Tene
    @giltene
    My #1 reason for wanting to support this officially is simple: Regardless of the public library semantics, I fully intend to create an under-the-hood String implementation that makes a best-effort attempt at keeping the char[] as an effective @Intrinsic PrimitiveCharArray. Such an implementation would not directly contradict any existing behaviors, but it would be nice to use well defined semantics for it.
    Michael Barker
    @mikeb01
    My concern around have visible public members on the inherited *Array classes actually came from trying to implement a String class using PrimitiveCharArray. It made for an implementation that would of been very confusing for a user as it would have a bunch of set methods that required overriding to UnsupportedOperationException if the immutable String semantics were to be maintained. If we can do the array at end with annotations that it definitely better (IMO). As for the difference between option (A) and (B), I think probably option (B). As this makes more sense with the vanilla implementation. As long as the rules are reasonably clear about which one will get flattened. You could probably do (A) and move to (B) later, but not vice versa. With option (B) you could probably support a simple choice (e.g. most specific class is flattened) and later add more advanced choices later (priority flag on the annotation or command line options).
    Gil Tene
    @giltene
    One key challenge for the "has a" variable size member thing, the delicate dance with constructors is still needed: the classes involved must use factories, and the factory must be able to determine length without calling any constructors. It is possible to shield the caller form direct knowledge of what "length" is (e.g. by an OLString class's newInstance computing the needed length from other parameters prior to calling the super-class's newInstance), but the fact that a shift to factories is involved makes this hard to "add late" to an existing implementation that already uses constructors directly.
    Gil Tene
    @giltene
    Even when factories are used (or if we ever find a magic way to convert "new + invokespecial" to a method handle call on a constructor, effectively making it a factory), we still have a challenge around "spelling" for providing the length to variable length member(s). In option A (only a single variable length member allowed), you can use implicit notions for this (someone must supply "length", but no need to name or associate that length with a field because there is exactly one receiver). In option B, some amount of static coding would probably be needed ("declaring" each variable-length intrinsic member for the class ahead of any instantiations, along with ways for factories to determine their needed lengths prior to allocation).
    In any case, I think this (variable length intrinsic members) is "too hard" to tackle in the first incarnation.
    As for the protected/public field access issue: I see a problem going with either one: On the one hand, natural uses of the arrays without added subclassing expose the notion of length to their factory callers, and tend to always want a get(), making it cumbersome for the sub-classer to have to expose those as public. On the other hand (as @mikeb01 notes), attempts to build things that use arrays internally without exposing their contents to mutation (or even to direct bypassing access) make it cumbersome of the sub-classer to have to protect all the public-by-default methods.
    Gil Tene
    @giltene
    How about this (yuck?): We can add protected variants of all the array classes (ProtectedStructuredArray, ProtectedReferenceArray, etc.), which would be used by sub-classing implementations that wish protected-by-default behavior, but keep the public variants?
    Logically, the protected variants would sit between the abstract and public variants in the hierarchy (the public variants would inherit from them, and expose what they need to as public-by-default).
    Would that work?
    Michael Barker
    @mikeb01
    That would work and give the user the option to choose their poison.
    Gil Tene
    @giltene
    Ok. Giving it a try with StructuredArray extends ProtectedStructuredArray... Running into some "questions":
    e.g.: Should ProtectedStructuredArray be Iterable? Probably not. (so Iterator support stays in StructuredArray)...
    Michael Barker
    @mikeb01
    Yep.
    Gil Tene
    @giltene
    And StructuredArrayBuilder: Is there a need for ProtectedStructuredArrayBuilder?
    Michael Barker
    @mikeb01
    Not sure, I haven't used the builder in any of my examples to see how much more useful it is and whether I would miss it.
    I would go without it for now. Can this be added in later if required?
    Gil Tene
    @giltene
    Ok. I just pushed a change with some underlying restructuring, and a ProtectedReferenceArray and ProtectedStructuredArray that I think make sense. I'm holding off on doing the rest of the primitive arrays until we kick this around a bit and work the kinks out (if any), because the various primitive arrays create a bunch of manual copy/re-implent effort I'd rather not repeat to many times.
    @mikeb01 : Can you take a look and think hard about my choices? E.g. I chose to make EVERYTHING protected in these variants. Including newInstance. (only the class is public, nothing else in it). In addition, there is a ProtectedStructuredArrayBuilder that includes various public APIs, but will not create a ProtectedStructuredArray without legitimate access to it's constructors, so I think it is ok on that side of the line.
    @kerbymart
    @kerbymart
    @giltene hi