by

Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
    hokiegeek2
    @hokiegeek2
    @e-kayrakli yup, if that's possible, a distributed string is exactly what's needed!
    @bradcray cool, excellent explanation, thanks!
    Engin Kayraklioglu
    @e-kayrakli
    @bradcray — Yeah, it is pretty much just that
    What I imagine is something like:
    record distString {
      forwarding var s: string;
      // some other fields to describe “distribution”
    
      proc writeToHDF5() {
        coforall l in Locales do on l {
           // get local slice, something like
           const myLocalSlice = s[this.localSliceDescriptorRangeWhatever];
           // now just write myLocalSlice locally somehow
        }
      }
    }
    hokiegeek2
    @hokiegeek2
    @bradcray @e-kayrakli What I was thinking, if we stick with the current construct of arrays of things, where, in this case, strings=things, my thought was to key on the delimiter (in the case of strings, I think is always 0:uint(8)), and basically have a max BlockDist size and, if the next sequence of non-null characters > max BlockDist size, write to the next one
    I don't think this approach is that complicated, but I am definitely a Chapel novice, so I may be way off, dunno
    Brad Chamberlain
    @bradcray
    I think “array of strings” is easy to get partitioned correctly, but will have each string in its own heap buffer (where the array elemenst are essentially the string meta-data pointing to the heap). Where Arkouda gets complicated is that it uses arrays of uints and then imposes a string interpretation on it, so the language knows nothing about the strings; it’s all embedded in the application-level meta-data.
    hokiegeek2
    @hokiegeek2
    @bradcray Yeah, gotcha. I think the key thing is that I need to handle the use case where a user just reads in one file
    Brad Chamberlain
    @bradcray
    Having a block distribution where the partitions are sensitive to the data in the array is complicated because it’s a chicken and egg problem: “Make me a 100000-element array that’s block distributed based on running this function over the 100000 elements.” If you could store all the elements to think about them, you wouldn’t need to make the array distributed to begin with. And when you can’t, it’s hard to know how to partition the array since you’d essentially be doing it on the fly as you populated its elements. Tricky.
    What I like about Engin’s proposal is that it leaves the array as-is and then does the localization lazily / on-the-fly / when needed at the I/O boundaries.
    hokiegeek2
    @hokiegeek2
    And that's why I am asking. If that's not a valid use case, then we may be able to emulate what happens when the array of "strings" AKA uint(8) sequences are written to openmem(), which is proven to work just fine.
    @bradcray yeah, if what @e-kayrakli is proposing is doable, that's definitely they way to go
    Engin Kayraklioglu
    @e-kayrakli
    I also want to add that I don’t know how Arkouda strings work. I just know a bit about Chapel strings :) In other words, I don’t see any major roadblock in what I am suggesting, but I don’t know how applicable it is to your use case
    hokiegeek2
    @hokiegeek2
    and @e-kayrakli proposal I am doing on the fly with grabbing the local slices and manipulating 'em. But....the key thing is that his approach should cover all of my use cases, including when the user wants to read from 1..n locales, not the entire dataset
    Engin Kayraklioglu
    @e-kayrakli
    And get a string that is only part of the data?
    hokiegeek2
    @hokiegeek2
    @e-kayrakli AFAIK, Arkouda strings = Chapel strings = uint(8) arrays?
    Engin Kayraklioglu
    @e-kayrakli
    I remember something about segmented arrays (?) at somepoint that made Arkouda strings = special uint(8) arrays. But I can be wrong
    hokiegeek2
    @hokiegeek2
    @e-kayrakli Yup, you're correct re: Arkouda strings = segmented arrays. I don't, however, have to get one string. This is for writing "strings" to/reading from hdf5, and I am thinking I need to make it so each string AKA array of uint(8) elements go to the same locale if I need to enable reading strings from one locale only.
    Brad Chamberlain
    @bradcray
    @hokiegeek2: Not really, unfortunately. I mean, Chapel strings are buffers of uint(8)s at their core, but there’s metadata that describes them and implements their string-ness which isn’t present in Arkouda.
    hokiegeek2
    @hokiegeek2
    @e-kayrakli you are correct! Given you answer, I am guessing Chapel strings ain't the same
    @bradcray gotcha, okay
    Brad Chamberlain
    @bradcray
    Think of a Chapel string as a record pointing to a uint(8) buffer. An array of Chapel strings is an array of records, each of which points off to somewhere else. So the string metadata is consecutive in memory and blocked such that no string is spanning locales, but the strings themselves are all disjoint on the heap.
    Arkouda’s strings are block-distributed arrays of uint(8)s which Arkouda interprets as strings by storing segments of where each string begins. So all the string data is consecutive in memory; but a string may be split between locales.
    Each approach has tradeoffs, obviously.
    hokiegeek2
    @hokiegeek2
    @bradcray ah, yes, excellent summary, thank you!
    hokiegeek2
    @hokiegeek2
    @bradcray @e-kayrakli thank you both for allowing me to borrow your respective brains -> excellent discussion that really helped
    So @e-kayrakli, when are you getting started, LOL
    Engin Kayraklioglu
    @e-kayrakli
    :) FWIW, looking at Brad’s description of Arkouda strings, you may need to adapt what I was suggesting to Arkouda strings. i.e. keep storing them using regular Block dist as it already is, but locally buffer and write that buffer when you need to write it. Reading may get a bit complicated though. i.e. I am not sure how to answer the question “I read these bytes here, but where do they go??”, but it sounds like you may have an answer by looking at Arkouda’s string implementation
    hokiegeek2
    @hokiegeek2
    @e-kayrakli Gotcha, cool, excellent info, will ponder this a bit more. Thanks again to both you and @bradcray!
    hokiegeek2
    @hokiegeek2
    @bradcray @e-kayrakli I think there's a way to do this in a straightforward and deterministic way. In the first screenshot there's the entire array of strings in the form of a byte array where the strings are separated by null bytes. Also in this first screenshot is the list of indices, in other words, where the string segments start as well as the overall length of the byte array. The list of indices I think provides a means of manually partitioning the byte data so that complete strings are written to the locales.
    image.png
    hokiegeek2
    @hokiegeek2
    The second screenshot show the array data slice, the domain corresponding to the array data slice, and the slice data type, uint(8). If the byte array was partitioned as I just described, there would be a zero at the end of each locale slice, which would ensure each string is written in it's entirety to the same locale.
    image.png
    In this case, the second "tech" string and the "foster" string are split between two locales as shown in the python return values where the variable new_strings variable points to the Arkouda/Chapel array and the nsr variable points to the Arkouda/Chapel array read out of hdf5 files, in this case, three of them since I am running Arkouda with three locales.
    image.png
    hokiegeek2
    @hokiegeek2
    I am hoping there's a hook in the codebase involving a BlockDist where I can manually partition the byte array based upon indices into separate locales. With the indices there's a deterministic means of partitioning chunks of bytes to locales. In this example, the locales would need to be {0...31}, {32...59}, and {60...82}
    hokiegeek2
    @hokiegeek2
    It not, perhaps the partitioning logic can be refactored so one could pass in a first-class function a la Python or Scala to define what chunks are assigned to which locale. @bradcray @e-kayrakli Are the hooks there to pass in or leverage custom partitioning logic? Re-reading what @bradcray wrote above, I am guessing not, but just want to double-check, thanks!
    hokiegeek2
    @hokiegeek2
    @bradcray is it the boundingBox that would have to be customized so that the values are not spread evenly across locales?
    image.png
    Thomas Rolinger
    @thomasrolinger
    @hokiegeek2 A problem with blocks in a distribution that are not all the same size is how the underlying indexing works. When all blocks are the same size, you can rely on div/mod tricks to figure out where the “logical” index is mapped to within the distribution. When there are blocks with different sizes, you are basically left with performing a binary search to find out where your element is. GlobalArrays supports variable size blocks and does just that. But doing that is very expensive compared to what happens when all blocks are the same size.
    hokiegeek2
    @hokiegeek2
    @thomasrolinger wow, gotcha, great point
    Thomas Rolinger
    @thomasrolinger
    “Problem” may not be the best way to describe it, it’s more of a consideration i suppose. I imagine it is something that you can extend/customize but it’s a lot more involved and will not give you the same performance.
    hokiegeek2
    @hokiegeek2
    @thomasrolinger so this is really a consideration for when a strings array is written out to hdf5. Is it possible to read in from disk and then "rebalance" so the locales are in the same sized blocks?
    In other words, read into a sub-optimal locale setup and then rebalance? Dunno if there's such a notion in Chapel (or if there should be!). The analogous thing is Spark or Dask is something to avoid, so it's likely the same in Chapel
    Thomas Rolinger
    @thomasrolinger
    I’m not sure if there is a built-in mechanism in Chapel for that. But it seems like something you could do as a pre-processing step prior to creating the distribution, or perhaps by introducing some sort of padding to “balance” things. I’m not all to familiar with distributions in Chapel but have spent a good amount of time doing similar things in other frameworks. Hopefully someone with a bit more knowledge will chime in
    hokiegeek2
    @hokiegeek2
    I just thought of something that I believe is important. The Arkouda approach to arrays of strings to persist the strings as a contiguous array of uint(8) elements with a corresponding indices array, the latter of which is used to access the string elements from each locale. Since the indices are there, I would there still be the binary search requirement?
    Thomas Rolinger
    @thomasrolinger
    If I understand correctly, there is an array that tells you the index of each element? If so, then you wouldn’t need to do a search to find an element, as you essentially have a pre-computed mapping. The only downside to that is memory/space.
    hokiegeek2
    @hokiegeek2
    Yep, the's an array with indices
    Cool, thanks for the confirmation!
    Michael Merrill
    @mhmerrill
    Is there a preferred way to have multi and single locale coexist? is it just as simple as setting export CHPL_COMM=none ?