by

Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
    hokiegeek2
    @hokiegeek2
    image.png
    hokiegeek2
    @hokiegeek2
    The second screenshot show the array data slice, the domain corresponding to the array data slice, and the slice data type, uint(8). If the byte array was partitioned as I just described, there would be a zero at the end of each locale slice, which would ensure each string is written in it's entirety to the same locale.
    image.png
    In this case, the second "tech" string and the "foster" string are split between two locales as shown in the python return values where the variable new_strings variable points to the Arkouda/Chapel array and the nsr variable points to the Arkouda/Chapel array read out of hdf5 files, in this case, three of them since I am running Arkouda with three locales.
    image.png
    hokiegeek2
    @hokiegeek2
    I am hoping there's a hook in the codebase involving a BlockDist where I can manually partition the byte array based upon indices into separate locales. With the indices there's a deterministic means of partitioning chunks of bytes to locales. In this example, the locales would need to be {0...31}, {32...59}, and {60...82}
    hokiegeek2
    @hokiegeek2
    It not, perhaps the partitioning logic can be refactored so one could pass in a first-class function a la Python or Scala to define what chunks are assigned to which locale. @bradcray @e-kayrakli Are the hooks there to pass in or leverage custom partitioning logic? Re-reading what @bradcray wrote above, I am guessing not, but just want to double-check, thanks!
    hokiegeek2
    @hokiegeek2
    @bradcray is it the boundingBox that would have to be customized so that the values are not spread evenly across locales?
    image.png
    Thomas Rolinger
    @thomasrolinger
    @hokiegeek2 A problem with blocks in a distribution that are not all the same size is how the underlying indexing works. When all blocks are the same size, you can rely on div/mod tricks to figure out where the “logical” index is mapped to within the distribution. When there are blocks with different sizes, you are basically left with performing a binary search to find out where your element is. GlobalArrays supports variable size blocks and does just that. But doing that is very expensive compared to what happens when all blocks are the same size.
    hokiegeek2
    @hokiegeek2
    @thomasrolinger wow, gotcha, great point
    Thomas Rolinger
    @thomasrolinger
    “Problem” may not be the best way to describe it, it’s more of a consideration i suppose. I imagine it is something that you can extend/customize but it’s a lot more involved and will not give you the same performance.
    hokiegeek2
    @hokiegeek2
    @thomasrolinger so this is really a consideration for when a strings array is written out to hdf5. Is it possible to read in from disk and then "rebalance" so the locales are in the same sized blocks?
    In other words, read into a sub-optimal locale setup and then rebalance? Dunno if there's such a notion in Chapel (or if there should be!). The analogous thing is Spark or Dask is something to avoid, so it's likely the same in Chapel
    Thomas Rolinger
    @thomasrolinger
    I’m not sure if there is a built-in mechanism in Chapel for that. But it seems like something you could do as a pre-processing step prior to creating the distribution, or perhaps by introducing some sort of padding to “balance” things. I’m not all to familiar with distributions in Chapel but have spent a good amount of time doing similar things in other frameworks. Hopefully someone with a bit more knowledge will chime in
    hokiegeek2
    @hokiegeek2
    I just thought of something that I believe is important. The Arkouda approach to arrays of strings to persist the strings as a contiguous array of uint(8) elements with a corresponding indices array, the latter of which is used to access the string elements from each locale. Since the indices are there, I would there still be the binary search requirement?
    Thomas Rolinger
    @thomasrolinger
    If I understand correctly, there is an array that tells you the index of each element? If so, then you wouldn’t need to do a search to find an element, as you essentially have a pre-computed mapping. The only downside to that is memory/space.
    hokiegeek2
    @hokiegeek2
    Yep, the's an array with indices
    Cool, thanks for the confirmation!
    Michael Merrill
    @mhmerrill
    Is there a preferred way to have multi and single locale coexist? is it just as simple as setting export CHPL_COMM=none ?
    Ben Albrecht
    @ben-albrecht
    @mhmerrill - What do you mean by coexist? Having 2 built binaries of CHPL_COMM={gasnet || ugni} and CHPL_COMM=none?
    Michael Merrill
    @mhmerrill
    yes comm=none vs comm=gasnet
    i built a comm=none first but then I needed to simulate multi-locale
    oh i wasn't clear, sorry, I meant I built the chapel compiler with CHPL_COM=none then I needed to build it again with CHPL_comm=gasnet
    Ben Albrecht
    @ben-albrecht
    OK, the Chapel compiler can be built with multiple configurations simultaneously, that you can switch configuration on the fly with env vars or compiler flags. Switching the env var or using the compiler flag —comm={none, gasnet} (and using different -o names to avoid overwriting each other obviously) is a reasonable approach.
    Michael Merrill
    @mhmerrill
    ok, i was afraid I was going to have to make clean and rebuild completely
    Brad Chamberlain
    @bradcray
    @mhmerrill: To put a bit more detail into Ben’s response: The chapel compiler itself is independent of whether it’s built with CHPL_COMM=none or gasnet or whatever. The runtime is not. So if you look under your $CHPL_HOME/bin/ directory, you’ll see a small amount of uniquification due to platform, architecture, etc. but most configurations will share the same path to chpl and binary. Whereas in the $CHPL_HOME/lib directory, you’ll see a lot more diversity in paths based on each setting of CHPL_COMM, CHPL_TASKS, etc.
    @hokiegeek2: Sorry that I missed your questions last week (I took Th-Fri off). Your notion of wanting to be able to specify where the blocks should start/stop is similar to the “cut” distribution that I mentioned which we’ve had a user proof-of-concept, yet don’t support on master. The big difference between how “cut” and “block” would work, as @thomasrolinger notes, is that “cut” would need to store some sort of "directory structure" indicating how to map from indices to locales whereas for “block” it’s “just math.” This is not a huge deal to implement, simply not something we have today.
    @thomasrolinger’s suggestion about potentially padding the strings such that they block up evenly would be another way to get strings to not be split across locales, but has that tricky chicken-and-egg problem where you have to know your data pretty intimately in order to pad them correctly.
    Brad Chamberlain
    @bradcray
    The other approach (which may or may not be appropriate, depending on your context and constraints) would be Engin’s notion of just letting the strings fall where they may, and then when you do an operation that wants them all to be on a single locale, make sure to get them there by hook or by crook (e.g., “this string is all local, so I’ll just do what I want to it” vs. “this string is split, so let me make a temporary buffer locally to store the whole thing and concatenate my local portion with my neighbor’s portion into that buffer before operating on it”).
    hokiegeek2
    @hokiegeek2
    @bradcray gotcha, cool, good info, thanks for the follow-up!
    Michael Merrill
    @mhmerrill
    has anyone build chapel on the new raspberry pi 4 with 8 GB of RAM???
    Elliot Ronaghan
    @ronawho
    Not on raspberry pi, but we have some nightly testing that builds on machines with 2 and 4 GB RAM
    I don't think arkouda will build with 8GB of RAM though, I think you need 11-12 currently
    Elliot Ronaghan
    @ronawho
    Brian G ran on the 1G boards a few years ago -- https://chapel-lang.org/CHIUW/2015/talks/CHIUW-Chapel_on_ARM_final.pdf
    Damon Kwok
    @damon-kwok
    Why not use zulip?
    Brad Chamberlain
    @bradcray
    Gitter’s worked well for us?
    Damon Kwok
    @damon-kwok
    @bradcray chapel-mode has been completed. Can you give me some feedback?
    Brad Chamberlain
    @bradcray
    I was stuck in meetings most of the day today and didn’t have much chance for experimentation, but will try to download and give it a try tomorrow, if not later tonight.
    Damon Kwok
    @damon-kwok
    NP
    Brad Chamberlain
    @bradcray
    @damon-kwok: Maybe here’s the motivation I needed to do so: This is another case that our current mode does badly with (unparenthesized conditionals):
    if loc == here {
                     writeln(“Hi”);
    }
    Does yours handle this case well?
    Damon Kwok
    @damon-kwok
    @bradcray Yes, the new chapel-mode handles it well.
    It will surprise you
    When you encounter any problem, please let me know and I will solve it quickly
    Brad Chamberlain
    @bradcray
    OK, cool. I’ll use this as motivation to give it a try now.
    Damon Kwok
    @damon-kwok
    @bradcray Fixed: (substitute-key-definition 'c-electric-brace nil chapel-mode-map)
    The same applies to chpl-mode
    Brad Chamberlain
    @bradcray
    @damon-kwok: Is there a way, within an emacs session, to validate that I’m using your emacs mode rather than ours? My emacs-fu is not strong enough to know.
    Damon Kwok
    @damon-kwok
    M-x chapel-mode