I’m almost done, just have to fix some linting errors. Will put in the PR once stdlib-js/stdlib#354 is resolved. I’m not sure if I can create 2 parallel PRs.
Will take a look again later today.
Also, regarding refactoring
codePointAt into its own module, do you want me to take that up?
@ShraddheyaS Looking at the
codePointAt utility. I see it differs slightly from the polyfill outlined on the MDN docs (https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/String/codePointAt). Any reason you can think of as to why they wouldn’t also consider whether the indexed character is a low surrogate? My sense is that your implementation is correct, but I want to sanity check.
Interestingly, the ECMAScript spec simply returns the low surrogate if the provided index does not mark the start of a surrogate pair: https://www.ecma-international.org/ecma-262/#sec-string.prototype.codepointat
Was discussed back in 2012 to simply return the low surrogate. Not clear why: https://esdiscuss.org/topic/march-24-meeting-notes#content-24
I am fine with deviating from the ECMAScript spec provided we can justify our reasoning. @ShraddheyaS Perhaps you have some thoughts here?
In thinking about it, I suppose one reason to have
codepointat be only forward looking is for during iteration. For example, if I attempt to iterate over a string while looking forward and backward, I would have to account for repeated surrogate pairs (i.e., every surrogate pair would be returned twice). If the algorithm instead is only forward looking, then I can check for a low surrogate at each iteration, potentially skipping along to the next code point.
I think we have a few of options here. 1) we can keep as is, with the need to justify our algorithm. 2) we refactor to only be forward looking (similar to the polyfill). 3) we can maintain the ability to look backward, but only when an added (third) parameter indicates that looking backward is desired.
Naively, I suppose (3) could be appealing in the scenario where I randomly want code points from a string. However, that is also problematic because then surrogate pairs would be overrepresented among other characters.
So in short, I am leaning toward (2).
Was discussed back in 2012 to simply return the low surrogate. Not clear why: https://esdiscuss.org/topic/march-24-meeting-notes#content-24
This is probably because one can always get the high surrogate using charCodeAt
@ShraddheyaS May also be useful to create a string utility
@stdlib/string/code-point-unit-count which takes a code point and returns the number of code units. This way, when we want to iterate over a string to return code points, we can first invoke codePointAt to get a code point and then codePointUnitCount to advance the index.
Once while deciphering that we have landed at a low surrogate, hence moving backwards in the iteration and then after iterator is at correct start of surrogate pair, another access for returning the appropriate code point after assimilation
The default is for
backward to be false.
Feel free to create a PR for that utility. Been on our internal TODO list for a while!
@ShraddheyaS Should it be
next-extended-grapheme-cluster-break? Not clear to me, based on this thread (https://bugs.python.org/issue30717), whether we’d also want to have a utility for non-extended breaks.
@ShraddheyaS You have any thoughts on whether we should consider updating our implementation (in the future) to use a trie? https://github.com/foliojs/unicode-trie
Original implementation can be found here: https://github.com/unicode-org/icu/blob/5a3ea669aaf3f2c3984e1a10b3998d4b736621bf/icu4c/source/common/unicode/ucptrie.h#L42
It’s possible that we may not want to care about “legacy” clusters. If we do, we could always do
next-legacy-grapheme-cluster-break.
Yeah, the Unicode report explicitly states "the legacy grapheme cluster boundaries are maintained primarily for backwards compatibility with earlier versions of this specification". https://unicode.org/reports/tr29/
Okay. So, let’s not worry about “legacy” boundaries for now. We can just state in the relevant package README that the package applies to extended grapheme clusters.
Re: trie. I think the trie is supposed to actually address the fact that there are multiple ways to encode the same grapheme. But would need to confirm.
Regardless, I think, for now, we can simply go with your implementation and create an issue after the fact that suggests considering using a trie for “fast” lookup.
For example,
In [8]: numGraphemeClusters( 'Z͑ͫ̓ͪ̂ͫ̽͏̴̙̤̞͉͚̯̞̠͍A̴̵̜̰͔ͫ͗͢L̠ͨͧͩ͘G̴̻͈͍͔̹̑͗̎̅͛́Ǫ̵̹̻̝̳͂̌̌͘!͖̬̰̙̗̿̋ͥͥ̂ͣ̐́́͜͞' )
Out[8]: 6
If anyone would like to chime in on strided array function interfaces (similar in concept to NumPy’s ufuncs), here is a tracking issue. :) stdlib-js/stdlib#364
Note that this proposal is a bit more limited and lower level, but would provide an important stepping stone for eventual functionality with NumPy feature parity.