Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
Anton Latukha
@Anton-Latukha

It is interesting that Frames context stack collection already produces that much overhead, we not even integrated it much into the system (most messages and exceptions are send without Frames use). We probably would be able to shave a lot in Frames processes somewhere.

And in any case with Frames cost - in the end we would dodge the Frames cost by not gathering them where the debugging is not needed (in the quiet mode at very least), which would make those modes run much faster.

Anton Latukha
@Anton-Latukha

Talking about Frame messaging system:

Is anyone aware of the current importance of the reinvented messaging classification/verbosity implementation in Nix, (it is where there are 5 levels of errors). AFAIK it was not working by its design, and Nix own messages were not ever following that gradation.

If DevOps infrastructures not really use it much - we better switch to syslog and pretend that under "levels of errors" one meant "Debug, Informational, Notice, Warning, Error" gradation and deduplicate the Nix's -v5 and make it obey the syslog standart.

Anton Latukha
@Anton-Latukha
@Infinisil, what is your exp about the Nix message system :arrow_up:, how much you encountered the use of it.
Ryan Trinkle
@ryantrinkle
@Anton-Latukha The thunk forcing is probably not super easy to change - it's gotta match how Nix does it
But maybe we can make it more efficient
Anton Latukha
@Anton-Latukha

Of course. I hope we arrive to golden tests soon and use Nix as a generator.

The thunk work you do of course relates to it.

Ryan Trinkle
@ryantrinkle
@Anton-Latukha Yeah, I have two particular goals to (eventually) get to on thunks
1) being able to roll back thunks
i.e. to their unevaluated state
this is necessary if we want to be able to evaluate all of nixpkgs without massive amounts of RAM
we can use an LRU policy for which thunks get to stay evaluated
2) being able to do all evaluation in parallel
with Threadripper being a thing these days, even if we're 20x slower on the CPU, we can still be 3x faster on the clock ;)
Anton Latukha
@Anton-Latukha

+====

I keep updating: https://github.com/haskell-nix/hnix/wiki/Design-of-the-HNix-code-base

So, if you are interested in the project, pop in its Wiki from time to time.

The gist of the core data type language (F-algebras for recursion shemes) of HNix:

data NValueF p m r  -- Functors for their F-algebras
    = NVConstantF NAtom
    | NVStrF NixString
    | NVPathF FilePath
    | NVListF [r]
    | NVSetF (AttrSet r) (AttrSet SourcePos)
    | NVClosureF (Params ()) (p -> m r)
    | NVBuiltinF String (p -> m r)

newtype NValue' t f m a =
  NValue
    { _nValue :: f (NValueF (NValue t f m) m a)  -- Applying F-algebra functor to the F-algebra carrier, aka `F (A)` OR `f a`
    }

--   The 'Free' structure is used here to represent the possibility that
--   Nix language cycles may appear during normalization.
type NValue t f m = Free (NValue' t f m) t

Soon would have insight enough to add documentation whenever arrive in the project, aka start teaching in the documentation how the project's source code goes together.
John writes in a very direct insightful pleasant manner, but people who does not know what he is talking about - would not understand. I can not match his manner, but can unfold the semantics of the code and the docs to explain to people. After all, data can be transmitted only wrapped with the addition of the error correction information (error correction code overhead) and unfold metadata overhead, even when having the realistically full knowledge of the context of data transmission.

In other words: When you can not transmit being direct - unfolding is what helps. All protocols and transmission encodings are the unfolds (ana) on the pure entropy of the data.

Anton Latukha
@Anton-Latukha
It is just as we needed to wrap the pure chaos into something to pass it around to process, or something. Such a strange burrito.
Anton Latukha
@Anton-Latukha

+====

Yesterday merged the haskell-nix/hnix#805 where a lot of instances found their parents. ~35 instances overall. So as you can guess, the HNix compilation became faster, and dependent project compilations also should become faster.

Anton Latukha
@Anton-Latukha

+====

Minor HNix-Store-{Core,Remote} releases:


hnix-store-core

hnix-store-remote

Anton Latukha
@Anton-Latukha

@marsam, releases are out.

There are still things to do but what important to Nixpkgs and macOS users is done.

Also GitHub shenanigans being weird, though it blocked me by a strange activity spike in the project or so, but seems it is infrastructure shenanigans, blocked messages appear 10 minutes later.

With this GitHub state the Haskell Nixpkgs maintenance can be difficult. Since GitHub even blocks SSH source uploads.
Screenshot-2021-03-12-20:41:53.png
Anton Latukha
@Anton-Latukha

+====

GHC 8.10.4 is the default in Nixpkgs.

Anton Latukha
@Anton-Latukha

+====

Would note that I refactored the class MonathThunk & demand & force to more lightweight & straight-forward versions.

Thou, I just now understood that they were written in a (continuation-passing style) way that was (probably) allowing to construct recursive scheme transformations from them. Ooops. Thou, seems internally we were not thinking (not mentioned anywhere) nor using them in those terms.

Thou, had enough sanity & theoretical understanding to preserve those unfolded versions in demandF & forceF - so there they are.

Thou (again), reduced that in Normal module normalizeValue. Would reintroduce the "F to pay respects" version there, before the release.

Overall, interested to know from people - where people think in the sort of language implementation & interpretation the base functor versions are (really) needed.

The release is a chunker one, it would stump in soon, since all that is a side effect of my hands (& mind I hope) - if something, write to me, would support transition.

Anton Latukha
@Anton-Latukha

+====

(diff) 0.13.0 (2021-05-10)

  • Breaking:

    • (link) (link) (link) Nix.Thunk: class MonadThunk t m a | t -> m, t -> a. Class was initially designed with Kleisli arrows (v -> m a) in mind, which where put to have the design open and inviting customization & integration. Those functional arguments are for custom implementation, so which in reality of the project were never used and HNax just "essentially" (simplifying, because pure was mixed with monadic binds to f) was passing pure into them (actually, f <=< pure). These Kliesli functors got arguments sorted properly and were moved to a MonadThunkF class and names gained *F. And MonadThunk now does only what is needed, for example force gets the thunk and computes it. All MonadThunk{,F} functions become with a classic Haskell arguments order, specialized, and got more straigh-forward to understand and use, and so now they tail recurse also.

      Now, for example, instead of force t f use it as v <- force t f =<< force t, or f <=< force.

      tl;dr: results:

      ```haskell
      class MonadThunkId m => MonadThunk t m a | t -> m, t -> a where

      thunkId  :: t -> ThunkId m
      
      thunk    :: m a -> m t
      
      queryM   :: m a -> t -> m a
      -- old became `queryMF`
      
      force    :: t -> m a
      -- old became `forceF`
      
      forceEff :: t -> m a
      -- old became `forceEffF`
      
      further  :: t -> m t
      -- old became `furtherF`
  -- | Class of Kleisli functors for easiness of customized implementation developlemnt.
  class MonadThunkF t m a | t -> m, t -> a where

    queryMF   :: (a   -> m r) -> m r -> t -> m r
    -- was :: t -> m r -> (a   -> m r) -> m r

    forceF    :: (a   -> m r) -> t   -> m r
    -- was :: t   -> (a   -> m r) -> m r

    forceEffF :: (a   -> m r) -> t   -> m r
    -- was :: t   -> (a   -> m r) -> m r

    furtherF  :: (m a -> m a) -> t   -> m t
    -- was :: t   -> (m a -> m a) -> m t

  ```
  • (link) (link) (link) (link) (link) Nix.Value.Monad: class MonadValue v m: instances became specialized, Kleisli versions unflipped the arguments of methods into a classical order and moved to the class MonadValueF. As a result, demand now gets optimized by GHC and also tail recurse. Please, use f =<< demand t, or just use demandF, while demandF in fact just kleisli =<< demand t.

    ```haskell
    class MonadValue v m where

    demand :: v       ->         m v
    -- old became `demandF`
    
    inform :: v -> m v
    -- old became `informF`
  class MonadValueF v m where

    demandF :: (v -> m r) -> v -> m r
    -- was :: v -> (v -> m r) -> m r

    informF :: (m v -> m v) -> v -> m v
    -- was :: v -> (m v -> m v) -> m v
  ```
  • (link) Nix.Normal: normalizeValue removed first functional argument that was passing the function that did the thunk forcing. Now function provides the thunk forcing. Now to normalize simply use normalizeValue v. Old implementation now is normalizeValueF.

  • (link) Nix.Value.Equal: valueEqM: freed from RankNTypes: forall t f m ..

  • (link) Nix.Strings: all hacky* functions replaced with lawful implemetations, because of that all functions become lawful - dropped the principled suffix from functions:

    • Nix.String:

        hackyGetStringNoContext          ->
             getStringNoContext
        hackyStringIgnoreContext         ->
             stringIgnoreContext
        hackyMakeNixStringWithoutContext ->
             makeNixStringWithoutContext
      
        principledMempty        -> mempty
        principledStringMempty  -> mempty
        principledStringMConcat -> mconcat
        principledStringMappend -> mappend
      
        principledGetContext                        ->
                  getContext
        principledMakeNixString                     ->
                  makeNixString
        principledIntercalateNixStrin               ->
                  intercalateNixString
        principledGetStringNoContext                ->
                  getStringNoContext
        principledStringIgnoreContext               ->
                  stringIgnoreContext
        principledMakeNixStringWithoutContext       ->
                  makeNixStringWithoutContext
        principledMakeNixStringWithSingletonContext ->
                  makeNixStringWithSingletonContext
        principledModifyNixContents                 ->
                  modifyNixContents
  • (link):

    • Data type: MonadFix1T t m: Nix.Standard -> Nix.Utils.Fix1
    • Children found their parents:

        Binary   NAtom :: Nix.Expr.Types -> Nix.Atoms
        FromJSON NAtom :: Nix.Expr.Types -> Nix.Atoms
        ToJSON   NAtom :: Nix.Expr.Types -> Nix.Atoms
      
        Eq1 (NValueF p m)     :: Nix.Value.Equal -> Nix.Value
      
        Eq1 (NValue' t f m a) :: Nix.Value.Equal -> Nix.Value 
      
        HasCitations m v (NValue' t f m a) :: Nix.Pretty -> Nix.Cited
        HasCitations m v (NValue  t f m)   :: Nix.Pretty -> Nix.Cited
      
        when
          (package hashable >= 1.3.1) -- gained instance
          $ Hashable1 NonEmpty:: Nix.Expr.Types -> Void -- please use upstreamed instance
      
        -- | Was upstreamed, released in `ref-tf >= 0.5`.
        MonadAtomicRef   (Fix1T t m) :: Nix.Standard -> Void
      
        MonadRef         (Fix1T t m) :: Nix.Standard -> Nix.Utils.Fix1
        MonadEnv         (Fix1T t m) :: Nix.Standard -> Nix.Effects
        MonadExec        (Fix1T t m) :: Nix.Standard -> Nix.Effects
        MonadHttp        (Fix1T t m) :: Nix.Standard -> Nix.Effects
        MonadInstantiate (Fix1T t m) :: Nix.Standard -> Nix.Effects
        MonadIntrospect  (Fix1T t m) :: Nix.Standard -> Nix.Effects
        MonadPaths       (Fix1T t m) :: Nix.Standard -> Nix.Effects
        MonadPutStr      (Fix1T t m) :: Nix.Standard -> Nix.Effects
        MonadStore       (Fix1T t m) :: Nix.Standard -> Nix.Effects
        MonadFile        (Fix1T t m) :: Nix.Standard -> Nix.Render
      
        MonadEnv         (Fix1 t)    :: Nix.Standard -> Nix.Effects
        MonadExec        (Fix1 t)    :: Nix.Standard -> Nix.Effects
        MonadHttp        (Fix1 t)    :: Nix.Standard -> Nix.Effects
        MonadInstantiate (Fix1 t)    :: Nix.Standard -> Nix.Effects
        MonadIntrospect  (Fix1 t)    :: Nix.Standard -> Nix.Effects
        MonadPaths       (Fix1 t)    :: Nix.Standard -> Nix.Effects
        MonadPutStr      (Fix1 t)    :: Nix.Standard -> Nix.Effects
  • (link) Nix.Value: nvSet{,',P}: got unflipped, now accept source position argument before the value.

  • (link) Nix.Pretty: mkNixDoc: got unflipped.

  • (link) Nix.Value: Data constructor for NValue' t f m a changed (NValue -> NValue').

  • (link) Nix.Parser: Parser: Data type was equivalent to Either, so became a type synonim for Either.

  • (link) Nix.Thunk.Basic: instance MonadThunk (NThunkF m v) m v: queryM: implementation no longer blocks the thunk resource it only reads from.

  • (link): Migrated (String -> Text):

    • Nix.Value: {NValueF, nvBuiltin{,'}, builtin{,2,3}, describeValue}
    • Nix.Eval: MonadNixEval
    • Nix.Render.Frame: render{Expr,Value}
    • Nix.Type: TVar
    • Nix.Thunk: ThunkLoop
    • Nix.Exec: {nvBuiltinP, nixInstantiateExpr, exec}
    • Nix.Effects:
      • class:
        • MonadExec: exec'
        • MonadEnv: getEnvVar
        • MonadInstantiate: instatiateExpr
      • parseStoreResult
    • Nix.Effects.Derivation: renderSymbolic
    • Nix.Lint: {NTypeF, symerr}
  • Additional:
    • haskell-nix/hnix@7e6cd97 cabal.project: freed from the cryptohash-sha512 override, Hackage trustees made a revision.
    • (link) To be more approachable for user understanding, the thunk representation in outputs changed from "<CYCLE>" -> "<expr>".
    • (link) The Nix evaluation cycle representation changed "<CYCLE>" -> "<cycle>".
    • haskell-nix/hnix@51a3ff9 Nix.Expr.Types: added hacky implementation of liftTyped for instance Lift (Fix NExprF).
    • haskell-nix/hnix@51a3ff9 Nix.Builtins: derivation primOp internal code always fully evaluated, so GHC now always ships only fully compiled version in the bytecode.
    • haskell-nix/hnix#890: Project switched to prelude relude.
    • A bunch of other stuff that is not user-facing.
sorki
@sorki:vesp.cz
[m]
🚀
Anton Latukha
@Anton-Latukha
:arrow_up: Just released, but well, the date mentions yesterday.

@sorki:vesp.cz Yes. Quite a chonky chonk.
So if someone has difficulty updating - awailable to help.

And would help with updates who promised to.

sorki
@sorki:vesp.cz
[m]
well done @Anton-Latukha 👍️
Anton Latukha
@Anton-Latukha

Well, because of a lot of self-merging expect there are bugs in it, so debugging is ahead.

Stronger test suite is needed, but we are definitely moving.

sorki
@sorki:vesp.cz
[m]
cool, I agree with more tests. what are you going to focus on now?
1 reply
I would like to finish the repl PR, still a bit lost in emacs world but at least I've settled on doom-emacs now but not all projects are converted so they work out of box (with LSP and nix)
Anton Latukha
@Anton-Latukha

+====

New HNix revision is provided with bytestring 0.11.

Anton Latukha
@Anton-Latukha

+====

Just making an in advance announcement.

It already was stated in ChangeLog for previous release 0.12.

& 0.13 had a very big API change - I awaited some report/chat activity in that regard. The release showed that big detailly documented API breakage was not used. I vent downstream and looked at what reverse depenencies use & was surprised that they use only a small section of the API that is currently opened. Which is by empirical evidence only proved the point, that library can/should close big chunks of API, so API would stop be cluttered & undocumented, to provide clear API & documentation for main&in use things first, while that would allow to do work/transformations/development/refactoring to the internal code (preparing project for split-package), due to regulated API the GHC would also be able to optimize the code & expose more concise interfaces. Then to assume everyone needs everything which limits the project development itself. I'd rather have the ability and knowledge of what is needed & what people use & to collaborate and to provide API that people actually need.

If anyone wanted to make any drastic changes - it is a good time.

Tony O
@bqv
hot stuff
gitter over irc
:D
sorki
@sorki:vesp.cz
[m]
here it's selfhosted matrix homeserver to gitter 😂
Tony O
@bqv
Ha
Anton Latukha
@Anton-Latukha
+====

(diff) 0.13.1 (2021-05-22)

  • (link) Nix.Parser: annotateLocation: Fix source location preservation.
  • (link) Require Cabal dependency relude >= 1.0: since imports & Cabal file got cleaned-up & that clean-up depends on relude reimports introduced in aforementioned version.
  • Refactors, reorganization in some modules, docs, clean-ups.
Anton Latukha
@Anton-Latukha

+====

Gathering to discuss the main question & to publish a more deep introduction (aka propaedeutics) into the project

Preamble

Working on the project, decided that did enough learning of it through the code reading/refactoring. I did enough of it to get an understanding of a whole project and approached some additional deeper understanding boundary, there are information & scope of questions that only John can cover.

There are hard programming themes in the project, that have a deep vast impact on the project, in particular: laziness, recursive schemes, abstracting definitional interpreters - these are the main & advanced programming paradigms the project is built on and they form the design, code, compilation & execution of the project

Questions

Questions that see particularly beneficial to ask:

  • History: How long you did what and volume of effort HNix took. How long it took to study/implement topics/parts & properties of the system.
  • Shortcomings in/of the design he knows of or think of as important to solve upfront.
  • Thoughts on the idea to migrate HNix to recursion-schemes library.
    • Thoughts on the current code (considering there were no changes to the recursion schemes part of it).
    • The desired scope of recursion schemes application in the code.
    • Maybe look particularly into Utils.Fix1T & Value and give thoughts on how to put them more classically-defaulty/migrate them to recursion-schemes.
  • Abstracting Definitional Interpreters brief.
    • Consequences of the use of the ADI on HNix.
      • Assumption of the scope it currently works at (considering there were no changes to it):
  • What companies you know that use Nix and do they use Haskell. (It is of personal interest, since looking for a place to work at, but also it is useful to know to the project contributors & to maintainers).
  • From what he sees, any improvement thoughts to translate.

Time windows:

John Wiegley response:
"
I am free the following slots:

Schedule in UTC:

Mon 20:00-23:00 UTC
Tue 20:00-23:00 UTC
Wed 03:00-05:00 UTC
Thu 20:00-23:00 UTC
Fri 03:00-05:00 UTC && 17:00-20:00 UTC

Schedule in PST:

Mon 13:00-16:00 PST
Tue 13:00-16:00 PST && 20:00-22:00 PST
Wed
Thu 13:00-16:00 PST && 20:00-22:00 PST
Fri 10:00-13:00 PST

If none of these times work for anyone else, I can juggle some things around to find more available for you.
"

I marked dates that suit me in the voting.

Picking a time window

Please vote:

Multichoice is possible. Check the dates, check the timezone.

1-click - "I vote here."
2-clicks - "If would be needed."
3-clicks - "Can not."

https://doodle.com/poll/b6k8cwd738eikb9c?utm_source=poll&utm_medium=link

The central thread would be: https://github.com/haskell-nix/hnix/discussions/944, info on when & how we would gather, please look there.
Anton Latukha
@Anton-Latukha

+====

HNix-Store-{Core,Remote} fix releases were made.
Several dependency situations were solved.

0.4.3.0 2021-05-30

sorki
@sorki:vesp.cz
[m]
@Anton-Latukha: Noticed a CI caching issue .. https://github.com/expipiplus1/update-nix-fetchgit/runs/2648533240?check_suite_focus=true#step:27:4 looks like it's an oversight and it should probably use actions/cache@v2 like hnix
Anton Latukha
@Anton-Latukha

+====

Thank you for a live discussion.

At least I would remember it fondly.

Anton Latukha
@Anton-Latukha

+====

HNix-Store-Core major release: 0.5.0.0 2021-06-10

Highlights: migration to cryptonite library to get the support of GHC 9.0.

  • Breaking:

    • System.Nix.Hash:
      • (link) Migration from packages cryptohash- -> cryptonite:
        • rm newtype Digest in favour of cryptonite: newtype Digest
        • rm data HashAlgorithm in favour of cryptonite: class HashAlgorithm
        • rm class ValidAlgo in favour of cryptonite: class HashAlgorithm.
        • class NamedAlgo removed hashSize in favour of cryptonite: class HashAlgorithm: hashDigestSize. Former became a subclass of the latter.
        • rm hash in favour of cryptonite: hash
        • rm hashLazy in favour of cryptonite: hashlazy
      • (link) Base encoding/decoding function for hashes (digests) changed (due to changes in type system & separation of specially truncated Nix Store hasing):
        • encode(InBase -> DigestWith)
        • decode(Base -> DigestWith)
    • (link) System.Nix.StorePath:
      • rm type StorePathHashAlgo = 'Truncated 20 'SHA256 in favour of StorePathHashPart & mkStorePathHashPart.
      • rm unStorePathName, please use GHC: coerce for StorePathName <-> Text, StorePathName data constructor is provided.
    • Internal modules now have export lists, if something, please contact.
  • Additional:

    • (link) Support of GHC 9.0.

    • (link) System.Nix.StorePath:

      • exposed StorePathName data constructor to API.
      • added newtype StorePathHashPart = StorePathHashPart ByteString.
        • added builder mkStorePathHashPart :: ByteString -> StorePathHashPart
    • (link) System.Nix.Hash:
      • Nix store (which are specially truncated) hashes are now handled separately from other hashes:
        • add mkStorePathHash - a function to create a content into Nix storepath-style hash:
          mkStorePathHash :: HashAlgorithm a => ByteString -> ByteString
          but recommend to at once use mkStorePathHashPart.

HNix-Store-Remote major release: 0.5.0.0 2021-06-11

Breakages in hash type system API due to migration of Core to cryptonite library.
  • Breaking:
    • haskell-nix/hnix-store@3b948d1 System.Nix.Store.Remote:
      • addToStore: constraint of ValidAlgo a removed in favour of constraint on cryptonite: HashAlgorithm a through constraint NamedAlgo a.
      • queryPathFromHashPart: 1st arg changed from Digest StorePathHashAlgo to StorePathHashPart, for details: hnix-store-core 0.5.0.0 ChangeLog.

+====

Kinda forgot to post the release notes here. Here we go.

Anton Latukha
@Anton-Latukha

+====

I have important family stuff. For a week I going to be away.
Anyway, I need a good digital detox.

  1. I would be glad, is someone gets to provide GHC 9.0 support & makes a release.

  2. We checked what dependent projects are used & determined that approached a point where ChangeLog is not that important & holding it is not manageable, it is not even important to note GHC 9.0 support, just tell DHall folks GHC 9.0 is ready, they wait for it.

DHall uses only the API of the Shorthands module.

  1. After release, please, contribute the HNIx{,Remote-{Core,Remote}}-related patches into Nixpkgs: haskell-updates branch. Guys froze the {Core,Remote} versions until HNix releases support for them.
sorki
@sorki:vesp.cz
[m]
Enjoy your time off! 🙂