dart-lang/build

Discussion about all things package:build[_*] – including source_gen

Simon Binder

@simolus3

The easiest way to not run other builders is to not depend on them :D But I assume you probably depend on them for a reason? Inside a build.yaml file, you can configure builders to only run on some sources with generate_for. Perhaps that helps to reduce build times already?

Rich Coutts

@buttonsrtoys

@simolus3 , thanks for the tip. I checked out our build.yaml and you're right--someone since added several dependencies for packages we've installed. We need those other packages but their independent of the one I wrote.

When I run pub run builder_runner build lots of code generation happens, including for json_serializableable and some GraphQL packages. But when we generate files for the gen_lang package we run pub run gen_lang:generate and only the gen_lange files are generated and it takes a second or two. That's what I'm after. To run code generation for my package and not run all the other packages.

Below is our build.yaml. My package is specified at the bottom and called gen_keys

targets:
  $default:
    builders:
      gql_build|schema_builder:
        enabled: true
      gql_build|ast_builder:
        enabled: true
      gql_build|data_builder:
        enabled: true
        options:
          schema: ewp|lib/common/graphql/schema.graphql
      gql_build|var_builder:
        enabled: true
        options:
          schema: ewp|lib/common/graphql/schema.graphql
      gql_build|serializer_builder:
        enabled: true
        options:
          schema: ewp|lib/common/graphql/schema.graphql

      ferry_generator|req_builder:
        enabled: true
        options:
          schema: ewp|lib/common/graphql/schema.graphql

      ewp|keys_file_builder:
        enabled: true

builders:
  ewp|keys_file_builder:
    import: 'package:ewp/common/gen_key/builders/keys_file_builder.dart'
    builder_factories: ['keysFileBuilder']
    build_extensions: { ".dart": [".keys.dart"] }
    auto_apply: dependents
    build_to: source

arcticfox

@arcticfox1919

@simolus3 My configuration is auto_apply: root_package, I'll try all_packages later, thanks for your reply!

wujek-srujek

@wujek-srujek

Hi there. I'm looking for information and examples how targets other than $default work within the build ecosystem. I would like to have a target with a builder with custom sources.

wujek-srujek

@wujek-srujek

More specifically - I'm writing a builder that will need to process files in a custom binary format, generate assets from them (and later a source file providing easy access to these assets, kind of like the R class on Android). The input files are in a custom directory in the root project directory and they are not picked up by the build by default because it is restricted to some defaults (https://github.com/dart-lang/build/blob/master/build_runner_core/lib/src/generate/options.dart#L37-L53)

So I want to create a target with my builder which would get this directory in its sources - but if I do it in $default, it seems I need to add all these default folders as well so that my other builders get the inputs they expect (so at the very least I need to repeat assets/, lib/, $package$ is enforced according to output from the build:
The missing sources are:
- $package$
- lib/$lib$
). Is there any other way? I thought of creating a custom target for this one builder, but there are issues - I don't know how to execute this target, and doctor says that outputs overlap because my custom builder is in my custom target and in the $default target as well - as the package uses auto_apply: dependents.

wujek-srujek

@wujek-srujek

ok I think I got it:

targets:
  $default:
    builders:
      my_builder:builder:
        enabled: false # disable the builder to prevent overlapping outputs error
  my_package:other_target:
    sources:
      - data/**

It seems as though all targets are always processed so I don't need to specify anywhere which to execute?

Jacob MacDonald

@jakemac53

Correct, you can't just build a single target. They are really just bundles of sources and code generators to run on them. Maybe it would be interesting to be able to do that, but targets in general are quite difficult and confusing to use properly as you have discovered... so we generally don't really encourage their use unless you have to use them. But what you have done here is one reasonable use.

In particular they get difficult to use if you have any aggregate builders (builders using $package$ etc). Since those can only exist in one target generally.

Kirill

@rotorgames

Hi there. Looks like build_runner watch doesn't work for assets folder.

Does someone know how to enable assets folder watching?

Thanks

Jacob MacDonald

@jakemac53

https://github.com/dart-lang/build/blob/master/docs/faq.md#how-can-i-include-additional-sources-in-my-build

Kirill

@rotorgames

@jakemac53 Thank you

Kirill

@rotorgames

@jakemac53 I tried to include additional sources but it doesn't work properly. So, after any changes in assets folder, all Builders are creating again, so if we use freezed or json_serializable, it take a lot of time. Also, if I renamed one file several times, it stoped working after 2-3 renames. So, I don't think that it's a good idea.

I found a way by using DirectoryWatcher, it looks good, but I don't know it is a good practice or not.

@jakemac53 how do u think, it it good idea to use DirectoryWatcher in the builder? I keep build method empty because all I need it's starting my code by build_runner.

Simon Binder

@simolus3

You definitely shouldn't use a DirectoryWatcher or any direct file system APIs in a builder. If all builders are invalidated due to a change in assets, they either read from there in some indirect way or the assets aren't being cached properly for some reason. Can you post your build.yaml?

Kirill

@rotorgames

@simolus3 Thank you for your answer. I found a reason why build method stopped invoking after several renaming. It was because of graph cache. Now, I have another problem. I can’t detect any file removing. build method is not invoking when I remove a file

Simon Binder

@simolus3

@rotorgames Builders only operate on inputs that exist, so they can't really get notified for deleted files. The intended behavior for the build system is to delete assets that depend on the input as well. So if you had a .dart -> .g.dart builder and you delete a foo.dart file, the build system should automatically delete the foo.g.dart file as well.

Kirill

@rotorgames

@simolus3 That’s a reason why I wanted to use DirectoryWatcher because build_runner has a lot of limitations and filters. What if I would like to generate an one file for a folders tree? It looks like impossible for build_runner

Simon Binder

@simolus3

It's possible with an aggregate builder: https://github.com/dart-lang/build/blob/master/docs/writing_an_aggregate_builder.md

In general, the build systems limitation on what files you can output is also the reason it works so well. Since it's able to reason about which builder might write which files, build ordering and incremental rebuilds can be much more efficient. If you used a DirectoryWatcher manually, you'd also have to take care of how file changes might propagate through intermediate steps and so on. If you write a builder you can just read the assets that you depend on, write outputs only depending on them and the build system will take care of everything else.

Kirill

@rotorgames

@simolus3 aggregate_builder supports only
lib/$lib$
$package$

I need an asset folder :(

Simon Binder

@simolus3

You can make a builder that has $package$ as a declared input and then does buildStep.findAssets(Glob('assets/**')) to consider all files in assets/ @rotorgames

Kirill

@rotorgames

@simolus3 and I’ll get notifications about file removing?

Simon Binder

@simolus3

@rotorgames Well, your builder would run again and if you then call findAssets again the deleted file would no longer be included.

Kirill

@rotorgames

@simolus3 amazing, that’s exactly I need

Simon Binder

@simolus3

Nice! :+1:

Kirill

@rotorgames

I’ll try tomorrow

Kirill

@rotorgames

@simolus3 I tested it. Looks like it works fine. Thank you!

Jacob MacDonald

@jakemac53

Yeah, the name of the game here is to never use dart:io at all basically. The build system won't know about anything you do using those apis, and won't set up the proper dependencies.

It will track all file events as long as you go through the BuildStep apis for all file interactions

Kirill

@rotorgames

@simolus3 @jakemac53 So, build_runner works unclear to me.

If i run build_runner build at the first time, build method in my builder is invoking.
But if I change pubspec.yaml file and run build_runner build again, build method in my builder is NOT invoking.

Also, if I call findAssets(‘assets/**') inside build method, then build method is invoking after changing something inside assets/** but if I don’t call findAssets then build method will not be invoked after changes in assets/**. It’s really unclear :(

So, $package$ doesn’t work properly as well. I can’t detect changes in pubspec.yaml and etc. :(

Kirill

@rotorgames

@jakemac53 @simolus3 So, I found a solution. To watch pubspec.yaml changes, we need to do this:

final pubspecAsset = await buildStep.findAssets(Glob(‘pubspec.yaml')).single;

await buildStep.digest(pubspecAsset);

digest method lets build_runner know that we would like to subscribe the file changes. Also, we can compare digest result with the previous digest to know if the file was changed.

My result: FlutterGen/flutter_gen#221

Simon Binder

@simolus3

@rotorgames The idea is that builders are only invoked again if there's a chance for their output to change. So if an asset you didn't read changes, that can't change the behavior of your builder (ideally) and hence the build system is smart enough to not run it again. This is a essentially a performance optimization.

By the way, you can also use buildStep.canRead(AssetId(buildStep.inputId.package, 'pubspec.yaml')) if you want your builder to be re-invoked after the pubspec.yaml changes.

Jacob MacDonald

@jakemac53

Yeah, basically what Simon said.

You MUST go through the build step apis for all file system access. If your builder never globs the assets directory, then your builder can't depend on its contents, unless you were going outside the build system and directly listing it yourself, or invoking an external process which was itself listing the file system.

If invoking an external process, I would encourage the use of the scratch_space package if possible for your use case. That makes it a lot harder to do the wrong thing.

Note that when you use findAssets, that will only re-run your builder if the results of the Glob changes. It will not re-run your builder if the contents of one of those files changes (because you haven't read the file).

You can use the canRead api as Simon suggests to depend on the contents of any given asset, or you can similarly read the digest or actual file contents. In theory some day we could optimize canRead to only re-run your builder if the visibility of the file changed, but it is unlikely (lots of builders depend on using it as a way of getting a dependency on the file contents).

Kirill

@rotorgames

@jakemac53 @simolus3 Thank you guys for the explanations. Anyway, I still think that using findAssets, canRead and digest unclear for new developers (as me). I found that digest method affects on build listeners only when I passed through the source code of the build_runner

Jacob MacDonald

@jakemac53

The golden rule is just, don't import dart:io. If you do, you are doing something that won't work as you expect, at least not without some additional work to inform the build system of what you did.

(and that rule applies to your dependencies too, other than build_runner itself)

wujek-srujek

@wujek-srujek

I have a question about dynamically defining build outputs (i.e. the are defined as part of a builder configuration in a target). This comment https://github.com/dart-lang/build/issues/3295#issuecomment-1116490222 confirms that doing this can break builder ordering etc. On the other hand, source_gen allows users to override build_extensions in builder configurations in a target (https://github.com/dart-lang/source_gen/tree/master/source_gen#generating-files-in-different-directories), essentially meaning the static build_extensions defined in the builder 'provider' package will be different than what the builder really uses and generates. It's unclear to me what this means - how bad is it? What will it break and influence?

Simon Binder

@simolus3

@wujek-srujek In virtually all instances, builder ordering only depends on file extensions. E.g. if you have one builder that needs .dart files as an input (typically a compiler), it should run after code-generating builders of course. But for that ordering to work, it doesn't matter whether source_gen has extensions defined as '.dart': ['.g.dart'] or '^lib/{{}}.dart': ['lib/generated/{{}}.g.dart']. In both cases, the only thing relevant for ordering is that the extension ends with .dart. You can contrive examples where this breaks, but I'll claim that as long as the extension stays the same you're in the clear.

I would avoid writing builders that dynamically change build extensions a lot (so for instance a builder having different outputs in dev or release mode) because that invalidates build phases for no reason. But defining your build extensions in build.yaml to include the basic file extensions and then letting users customize that is fine.

wujek-srujek

@wujek-srujek

Thanks for the explanation @simolus3

wujek-srujek

@wujek-srujek

Hi dart pub redirected me here for discussions regarding source_gen. I hope it I'm not violating any rules by asking if someone could help me with this: https://stackoverflow.com/questions/72261095/how-to-get-annotation-field-names-in-when-generating-code-with-source-gen

Kirill

@rotorgames

Hi there. What if in build.yaml file I would like to include my custom folder in the sources but I don’t want to exclude all default folders? If I include my own folder, all default folders as web, tools and etc are excluded automatically. I would like to prevent this behavior. Any ideas?

Jacob MacDonald

@jakemac53

There is not a way to do that directly.

You might be able to create a new target to kind of do the same thing... but you will run into other issues that will make that harder to maintain than just manually setting up all the default sources.

Kirill

@rotorgames

@jakemac53 Thank you for a quick response.

Currently, we discuss the best solution here: dart-lang/build#3311

So if someone wants to vote or suggest your own solution please, join us in the discussion of the PR.
Thanks.

Where communities thrive

People

Repo info

Activity