Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • 04:03
    github-actions[bot] labeled #685
  • 04:03
    github-actions[bot] commented #685
  • 04:03
    github-actions[bot] labeled #670
  • 04:03
    github-actions[bot] commented #670
  • 04:03
    github-actions[bot] labeled #669
  • 04:03
    github-actions[bot] commented #669
  • 04:03
    github-actions[bot] labeled #659
  • 04:03
    github-actions[bot] commented #659
  • 04:03
    github-actions[bot] labeled #658
  • 04:03
    github-actions[bot] commented #658
  • 04:03
    github-actions[bot] labeled #657
  • 04:03
    github-actions[bot] commented #657
  • 04:03
    github-actions[bot] labeled #656
  • 04:03
    github-actions[bot] commented #656
  • 04:03
    github-actions[bot] labeled #651
  • 04:03
    github-actions[bot] commented #651
  • 04:03
    github-actions[bot] labeled #649
  • 04:03
    github-actions[bot] commented #649
  • 04:03
    github-actions[bot] labeled #646
  • 04:03
    github-actions[bot] commented #646
Ray Donnelly
@mingwandroid
It might be an idea to split the package into package and package-tests too. Then make the test for package depend on package-tests. That way people who do not care about the tests do not suffer the costs. This would also be a good way to handle tests that are larger (and maybe take longer) .. I would love to find the time to implement multi-level testing too or perhaps giving tests some metadata against which you (or tooling) can filter them (needs_network, needs_super_user, slow_test, heavy_test, etc).
Isuru Fernando
@isuruf
Why does conda-build print warning messages twice? For eg:
No numpy version specified in conda_build_config.yaml.  Falling back to default numpy value of 1.11
WARNING:conda_build.metadata:No numpy version specified in conda_build_config.yaml.  Falling back to default numpy value of 1.11
sometimes 3 times
Multiple meta.yaml files found. The meta.yaml file in the base directory will be used.
WARNING:conda_build.utils:Multiple meta.yaml files found. The meta.yaml file in the base directory will be used.
WARNING conda_build.utils:find_recipe(1219): Multiple meta.yaml files found. The meta.yaml file in the base directory will be used.
Ray Donnelly
@mingwandroid
It'd be nice to fix I agree. No idea, the stdout/err handling is a bit complicated (mostly due to python 2 and unicode concerns and capturing and filtering..)
Ray Donnelly
@mingwandroid
I think I would like to remove this warning from conda-build
"UserWarning: The environment variable 'CONDA_FORGE' is being passed through with value 'yes'. If you are splitting build and test phases with --no-test, please ensure that this value is also set similarly at test time."
reasons: 1. these vars are passed specifically to modify build behavior (as I use them at least!), and have no bearing on tests ..
  1. If we do want this warning, well, conda-build knows if we're using --no-test or not, and we could at least keep quiet when that isn't in play.
C:\opt\conda-win-32\lib\site-packages\conda_build\environ.py:447: UserWarning: The environment variable 'PY_INTERP_LINKAGE_NATURE' is being passed through with value ''.  If you are splitting build and test phases with --no-test, please ensure that this value is also set similarly at test time.
  warnings.warn(
C:\opt\conda-win-32\lib\site-packages\conda_build\environ.py:447: UserWarning: The environment variable 'PY_INTERP_DEBUG' is being passed through with value 'no'.  If you are splitting build and test phases with --no-test, please ensure that this value is also set similarly at test time.
  warnings.warn(
C:\opt\conda-win-32\lib\site-packages\conda_build\environ.py:447: UserWarning: The environment variable 'OPENSSL_DIR' is being passed through with value '%PREFIX%\Library'.  If you are splitting build and test phases with --no-test, please ensure that this value is also set similarly at test time.
  warnings.warn(
C:\opt\conda-win-32\lib\site-packages\conda_build\environ.py:447: UserWarning: The environment variable 'SQLITE3_DIR' is being passed through with value '%PREFIX%\Library'.  If you are splitting build and test phases with --no-test, please ensure that this value is also set similarly at test time.
  warnings.warn(
C:\opt\conda-win-32\lib\site-packages\conda_build\environ.py:447: UserWarning: The environment variable 'CONDA_FORGE' is being passed through with value 'yes'.  If you are splitting build and test phases with --no-test, please ensure that this value is also set similarly at test time.
  warnings.warn(
INFO:conda_build.metadata:Attempting to finalize metadata for python
two ones, no two, oh well.
Isuru Fernando
@isuruf
since we started adding rpath on macos, all shared libraries and executables needs relocation whereas on Linux due to the use of $ORIGIN, it's not needed. is there something we can do on macos?
Cheng H. Lee
@chenghlee
IIRC, macOS has @loader_path and @executable_path which are roughly analogous to $ORIGIN, but it's been a while since I've looked at the developer docs.
Ray Donnelly
@mingwandroid
Indeed, we don't rely on any runtime relocation of macOS binaries AFAIK
We patch stuff to be exe-relative
Isuru Fernando
@isuruf
Alastair Rankine
@randomphrase

Hi all, hoping someone can help me, struggling a bit with conda-build. I am trying to package a closed-source c++ library which is distirbuted as tarball. I want to accomplish a (seemingly) simple task: produce a conda package from a the headers and static libraries from this tarball. I have the following meta.yaml file:

package:
  name: my-package
  version: 1.0
source:
  url: http://path/to/tar.gz
  sha256: here
build:
  number: 0
outputs:
  - name: my-package
    script: dev-files.py

The dev-files.py file script selects the header files from the current (ie work) directory and copies them into the corresponding $PREFIX directory (I assume this is correct - it's not documented). The copy task seems to work but then the packaging job fails with this error:

Failed to rename host env directory despite sleeping and retrying. This is some Windows file locking mis-bahaviour.

Which is interesting because I'm not on Windows. Any help appreciated.

16 replies
Alastair Rankine
@randomphrase
With reference to the above issue I now believe this is an entirely spurious error message. From reading the code in question, it will always produce this error message even when the underlying move operation is entirely successful. I'll raise an issue/PR when I get some time. Thanks @nehaljwani for your help.
Ray Donnelly
@mingwandroid
A new conda-build release (3.20.4) is coming up, so if anyone wants me to review PRs for inclusion (or have bugs they feel really should be fixed) then please holler!
Isuru Fernando
@isuruf
@mingwandroid, conda/conda-build#4080
Alastair Rankine
@randomphrase
conda/conda-build#4087 but CLA not signed yet
Ray Donnelly
@mingwandroid
@randomphrase I have a fix for that too. @Isuruf, should be fine, thanks.
Matthew R. Becker
@beckermr
do we want this one @isuruf @mingwandroid @jjhelmus : conda/conda-build#4052
Ray Donnelly
@mingwandroid
Probably? I don't know a load about the current_metadata.yaml stuff though
Krishna Bhogaonker
@00krishna

Hey folks. I am getting a strange build error when using conda build. It has not happened before and googling did not yield any results. I built my package, but when uploading to anaconda.org I get this error.

Uploading nyuki-0.0.3-py_0.tar.bz2 to anaconda.org
Traceback (most recent call last):
  File "/media/krishnab/lakshmi/anaconda3/bin/conda-build", line 11, in <module>
    sys.exit(main())
  File "/media/krishnab/lakshmi/anaconda3/lib/python3.7/site-packages/conda_build/cli/main_build.py", line 474, in main
    execute(sys.argv[1:])
  File "/media/krishnab/lakshmi/anaconda3/lib/python3.7/site-packages/conda_build/cli/main_build.py", line 465, in execute
    verify=args.verify, variants=args.variants)
  File "/media/krishnab/lakshmi/anaconda3/lib/python3.7/site-packages/conda_build/api.py", line 210, in build
    notest=notest, variants=variants)
  File "/media/krishnab/lakshmi/anaconda3/lib/python3.7/site-packages/conda_build/build.py", line 3155, in build_tree
    handle_anaconda_upload(tarballs, config=config)
  File "/media/krishnab/lakshmi/anaconda3/lib/python3.7/site-packages/conda_build/build.py", line 3236, in handle_anaconda_upload
    subprocess.call(cmd + [package])
  File "/media/krishnab/lakshmi/anaconda3/lib/python3.7/subprocess.py", line 339, in call
    with Popen(*popenargs, **kwargs) as p:
  File "/media/krishnab/lakshmi/anaconda3/lib/python3.7/subprocess.py", line 800, in __init__
    restore_signals, start_new_session)
  File "/media/krishnab/lakshmi/anaconda3/lib/python3.7/subprocess.py", line 1465, in _execute_child
    executable = os.fsencode(executable)
  File "/media/krishnab/lakshmi/anaconda3/lib/python3.7/os.py", line 812, in fsencode
    filename = fspath(filename)  # Does type-checking of `filename`.
TypeError: expected str, bytes or os.PathLike object, not list

So it is execting a string or os.PathLike object, and not a list. I am not sure where this error could come coming from. Does it seem familiar to anyone. The only other thing I found in the entire build stdout is a reference to that ERROR:conda_build.utils:Failed to rename work directory despite sleeping and retrying. This is some Windows file locking mis-bahaviour. errro that someone mentioned yesterday or a few days ago.

I can share the meta.yaml file if it would be useful. Seems like the build process completed, but this message arises when uploading to anaconda.org.
Lucas Hosseini
@beauby
Any general advice for distributing nightly packages through conda? I'm leaning towards having a separate package named foo-nightly with versions 1.0.0-nightly-AAAAMMDD, but maybe there is a better solution? Ideally, I'd like the -nightly package to be incompatible with the real releases, to avoid conflicts
An other option being having a separate channel
Ray Donnelly
@mingwandroid
You can use 'labels' on anaconda.org, it's just like nested channels really
But you probably do want different filenames. You can see how we address that in the AD python-recipe here: https://github.com/AnacondaRecipes/python-feedstock/blob/master/recipe/meta.yaml
follow PY_INTERP_DEBUG
Lucas Hosseini
@beauby
@mingwandroid Thanks, I'll look into that! Would I be correct to assume that conda install foo -c mychannel would install only packages tagged with "main", and conda install foo -c mychannel/labels/nightly would install only packages tagged with "nightly"?
Ray Donnelly
@mingwandroid
Yeah, I think so. I don't use labels myself though, probably I should
Lucas Hosseini
@beauby
Pretty cool, if this works the way I think, it's exactly what I'm looking for. Thanks for pointing it out
Ray Donnelly
@mingwandroid
No problem!
Lucas Hosseini
@beauby
Yeah I'm not 100% sure what the role of the build string is – it has to be unique per version, but does it serve a technical purpose?
Cheng H. Lee
@chenghlee
It serves to distinguish multiple builds of the same version of a package. These may be due to changes in recipes to fix build problems, to variants of the same base recipes (e.g., choosing between py36, py37, or py38 builds of numpy), or to whatever other purpose the package maintainer chooses.
Lucas Hosseini
@beauby
@chenghlee Right, but the build string itself is not used to decide which package is "more current", right?
Ray Donnelly
@mingwandroid
There's a longstanding bug in conda here that build/string also helps to avoid. The package cache can have conflicts between e.g. win-32 and win-64 (though we've mitigated that some in recent conda-builds).
Cheng H. Lee
@chenghlee
All other things being equal, the solver considers build 1 more current than build 0.
Lucas Hosseini
@beauby
@chenghlee Right, but by "build 1" you mean the build number is "1", right?
Ray Donnelly
@mingwandroid
If the win-32 package has the same file name as the win-64 package (for example, but it can also happen outside of different arches) then conda will happily unpack one on top of the other and that breaks things sometimes.#
Lucas Hosseini
@beauby
Btw @chenghlee Do you have a link to the bug tracker issue you mentioned the other day (regarding the bug in conda cache)?
2 replies
@mingwandroid I see, so keeping it unique probably still matters quite a bit, but it does not have to be lexicographically increasing between versions/builds
Ray Donnelly
@mingwandroid
If I were doing e.g. a release and debug python package I'd keep the build numbers the same when built from the same source and add -dbg or something like that to the build string to ensure no package cache conflicts, then labels also to facilitate selection.
the package is still then called "python" for both so the solver will be ok.
Lucas Hosseini
@beauby
Right, in my scheme I do add nightly as well to the nightly builds
Cheng H. Lee
@chenghlee
Looking through the solver code, the build string is used as the tie-breaker of last resort; i.e., if two packages somehow end up with the same channel priority, version, build number, sub-directory (noarch vs platform-specific), and build timestamps.
Lucas Hosseini
@beauby
I see, but then it's quite surprising to use a hash in the build string by default