Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • 04:02
    github-actions[bot] labeled #3165
  • 04:02
    github-actions[bot] commented #3165
  • 04:02
    github-actions[bot] labeled #3161
  • 04:02
    github-actions[bot] commented #3161
  • 04:02
    github-actions[bot] labeled #3111
  • 04:02
    github-actions[bot] unlabeled #3111
  • 04:02
    github-actions[bot] labeled #3080
  • 04:02
    github-actions[bot] unlabeled #3080
  • 04:02
    github-actions[bot] closed #2997
  • 04:02
    github-actions[bot] labeled #2997
  • 04:02
    github-actions[bot] closed #2996
  • 04:02
    github-actions[bot] labeled #2996
  • 04:02
    github-actions[bot] closed #2989
  • 04:02
    github-actions[bot] labeled #2989
  • Dec 03 23:41
    mbargull commented #4665
  • Dec 03 23:36
    mbargull ready_for_review #4665
  • Dec 03 23:36
    mbargull edited #4665
  • Dec 03 23:33
    mbargull synchronize #4665
  • Dec 03 22:33
    mbargull synchronize #4665
  • Dec 03 22:18
    mbargull synchronize #4665
Krishna Bhogaonker
@00krishna
hello conda-build folks. I had a tricky issue that I was hoping someone might know how to resolve. So I have a noarch package that will work on python 3.6, 3.7, and 3.8. Note the package uses geospatial packges like geopandas and rasterio. So the package builds just fine and imports with no error. The problem is that is will only import properly for python=3.8 now, and if I try to import it with a python 3.6 or python 3.7 environment, then it will give me these CUDA driver errors - feature:/linux-64::__cuda==10.0=0. So is the package perhaps not really noarch, since CUDA is a compiled extension for one of the dependencies probably.
Here is a snippet from the build section of the meta.yaml
build:
  noarch: python
  number: 0
  skip: true  # [py<36 ]
Note that the conda linter will tell me that selectors are not supposed to be used with noarch python. So not sure of the right place to look for the answer in the documentation. I checked https://docs.conda.io/projects/conda-build/en/latest/resources/define-metadata.html , but it really details how to use noarch, but not what to do if noarch does not work.
Krishna Bhogaonker
@00krishna
Oops looks like I might have fallen into this error. conda/conda-build#3956 . I can see that there is a dependency for - python_abi 3.8.* *_cp38 in the built package metadata that should not be there. I read the issues, but did not see any resolution. If anyone knows how to remove this odd dependency, please LMK.
Jonathan J. Helmus
@jjhelmus
Either do not use conda-forge packages or update to a version of conda-build>=3.18.12
Krishna Bhogaonker
@00krishna
@jjhelmus Oh excellent. yeah looks like I am at 3.18.11 right now. Man, it is that easy :). Let me see if that works. I have been working on this all day. Thanks @jjhelmus
Jonathan J. Helmus
@jjhelmus
Note that defaults does not have 3.18.12 so you will need to use a conda-build from conda-forge if you build using conda-forge packages
Krishna Bhogaonker
@00krishna
Yep, thanks for the tip. I must have installed conda-build before adding conda-forge to my list of channels in condarc. So now I will rectify that.
Brian Miller
@bkmdev
hi, conda-build v3.17.8, when I attempt to do conda build --no-include-recipe meta.yaml for a python sdk app, I get ~187 of the following odd errors (warnings?). Why? Is this something to be concerned about? If so, how does one fix this? :
[...]

Packaging my_sdk
INFO:conda_build.build:Packaging my_sdk
Packaging my_sdk-w.x.y.z-py_0
INFO:conda_build.build:Packaging my_sdk-w.x.y.z-py_0
number of files: 112
/Library/Developer/CommandLineTools/usr/bin/nm: /anaconda3/conda-bld/my_sdk_1596598276840/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_plac/lib/libpython3.8.a(getbuildinfo.o) Invalid value (Producer: 'LLVM10.0.0' Reader: 'LLVM APPLE_1_900.0.38_0')

/Library/Developer/CommandLineTools/usr/bin/nm: /anaconda3/conda-bld/my_sdk_1596598276840/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_plac/lib/libpython3.8.a(acceler.o) Invalid value (Producer: 'LLVM10.0.0' Reader: 'LLVM APPLE_1_900.0.38_0')

/Library/Developer/CommandLineTools/usr/bin/nm: /anaconda3/conda-bld/my_sdk_1596598276840/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_plac/lib/libpython3.8.a(grammar1.o) Invalid value (Producer: 'LLVM10.0.0' Reader: 'LLVM APPLE_1_900.0.38_0')

[...]

/Library/Developer/CommandLineTools/usr/bin/nm: /anaconda3/conda-bld/my_sdk_1596598276840/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_plac/lib/libpython3.8.nolto.a(xxsubtype.o) Invalid value (Producer: 'LLVM10.0.0' Reader: 'LLVM APPLE_1_900.0.38_0')

/Library/Developer/CommandLineTools/usr/bin/nm: /anaconda3/conda-bld/my_sdk_1596598276840/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_plac/lib/libpython3.8.nolto.a(frozen.o) Invalid value (Producer: 'LLVM10.0.0' Reader: 'LLVM APPLE_1_900.0.38_0')
This previously never occurred during builds and I think happened only after (Home)brew decided to upgrade XCode CLI tools?
Brian Miller
@bkmdev
I'm also concerned about this strange ..._placehold_placehold... coming from $PREFIX presumably?
(FWIW, I'm on OSX 10.13.X + XCode 9.2)
Krishna Bhogaonker
@00krishna
@bkmdev The placehold stuff is standard. That happens to everyone. I don't know the exact details, but basically it is a way to ensure that when installing the package, they have all of that space reserved for the paths in the package. I think the conda developers used to have shorter path and then that led to problems with not having the space for longer paths on some OSs, but I am not the expert on that.
Also is there a reason you are using v3.17.8? I think the current version is up to v3.19.3 I believe.
Brian Miller
@bkmdev
@00krishna thanks for the reply. Re. the placehold stuff, I think you're right, I think I may have even asked this here a long time back & a dev commented, although the issue/solution still seems strange to me. Re. the version: unless there's a good reason to (ex: specific fix, necessary feature, severe security vulnerability, etc.) I don't like to upgrade, as things tend to break. Case in point, I don't think the nm issues above occurred until Homebrew forced the re-installation of XCode CLI tools :confused:
mattchan-tencent
@mattchan-tencent
Hi, I got an unexpected dependency error when specifying the python version in conda-build. The package builds properly, but in the test phase it fails to install the dependencies (pytest). It's an extremely simple package. No deps except python 3.7.* in the build/host/run, and pytest (version not pinned) in test/requires.
This is the error:
Output in format: Requested package -> Available versions

Package ca-certificates conflicts for:
python[version='>=3.7,<3.8.0a0'] -> openssl[version='>=1.1.1g,<1.1.2a'] -> ca-certificates
pytest -> python[version='>=2.7,<2.8.0a0'] -> ca-certificates

Package python conflicts for:
pytest -> attrs[version='>=17.4.0'] -> python[version='>=3.4']
eqasm-parse==0.1=openmp -> python[version='>=3.8,<3.9.0a0']
pytest -> python[version='>=2.7,<2.8.0a0|>=3.6,<3.7.0a0|>=3.7,<3.8.0a0|>=3.8,<3.9.0a0|>=3.5,<3.6.0a0']
python[version='>=3.7,<3.8.0a0']
mattchan-tencent
@mattchan-tencent
Well, this is seriously weird. I fixed it by removing the build string from meta.yaml
I thought that had no effect on the package build?
Krishna Bhogaonker
@00krishna
Has anyone figured out a workaround for this issue conda/conda#7758. The problem occurs when building a package in CI using conda-build. So the package gets build when when trying to install that package into the test environment, conda cannot find the newly built package. So the command according to the conda build docs is conda install --use-local <packagename>. That did not work, nor did the alternative conda install -c local <packagename>. I was hunting through feedstocks but did not see anyone using this code, so hopefully someone knows the answer.
I have a travis script running, as per this doc. https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/use-conda-with-travis-ci.html . Are you able to see the log if I give it to you this way. https://api.travis-ci.org/v3/job/715572047/log.txt
@isuruf indicated: @isuruf
You are using conda-build from a non-root environment and trying to install the package using conda from root environment
@isuruf so you suggest being in the base environment for the build, then activating the test environment and doing the local install?
Isuru Fernando
@isuruf
Either,
  1. Install conda-build in root env
  2. Use conda from non-root env(this is tricky to do)
  3. Try giving -c file://home/travis/miniconda/envs/test-environment/conda-bld (Not sure what the exact command is
Krishna Bhogaonker
@00krishna
Okay cool. I am trying this as you suggest in 1. install to root env.
  - bash miniconda.sh -b -p $HOME/miniconda
  - source "$HOME/miniconda/etc/profile.d/conda.sh"
  - hash -r
  - conda config --set always_yes yes --set changeps1 no
  - conda update -q conda
  - conda info -a
  - conda install conda-build
  - conda create -q -n test-environment python=$TRAVIS_PYTHON_VERSION conda pytest six pytest-cov pytest-mock 
  - conda activate test-environment
  - conda build packaging/ 
  - conda install --use-local <packagename>
Krishna Bhogaonker
@00krishna
@isuruf Excellent, this worked. Wow, thanks so much.
Is it possible for me to submit a PR for this section of the documentation?
Just to add a note that conda-build should be in the root environment.
Jonathan J. Helmus
@jjhelmus
Krishna Bhogaonker
@00krishna
@jjhelmus I can do that. Should I open an issue first and then submit the PR, or can I just submit the PR directly.
Jonathan J. Helmus
@jjhelmus
A PR directly is fine
Krishna Bhogaonker
@00krishna
:thumbsup:
Brian Miller
@bkmdev
hi, is it really possible to distribute JVM-based apps with conda? I know the docs claim "Package, dependency and environment management for any language—Python, R, Ruby, Lua, Scala, Java, ..." but conda-forge/conda-forge.github.io#590 is making wonder if there is more to it, ie: re. .jar dependencies? (I assume native jar dependencies currently aren't handled and would need to just be bundled with the JVM-based app itself?)...
Cheng H. Lee
@chenghlee
@bkmdev: it really depends on the complexity of your jar dependencies. In the simplest cases, you can dump the jars for your application into a single conda package; I have an example here: https://github.com/biobuilds/biobuilds/tree/master/picard/2.15.0
for more complex cases, you'll likely have to build conda packages for each of your dependencies.
Krishna Bhogaonker
@00krishna
Hey folks, I had a question about conda-build and requirements.txt. As I understand it, we are supposed to keep conda packages/environments separate from pip stuff, as much as possible--I read @jjhelmus blog post on this. Now one thing that was not clear in the docs was the relationship between the Requirements section of meta.yaml and the requirements.txt. I have packages listed in requirements.txt so that I can test them in Travis for regular CI -- where I can use conda install --file requirements.txt and it will find the packages in the conda channel. But for package building, if I have the same packages listed in the Host and Runtime section of meta.yaml and in requirements.txt, will that cause issues? Like will the package--say numpy for example--get installed by conda and then upgraded by pip, or some other weird behavior? Or if a package is listed in meta.yaml then will conda-build skip that same reference in requirements.txt? I was trying to find a good example feedstock for this, but was not sure where to look.
Krishna Bhogaonker
@00krishna
Note, I am associating requirements.txt with pip and meta.yaml with conda package manager, and perhaps things are not so separated. But, that is why I was looking for some clarification. Thanks.
Cheng H. Lee
@chenghlee
conda-build does not process anything in requirements.txt. When building packages, all the requirements should be listed in the appropriate section of recipe's meta.yaml.
Krishna Bhogaonker
@00krishna
@chenghlee Oh thanks for the info. So conda-build does not bother with requirements.txt, that is helpful. So then if I list non-conda packages in the meta.yaml file, then conda-build will know to use pip to install those, or do I have to specify this somewhere? Sorry, just trying to make sure I am thorough in my understanding. Thanks again.
Cheng H. Lee
@chenghlee
I'd need to check to be 100% sure, but I don't think conda-build is able to directly install requirements using pip. Generally, the best practice is to build your requirements as conda packages first.
You could, in theory, run pip install --no-deps with your build script (build.sh and/or bld.bat), but that's a not 100% reliable option.
Krishna Bhogaonker
@00krishna
@chenghlee Hmm, okay. I have 2 dependencies which are not in conda now, those are dhash and pytest-cases. So if I wanted to install those in my conda built package, what would be the right way to do that. Would I include a script that does a pip install dhash --no-deps or something like that?
Cheng H. Lee
@chenghlee
Are these dependencies only needed for the build process, or will they be needed to run as well?
Krishna Bhogaonker
@00krishna
I need to use dhash during run time. pytest-cases is only for testing so not needed to run.
Cheng H. Lee
@chenghlee
If they're needed to run, you will need to create a dhash conda package first.
I don't believe there's currently a way for a conda package to depend on a pip package.
Krishna Bhogaonker
@00krishna
Oh okay. Yeah, since dhash is in pypi, I can probably use conda skeleton to build the package right.
Cheng H. Lee
@chenghlee
Yes