Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • 04:02
    github-actions[bot] labeled #3165
  • 04:02
    github-actions[bot] commented #3165
  • 04:02
    github-actions[bot] labeled #3161
  • 04:02
    github-actions[bot] commented #3161
  • 04:02
    github-actions[bot] labeled #3111
  • 04:02
    github-actions[bot] unlabeled #3111
  • 04:02
    github-actions[bot] labeled #3080
  • 04:02
    github-actions[bot] unlabeled #3080
  • 04:02
    github-actions[bot] closed #2997
  • 04:02
    github-actions[bot] labeled #2997
  • 04:02
    github-actions[bot] closed #2996
  • 04:02
    github-actions[bot] labeled #2996
  • 04:02
    github-actions[bot] closed #2989
  • 04:02
    github-actions[bot] labeled #2989
  • Dec 03 23:41
    mbargull commented #4665
  • Dec 03 23:36
    mbargull ready_for_review #4665
  • Dec 03 23:36
    mbargull edited #4665
  • Dec 03 23:33
    mbargull synchronize #4665
  • Dec 03 22:33
    mbargull synchronize #4665
  • Dec 03 22:18
    mbargull synchronize #4665
Krishna Bhogaonker
@00krishna
Yep, thanks for the tip. I must have installed conda-build before adding conda-forge to my list of channels in condarc. So now I will rectify that.
Brian Miller
@bkmdev
hi, conda-build v3.17.8, when I attempt to do conda build --no-include-recipe meta.yaml for a python sdk app, I get ~187 of the following odd errors (warnings?). Why? Is this something to be concerned about? If so, how does one fix this? :
[...]

Packaging my_sdk
INFO:conda_build.build:Packaging my_sdk
Packaging my_sdk-w.x.y.z-py_0
INFO:conda_build.build:Packaging my_sdk-w.x.y.z-py_0
number of files: 112
/Library/Developer/CommandLineTools/usr/bin/nm: /anaconda3/conda-bld/my_sdk_1596598276840/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_plac/lib/libpython3.8.a(getbuildinfo.o) Invalid value (Producer: 'LLVM10.0.0' Reader: 'LLVM APPLE_1_900.0.38_0')

/Library/Developer/CommandLineTools/usr/bin/nm: /anaconda3/conda-bld/my_sdk_1596598276840/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_plac/lib/libpython3.8.a(acceler.o) Invalid value (Producer: 'LLVM10.0.0' Reader: 'LLVM APPLE_1_900.0.38_0')

/Library/Developer/CommandLineTools/usr/bin/nm: /anaconda3/conda-bld/my_sdk_1596598276840/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_plac/lib/libpython3.8.a(grammar1.o) Invalid value (Producer: 'LLVM10.0.0' Reader: 'LLVM APPLE_1_900.0.38_0')

[...]

/Library/Developer/CommandLineTools/usr/bin/nm: /anaconda3/conda-bld/my_sdk_1596598276840/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_plac/lib/libpython3.8.nolto.a(xxsubtype.o) Invalid value (Producer: 'LLVM10.0.0' Reader: 'LLVM APPLE_1_900.0.38_0')

/Library/Developer/CommandLineTools/usr/bin/nm: /anaconda3/conda-bld/my_sdk_1596598276840/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_plac/lib/libpython3.8.nolto.a(frozen.o) Invalid value (Producer: 'LLVM10.0.0' Reader: 'LLVM APPLE_1_900.0.38_0')
This previously never occurred during builds and I think happened only after (Home)brew decided to upgrade XCode CLI tools?
Brian Miller
@bkmdev
I'm also concerned about this strange ..._placehold_placehold... coming from $PREFIX presumably?
(FWIW, I'm on OSX 10.13.X + XCode 9.2)
Krishna Bhogaonker
@00krishna
@bkmdev The placehold stuff is standard. That happens to everyone. I don't know the exact details, but basically it is a way to ensure that when installing the package, they have all of that space reserved for the paths in the package. I think the conda developers used to have shorter path and then that led to problems with not having the space for longer paths on some OSs, but I am not the expert on that.
Also is there a reason you are using v3.17.8? I think the current version is up to v3.19.3 I believe.
Brian Miller
@bkmdev
@00krishna thanks for the reply. Re. the placehold stuff, I think you're right, I think I may have even asked this here a long time back & a dev commented, although the issue/solution still seems strange to me. Re. the version: unless there's a good reason to (ex: specific fix, necessary feature, severe security vulnerability, etc.) I don't like to upgrade, as things tend to break. Case in point, I don't think the nm issues above occurred until Homebrew forced the re-installation of XCode CLI tools :confused:
mattchan-tencent
@mattchan-tencent
Hi, I got an unexpected dependency error when specifying the python version in conda-build. The package builds properly, but in the test phase it fails to install the dependencies (pytest). It's an extremely simple package. No deps except python 3.7.* in the build/host/run, and pytest (version not pinned) in test/requires.
This is the error:
Output in format: Requested package -> Available versions

Package ca-certificates conflicts for:
python[version='>=3.7,<3.8.0a0'] -> openssl[version='>=1.1.1g,<1.1.2a'] -> ca-certificates
pytest -> python[version='>=2.7,<2.8.0a0'] -> ca-certificates

Package python conflicts for:
pytest -> attrs[version='>=17.4.0'] -> python[version='>=3.4']
eqasm-parse==0.1=openmp -> python[version='>=3.8,<3.9.0a0']
pytest -> python[version='>=2.7,<2.8.0a0|>=3.6,<3.7.0a0|>=3.7,<3.8.0a0|>=3.8,<3.9.0a0|>=3.5,<3.6.0a0']
python[version='>=3.7,<3.8.0a0']
mattchan-tencent
@mattchan-tencent
Well, this is seriously weird. I fixed it by removing the build string from meta.yaml
I thought that had no effect on the package build?
Krishna Bhogaonker
@00krishna
Has anyone figured out a workaround for this issue conda/conda#7758. The problem occurs when building a package in CI using conda-build. So the package gets build when when trying to install that package into the test environment, conda cannot find the newly built package. So the command according to the conda build docs is conda install --use-local <packagename>. That did not work, nor did the alternative conda install -c local <packagename>. I was hunting through feedstocks but did not see anyone using this code, so hopefully someone knows the answer.
I have a travis script running, as per this doc. https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/use-conda-with-travis-ci.html . Are you able to see the log if I give it to you this way. https://api.travis-ci.org/v3/job/715572047/log.txt
@isuruf indicated: @isuruf
You are using conda-build from a non-root environment and trying to install the package using conda from root environment
@isuruf so you suggest being in the base environment for the build, then activating the test environment and doing the local install?
Isuru Fernando
@isuruf
Either,
  1. Install conda-build in root env
  2. Use conda from non-root env(this is tricky to do)
  3. Try giving -c file://home/travis/miniconda/envs/test-environment/conda-bld (Not sure what the exact command is
Krishna Bhogaonker
@00krishna
Okay cool. I am trying this as you suggest in 1. install to root env.
  - bash miniconda.sh -b -p $HOME/miniconda
  - source "$HOME/miniconda/etc/profile.d/conda.sh"
  - hash -r
  - conda config --set always_yes yes --set changeps1 no
  - conda update -q conda
  - conda info -a
  - conda install conda-build
  - conda create -q -n test-environment python=$TRAVIS_PYTHON_VERSION conda pytest six pytest-cov pytest-mock 
  - conda activate test-environment
  - conda build packaging/ 
  - conda install --use-local <packagename>
Krishna Bhogaonker
@00krishna
@isuruf Excellent, this worked. Wow, thanks so much.
Is it possible for me to submit a PR for this section of the documentation?
Just to add a note that conda-build should be in the root environment.
Jonathan J. Helmus
@jjhelmus
Krishna Bhogaonker
@00krishna
@jjhelmus I can do that. Should I open an issue first and then submit the PR, or can I just submit the PR directly.
Jonathan J. Helmus
@jjhelmus
A PR directly is fine
Krishna Bhogaonker
@00krishna
:thumbsup:
Brian Miller
@bkmdev
hi, is it really possible to distribute JVM-based apps with conda? I know the docs claim "Package, dependency and environment management for any language—Python, R, Ruby, Lua, Scala, Java, ..." but conda-forge/conda-forge.github.io#590 is making wonder if there is more to it, ie: re. .jar dependencies? (I assume native jar dependencies currently aren't handled and would need to just be bundled with the JVM-based app itself?)...
Cheng H. Lee
@chenghlee
@bkmdev: it really depends on the complexity of your jar dependencies. In the simplest cases, you can dump the jars for your application into a single conda package; I have an example here: https://github.com/biobuilds/biobuilds/tree/master/picard/2.15.0
for more complex cases, you'll likely have to build conda packages for each of your dependencies.
Krishna Bhogaonker
@00krishna
Hey folks, I had a question about conda-build and requirements.txt. As I understand it, we are supposed to keep conda packages/environments separate from pip stuff, as much as possible--I read @jjhelmus blog post on this. Now one thing that was not clear in the docs was the relationship between the Requirements section of meta.yaml and the requirements.txt. I have packages listed in requirements.txt so that I can test them in Travis for regular CI -- where I can use conda install --file requirements.txt and it will find the packages in the conda channel. But for package building, if I have the same packages listed in the Host and Runtime section of meta.yaml and in requirements.txt, will that cause issues? Like will the package--say numpy for example--get installed by conda and then upgraded by pip, or some other weird behavior? Or if a package is listed in meta.yaml then will conda-build skip that same reference in requirements.txt? I was trying to find a good example feedstock for this, but was not sure where to look.
Krishna Bhogaonker
@00krishna
Note, I am associating requirements.txt with pip and meta.yaml with conda package manager, and perhaps things are not so separated. But, that is why I was looking for some clarification. Thanks.
Cheng H. Lee
@chenghlee
conda-build does not process anything in requirements.txt. When building packages, all the requirements should be listed in the appropriate section of recipe's meta.yaml.
Krishna Bhogaonker
@00krishna
@chenghlee Oh thanks for the info. So conda-build does not bother with requirements.txt, that is helpful. So then if I list non-conda packages in the meta.yaml file, then conda-build will know to use pip to install those, or do I have to specify this somewhere? Sorry, just trying to make sure I am thorough in my understanding. Thanks again.
Cheng H. Lee
@chenghlee
I'd need to check to be 100% sure, but I don't think conda-build is able to directly install requirements using pip. Generally, the best practice is to build your requirements as conda packages first.
You could, in theory, run pip install --no-deps with your build script (build.sh and/or bld.bat), but that's a not 100% reliable option.
Krishna Bhogaonker
@00krishna
@chenghlee Hmm, okay. I have 2 dependencies which are not in conda now, those are dhash and pytest-cases. So if I wanted to install those in my conda built package, what would be the right way to do that. Would I include a script that does a pip install dhash --no-deps or something like that?
Cheng H. Lee
@chenghlee
Are these dependencies only needed for the build process, or will they be needed to run as well?
Krishna Bhogaonker
@00krishna
I need to use dhash during run time. pytest-cases is only for testing so not needed to run.
Cheng H. Lee
@chenghlee
If they're needed to run, you will need to create a dhash conda package first.
I don't believe there's currently a way for a conda package to depend on a pip package.
Krishna Bhogaonker
@00krishna
Oh okay. Yeah, since dhash is in pypi, I can probably use conda skeleton to build the package right.
Cheng H. Lee
@chenghlee
Yes
Krishna Bhogaonker
@00krishna
Nice. That gives me a plan. Thanks for your help.
Cheng H. Lee
@chenghlee
You're welcome.
Philip Austin
@phaustin
A compromise for pure python packages between a pypi install and writing your own conda recipe is to build a whl file (which essentially locks the dependency) and install that using build.sh:
https://docs.conda.io/projects/conda-build/en/latest/user-guide/wheel-files.html
matrixbot
@matrixbot
gabm I noticed that conda-build filters the .git folder from being packaged... I actually would like to package the whole .git folder along with the files present.. is there any way around that? https://github.com/conda/conda-build/blob/master/conda_build/utils.py#L1427
Jan Pöppel
@jpoeppel

Hi everyone, I may have mentioned this problem before, but while I did get it to work briefly, I do not belief I found a proper solution. My problem is related to build variants in combination with multiple outputs in a recipe.
As an example, I created a minimal example recipe:

package:
        name: outputvariants
        version: "0.1"

build:
        number: 0

requirements:
        run:
                - out_a
                - out_b

outputs:
        - name: out_a
          script: install_A.sh        # [unix]
          requirements:
                  build:
                        - {{ compiler('c') }}
                        - {{ compiler('cxx') }}
                        - cmake >=3.10
                  host:
                        - boost-cpp {{ boostcpp }}
                  run:
                        - boost-cpp {{ boostcpp }}

        - name: out_b
          noarch: python
          script: install_B.sh
          requirements:
                  host:
                          - python
                  run:
                          - python

with minimal build scripts that only do `touch out[A|B]file.
If i setup my conda_build_config.yaml with at most 2 variants, e.g. as such:

boostcpp: 
   - 1.68
   - 1.69

it builds as expected, but if I add more boost variants, or something else, liek so:

boostcpp: 
   - 1.68
   - 1.69
   - 1.70
   - 1.71
python:
   - 3.6
   - 3.7
   - 3.8

I only get a IndexError:

...
File "/Users/jpoeppel/miniconda3/lib/python3.7/site-packages/conda_build/metadata.py", line 2173, in _get_used_vars_meta_yaml_helper
    apply_selectors=apply_selectors))
  File "/Users/jpoeppel/miniconda3/lib/python3.7/site-packages/conda_build/metadata.py", line 1698, in extract_single_output_text
    output = output_matches[output_index] if output_matches else ''
IndexError: list index out of range

Since I cannot imaging being the only one trying to have multiple outputs build with variants, I must be doing something wrong, but I do not quite get what the problem is. I would be happy about any help!

Jan Pöppel
@jpoeppel
Digging into the code, the error appears to occur, because conda build tries to find an output named according to the implizit meta-package ("outputvariants" in this case), which is added to the output_tuples in the function, but its index can obviously not be found in the "output_matches" variable, which only contains the list of actual outputs.
However, I have no idea, why this does not occur, when the variants are 2 or less...
Ray Donnelly
@mingwandroid
@jpoeppel Please file a bug at github.com/conda/conda-build with full reproduction steps