by

Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • 08:35
    NEGU93 edited #4060
  • Sep 19 06:30
    anaconda-issue-bot labeled #4063
  • Sep 19 06:30
    isuruf opened #4063
  • Sep 18 21:39
    beckermr edited #4062
  • Sep 18 21:38
    beckermr opened #4062
  • Sep 18 21:36
    anaconda-issue-bot labeled #4062
  • Sep 18 21:35
    beckermr opened #4061
  • Sep 18 15:19
    gilescope commented #4059
  • Sep 18 15:10
    gilescope commented #4059
  • Sep 18 14:03
    mingwandroid closed #4059
  • Sep 18 14:03
    mingwandroid commented #4059
  • Sep 18 12:42
    NEGU93 edited #4060
  • Sep 18 12:39
    NEGU93 opened #4060
  • Sep 18 11:48
    gilescope opened #4059
  • Sep 18 08:43

    conda-bot on gh-pages

    docs for pr 4058 (compare)

  • Sep 18 08:43
    anaconda-issue-bot commented #4058
  • Sep 18 08:43
    Juke34 opened #4058
  • Sep 17 18:43
    BrenBarn commented #4056
  • Sep 17 10:31
    wilsonmr opened #4057
  • Sep 17 08:59
    pingvar commented #4056
Krishna Bhogaonker
@00krishna
Hey folks, I had a question about conda-build and requirements.txt. As I understand it, we are supposed to keep conda packages/environments separate from pip stuff, as much as possible--I read @jjhelmus blog post on this. Now one thing that was not clear in the docs was the relationship between the Requirements section of meta.yaml and the requirements.txt. I have packages listed in requirements.txt so that I can test them in Travis for regular CI -- where I can use conda install --file requirements.txt and it will find the packages in the conda channel. But for package building, if I have the same packages listed in the Host and Runtime section of meta.yaml and in requirements.txt, will that cause issues? Like will the package--say numpy for example--get installed by conda and then upgraded by pip, or some other weird behavior? Or if a package is listed in meta.yaml then will conda-build skip that same reference in requirements.txt? I was trying to find a good example feedstock for this, but was not sure where to look.
Krishna Bhogaonker
@00krishna
Note, I am associating requirements.txt with pip and meta.yaml with conda package manager, and perhaps things are not so separated. But, that is why I was looking for some clarification. Thanks.
Cheng H. Lee
@chenghlee
conda-build does not process anything in requirements.txt. When building packages, all the requirements should be listed in the appropriate section of recipe's meta.yaml.
Krishna Bhogaonker
@00krishna
@chenghlee Oh thanks for the info. So conda-build does not bother with requirements.txt, that is helpful. So then if I list non-conda packages in the meta.yaml file, then conda-build will know to use pip to install those, or do I have to specify this somewhere? Sorry, just trying to make sure I am thorough in my understanding. Thanks again.
Cheng H. Lee
@chenghlee
I'd need to check to be 100% sure, but I don't think conda-build is able to directly install requirements using pip. Generally, the best practice is to build your requirements as conda packages first.
You could, in theory, run pip install --no-deps with your build script (build.sh and/or bld.bat), but that's a not 100% reliable option.
Krishna Bhogaonker
@00krishna
@chenghlee Hmm, okay. I have 2 dependencies which are not in conda now, those are dhash and pytest-cases. So if I wanted to install those in my conda built package, what would be the right way to do that. Would I include a script that does a pip install dhash --no-deps or something like that?
Cheng H. Lee
@chenghlee
Are these dependencies only needed for the build process, or will they be needed to run as well?
Krishna Bhogaonker
@00krishna
I need to use dhash during run time. pytest-cases is only for testing so not needed to run.
Cheng H. Lee
@chenghlee
If they're needed to run, you will need to create a dhash conda package first.
I don't believe there's currently a way for a conda package to depend on a pip package.
Krishna Bhogaonker
@00krishna
Oh okay. Yeah, since dhash is in pypi, I can probably use conda skeleton to build the package right.
Cheng H. Lee
@chenghlee
Yes
Krishna Bhogaonker
@00krishna
Nice. That gives me a plan. Thanks for your help.
Cheng H. Lee
@chenghlee
You're welcome.
Philip Austin
@phaustin
A compromise for pure python packages between a pypi install and writing your own conda recipe is to build a whl file (which essentially locks the dependency) and install that using build.sh:
https://docs.conda.io/projects/conda-build/en/latest/user-guide/wheel-files.html
matrixbot
@matrixbot
gabm I noticed that conda-build filters the .git folder from being packaged... I actually would like to package the whole .git folder along with the files present.. is there any way around that? https://github.com/conda/conda-build/blob/master/conda_build/utils.py#L1427
Jan Pöppel
@jpoeppel

Hi everyone, I may have mentioned this problem before, but while I did get it to work briefly, I do not belief I found a proper solution. My problem is related to build variants in combination with multiple outputs in a recipe.
As an example, I created a minimal example recipe:

package:
        name: outputvariants
        version: "0.1"

build:
        number: 0

requirements:
        run:
                - out_a
                - out_b

outputs:
        - name: out_a
          script: install_A.sh        # [unix]
          requirements:
                  build:
                        - {{ compiler('c') }}
                        - {{ compiler('cxx') }}
                        - cmake >=3.10
                  host:
                        - boost-cpp {{ boostcpp }}
                  run:
                        - boost-cpp {{ boostcpp }}

        - name: out_b
          noarch: python
          script: install_B.sh
          requirements:
                  host:
                          - python
                  run:
                          - python

with minimal build scripts that only do `touch out[A|B]file.
If i setup my conda_build_config.yaml with at most 2 variants, e.g. as such:

boostcpp: 
   - 1.68
   - 1.69

it builds as expected, but if I add more boost variants, or something else, liek so:

boostcpp: 
   - 1.68
   - 1.69
   - 1.70
   - 1.71
python:
   - 3.6
   - 3.7
   - 3.8

I only get a IndexError:

...
File "/Users/jpoeppel/miniconda3/lib/python3.7/site-packages/conda_build/metadata.py", line 2173, in _get_used_vars_meta_yaml_helper
    apply_selectors=apply_selectors))
  File "/Users/jpoeppel/miniconda3/lib/python3.7/site-packages/conda_build/metadata.py", line 1698, in extract_single_output_text
    output = output_matches[output_index] if output_matches else ''
IndexError: list index out of range

Since I cannot imaging being the only one trying to have multiple outputs build with variants, I must be doing something wrong, but I do not quite get what the problem is. I would be happy about any help!

Jan Pöppel
@jpoeppel
Digging into the code, the error appears to occur, because conda build tries to find an output named according to the implizit meta-package ("outputvariants" in this case), which is added to the output_tuples in the function, but its index can obviously not be found in the "output_matches" variable, which only contains the list of actual outputs.
However, I have no idea, why this does not occur, when the variants are 2 or less...
Ray Donnelly
@mingwandroid
@jpoeppel Please file a bug at github.com/conda/conda-build with full reproduction steps
Since I cannot imaging being the only one trying to have multiple outputs build with variants
indeed, far from it
everything we build for AD is built against a large matrix.
Jan Pöppel
@jpoeppel
@mingwandroid Thanks for the reply. While creating the bug report (#4023), I did figure out that it seems to be a problem with the Python variants. I assume that python variants and/or requirements are handled in a different way than other requirements, causing the error. But I may also just be doing something I am not supposed to with Python variants, but as they are mentioned in the docs, I thought it should be fine.
Mike Boyle
@moble
Hi everyone. Is there any way to build python C extensions on windows with gcc? I tried really hard, but exhausted all hope of the getting the code I'm building working with MSVC (via the feedstock PR conda-forge/spinsfast-feedstock#9). I saw these comments by @duncanmmacleod giving the most helpful suggestions so far. But at link time ld complains that it can't find "-lmsvcr140". Duncan said it wasn't showing up for him in azure, but it is for me. (I don't have any other access to windows machines.) There's some noise on the internet about hacking distutils to point it at vcruntime140 instead. But surely there's a more standard way to handle this, right???
Ray Donnelly
@mingwandroid
@jpoeppel there are some hacks in conda-build to do with python but if you use variants properly (i.e. have a zip_key containing python in it, probably) then those hacks are skipped.
Jan Pöppel
@jpoeppel
@mingwandroid I have not tried using zip_keys yet, but from how I understand the documentations, zip_keys would cause only specific pairings of versions, e.g. in the example above, one boost version with one python version. And it would also not work with 4 boost versions, but only 3 python versions, as the lists would need to be identical lengths, or am I missunderstanding something? Thanks a lot for the help regardless!
Jonathan J. Helmus
@jjhelmus
conda-build 3.20.0 has been tagged, https://github.com/conda/conda-build/releases/tag/3.20.0. Expect packages in the conda-canary channel tomorrow. This release supports Python 2.7 but support for running conda-build in a Python 2.7 or <3.6 environments will be dropped in a 4.0.0 release in the near future. See conda/conda-build#4024 for more discussion.
Dave Hirschfeld
@dhirschfeld
Looking forward to kicking the tyres! :rocket:
Faustin Carter
@FaustinCarter
Hey all, can anyone help shed some light on whether channeldata.json is actually used for anything? The only mentions of it in the docs are here: https://docs.conda.io/projects/conda-build/en/latest/concepts/generating-index.html?highlight=channeldata.json#parts-of-a-channel, but no mention of actual utility.
I'm wondering what negative downstream consequences there are if that file does not exist in a channel (but all the repodata and current_repodata files are in place).
nvm, found it
Anthony Scopatz
@scopatz
Hey Folks! I vaguely recall there was some way to use ccache with conda-build can anyone point me in the right direction here?
Blog posts, docs, etc
Jonathan J. Helmus
@jjhelmus
Anthony Scopatz
@scopatz
Thanks!
Fabien Celier
@fabiencelier
Hello,
Is it possible to build a package that is OS specific (win-64/osx-64/linux-64) but cross Python version ?
I always end up having my Python version pinned in "lib/python3.8/site-packages"
Nehal J Wani
@nehaljwani
26 replies
Matthew R Becker
@beckermr
Does conda need the repodata.json files for anything?
or does it only pull the compressed ones?
Ray Donnelly
@mingwandroid
Hey Folks! I vaguely recall there was some way to use ccache with conda-build can anyone point me in the right direction here?
Hi @scopatz , conda-build 3.20.x have support for the jinja2 function "ccache(method)" where method can be symlinks, env_vars or native.
Defaults has fairly robust ccache exes, but ccache is moving fast, esp. on Windows support so it might be worth making sure that we're up to date with our ccaches
To see that it's working, you want to do e.g. ls -lrt ~/.ccache during builds and see that the order changes around lots! or run ccache -s to see some stats
probably I want to clear the stats at the start of a build and print them at the end of it, PRs and any testing/fixing very welcome!
Oops, missed Jonathan's comment. I'd like people to kick this feature around a bit for now, I don't think it's quite ready yet.
Anthony Scopatz
@scopatz
Yeah, I am just not even sure how to use it :laughing:
I have been trying to use it on windows, where everything is weird
Ray Donnelly
@mingwandroid
I'll try to kick the tyres a bit later, for sure it'll work better on the other OSes and it is still a bit of an experiment. Isuru didn't want it integrated into conda-build really but I want conda-build's stats reporting to tie into it via calling ccache -s. It can then report if using ccache actually did anything useful for your build or not so you can investigate that (maybe try the other methods, or modify bld.bat/build.sh etc)