by

Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
    Renato Alves
    @unode
    Depending on test requirements we can have "on demand" tests setup with GitLab. CI runtime limit there is customizable. Currently set to 2 hours but can be customized per runner.
    This means we could have a runner setup on a "big enough" machine that can run longer jobs.
    "On demand" means, clicking on gitlab's web interface. See for instance the last target in: https://gitlab.com/ngless/ngless/pipelines/121509238 .
    I believe it's also possible to bind a test target to a specific runner to prevent jobs from starting if the runner is offline or unavailable for some reason.
    Luis Pedro Coelho
    @luispedro
    Even just running the gut-demo example from scratch takes a while and does a lot of compute/diskspace. 2 hours is not enough
    Renato Alves
    @unode
    Yes, I'm also not sure if the CI machines behind gitlab are big enough in terms of disk/RAM. The option of having a custom runner is still viable though.
    Conditional to being able to allocate a machine to run it... Doesn't need to be dedicated though.
    Renato Alves
    @unode
    The main advantage of this setup is the simplicity of triggering/queueing a test run for one or more commits in the past.
    Luis Pedro Coelho
    @luispedro
    If it's easy to set up, but requires payment, I think this is an allowable research expense for me
    Renato Alves
    @unode
    If you have some old hardware running under the desk this is also an option. I might be able to scavenge some old machines here too.
    Will need to ask
    Luis Pedro Coelho
    @luispedro
    Frankly, it's likely to be easier to just add another AWS machine to the bill
    (at least from my side)
    Renato Alves
    @unode
    I don't have that flexibility here. There are also a few clouds that could be used, including a new kubernetes system currently in testing but usage is still unclear.
    I can help setup the runner if you can provide SSH credentials.
    I'm not familiar with AWS so can't estimate beforehand how the bill would look like.
    Luis Pedro Coelho
    @luispedro
    It really depends. A small machine is cheap, but it does go up fast
    Luis Pedro Coelho
    @luispedro
    @/all As conversations sometimes get lost here or for longer messages, I created an NGLess devel Google Group: https://groups.google.com/forum/?nomobile=true#!forum/ngless-devel
    Luis Pedro Coelho
    @luispedro
    @sureyeaah : the JIT idea can be fleshed out in different ways. I see you have been trying to understand the system better. Our initial thought was that the block calls like preprocess(...) using |r|: ... could be JIT compiled for speed
    Divyanshu Mahajan
    @dmahajan980
    @luispedro the group link that you shared shows that either we don't have permission to access the group or the group doesn't exist.
    Luis Pedro Coelho
    @luispedro
    Sorry, try again
    Luis Pedro Coelho
    @luispedro
    (I had set up "Anyone can join", but I didn't realize that "Anyone can see the group" is a separate setting)
    Divyanshu Mahajan
    @dmahajan980
    Great! It's working now.
    Renato Alves
    @unode
    Hi all. Some of you tried following the gut-short demo but due to the high memory/disk requirements haven't managed to finish the more computationally intensive steps. If you want to have a look at the final files you can download http://ngless.embl.de/resources/Demos/gut-short-results.tar.xz
    Sajjal Nair
    @nairsajjal
    Hi Renato, I looked at the final files, can I know how can I generate the HTML view of "gut-short-demo" because that would come handy in creating a sample wireframe for the outputs.
    Muhammad_M
    @mm2774
    Hi everyone, my name is Muhammad and you can refer to me as Mo if you’d like. I’m a student at Cornell University in the U.S. and I’m interested in contributing to “NGLess: Expanding data types” through Haskell for GSoC 2020.
    I have experience in functional programming and OCaml through an intermediate, one semester course. I’m a newbie to Haskell and open source - but I’m very much interested in learning both. I see that the project is designated as “Medium to Hard” in terms of difficulty, so is this doable as a summer project considering my experience, or will it be too much to take on since I’m new to Haskell?
    Edmund Wu
    @eadwu
    Hi, I'm Edmund, currently attending University at Buffalo as a first year student. Currently I'm interested in "Integrating with nixpkgs". All experience I have with Nix is basically as a hobbyist who runs NixOS on their daily machine.
    Damn, formal talk always looks so unnatural to me
    Luis Pedro Coelho
    @luispedro
    @mm2774 Hi Mo, if you are truly a newbie to Haskell it may be a bit too ambitious as you'd be learning the language at the same time as learning the project. On the other hand, there is still some time until the summer and even until the GSoC submission, so you can also try to learn some of it in preparation. Or if you have bioinformatics knowledge, then that's also a good preparation (you don't mention it). If it is all new at the same time (the language, the project, the bioinformatics settings), it may be too much, though
    @eadwu Do you have experience with nixpkgs also by writing some of your own packages/config?
    Edmund Wu
    @eadwu
    @luispedro Yeah I do maintain some packages and regularly contribute to nixpkgs (if I need packages, etc), though most of the regular updates are just version bumps
    matrixbot
    @matrixbot
    @eadwu:matrix.org Test
    Luis Pedro Coelho
    @luispedro
    @eadwu Then, you likely have the basic nix/bash knowledge needed for the project. Do you understand the goals of the project? Do you have some bioinformatics experience?
    Basically, the tl;dr is that we would want to use nix to do perfectly reproducible pipelines, wrapped up with NGLess so that 95% of users wouldn't ever have to even know that nix is a thing that exists
    Luis Pedro Coelho
    @luispedro
    For a longer discussion, we can also move this to email and use the devel mailing list (https://groups.google.com/forum/m/#!forum/ngless-devel)
    Muhammad Shahzaib Shoaib
    @mshahzaib101
    hello, My name is Shahzaib. Currently Iam Full Stack developer and done many projects using React.js and React Native.
    I would like to contribute in "Improved reporting of results".
    I have worked with graphs/charts using javascript and am confident that I can handle this one. But iam bit confused , did I will be building a new website for charts representation or working in the existing project repo ?
    thankyou
    Luis Pedro Coelho
    @luispedro

    Since this has gotten a bit noisy, as an experiment, I invite you to do join my open office hours tomorrow @ 2pm GMT

    https://us04web.zoom.us/j/181765033

    Sanjiban Sengupta
    @kahanikaar

    This is Sanjiban Sengupta, sophomore in Computer Engineering from IIIT Bhubaneswar, India, would like to contribute to your org for GSoC'20, I have practical and working knowledge of C, C++, Python and Java, for web, I am familiar with HTML, CSS, JS, Bootstrap and frameworks such as ReactJS and NodeJS, also i am acquainted with concepts of ML and AI and know the technicalities to apply these to solve modern real life problems.

    Being a beginner at GSOC'20, it will be very grateful if you could provide the required information for beginning processes for the development.

    Luis Pedro Coelho
    @luispedro

    We had a useful zoom discussion today, until we ran out of time, so I am scheduling another round for next week. Open to anyone in the world

    Luis Pedro Coelho is inviting you to a scheduled Zoom meeting.

    Topic: Open Office Hours
    Time: Mar 18, 2020 02:00 PM Greenwich Mean Time

    Join Zoom Meeting
    https://us04web.zoom.us/j/756695245

    Meeting ID: 756 695 245

    Muhammad_M
    @mm2774
    Sounds good, also they completely suspended classes until April, so I can focus on the project once I have logistics/moving out done this weekend
    Sadeep Weerasinghe
    @sadeepdarshana

    Hi everyone,

    Sadeep here, from Sri Lanka.
    Final year student in CSE, University of Moratuwa.

    For my final year project, I am working on "reducing higher dimensional genomic data (we take oligonucleotide frequencies as of now) using Auto Encoders". I am using C to pre-process FASTA and FASTQ files to feed them to the neural network implemented in Python.

    I am good in C/C++. I come from a strong algorithm background, having won the Sri Lankan Olympiad and represented Sri Lanka at the International Olympiad in Inforrmatics (https://www.ioi2014.org/ioi-2014.html).

    For GSoC, I love to work on a genomics projects since my learning curve would be low.

    Is there a real chance of getting selected to an NGLess project since its a little late to the party?

    ishan kunkolikar
    @kunkolikar.ishan_gitlab
    Hi ! Ishan from India. I was interested in one of the project ideas , should I contribute to the project first before starting work on an idea ?
    Luis Pedro Coelho
    @luispedro
    Hi @kunkolikar.ishan_gitlab and @sadeepdarshana : contributions are, obviously, always welcome, but do spend some time learning about the project to see if your ideas would be a good fit
    ishan kunkolikar
    @kunkolikar.ishan_gitlab
    Hi @luispedro should we start with an idea on gsoc website that we like and the mentors will recommend and ask for necessary changes ?
    Luis Pedro Coelho
    @luispedro
    Open office hours right now

    Topic: Open Office Hours
    Time: Mar 18, 2020 02:00 PM Greenwich Mean Time

    Join Zoom Meeting
    https://us04web.zoom.us/j/756695245

    Renato Alves
    @unode
    Hi everyone, apologies for the silence these past weeks but between sickness and an overwhelming amount of work and life changes due to the COVID-19 situation I haven't been able to provide feedback.
    As the project submission deadline is approaching quickly (next Tuesday 31st March ), please let us know if there is anything we can help you with as early as possible. We'll try our best given current limitations.
    We are still looking forward to hear from and work with you.
    Praveen Kumar
    @pkspyder007
    Can anyone tell me how ngless produce results in html?
    Sajjal Nair
    @nairsajjal

    After downloading the gut-short demo. The script "gut-demo.ngl " looks like this

    ngless "1.1"
    import "parallel" version "1.0"
    import "mocat" version "1.0"
    import "motus" version "0.1"
    import "igc" version "1.0"

    igc module version 1.0, causes unexpected results because of its size of 45 bytes at https://ngless.embl.de/resources/Demos/.
    As an alternative I changed the version from 1.0 to 0.9, the script works fine.
    Since this is not an error in the source code, but simply an availability issue, a github issue would not suffice.

    Also,
    The documentation present at https://ngless.embl.de/tutorial-gut-metagenomics.html#
    would require minor documentation updates like

    ngless "1.0" (to)------ "1.1"
    import "parallel" version "0.6" ------ "1.0"
    import "mocat" version "0.0" ------ "1.0"
    import "motus" version "0.1"
    import "igc" version "0.0" ------ "0.9"

    As an idea, I suggest we could make a small mechanism for linking the documentation with the files present on the server, hence there would be dynamic changes to the scripts present at documentation and hence we can reduce human interaction(manual changes) with the documentation.
    Its an interesting prospect.

    Luis Pedro Coelho
    @luispedro
    NGLess v1.1.1 was just released
    This is a bugfix release
    Sajjal Nair
    @nairsajjal

    Hey everybody,
    I hope everyone is SAFE and doing well!

    I have been thinking about the compute problem, and my main aim was to reduce the processing time(hours) taken by ngless even for tests like "gut-short" and "ocean genomics".
    Processing on a local machine limited by our traditional infrastructure would always require much time, I did quite some digging and I am onto something.

    Possible Alternative: Using Google Cloud Life-sciences : It enables the life sciences community to process biomedical data at scale. Cost effective and supported by a growing partner ecosystem, Cloud Life Sciences lets you focus on analyzing data and reproducing results while GCP takes care of the rest.
    Its a "High Performance Computing" solution towards using NGless and analyzing all the modules in seconds/minutes(not hours). That is a big jump and can mean much time would be saved to do better tasks.
    Bioinformaticians can build what they want, not just what they need, using open standards. Researchers can speed up your research, ask new questions, and share data in a secure online environment. IT professionals, rest easy knowing you have the resources you need to meet computational demand, secure data, and ensure system reliability.

    I would be exploring more about this solution in the coming days, and if I find a solution that seems useful and would benefit everyone, I will make a medium article(step-by-step guide) and post it here, letting everyone know.

    Meanwhile, you may see the documentation for Google Life Sciences,
    here: https://cloud.google.com/life-sciences

    Genomic Analyses on Google Cloud Platform: https://www.youtube.com/watch?v=27tSivxnQ_E&feature=emb_logo

    Case Study by "Color": https://cloud.google.com/customers/color (Color: Enabling scientists to do real-time genomic data exploration)

    Luis Pedro Coelho
    @luispedro
    I have just released NGLess 1.2.0 on Github. It is still building on bioconda, so I am waiting for that to announce it on the mailing-list, but I ran some long term tests over the week-end and it was passing them all