Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
  • 03:33

    pdxjohnny on gh-pages

    docs: Mon Apr 12 03:33:55 UTC 2… (compare)

  • Apr 11 03:25

    pdxjohnny on gh-pages

    docs: Sun Apr 11 03:25:49 UTC 2… (compare)

  • Apr 10 03:26

    pdxjohnny on gh-pages

    docs: Sat Apr 10 03:25:58 UTC 2… (compare)

  • Apr 09 09:06

    pdxjohnny on gh-pages

    docs: Fri Apr 9 09:06:22 UTC 2… (compare)

  • Apr 08 03:48

    pdxjohnny on snyk-fix-98742e62125ff7eb5388f6071fd2b321

    fix: operations/image/Dockerfil… (compare)

  • Apr 08 03:48

    pdxjohnny on snyk-fix-98742e62125ff7eb5388f6071fd2b321

    (compare)

  • Apr 08 03:34

    pdxjohnny on gh-pages

    docs: Thu Apr 8 03:34:30 UTC 2… (compare)

  • Apr 08 02:12

    pdxjohnny on snyk-fix-4bb1b9c4d21e4f4516cf5e3e12ac08c7

    fix: operations/deploy/Dockerfi… (compare)

  • Apr 08 02:12

    pdxjohnny on snyk-fix-4bb1b9c4d21e4f4516cf5e3e12ac08c7

    (compare)

  • Apr 08 01:47

    pdxjohnny on snyk-fix-33ebe7b2b746d659c5dfc3c13083dda2

    fix: dffml/skel/operations/Dock… (compare)

  • Apr 08 01:47

    pdxjohnny on snyk-fix-33ebe7b2b746d659c5dfc3c13083dda2

    (compare)

  • Apr 08 01:24

    pdxjohnny on gh-pages

    docs: Thu Apr 8 01:24:05 UTC 2… (compare)

  • Apr 08 01:08

    pdxjohnny on master

    service: dev: Port scripts/docs… (compare)

  • Apr 08 00:11

    pdxjohnny on snyk-fix-0d54f09ae78d71dd0a0ce473ce7ca45f

    fix: operations/nlp/Dockerfile … (compare)

  • Apr 08 00:11

    pdxjohnny on snyk-fix-0d54f09ae78d71dd0a0ce473ce7ca45f

    (compare)

  • Apr 07 21:48

    pdxjohnny on snyk-fix-2aaa92e9ffe02c2d6cd25d10d023067f

    fix: operations/binsec/Dockerfi… (compare)

  • Apr 07 21:48

    pdxjohnny on snyk-fix-2aaa92e9ffe02c2d6cd25d10d023067f

    (compare)

  • Apr 07 03:26

    pdxjohnny on gh-pages

    docs: Wed Apr 7 03:26:10 UTC 2… (compare)

  • Apr 05 03:48

    pdxjohnny on snyk-fix-f790ca3982aed8c5df308a329cd0f1ec

    fix: operations/image/Dockerfil… (compare)

  • Apr 05 03:48

    pdxjohnny on snyk-fix-f790ca3982aed8c5df308a329cd0f1ec

    (compare)

Aryan Kenchappagol
@aryanxk02
issue : intel/dffml#1007
I think I can solve this. Shall I make a PR??
Hashim
@mHash1m
@aryanxk02 I commented on your issue with a possible fix. Seems more of a local issue to me. :)
Aryan Kenchappagol
@aryanxk02
okay thank you @mHash1m I just checked and sorry for that :/
Seunghyeon Shin
@jacob771
@pdxjohnny As a follow-up question from your previous answer, I am first intending to modify logistic regression model on scratch folder that already existed. I am intending to change algorithm to be more efficient, and one effort that I made (I made a PR on it) is to use "from scipy.stats import multivariate_normal as mvn" to generate dataset rather than taking dataset.sh that was prepared. My intention is to build logistic regression model from scratch into more self-sustained, but I hope your criticisms to understand if there are problems that I missed. And you previously mentioned that you wish not to take entire dataset into the memory while training the dataset, and it is my idea that we could generate "random dataset" rather than taking from dataset.sh.
Hashim
@mHash1m
@aryanxk02 anytime :)
10 replies
Sanjiban Sengupta
@kahanikaar
Hello everyone!
This is Sanjiban Sengupta, a junior undergrad from IIIT Bhubaneswar, India, pursuing Computer Engineering. I would like to contribute to dffml.
I started with solving the issue: intel/dffml#1002
and made the PR intel/dffml#1005
For the issue, I have commented out the property decorator for the key in Record class and added the self.key= str(key) in the init function.
Currently, 8 checks are failing, some of which states having this issue within the tests,
File "/home/runner/work/dffml/dffml/model/daal4py/tests/test_lr.py", line 77, in test_02_predict correct = self.test_data[i]["Y"] TypeError: list indices must be integers or slices, not str
Hashim
@mHash1m
Hey @kahanikaar, since you are forcing the key to be str and the tests expect the key(the i in this case) to be int, it throws the error. Perhaps a bit redundant, but you could handle this by just casting the type back into int, in the tests that are failing.
So the /home/runner/work/dffml/dffml/model/daal4py/tests/test_lr.py", line 77 should be changed to: correct = self.test_data[int(i)]["Y"] and similarly in the model/scratch failing tests.
5 replies
Hashim
@mHash1m
Since the key is also being used as index of a list in the tests.
siddhant sharma
@iamapickle
Help Needed! while installing dffml-model-pytorch; I'm getting an error ERROR: Could not find a version that satisfies the requirement torch==1.7.1+cu110; platform_system == "Windows" and python_version == "3.7" (from dffml-model-pytorch)
ERROR: No matching distribution found for torch==1.7.1+cu110; platform_system == "Windows" and python_version == "3.7"
how to resolve it?
John Andersen
@pdxjohnny
@iamapickle You'll need to add -f https://download.pytorch.org/whl/torch_stable.html to the end of the pip install command
We should document this. Can you open an issue please?
2 replies
Aryan Kenchappagol
@aryanxk02
I was going through the quick-start page of the documentation and looking the command line section. While creating the training data cat > training.csv << EOF
Years,Expertise,Trust,Salary
0,1,0.1,10
1,3,0.2,20
2,5,0.3,30
3,7,0.4,40
EOF
What is cat and EOF ??
Pratik Dey
@Pratikrocks
cat is a Unix command
Eof is end of file delimiter
Aryan Kenchappagol
@aryanxk02
alright thank you @Pratikrocks
Pratik Dey
@Pratikrocks
✌️
John Andersen
@pdxjohnny
Today's Meeting Recording: https://youtu.be/l7tjBiautyU
Pratik Dey
@Pratikrocks
Hi all can anyone help me why I am getting the following error in the ci test checks https://github.com/intel/dffml/pull/1015/checks?check_run_id=1967979265
I the following pull I created a new test case , for that I created a file in the test directory called test_styles
And then I inherited the base class IntegrationCLITestCase
Pratik Dey
@Pratikrocks
Do we need to do anything more to add a test ?
Hashim
@mHash1m
Hey @Pratikrocks , the failing tests on your PR can be fixed by:
1: Adding an entry in the changelog.
2: Using black to style your code as it is the format of the whole code base.
You can find more about these CI tests and fixes in the 'Contributing -> Git and Github -> How to Read the CI' documentation section.
Pratik Dey
@Pratikrocks
thank you ++ @mHash1m
alex
@frecklebars
Hello, I have a flask web app with an upload page where I can upload a train and a test csv file that I use to train a model and then redirect to a "trained" page that shows the accuracy. The issue is that when i go back to the upload page and upload a different set of csv files for a different problem, while the model trains, the accuracy method returns a NoRecordsWithMatchingFeatures exception, as if the test.csv file was never changed. I checked if the files are being uploaded correctly, which they are, overwriting the old files. The train function seems to work fine but I can't seem to get the accuracy function to work unless I either restart the server, or name the test file differently (like test2.csv for example) which I don't want to. Is there any way to get this to work fine? I believe this wasn't happening while using dffml 0.3.7
Hashim
@mHash1m
@Pratikrocks np :)
Pratik Dey
@Pratikrocks
I was looking at the source of the csv file I came cross the following config params https://github.com/intel/dffml/blob/38ddb55ce0276b4590a9afc89dad47ba8cab1e5e/dffml/source/csv.py#L42 Can anyone who have worked on it give me a jist of what role does the following CSV_SOURCE_CONFIG_DEFAULT_tag CSV_SOURCE_CONFIG_DEFAULT_KEY params does?
John Andersen
@pdxjohnny
@frecklebars Can you point us to some code?
Also, what model are you using?
Any other information that would help us re-produce the issue would be helpful
Shaurya Puri
@spur19
@Pratikrocks key is a parameter that lets us access Records in the CSV file.
Refer https://intel.github.io/dffml/master/api/record.html
Pratik Dey
@Pratikrocks
okay @spur19
John Andersen
@pdxjohnny
Check it you! Things are working in colab! https://colab.research.google.com/drive/13SYqe0yQJjUYdLRvC1yajyFoyNTEASvP#scrollTo=EBy6_FQ1-ZYC I haven't used colab much before, but I think if you go to "File" -> "Save a Copy in Drive" then you can start messing around with it.
We can try tracking google colab realted stuff in this discussion: https://github.com/intel/dffml/discussions/1018 or also on Gitter as usual
Pinging @yashlamba ^
Nitesh yadav
@nitesh585
@pdxjohnny Access denied! while opening the colab's link.
John Andersen
@pdxjohnny
@nitesh585 Thank you! Should be fixed now :)
alex
@frecklebars
@pdxjohnny
from dffml import Features, Feature, train, accuracy, CSVSource
from dffml_model_scikit import LinearRegressionModel
import asyncio


salarymodel = LinearRegressionModel(
    features=Features(
        Feature("Years", int, 1),
        Feature("Expertise", int, 1),
        Feature("Trust", float, 1),
    ),
    predict=Feature("Salary", int, 1),
    directory="tempmodel"
)

irismodel = LinearRegressionModel(
    features=Features(
        Feature("sepal_length", float, 1),
        Feature("sepal_width", float, 1),
        Feature("petal_length", float, 1)
    ),
    predict=Feature("petal_width", float, 1),
    directory="tempmodel"
)

async def main():
    s = open("data/salary.csv", "r")
    salary = s.read()
    s.close()

    st = open("data/salarytest.csv", "r")
    salarytest = st.read()
    st.close()

    ir = open("data/iris.csv", "r")
    iris = ir.read()
    ir.close()

    irt = open("data/iristest.csv", "r")
    iristest = irt.read()
    irt.close()

    # write the salary.csv and salarytest.csv files in the train.csv and test.csv files
    trainfile = open("data/train.csv", "w")
    trainfile.write(salary)
    trainfile.close()

    testfile = open("data/test.csv", "w")
    testfile.write(salarytest)
    testfile.close()

    # train first model 
    await train(salarymodel, "data/train.csv")
    acc = await accuracy(salarymodel, CSVSource(filename="data/test.csv"))
    print("accuracy:", acc)

    # rewrite the train.csv and test.csv files to contain a separate dataset
    trainfile = open("data/train.csv", "w")
    trainfile.write(iris)
    trainfile.close()

    testfile = open("data/test.csv", "w")
    testfile.write(iristest)
    testfile.close()

    # train different model
    await train(irismodel, "data/train.csv")
    # NoRecordsWithMatchingFeatures exception occurs, looking for the features in the first dataset despite changing the contents of the train.csv file
    acc = await accuracy(irismodel, CSVSource(filename="data/test.csv"))
    print("accuracy:", acc)



if __name__ == "__main__":
    asyncio.run(main())
i recreated the issue here
John Andersen
@pdxjohnny
This message was deleted
John Andersen
@pdxjohnny
Ah that's a bug
If you make them different file names it will work. Otherwise we'll fix that in the 0.4.1 release.
John Andersen
@pdxjohnny
It's because the CSV source is keeping files in memory, key'd off of their filename. We have an issue somewhere to stop doing this...
Pratik Dey
@Pratikrocks
If I am supposed to install a module in dffml (say openpyxl) then is it sufficient to mention it in requirements-dev.txt and then run the cmd pip install -e .[dev] dffml service dev install

``
from openpyxl import Workbook, load_workbook
ModuleNotFoundError: No module named 'openpyxl'
^Z

```

Pratik Dey
@Pratikrocks
Sectional test cases are pssing as expected but the overall test are not pssing
Yash Lamba
@yashlamba

@Pratikrocks dffml is a package, just like openpyxl. You don't install a module in dffml and for using another package, you don't need to mention it in requirements unless it's a dependency (a package required for the dependent package to work).

Now for using openpyxl, just try pip install openpyxl.

Another thing, since you are using -e to install dffml, you won't need to run pip install -e .[dev] again until a core requirement is changed.