Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
  • Apr 12 03:33

    pdxjohnny on gh-pages

    docs: Mon Apr 12 03:33:55 UTC 2… (compare)

  • Apr 11 03:25

    pdxjohnny on gh-pages

    docs: Sun Apr 11 03:25:49 UTC 2… (compare)

  • Apr 10 03:26

    pdxjohnny on gh-pages

    docs: Sat Apr 10 03:25:58 UTC 2… (compare)

  • Apr 09 09:06

    pdxjohnny on gh-pages

    docs: Fri Apr 9 09:06:22 UTC 2… (compare)

  • Apr 08 03:48

    pdxjohnny on snyk-fix-98742e62125ff7eb5388f6071fd2b321

    fix: operations/image/Dockerfil… (compare)

  • Apr 08 03:48

    pdxjohnny on snyk-fix-98742e62125ff7eb5388f6071fd2b321

    (compare)

  • Apr 08 03:34

    pdxjohnny on gh-pages

    docs: Thu Apr 8 03:34:30 UTC 2… (compare)

  • Apr 08 02:12

    pdxjohnny on snyk-fix-4bb1b9c4d21e4f4516cf5e3e12ac08c7

    fix: operations/deploy/Dockerfi… (compare)

  • Apr 08 02:12

    pdxjohnny on snyk-fix-4bb1b9c4d21e4f4516cf5e3e12ac08c7

    (compare)

  • Apr 08 01:47

    pdxjohnny on snyk-fix-33ebe7b2b746d659c5dfc3c13083dda2

    fix: dffml/skel/operations/Dock… (compare)

  • Apr 08 01:47

    pdxjohnny on snyk-fix-33ebe7b2b746d659c5dfc3c13083dda2

    (compare)

  • Apr 08 01:24

    pdxjohnny on gh-pages

    docs: Thu Apr 8 01:24:05 UTC 2… (compare)

  • Apr 08 01:08

    pdxjohnny on master

    service: dev: Port scripts/docs… (compare)

  • Apr 08 00:11

    pdxjohnny on snyk-fix-0d54f09ae78d71dd0a0ce473ce7ca45f

    fix: operations/nlp/Dockerfile … (compare)

  • Apr 08 00:11

    pdxjohnny on snyk-fix-0d54f09ae78d71dd0a0ce473ce7ca45f

    (compare)

  • Apr 07 21:48

    pdxjohnny on snyk-fix-2aaa92e9ffe02c2d6cd25d10d023067f

    fix: operations/binsec/Dockerfi… (compare)

  • Apr 07 21:48

    pdxjohnny on snyk-fix-2aaa92e9ffe02c2d6cd25d10d023067f

    (compare)

  • Apr 07 03:26

    pdxjohnny on gh-pages

    docs: Wed Apr 7 03:26:10 UTC 2… (compare)

  • Apr 05 03:48

    pdxjohnny on snyk-fix-f790ca3982aed8c5df308a329cd0f1ec

    fix: operations/image/Dockerfil… (compare)

  • Apr 05 03:48

    pdxjohnny on snyk-fix-f790ca3982aed8c5df308a329cd0f1ec

    (compare)

Pratik Dey
@Pratikrocks
cat is a Unix command
Eof is end of file delimiter
Aryan Kenchappagol
@aryanxk02
alright thank you @Pratikrocks
Pratik Dey
@Pratikrocks
✌️
John Andersen
@pdxjohnny
Today's Meeting Recording: https://youtu.be/l7tjBiautyU
Pratik Dey
@Pratikrocks
Hi all can anyone help me why I am getting the following error in the ci test checks https://github.com/intel/dffml/pull/1015/checks?check_run_id=1967979265
I the following pull I created a new test case , for that I created a file in the test directory called test_styles
And then I inherited the base class IntegrationCLITestCase
Pratik Dey
@Pratikrocks
Do we need to do anything more to add a test ?
Hashim
@mHash1m
Hey @Pratikrocks , the failing tests on your PR can be fixed by:
1: Adding an entry in the changelog.
2: Using black to style your code as it is the format of the whole code base.
You can find more about these CI tests and fixes in the 'Contributing -> Git and Github -> How to Read the CI' documentation section.
Pratik Dey
@Pratikrocks
thank you ++ @mHash1m
alex
@frecklebars
Hello, I have a flask web app with an upload page where I can upload a train and a test csv file that I use to train a model and then redirect to a "trained" page that shows the accuracy. The issue is that when i go back to the upload page and upload a different set of csv files for a different problem, while the model trains, the accuracy method returns a NoRecordsWithMatchingFeatures exception, as if the test.csv file was never changed. I checked if the files are being uploaded correctly, which they are, overwriting the old files. The train function seems to work fine but I can't seem to get the accuracy function to work unless I either restart the server, or name the test file differently (like test2.csv for example) which I don't want to. Is there any way to get this to work fine? I believe this wasn't happening while using dffml 0.3.7
Hashim
@mHash1m
@Pratikrocks np :)
Pratik Dey
@Pratikrocks
I was looking at the source of the csv file I came cross the following config params https://github.com/intel/dffml/blob/38ddb55ce0276b4590a9afc89dad47ba8cab1e5e/dffml/source/csv.py#L42 Can anyone who have worked on it give me a jist of what role does the following CSV_SOURCE_CONFIG_DEFAULT_tag CSV_SOURCE_CONFIG_DEFAULT_KEY params does?
John Andersen
@pdxjohnny
@frecklebars Can you point us to some code?
Also, what model are you using?
Any other information that would help us re-produce the issue would be helpful
Shaurya Puri
@spur19
@Pratikrocks key is a parameter that lets us access Records in the CSV file.
Refer https://intel.github.io/dffml/master/api/record.html
Pratik Dey
@Pratikrocks
okay @spur19
John Andersen
@pdxjohnny
Check it you! Things are working in colab! https://colab.research.google.com/drive/13SYqe0yQJjUYdLRvC1yajyFoyNTEASvP#scrollTo=EBy6_FQ1-ZYC I haven't used colab much before, but I think if you go to "File" -> "Save a Copy in Drive" then you can start messing around with it.
We can try tracking google colab realted stuff in this discussion: https://github.com/intel/dffml/discussions/1018 or also on Gitter as usual
Pinging @yashlamba ^
Nitesh yadav
@nitesh585
@pdxjohnny Access denied! while opening the colab's link.
John Andersen
@pdxjohnny
@nitesh585 Thank you! Should be fixed now :)
alex
@frecklebars
@pdxjohnny
from dffml import Features, Feature, train, accuracy, CSVSource
from dffml_model_scikit import LinearRegressionModel
import asyncio


salarymodel = LinearRegressionModel(
    features=Features(
        Feature("Years", int, 1),
        Feature("Expertise", int, 1),
        Feature("Trust", float, 1),
    ),
    predict=Feature("Salary", int, 1),
    directory="tempmodel"
)

irismodel = LinearRegressionModel(
    features=Features(
        Feature("sepal_length", float, 1),
        Feature("sepal_width", float, 1),
        Feature("petal_length", float, 1)
    ),
    predict=Feature("petal_width", float, 1),
    directory="tempmodel"
)

async def main():
    s = open("data/salary.csv", "r")
    salary = s.read()
    s.close()

    st = open("data/salarytest.csv", "r")
    salarytest = st.read()
    st.close()

    ir = open("data/iris.csv", "r")
    iris = ir.read()
    ir.close()

    irt = open("data/iristest.csv", "r")
    iristest = irt.read()
    irt.close()

    # write the salary.csv and salarytest.csv files in the train.csv and test.csv files
    trainfile = open("data/train.csv", "w")
    trainfile.write(salary)
    trainfile.close()

    testfile = open("data/test.csv", "w")
    testfile.write(salarytest)
    testfile.close()

    # train first model 
    await train(salarymodel, "data/train.csv")
    acc = await accuracy(salarymodel, CSVSource(filename="data/test.csv"))
    print("accuracy:", acc)

    # rewrite the train.csv and test.csv files to contain a separate dataset
    trainfile = open("data/train.csv", "w")
    trainfile.write(iris)
    trainfile.close()

    testfile = open("data/test.csv", "w")
    testfile.write(iristest)
    testfile.close()

    # train different model
    await train(irismodel, "data/train.csv")
    # NoRecordsWithMatchingFeatures exception occurs, looking for the features in the first dataset despite changing the contents of the train.csv file
    acc = await accuracy(irismodel, CSVSource(filename="data/test.csv"))
    print("accuracy:", acc)



if __name__ == "__main__":
    asyncio.run(main())
i recreated the issue here
John Andersen
@pdxjohnny
This message was deleted
John Andersen
@pdxjohnny
Ah that's a bug
If you make them different file names it will work. Otherwise we'll fix that in the 0.4.1 release.
John Andersen
@pdxjohnny
It's because the CSV source is keeping files in memory, key'd off of their filename. We have an issue somewhere to stop doing this...
Pratik Dey
@Pratikrocks
If I am supposed to install a module in dffml (say openpyxl) then is it sufficient to mention it in requirements-dev.txt and then run the cmd pip install -e .[dev] dffml service dev install

``
from openpyxl import Workbook, load_workbook
ModuleNotFoundError: No module named 'openpyxl'
^Z

```

Pratik Dey
@Pratikrocks
Sectional test cases are pssing as expected but the overall test are not pssing
Yash Lamba
@yashlamba

@Pratikrocks dffml is a package, just like openpyxl. You don't install a module in dffml and for using another package, you don't need to mention it in requirements unless it's a dependency (a package required for the dependent package to work).

Now for using openpyxl, just try pip install openpyxl.

Another thing, since you are using -e to install dffml, you won't need to run pip install -e .[dev] again until a core requirement is changed.

Pratik Dey
@Pratikrocks
Hi @yashlamba I already did pip3 install openpyxl
(.venv) pratik@pratik-HP-Pavilion-Notebook:~/dffml$ pip3 install openpyxl
Requirement already satisfied: openpyxl in ./.venv/lib/python3.8/site-packages (3.0.6)
Requirement already satisfied: et-xmlfile in ./.venv/lib/python3.8/site-packages (from openpyxl) (1.0.1)
Requirement already satisfied: jdcal in ./.venv/lib/python3.8/site-packages (from openpyxl) (1.4.1)
But on running overall test cases am getting err
Traceback (most recent call last):
  File "/usr/lib/python3.8/runpy.py", line 185, in _run_module_as_main
    mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
  File "/usr/lib/python3.8/runpy.py", line 144, in _get_module_details
    return _get_module_details(pkg_main_name, error)
  File "/usr/lib/python3.8/runpy.py", line 111, in _get_module_details
    __import__(pkg_name)
  File "/home/pratik/dffml/dffml/__init__.py", line 69, in <module>
    for import_name, module in modules(root, package_name, skip=skip):
  File "/home/pratik/dffml/dffml/__init__.py", line 30, in modules
    yield import_name, importlib.import_module(import_name)
  File "/usr/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "/home/pratik/dffml/dffml/source/xlsx.py", line 1, in <module>
    from openpyxl import Workbook, load_workbook
ModuleNotFoundError: No module named 'openpyxl'
Pratik Dey
@Pratikrocks
@yashlamba @pdxjohnny The above imports are failing in the CI too , https://github.com/intel/dffml/pull/1015/checks?check_run_id=1994391075 (I just imported the openpyxl in one of the file in my one PR just for the sake of testing weather its passing the CI's or not )
Yash Lamba
@yashlamba
In the above cases, I think the python environment is different.
Hashim
@mHash1m
I'm unable to run ci tests locally. I've tried a few things but to no avail.
Does anybody know a fix?
++ [[ x. == \x\f\e\a\t\u\r\e\/\g\i\t ]]
++ [[ x. == \x\o\p\e\r\a\t\i\o\n\s\/\d\e\p\l\o\y ]]
++ [[ x. == \x\. ]]
++ curl -sSL https://github.com/XAMPPRocky/tokei/releases/download/v9.1.1/tokei-v9.1.1-x86_64-unknown-linux-gnu.tar.gz
++ tar xvz -C /tmp/tmp.Vq0UaZ2vwk/.local/bin/
tokei
++ sudo apt-get update
./.ci/deps.sh: line 48: sudo: command not found
++ [[ x. == \x\s\o\u\r\c\e\/\m\y\s\q\l ]]
++ [[ x. == \x\. ]]
++ curl -fsSL https://download.docker.com/linux/ubuntu/gpg
++ sudo apt-key add -
./.ci/deps.sh: line 53: sudo: command not found
(23) Failed writing body
John Andersen
@pdxjohnny
@Pratikrocks You need to add the dependency to the install_requires list in setup.cfg https://intel.github.io/dffml/master/tutorials/sources/complex.html#register-your-source
Pratik Dey
@Pratikrocks
I actually created a file(xlsx.py) in the dffml/source just like we are currently having csv.py and then a single tests/source/test_xlsx.py file
Will it not work like this or we need to explicily create the pakage for it?
@pdxjohnny
John Andersen
@pdxjohnny
@Pratikrocks We should create it as a new plugin
cd /path/to/top/of/dffml/git/repo
cd source
dffml service dev create source dffml-source-xlsx
mv dffml-source-xlsx xlsx
John Andersen
@pdxjohnny
$ tree source/ -L 1
source/
├── xlsx
└── mysql

2 directories, 0 files
Pratik Dey
@Pratikrocks
Okay @pdxjohnny ✌️
Shaurya Puri
@spur19
@pdxjohnny Sorry, I won't be able to make it to the meeting today (I have a couple of tests tomorrow)
I'm working on the Dataframe Source and should hopefully have an update for you by the end of the week.