Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
  • Apr 12 03:33

    pdxjohnny on gh-pages

    docs: Mon Apr 12 03:33:55 UTC 2… (compare)

  • Apr 11 03:25

    pdxjohnny on gh-pages

    docs: Sun Apr 11 03:25:49 UTC 2… (compare)

  • Apr 10 03:26

    pdxjohnny on gh-pages

    docs: Sat Apr 10 03:25:58 UTC 2… (compare)

  • Apr 09 09:06

    pdxjohnny on gh-pages

    docs: Fri Apr 9 09:06:22 UTC 2… (compare)

  • Apr 08 03:48

    pdxjohnny on snyk-fix-98742e62125ff7eb5388f6071fd2b321

    fix: operations/image/Dockerfil… (compare)

  • Apr 08 03:48

    pdxjohnny on snyk-fix-98742e62125ff7eb5388f6071fd2b321

    (compare)

  • Apr 08 03:34

    pdxjohnny on gh-pages

    docs: Thu Apr 8 03:34:30 UTC 2… (compare)

  • Apr 08 02:12

    pdxjohnny on snyk-fix-4bb1b9c4d21e4f4516cf5e3e12ac08c7

    fix: operations/deploy/Dockerfi… (compare)

  • Apr 08 02:12

    pdxjohnny on snyk-fix-4bb1b9c4d21e4f4516cf5e3e12ac08c7

    (compare)

  • Apr 08 01:47

    pdxjohnny on snyk-fix-33ebe7b2b746d659c5dfc3c13083dda2

    fix: dffml/skel/operations/Dock… (compare)

  • Apr 08 01:47

    pdxjohnny on snyk-fix-33ebe7b2b746d659c5dfc3c13083dda2

    (compare)

  • Apr 08 01:24

    pdxjohnny on gh-pages

    docs: Thu Apr 8 01:24:05 UTC 2… (compare)

  • Apr 08 01:08

    pdxjohnny on master

    service: dev: Port scripts/docs… (compare)

  • Apr 08 00:11

    pdxjohnny on snyk-fix-0d54f09ae78d71dd0a0ce473ce7ca45f

    fix: operations/nlp/Dockerfile … (compare)

  • Apr 08 00:11

    pdxjohnny on snyk-fix-0d54f09ae78d71dd0a0ce473ce7ca45f

    (compare)

  • Apr 07 21:48

    pdxjohnny on snyk-fix-2aaa92e9ffe02c2d6cd25d10d023067f

    fix: operations/binsec/Dockerfi… (compare)

  • Apr 07 21:48

    pdxjohnny on snyk-fix-2aaa92e9ffe02c2d6cd25d10d023067f

    (compare)

  • Apr 07 03:26

    pdxjohnny on gh-pages

    docs: Wed Apr 7 03:26:10 UTC 2… (compare)

  • Apr 05 03:48

    pdxjohnny on snyk-fix-f790ca3982aed8c5df308a329cd0f1ec

    fix: operations/image/Dockerfil… (compare)

  • Apr 05 03:48

    pdxjohnny on snyk-fix-f790ca3982aed8c5df308a329cd0f1ec

    (compare)

Hashim
@mHash1m
@Pratikrocks np :)
Pratik Dey
@Pratikrocks
I was looking at the source of the csv file I came cross the following config params https://github.com/intel/dffml/blob/38ddb55ce0276b4590a9afc89dad47ba8cab1e5e/dffml/source/csv.py#L42 Can anyone who have worked on it give me a jist of what role does the following CSV_SOURCE_CONFIG_DEFAULT_tag CSV_SOURCE_CONFIG_DEFAULT_KEY params does?
John Andersen
@pdxjohnny
@frecklebars Can you point us to some code?
Also, what model are you using?
Any other information that would help us re-produce the issue would be helpful
Shaurya Puri
@spur19
@Pratikrocks key is a parameter that lets us access Records in the CSV file.
Refer https://intel.github.io/dffml/master/api/record.html
Pratik Dey
@Pratikrocks
okay @spur19
John Andersen
@pdxjohnny
Check it you! Things are working in colab! https://colab.research.google.com/drive/13SYqe0yQJjUYdLRvC1yajyFoyNTEASvP#scrollTo=EBy6_FQ1-ZYC I haven't used colab much before, but I think if you go to "File" -> "Save a Copy in Drive" then you can start messing around with it.
We can try tracking google colab realted stuff in this discussion: https://github.com/intel/dffml/discussions/1018 or also on Gitter as usual
Pinging @yashlamba ^
Nitesh yadav
@nitesh585
@pdxjohnny Access denied! while opening the colab's link.
John Andersen
@pdxjohnny
@nitesh585 Thank you! Should be fixed now :)
alex
@frecklebars
@pdxjohnny
from dffml import Features, Feature, train, accuracy, CSVSource
from dffml_model_scikit import LinearRegressionModel
import asyncio


salarymodel = LinearRegressionModel(
    features=Features(
        Feature("Years", int, 1),
        Feature("Expertise", int, 1),
        Feature("Trust", float, 1),
    ),
    predict=Feature("Salary", int, 1),
    directory="tempmodel"
)

irismodel = LinearRegressionModel(
    features=Features(
        Feature("sepal_length", float, 1),
        Feature("sepal_width", float, 1),
        Feature("petal_length", float, 1)
    ),
    predict=Feature("petal_width", float, 1),
    directory="tempmodel"
)

async def main():
    s = open("data/salary.csv", "r")
    salary = s.read()
    s.close()

    st = open("data/salarytest.csv", "r")
    salarytest = st.read()
    st.close()

    ir = open("data/iris.csv", "r")
    iris = ir.read()
    ir.close()

    irt = open("data/iristest.csv", "r")
    iristest = irt.read()
    irt.close()

    # write the salary.csv and salarytest.csv files in the train.csv and test.csv files
    trainfile = open("data/train.csv", "w")
    trainfile.write(salary)
    trainfile.close()

    testfile = open("data/test.csv", "w")
    testfile.write(salarytest)
    testfile.close()

    # train first model 
    await train(salarymodel, "data/train.csv")
    acc = await accuracy(salarymodel, CSVSource(filename="data/test.csv"))
    print("accuracy:", acc)

    # rewrite the train.csv and test.csv files to contain a separate dataset
    trainfile = open("data/train.csv", "w")
    trainfile.write(iris)
    trainfile.close()

    testfile = open("data/test.csv", "w")
    testfile.write(iristest)
    testfile.close()

    # train different model
    await train(irismodel, "data/train.csv")
    # NoRecordsWithMatchingFeatures exception occurs, looking for the features in the first dataset despite changing the contents of the train.csv file
    acc = await accuracy(irismodel, CSVSource(filename="data/test.csv"))
    print("accuracy:", acc)



if __name__ == "__main__":
    asyncio.run(main())
i recreated the issue here
John Andersen
@pdxjohnny
This message was deleted
John Andersen
@pdxjohnny
Ah that's a bug
If you make them different file names it will work. Otherwise we'll fix that in the 0.4.1 release.
John Andersen
@pdxjohnny
It's because the CSV source is keeping files in memory, key'd off of their filename. We have an issue somewhere to stop doing this...
Pratik Dey
@Pratikrocks
If I am supposed to install a module in dffml (say openpyxl) then is it sufficient to mention it in requirements-dev.txt and then run the cmd pip install -e .[dev] dffml service dev install

``
from openpyxl import Workbook, load_workbook
ModuleNotFoundError: No module named 'openpyxl'
^Z

```

Pratik Dey
@Pratikrocks
Sectional test cases are pssing as expected but the overall test are not pssing
Yash Lamba
@yashlamba

@Pratikrocks dffml is a package, just like openpyxl. You don't install a module in dffml and for using another package, you don't need to mention it in requirements unless it's a dependency (a package required for the dependent package to work).

Now for using openpyxl, just try pip install openpyxl.

Another thing, since you are using -e to install dffml, you won't need to run pip install -e .[dev] again until a core requirement is changed.

Pratik Dey
@Pratikrocks
Hi @yashlamba I already did pip3 install openpyxl
(.venv) pratik@pratik-HP-Pavilion-Notebook:~/dffml$ pip3 install openpyxl
Requirement already satisfied: openpyxl in ./.venv/lib/python3.8/site-packages (3.0.6)
Requirement already satisfied: et-xmlfile in ./.venv/lib/python3.8/site-packages (from openpyxl) (1.0.1)
Requirement already satisfied: jdcal in ./.venv/lib/python3.8/site-packages (from openpyxl) (1.4.1)
But on running overall test cases am getting err
Traceback (most recent call last):
  File "/usr/lib/python3.8/runpy.py", line 185, in _run_module_as_main
    mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
  File "/usr/lib/python3.8/runpy.py", line 144, in _get_module_details
    return _get_module_details(pkg_main_name, error)
  File "/usr/lib/python3.8/runpy.py", line 111, in _get_module_details
    __import__(pkg_name)
  File "/home/pratik/dffml/dffml/__init__.py", line 69, in <module>
    for import_name, module in modules(root, package_name, skip=skip):
  File "/home/pratik/dffml/dffml/__init__.py", line 30, in modules
    yield import_name, importlib.import_module(import_name)
  File "/usr/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "/home/pratik/dffml/dffml/source/xlsx.py", line 1, in <module>
    from openpyxl import Workbook, load_workbook
ModuleNotFoundError: No module named 'openpyxl'
Pratik Dey
@Pratikrocks
@yashlamba @pdxjohnny The above imports are failing in the CI too , https://github.com/intel/dffml/pull/1015/checks?check_run_id=1994391075 (I just imported the openpyxl in one of the file in my one PR just for the sake of testing weather its passing the CI's or not )
Yash Lamba
@yashlamba
In the above cases, I think the python environment is different.
Hashim
@mHash1m
I'm unable to run ci tests locally. I've tried a few things but to no avail.
Does anybody know a fix?
++ [[ x. == \x\f\e\a\t\u\r\e\/\g\i\t ]]
++ [[ x. == \x\o\p\e\r\a\t\i\o\n\s\/\d\e\p\l\o\y ]]
++ [[ x. == \x\. ]]
++ curl -sSL https://github.com/XAMPPRocky/tokei/releases/download/v9.1.1/tokei-v9.1.1-x86_64-unknown-linux-gnu.tar.gz
++ tar xvz -C /tmp/tmp.Vq0UaZ2vwk/.local/bin/
tokei
++ sudo apt-get update
./.ci/deps.sh: line 48: sudo: command not found
++ [[ x. == \x\s\o\u\r\c\e\/\m\y\s\q\l ]]
++ [[ x. == \x\. ]]
++ curl -fsSL https://download.docker.com/linux/ubuntu/gpg
++ sudo apt-key add -
./.ci/deps.sh: line 53: sudo: command not found
(23) Failed writing body
John Andersen
@pdxjohnny
@Pratikrocks You need to add the dependency to the install_requires list in setup.cfg https://intel.github.io/dffml/master/tutorials/sources/complex.html#register-your-source
Pratik Dey
@Pratikrocks
I actually created a file(xlsx.py) in the dffml/source just like we are currently having csv.py and then a single tests/source/test_xlsx.py file
Will it not work like this or we need to explicily create the pakage for it?
@pdxjohnny
John Andersen
@pdxjohnny
@Pratikrocks We should create it as a new plugin
cd /path/to/top/of/dffml/git/repo
cd source
dffml service dev create source dffml-source-xlsx
mv dffml-source-xlsx xlsx
John Andersen
@pdxjohnny
$ tree source/ -L 1
source/
├── xlsx
└── mysql

2 directories, 0 files
Pratik Dey
@Pratikrocks
Okay @pdxjohnny ✌️
Shaurya Puri
@spur19
@pdxjohnny Sorry, I won't be able to make it to the meeting today (I have a couple of tests tomorrow)
I'm working on the Dataframe Source and should hopefully have an update for you by the end of the week.
John Andersen
@pdxjohnny
Today's Meeting Link: https://meet.google.com/qum-ntwe-aoh
Hashim
@mHash1m
@pdxjohnny please upload the recording whenever you get the time.
Sanjiban Sengupta
@kahanikaar
@pdxjohnny Apologies sir! Wasn't there at the last meeting. Got some medical urgency.
I made a recent PR, do kindly review intel/dffml#1023
Also, I was working for the Darts model, just to mention, The current Darts package version 0.6.1 shows incompatibility with numpy version 1.19.2 and scipy version 1.4.1 which are currently used by packages of dffml, and requires numpy==1.19.0 and scipy==1.5.0. But, the u8darts[all]==0.2.0 seems to work fine with them.
Sudhanshu kumar
@sk-ip
@pdxjohnny , can we download this dataset https://www.ncdc.noaa.gov/cag/city/time-series/USW00023174/tavg/all/1/2020-2021?base_prd=true&begbaseyear=1901&endbaseyear=2000 for getting the temperature of a given city for the ice cream demo and similarly download dataset for other cities as well and merge them into one dataset
John Andersen
@pdxjohnny
It was a little circle button next to XML, just wanted to make sure you saw it
Sudhanshu kumar
@sk-ip
@pdxjohnny , yes saw that
John Andersen
@pdxjohnny
Sweet! :D
John Andersen
@pdxjohnny

This might be interesting for those working on sources, or just anyone who wants to know a bit more about sources and the plugin system in general: https://www.youtube.com/watch?v=79pN4Yy4-A0

Pinging @yashlamba, this is the recording from this morning's meeting. Branch is here: https://github.com/pdxjohnny/dffml/tree/iris_cached_dataset It may not be 100% at the moment

Nitesh yadav
@nitesh585

I getting the error given below while trying to run the HTTP: python example https://intel.github.io/dffml/master/plugins/service/http/python.html.

requests.exceptions.ConnectionError: HTTPSConnectionPool(host='127.0.0.1', port=8000): Max retries exceeded with URL:
 /configure/model/fake/mymodel?model=plugin&model=config (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f99a7278a50>: 
Failed to establish a new connection: [Errno 111] Connection refused'))

To run the example. In one terminal I have started the service by using the command below:

dffml service http server -insecure -cors '*' -addr 0.0.0.0 -port 8080

and in another terminal, I have just simply run the python_example -> "python3 python_example.py"

I am following the right steps or do I need to follow some other way?
John Andersen
@pdxjohnny
The port number you're connecting to HTTPSConnectionPool(host='127.0.0.1', port=8000) and the port you're hosting on is -port 8080
John Andersen
@pdxjohnny
Sorry I've been looking at the example, was in the middle of the iris thing
Nitesh yadav
@nitesh585

Now I have changed the port number - 8080 for both client and server.
But still getting an error simultaneously as server and client side.
Error at client side (Some SSLError may be we need .pem file ) :

requests.exceptions.SSLError: HTTPSConnectionPool(host='127.0.0.1', port=8080): 
Max retries exceeded with url: /configure/model/fake/mymodel?model=plugin&model=config (Caused by SSLError(SSLError(1, '[SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:1091)')))

Error at server side:

  File "/home/nitesh/.local/lib/python3.7/site-packages/aiohttp/web_protocol.py", line 314, in data_received
    messages, upgraded, tail = self._request_parser.feed_data(data)
  File "aiohttp/_http_parser.pyx", line 546, in aiohttp._http_parser.HttpParser.feed_data
aiohttp.http_exceptions.BadStatusLine: 400, message="Bad status line 'invalid HTTP method'"