We should document this. Can you open an issue please?
2 replies
What is cat and EOF ??
Eof is end of file delimiter
Hi all can anyone help me why I am getting the following error in the ci test checks https://github.com/intel/dffml/pull/1015/checks?check_run_id=1967979265
I the following pull I created a new test case , for that I created a file in the
test directory called test_styles
And then I inherited the base class
IntegrationCLITestCase
Hey @Pratikrocks , the failing tests on your PR can be fixed by:
1: Adding an entry in the changelog.
2: Using black to style your code as it is the format of the whole code base.
You can find more about these CI tests and fixes in the 'Contributing -> Git and Github -> How to Read the CI' documentation section.
1: Adding an entry in the changelog.
2: Using black to style your code as it is the format of the whole code base.
You can find more about these CI tests and fixes in the 'Contributing -> Git and Github -> How to Read the CI' documentation section.
Hello, I have a flask web app with an upload page where I can upload a train and a test csv file that I use to train a model and then redirect to a "trained" page that shows the accuracy. The issue is that when i go back to the upload page and upload a different set of csv files for a different problem, while the model trains, the accuracy method returns a
NoRecordsWithMatchingFeatures exception, as if the test.csv file was never changed. I checked if the files are being uploaded correctly, which they are, overwriting the old files. The train function seems to work fine but I can't seem to get the accuracy function to work unless I either restart the server, or name the test file differently (like test2.csv for example) which I don't want to. Is there any way to get this to work fine? I believe this wasn't happening while using dffml 0.3.7
I was looking at the
source of the csv file I came cross the following config params https://github.com/intel/dffml/blob/38ddb55ce0276b4590a9afc89dad47ba8cab1e5e/dffml/source/csv.py#L42 Can anyone who have worked on it give me a jist of what role does the following CSV_SOURCE_CONFIG_DEFAULT_tag CSV_SOURCE_CONFIG_DEFAULT_KEY params does?
Also, what model are you using?
Any other information that would help us re-produce the issue would be helpful
@Pratikrocks
Refer https://intel.github.io/dffml/master/api/record.html
key is a parameter that lets us access Records in the CSV file.Refer https://intel.github.io/dffml/master/api/record.html
Check it you! Things are working in colab! https://colab.research.google.com/drive/13SYqe0yQJjUYdLRvC1yajyFoyNTEASvP#scrollTo=EBy6_FQ1-ZYC I haven't used colab much before, but I think if you go to "File" -> "Save a Copy in Drive" then you can start messing around with it.
We can try tracking google colab realted stuff in this discussion: https://github.com/intel/dffml/discussions/1018 or also on Gitter as usual
Pinging @yashlamba ^
@pdxjohnny
from dffml import Features, Feature, train, accuracy, CSVSource
from dffml_model_scikit import LinearRegressionModel
import asyncio
salarymodel = LinearRegressionModel(
features=Features(
Feature("Years", int, 1),
Feature("Expertise", int, 1),
Feature("Trust", float, 1),
),
predict=Feature("Salary", int, 1),
directory="tempmodel"
)
irismodel = LinearRegressionModel(
features=Features(
Feature("sepal_length", float, 1),
Feature("sepal_width", float, 1),
Feature("petal_length", float, 1)
),
predict=Feature("petal_width", float, 1),
directory="tempmodel"
)
async def main():
s = open("data/salary.csv", "r")
salary = s.read()
s.close()
st = open("data/salarytest.csv", "r")
salarytest = st.read()
st.close()
ir = open("data/iris.csv", "r")
iris = ir.read()
ir.close()
irt = open("data/iristest.csv", "r")
iristest = irt.read()
irt.close()
# write the salary.csv and salarytest.csv files in the train.csv and test.csv files
trainfile = open("data/train.csv", "w")
trainfile.write(salary)
trainfile.close()
testfile = open("data/test.csv", "w")
testfile.write(salarytest)
testfile.close()
# train first model
await train(salarymodel, "data/train.csv")
acc = await accuracy(salarymodel, CSVSource(filename="data/test.csv"))
print("accuracy:", acc)
# rewrite the train.csv and test.csv files to contain a separate dataset
trainfile = open("data/train.csv", "w")
trainfile.write(iris)
trainfile.close()
testfile = open("data/test.csv", "w")
testfile.write(iristest)
testfile.close()
# train different model
await train(irismodel, "data/train.csv")
# NoRecordsWithMatchingFeatures exception occurs, looking for the features in the first dataset despite changing the contents of the train.csv file
acc = await accuracy(irismodel, CSVSource(filename="data/test.csv"))
print("accuracy:", acc)
if __name__ == "__main__":
asyncio.run(main())
i recreated the issue here
If you make them different file names it will work. Otherwise we'll fix that in the 0.4.1 release.
``
from openpyxl import Workbook, load_workbook
ModuleNotFoundError: No module named 'openpyxl'
^Z
```
@Pratikrocks dffml is a package, just like openpyxl. You don't install a module in dffml and for using another package, you don't need to mention it in requirements unless it's a dependency (a package required for the dependent package to work).
Now for using openpyxl, just try pip install openpyxl.
Another thing, since you are using -e to install dffml, you won't need to run pip install -e .[dev] again until a core requirement is changed.
Hi @yashlamba I already did
pip3 install openpyxl(.venv) pratik@pratik-HP-Pavilion-Notebook:~/dffml$ pip3 install openpyxl
Requirement already satisfied: openpyxl in ./.venv/lib/python3.8/site-packages (3.0.6)
Requirement already satisfied: et-xmlfile in ./.venv/lib/python3.8/site-packages (from openpyxl) (1.0.1)
Requirement already satisfied: jdcal in ./.venv/lib/python3.8/site-packages (from openpyxl) (1.4.1)
But on running overall test cases am getting err
Traceback (most recent call last):
File "/usr/lib/python3.8/runpy.py", line 185, in _run_module_as_main
mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
File "/usr/lib/python3.8/runpy.py", line 144, in _get_module_details
return _get_module_details(pkg_main_name, error)
File "/usr/lib/python3.8/runpy.py", line 111, in _get_module_details
__import__(pkg_name)
File "/home/pratik/dffml/dffml/__init__.py", line 69, in <module>
for import_name, module in modules(root, package_name, skip=skip):
File "/home/pratik/dffml/dffml/__init__.py", line 30, in modules
yield import_name, importlib.import_module(import_name)
File "/usr/lib/python3.8/importlib/__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "/home/pratik/dffml/dffml/source/xlsx.py", line 1, in <module>
from openpyxl import Workbook, load_workbook
ModuleNotFoundError: No module named 'openpyxl'
@yashlamba @pdxjohnny The above imports are failing in the CI too , https://github.com/intel/dffml/pull/1015/checks?check_run_id=1994391075 (I just imported the
openpyxl in one of the file in my one PR just for the sake of testing weather its passing the CI's or not )
I'm unable to run ci tests locally. I've tried a few things but to no avail.
Does anybody know a fix?
Does anybody know a fix?
++ [[ x. == \x\f\e\a\t\u\r\e\/\g\i\t ]]
++ [[ x. == \x\o\p\e\r\a\t\i\o\n\s\/\d\e\p\l\o\y ]]
++ [[ x. == \x\. ]]
++ curl -sSL https://github.com/XAMPPRocky/tokei/releases/download/v9.1.1/tokei-v9.1.1-x86_64-unknown-linux-gnu.tar.gz
++ tar xvz -C /tmp/tmp.Vq0UaZ2vwk/.local/bin/
tokei
++ sudo apt-get update
./.ci/deps.sh: line 48: sudo: command not found
++ [[ x. == \x\s\o\u\r\c\e\/\m\y\s\q\l ]]
++ [[ x. == \x\. ]]
++ curl -fsSL https://download.docker.com/linux/ubuntu/gpg
++ sudo apt-key add -
./.ci/deps.sh: line 53: sudo: command not found
(23) Failed writing body
@Pratikrocks You need to add the dependency to the
install_requires list in setup.cfg https://intel.github.io/dffml/master/tutorials/sources/complex.html#register-your-source
Will it not work like this or we need to explicily create the pakage for it?
@pdxjohnny