It is somewhat off-topic, but I figured that I would find some power users of search engines around here! I am working on a survey about search engines users and uses. I put together some questions at https://bimestriel.framapad.org/p/6phzfohmhs-9iot?lang=en
Please add relevant questions / topics you would like to see answered :)
I am working on a project where I am using ConceptNet database. I have a question. In the database, near the word in the brackets, there is a pos and a term. For example, Boston (n, location). Where can I find this relation (Boston is a location) in the ConceptNet database? I have looked at IsA relation and it is not the same.
Hi, I am new to conceptnet; Have set up the Amazon AMI, following instructions from here https://github.com/commonsense/conceptnet5/wiki/Running-your-own-copy
And as mentioned here, there is a timeout error issue with using AMI; though I tried out the solution for this problem, as mentioned in the documentation
Here's a command to warm up the entire disk, by accessing every byte of data on the disk:
sudo dd if=/dev/xvda of=/dev/null bs=16M
But it doesn't seem to work, facing issue of missing /dev/xvda directory
Can somebody help me out resolving this? Or any other solution for the timeout issue is welcome!
Error in rule convert_opensubtitles_ft:
jobid: 0
output: data/vectors/fasttext-opensubtitles.h5
RuleException:
CalledProcessError in line 674 of /home/conceptnet5/Snakefile:
Command 'set -euo pipefail; CONCEPTNET_DATA=data cn5-vectors convert_fasttext -n 2000000 data/raw/vectors/ft-opensubtitles.vec.gz data/vectors/fasttext-opensubtitles.h5' returned non-zero exit status 137.
File "/home/conceptnet5/Snakefile", line 674, in __rule_convert_opensubtitles_ft
File "/usr/lib/python3.6/concurrent/futures/thread.py", line 56, in run
Exiting because a job execution failed. Look above for error message
[Fri Nov 27 21:43:20 2020]
Finished job 183.
11 of 94 steps (12%) done
Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
Complete log: /home/conceptnet5/.snakemake/log/2020-11-27T213538.909921.snakemake.log
[Tue Dec 22 15:02:54 2020]
rule join_propagate:
input: data/vectors/numberbatch-biased.h5.shard0, data/vectors/numberbatch-biased.h5.shard1, data/vectors/numberbatch-biased.h5.shard2, data/v
ectors/numberbatch-biased.h5.shard3, data/vectors/numberbatch-biased.h5.shard4, data/vectors/numberbatch-biased.h5.shard5
output: data/vectors/numberbatch-biased.h5
jobid: 18
resources: ram=24
Unable to open/create file 'data/vectors/numberbatch-biased.h5.shard5'
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/conceptnet/env/bin/cn5-vectors", line 33, in <module>
sys.exit(load_entry_point('ConceptNet', 'console_scripts', 'cn5-vectors')())
File "/usr/lib/python3/dist-packages/click/core.py", line 764, in __call__
return self.main(*args, **kwargs)
File "/usr/lib/python3/dist-packages/click/core.py", line 717, in main
rv = self.invoke(ctx)
File "/usr/lib/python3/dist-packages/click/core.py", line 1137, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/lib/python3/dist-packages/click/core.py", line 956, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/lib/python3/dist-packages/click/core.py", line 555, in invoke
return callback(*args, **kwargs)
File "/home/conceptnet/conceptnet5/conceptnet5/vectors/cli.py", line 339, in run_join_shard_files
join_shards(filename, nshards, sort=sort)
File "/home/conceptnet/conceptnet5/conceptnet5/vectors/retrofit.py", line 58, in join_shards
shard = load_hdf(output_filename + '.shard%d' % i)
File "/home/conceptnet/conceptnet5/conceptnet5/vectors/formats.py", line 21, in load_hdf
return pd.read_hdf(filename, 'mat', encoding='utf-8')
File "/home/conceptnet/env/lib/python3.8/site-packages/pandas/io/pytables.py", line 389, in read_hdf
store = HDFStore(path_or_buf, mode=mode, errors=errors, **kwargs)
File "/home/conceptnet/env/lib/python3.8/site-packages/pandas/io/pytables.py", line 553, in __init__
self.open(mode=mode, **kwargs)
File "/home/conceptnet/env/lib/python3.8/site-packages/pandas/io/pytables.py", line 729, in open
raise IOError(str(err)) from err
OSError: HDF5 error back trace
File "H5F.c", line 509, in H5Fopen
unable to open file
File "H5Fint.c", line 1400, in H5F__open
unable to open file
File "H5Fint.c", line 1709, in H5F_open
unable to read root group
File "H5Groot.c", line 239, in H5G_mkroot
can't check if symbol table message exists
File "H5Omessage.c", line 883, in H5O_msg_exists
unable to protect object header
File "H5Oint.c", line 1100, in H5O_protect
unable to load object header
File "H5AC.c", line 1625, in H5AC_protect
H5C_protect() failed
File "H5C.c", line 2362, in H5C_protect
can't load entry
File "H5C.c", line 6712, in H5C_load_entry
incorrect metadatda checksum after all read attempts
File "H5Ocache.c", line 219, in H5O__cache_get_final_load_size
can't deserialize object header prefix
File "H5Ocache.c", line 1231, in H5O__prefix_deserialize
bad object header version number
End of HDF5 error back trace
Unable to open/create file 'data/vectors/numberbatch-biased.h5.shard5'
5 replies
Hi,
Anybody getting this error??
Please, it would be great if anyone can help
return _wf_cache[args]
KeyError: ('##', 'ja', 'best', 0.0)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/root/anaconda3/bin/cn5-vectors", line 11, in <module>
load_entry_point('ConceptNet', 'console_scripts', 'cn5-vectors')()
File "/root/anaconda3/lib/python3.7/site-packages/click/core.py", line 722, in call
return self.main(args, kwargs)
File "/root/anaconda3/lib/python3.7/site-packages/click/core.py", line 697, in main
rv = self.invoke(ctx)
File "/root/anaconda3/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/root/anaconda3/lib/python3.7/site-packages/click/core.py", line 895, in invoke
return ctx.invoke(self.callback, ctx.params)
File "/root/anaconda3/lib/python3.7/site-packages/click/core.py", line 535, in invoke
return callback(args, *kwargs)
File "/dados/conceptnet5/conceptnet5/vectors/cli.py", line 288, in run_miniaturize
mini = miniaturize(frame, other_vocab=othervocab, k=k)
File "/dados/conceptnet5/conceptnet5/vectors/miniaturize.py", line 51, in miniaturize
term for term in frame.index if '' not in term and termfreq(term) >= 1e-8
File "/dados/conceptnet5/conceptnet5/vectors/miniaturize.py", line 51, in <listcomp>
term for term in frame.index if '' not in term and term_freq(term) >= 1e-8
File "/dados/conceptnet5/conceptnet5/vectors/miniaturize.py", line 25, in term_freq
return wordfreq.word_frequency(term, lang)
File "/root/anaconda3/lib/python3.7/site-packages/wordfreq/init.py", line 301, in word_frequency
_wf_cache[args] = _word_frequency(args)
File "/root/anaconda3/lib/python3.7/site-packages/wordfreq/init.py", line 244, in _word_frequency
tokens = lossy_tokenize(word, lang)
File "/root/anaconda3/lib/python3.7/site-packages/wordfreq/tokens.py", line 313, in lossy_tokenize
tokens = tokenize(text, lang, include_punctuation, external_wordlist)
