AttributeError: Can't get attribute 'DocvecsArray' on <module 'gensim.models.doc2vec' from '/anaconda3/lib/python3.6/site-packages/gensim/models/doc2vec.py'>
it seems to pop up when I'm trying to load a pretrained model from the following project: https://github.com/sb1992/NETL-Automatic-Topic-Labelling-
the gensim GitHub issue that I found seems to suggest that the error should've been fixed in a later version (I have the latest version of gensim currently)
any help would be greatly appreciated!
Did someone encountred this issue before ?
@eugines that's an interesting project! I was not aware of that. Tweeted it out now :)
Regarding your question: there has been some refactoring in the past 2-3 releases. We tried to maintain backward compatibility, but if NETL used some older version, there may have been an issue. @gojomo do you know whether the
DocvecsArray class moved (and where)?
If so, we should at least keep an alias in the old location, to maintain backward compatibility.
Do we have any convenience function (in gensim) for getting a word vector's length? (as a proxy for its frequency in the data the embedding was trained over)
Of course this question is motivated by laziness....
@phalexo thanks for being agressive though
Now ignoring your snarky comment, I'll post another question for others if they care to respond
I am probably not getting the point of
Don't people typically want to deal with sparse matrices in language processing? e.g. when training classification models over sparse language data; for example you typically can't efficiently train/fit a model on a large bag-of-wordish dense matrix in sklearn, without huge memory. What am I missing about it?
corpus2dense().Don't people typically want to deal with sparse matrices in language processing? e.g. when training classification models over sparse language data; for example you typically can't efficiently train/fit a model on a large bag-of-wordish dense matrix in sklearn, without huge memory. What am I missing about it?
Greetings chat, it's nice to have joined you. I have a relative simple question I suppose. I want to use tf-idf , executed the example in comments and successfully the code returned a vector with the index of the word and the probability, how can I map the index of that word to the actual word? is there a function I am not aware of ? Can I implement it somehow ?
@matanster for NumPy vectors , you can get their length using
vector.shape. See the # numpy vector of a word example at https://radimrehurek.com/gensim/models/keyedvectors.html#gensim.models.keyedvectors.KeyedVectors
@matanster regarding
corpus2dense: it's true NLP produces sparse matrices using the bag-of-words representation. But some later transformations, such as LSI, transform the sparse vectors into a dense space. So using a dense representation actually saves you memory (less overhead than representing a dense matrix using sparse structures).
Plus, there are many other external tools in the Data Science ecosystem that still cannot handle sparse inputs. So you have to export/supply a dense structure => need
corpus2dense. Even scikit-learn didn't have sparse support for a while ;)
@estathop converting between words and ids is the job of the Dictionary: https://radimrehurek.com/gensim/corpora/dictionary.html
And see also the Tutorials: https://radimrehurek.com/gensim/tutorial.html
import gensim.downloader as api
from gensim.models import TfidfModel
from gensim.corpora import Dictionary
dataset = api.load("fake-news")
dct = Dictionary(dataset) # fit dictionary
corpus = [dct.doc2bow(line) for line in dataset] # convert corpus to BoW format
model = TfidfModel(corpus) # fit model
vector = model[corpus[0]] # apply model to the first corpus document
the vector seems to be empty and every element in corpus seems to be the same
@piskvorky and I do not envy those who need to convert a big corpus into a dense rep with
corpus2dense, I had it overflow the memory of a large machine on just 250,000 documents .....
Then again, it was a bag-of-words drill, with word vectors it might be harmless
Hi guys. I am trying to use gensim fasttext library to train custom word embeddings and then load it into a network. However, the embedding bins created by gensim seem not to work while facebook github version bin does. Why? Because whenever I use pretrained bin by gensim my loss does not drop, it stays the same every single time disregarding the different params I use for fasttext model. Facebook version gives me a x2 drop loss. I wonder if I use gensim.FastText correctly:
def read_dataset(fpath):
with open(fpath, "r") as f:
for line in f:
# <sentence creation logic>
yield sentence
model = FastText(min_count=1, size=50, window=5, workers=8, sg=1, word_ngrams=1, min_n=3, max_n=6, iter=5, negative=0)
model.build_vocab(read_dataset(args.fpath))
model.train(read_dataset(args.fpath), total_examples=model.corpus_count, epochs=model.iter)
model.save("custom_model")
word^
the antonym
Hey, I have a question, I've used gensim for a while now and was using the infer_vector for an application I wrote and started noticing vastly different values for the document vector (between dm=1 and dm=0). This is in version 3.5.0 but in all versions prior, the results are very different, I checked the logs and there was a learning rate bug fix in 3.5, could this be the reason for this vastly different values in document vectors?
@PranavSubramani_twitter this gitter is rarely visited by the Gensim devs. you may have a better chance at the mailing list, https://groups.google.com/forum/#!forum/gensim
is it possible to get the most similar words for some unknown word which might not have existed in the dictionary before?
