piskvorky/gensim

Topic Modelling for Humans

Matan Shenhav

@ixxie

yeah that's what I am starting to see.... better to try as many combinations as possible

Matan Shenhav

@ixxie

@menshikh-iv its a bit unclear how FastText is used to vectorize documents; the API reference mentions how to get word vector but not document vectors

same with LSI

oh with LSI I see now I can use the [ ] method to get the vector embedding

Ivan Menshikh

@menshikh-iv

LSI - embeddings for document

you can use model[['my', 'document']]

for FastText - infer vector for each word and calculate vector of document as average of word-vectors

Matan Shenhav

@ixxie

cheers @menshikh-iv

Matan Shenhav

@ixxie

@menshikh-iv what kind of corpus does LSI expect? I tried supplying a list of token lists and I get a value error

Ivan Menshikh

@menshikh-iv

@ixxie Corpus in bag-of-words format, look at tutorial https://github.com/RaRe-Technologies/gensim/blob/develop/docs/notebooks/Topics_and_Transformations.ipynb

@ixxie

from gensim.corpora import Dictionary
from gensim.models import LsiModel

data = [["a", "a", "b"], ["c", "d"]]
dictionary = Dictionary(data)
corpus = [dictionary.doc2bow(doc) for doc in data]

model = LsiModel(corpus, id2word=dictionary)
list(model[corpus])  # [[(0, 2.236067977499789)], [(1, -1.4142135623730951)]]

Matan Shenhav

@ixxie

@menshikh-iv thank you very much ^^

Ivan Menshikh

@menshikh-iv

@ixxie :+1:

phalexo

@phalexo

I have not followed Gensim threads for a while, was there ever progress made in porting Doc2Vec to run on GPUs?

I recently retrained Doc2Vec with a pretty large corpus around 30 million documents (about 1 page long on the average). I used 20 epochs and it took like 2.5-3 days. How many epochs are usually good enough? Thanks.

Ivan Menshikh

@menshikh-iv

@phalexo no, this doesn't parallelized on GPU in effective way (on CPU this already fast).
About number of epoch - you can evaluate with upstream task (for what you use Doc2Vec), if you want to receive better result - you can update model with one more epoch.

phalexo

@phalexo

Do you have any insight why the algo is so resistant to GPU porting? Where is the bottleneck? Communications between GPUs? Any algo that produces similar embeddings but may lend itself to better parallel execution?

Matan Shenhav

@ixxie

@menshikh-iv - I followed your approach for LSI but I get some empty elements in the resulting list

Ivan Menshikh

@menshikh-iv

@ixxie how looks input bag-of-words?

Matan Shenhav

@ixxie

@menshikh-iv a list of lists of strings, as you suggested it should

none of them empty

but they do vary in length

Ivan Menshikh

@menshikh-iv

@ixxie are you sure? maybe after conversion to bag-of-word format some documents is empty (because part of tokens was filered)

you don't pass list of list of strings to model directly, you pass list of list of (int, int)

Matan Shenhav

@ixxie

@menshikh-iv:

dictionary = Dictionary(docs)
corpus = [ dictionary.doc2bow(doc) for doc in docs ]
model = LsiModel(corpus, id2word=dictionary,  num_topics =10)
for doc in docs:
    if len(doc) ==0:
        print("doc: ",doc)
vecs = []
for doc in list(model[corpus]):
    x = [ vec[1] for vec in doc ]
    if len(x) == 0:
        print("vec: ", x)
    vecs.append(x)

vecs = np.array(vecs, float)

this produces:

vec:  []
vec:  []
vec:  []
vec:  []

I tried to follow your instructions as best I could

Matan Shenhav

@ixxie

@menshikh-iv we have discovered this error seems to occur precisely when the document is a token that occurs only once in the corpus

Ivan Menshikh

@menshikh-iv

@ixxie LSI filtered some values if this very low (almost zero), for this reason you can receive an empty vectors

Matan Shenhav

@ixxie

is there a way to change the threshold @menshikh-iv ?

Ivan Menshikh

@menshikh-iv

@ixxie no way for [corpus] (also, eps=1e-9, this have no sense)

Matan Shenhav

@ixxie

so there is an alternative approach to [corpus]?

Ivan Menshikh

@menshikh-iv

probably only "mokey-patch" full2sparce function https://github.com/RaRe-Technologies/gensim/blob/develop/gensim/models/lsimodel.py#L486 (but believe me, you do not need it, these values are filtered for a reason, too small values like a noize, this doesn't contain any useful information).

Matan Shenhav

@ixxie

@menshikh-iv so what vectors do you normally substitute for such missing vectors? If we use all zeros the clustering algorithm will put them together

Ivan Menshikh

@menshikh-iv

@ixxie exactly, better way - exclude this type of vectors (anyway, even if you disable "threshold", clustering algo will "glue" all of this vectors to one cluster).

Matan Shenhav

@ixxie

alright, thanks for all the help @menshikh-iv

phalexo

@phalexo

Does anyone know if there is a mechanism in Doc2Vec model to choose and keep some document (and/or word) vectors fixed while training all others?

Matan Shenhav

@ixxie

@phalexo you could just omit those documents from the training I guess?

phalexo

@phalexo

@ixxie That is not the goal here. I want these documents to be present in the model, but I want other document vectors trained with these constraints in place.

Matan Shenhav

@ixxie

@phalexo I guess I was thinking something like: train constrained docs -> keep vectors -> train other docs

but I am not sure that works for you

or at all xD

Matan Shenhav

@ixxie

I am getting:

AttributeError: 'FastText' object has no attribute 'syn0_vocab_lockf'

does anybody know what that means?

Matan Shenhav

@ixxie

@menshikh-iv - is this a bug due to a transition to a new API in 4.0?

Ivan Menshikh

@menshikh-iv

@ixxie can you share full code

phalexo

@phalexo

@ixxie I know what I want to do already. I need to know how to do it specifically with Gensim Doc2Vec. Models trained on different sets of documents will have no relationship to each other. And any later training will completely mutate vectors for any original documents.

The question is how to keep a certain set of vectors fixed while training.

Matan Shenhav

@ixxie

sure:

docs = [['this', 'is', 'some'], ['this', 'example'], ['example', 'is', 'this']]
model = FastText(workers=ncores, min_count=1)
corpus = { 'this': 1, 'is': 200, 'some': 5, 'example': 1 }
model.build_vocab_from_freq(corpus, corpus_count = len(docs))
model.train(docs, total_examples=model.corpus_count, epochs=model.iter)
vecs = []
for doc in docs:
    docVecs = []
    for token in doc:
        docVecs.append(model[token])
    vecs.append(np.mean(docVecs, axis=0))

Where communities thrive

People

Repo info

Activity