Hi @patil-suraj - you could probably concatenate <document text>, <query> and <answer> with a special token, and mask the loss to only consider the answer tokens. So ultimately the target would look like <document text><sep1><query><sep2><answer> and set the loss mask to only operate on the answer tokens. Does it make sense? @nkitaev is making a Reformer encoder and things will look like seq2seq with that.
Also scroll a little up where we discuss almost the same thing wrt summarization and see what Lukasz had to say as well
Just wanted to chime in to say we are really excited to be making use of Trax in the Project Clarify (https://github.com/projectclarify/clarify) codebase and would like to invite anyone with experience with Trax or Jax and interest in mentoring to come give a tutorial session at one of our upcoming hackathon/training days (Jan 25th and Feb 29): https://forms.gle/oFWkN7UuAxS7NUGJ9 Especially you @afrozenator, didn't have your email to send you an invite. Looking forward to adding value to Trax as we get up to speed with using it.
Right now
IDS and PAD_AMOUNT are global constants that are constructed by loading a txt file of Crime and Punishment. You can instead load multiple txt files, apply the tokenizer, and then generate corresponding token ids for each one
Also I've recently came across Trax and it seems very interesting and clean. But I'm wondering why do we need another DL library when we already have TF2.0, also what are the advantages of Trax over TF2.0 and what are the design goals for Trax. @afrozenator @lukaszkaiser would like to know your thoughts on this
Hi, thank you for making Reformer and Trax available.
I have a question regarding the TPU Crime and Punishment example. The language model obviously learns made-up words - scandlchedness , raggong, innatummed , quisten... Some great words there, but...
Is this an artifact of the hashing, or what do you think causes it?
It's very close to a character-level language model (only 320 basic tokens -- just enough to cover characters, character pairs, and maybe a few frequent words). It's also only training for 600 steps with minimal regularization.
For comparison, BERT trains for 1 million steps, with a fair bit of dropout, on more diverse data
One more question, if I may - are there any examples using Reformer for question answering?
I'm also very interested in using the Reformer for text generation. What would be your advice for feeding in textual data which is longer than the example given in the Colab? So far I ran into errors trying to modify the architecture even slightly (to test the limit of the TPU memory), and went for the option of selecting a random slice of the same length as the original input, even dropping the padding mechanism entirely. Is it how you would do it on a large corpus?
I'm also curious to know how easy it would be to train on several TPUs: would it work out of the box?
def my_inputs(n_devices):
while True:
file = random.choice(os.listdir('files'))
with GFile('/files/' + file) as f:
text = f.read()
IDS = TOKENIZER.EncodeAsIds(text)
As long as the total batch size is a multiple of the number of devices (8), the code will work. Half a million tokens is close to the limit of what fits in 8GB, so if you increase the batch size you'll almost certainly need to shorten the length
@NGamma The parameters of
MultifactorSchedule control the learning rate schedule, which only affects how long training takes and not how much memory is used. You can try running with a little more warmup steps, and more steps_per_cycle in the cyclic cosine schedule.
TypeError: reshape total size must be unchanged, got new_sizes (1, 281525, 256) for shape (1, 512, 1024, 256).
Any ideas, TYIA!
Storing a position embedding vector for each of 0.5M positions is too wasteful, so the approach is to pretend that the text has 2D 512x1024 shape, and concatenate an "x-embedding" and a "y-embedding"
@dimeldo You can try modifying the text generation demo. Redefining
my_inputs will let you feed in your own data, and you can tune the model hyperparameters as well
The problem is that output logits are size (281K, 16K), which is more than 8GB. They need to be chunked to fit in memory.
This is on my agenda because I'm dealing with the exact same problem with MT (vocab size 32K)
@nkitaev curious whether there are plans to scale up the text Reformer model and release a pre-trained model a. la. BERT / RoBERTa, etc? Is the primary concern availability of appropriate corpora to train on where long-term context is useful in reducing an MLM loss?
I'm dealing with a domain where I have a strong indicator that context more than 512 tokens away (long-form scanned documents) is useful and am interested in building off of the work of you and your collaborators.
I'm dealing with a domain where I have a strong indicator that context more than 512 tokens away (long-form scanned documents) is useful and am interested in building off of the work of you and your collaborators.
@patil-suraj : to clarify the TF2 question: some lower-level things are quite hard to do in TF. One of them is really memory-efficient reversible layers, hashing is not trivial too. Trax in general can run with the TF2 backend (and, e.g., Trax Transformer runs using TF), but some things we needed in Reformer were just very hard to do without JAX. That's why we went with JAX and it's been working really well for us!
@madisonmay : We'd really like to train on a larger data-set and release a model then. I think currently we're thinking of using the C4 dataset, what would you think? It's not very long context though...
@patil-suraj @nkitaev the error I encountered first had indeed to do with shapes, and if I remember correctly adding multiples of 256 to 1024 before multiplying with 512 worked.... but ran out of memory. The Colab example fits a single TPU memory really well, and it seems way easier to work with this input length in this context.
@lukaszkaiser thanks for the reply, and glad to hear you do have plans to train on a larger dataset! Barring introducing a new dataset, it seems like C4 or one of its subsets is one of the few decent options out there. The T5 paper also benchmarked on Toronto Book Corpus + Wikipedia -- I imagine because of TBC you'd have much longer avg. sequence length at the expense of operating over a much narrower domain.
I'm sure there would be serious interest from industry in something trained on the text of long-form PDFs from Common Crawl and it does feel like there is long-term information to exploit there (section headers + subheaders, page headers, references to defined terms and other clauses in legal docs, etc.), but that comes with the added warts of having to deal with poor OCR, poor detected reading order, and all sorts of other warts to be sorted out in preprocessing that mean it's probably more trouble than it's worth.