profile
viewpoint

allenai/PeerRead 306

Data and code for Kang et al., NAACL 2018's paper titled "A Dataset of Peer Reviews (PeerRead): Collection, Insights and NLP Applications"

allenai/propara 72

ProPara (Process Paragraph Comprehension) dataset and models

nikett/pytorch-pretrained-BERT 1

A PyTorch implementation of Google AI's BERT model provided with Google's pre-trained models, examples and utilities.

bhavanadalvi/allennlp 0

An open-source NLP research library, built on PyTorch.

bhavanadalvi/CoreNLP 0

Stanford CoreNLP: A Java suite of core NLP tools.

bhavanadalvi/deep_qa 0

A deep NLP library, based on Keras / tf, focused on question answering (but useful for other NLP too)

bhavanadalvi/nlptools 0

A toolkit that wraps various natural language processing implementations behind a common interface.

bhavanadalvi/nlpweb 0

A demonstration of various NLP tools.

issue openedgoogle-research/text-to-text-transfer-transformer

Continue pre-training or fine-tune?

Hello,

I wanted to get some clarity regarding train vs fine-tune options on existing pertained T5 model. Is there a difference between the two? E.g. train will adjust weights for all layers in the model whereas fine-tune will freeze all other layers and only tune weights in the last layer. In particular, we are interested in 1. Fine continue fine-tuning T5 on our domain corpus. It would be awesome if we have a control over the masking or kinds of queries. 2. Then fine-tune it further on our end task.

It is not clear how to do step-1 here. Should we use fine-tune option for both steps or use train for step-1 and fine-tune for step-2. Also, if we need to use "train" option, then can you please give an example of how the dataset should be formatted and what command options should be provided? Also, how can we control the masking strategies?

created time in 2 months

more