We evaluate on several downstream tasks and outperform multilingual BERT. In addition, we test the effect of the number of iterations using a checkpoint at 850k iterations.pic.twitter.com/5iYYf4RQGI
U tweetove putem weba ili aplikacija drugih proizvođača možete dodati podatke o lokaciji, kao što su grad ili točna lokacija. Povijest lokacija tweetova uvijek možete izbrisati. Saznajte više
We evaluate on several downstream tasks and outperform multilingual BERT. In addition, we test the effect of the number of iterations using a checkpoint at 850k iterations.pic.twitter.com/5iYYf4RQGI
Our gains in performance over multilingual BERT are due to (1) focusing on a single language, (2) improvements in training, but also (3) the quantity and especially quality of our training data which contains not just Wikipedia text, but also a large number of books.pic.twitter.com/wYgOuytb8r
The architecture of Bertje is the same as English BERT_base, but we use improved training objectives presented in recent work: we apply Sentence Order Prediction instead of Next Sentence Prediction, and the Masked Language Model predicts whole words.
Bertje is the latest installment in a series of language-specific BERT models, after French, German, Italian, Finish, and Japanese. Collecting better data and pretraining a monolingual model is clearly worth the effort compared to using multilingual BERT.
We intend to analyze the performance of Bertje in more detail and on more tasks. To be continued!
I just saw the CLIN abstracts for next year and there's a presentation titled "BERT-NL: a set of language models pre-trained on the Dutch SoNaR corpus" by Branden, Dirkson, Verberne, Sappelli, Manh Chu, Stoutjesdijk. Is it different from your implementation?
Yes they are different. Different data and different training setup.
Thanks for providing this model! I did a quick comparison of BERTje and the multilingual cased BERT model using a multi-task learing (multiple scalar weighting) setup that we used for German.
Twitter je možda preopterećen ili ima kratkotrajnih poteškoća u radu. Pokušajte ponovno ili potražite dodatne informacije u odjeljku Status Twittera.