Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context
Add an identifier
Thank You!
Abstract: Transformers have a potential of learning longer-term dependency, but are
limited by a fixed-length context in the setting of language modeling. We
propose a novel neural architecture Transformer-XL that enables learning
dependency beyond a fixed length without disrupting temporal coherence. It
consists of a segment-level recurrence mechanism and a novel positional
encoding scheme. Our method not only enables capturing longer-term dependency,
but also resolves the context fragmentation problem. As a res ...
0
votes
0
answers
108
views
Generalizing attention length beyond training data length
Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context
Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context
updated 14 months ago by
Admin User
1
•
written 14 months ago by
Dustin
Recent Awards •
All
Invitee to
Yuval Filmus
▴
15
Founding Contributor to
FrankS
▴
55
Invitee to
marco.fellous-asiani
▴
15
Founding Contributor to
Ken
▴
45
Recent Replies
Answer: Implementing an algorithm
by
Andy
46
Use a distibution that's symmetric around the origin and normalize the results so they lie on the sphere. E.g. you can use a Gaussian. He…
Answer: Application of Property RD
by
Dustin
125
It is used to prove the bound in proposition 4.6 which is then used crucially in proposition 4.7 to prove the uniform boundedness of the op…
Traffic: 1 users visited in the last hour