Is there anything that shows what different layers of an NLP model learns, as well as Zeiler and Fergus showed what different layers of an image model learns? arxiv.org/abs/1311.2901

I'm looking for something to help my students understand model structure and fine-tuning for pretrained NLP models.

@jeremyphoward For LSTMs I really liked visualisations in Karpathy's "unreasonable effectiveness" blogpost and paper: karpathy.github.io/2015/05/21/rnn…

@jeremyphoward Could be relevant (off the top of my head): Liu et al., 2019 (arxiv.org/pdf/1903.08855…), Peters et al., 2018 (arxiv.org/pdf/1808.08949…), Voita et al., 2019 (arxiv.org/pdf/1909.01380…).

@jeremyphoward I’m currently preparing a paper on this, I would be happy to find some time to talk to you about it this week or next.

@jeremyphoward great post about an emnlp paper, by elena voita: "Evolution of Representations in the Transformer" lena-voita.github.io/posts/emnlp19_…

@jeremyphoward What came to mind was this by @ch402, @catherineols, et al.: transformer-circuits.pub/2021/framework…

@jeremyphoward There was the work on CLIP of @ch402 and collaborators distill.pub/2021/multimoda…

@jeremyphoward Just found the implementation on Github: github.com/tetrachrome/su…

@jeremyphoward I think this could be useful github.com/jessevig/bertv…. It is referred in this book amazon.com/Natural-Langua…

@jeremyphoward The logit lens work from nostalgebraist is really nice: lesswrong.com/posts/AcKRB8wD… Work by Anthropic on induction heads was very interesting (transformer-circuits.pub/2022/in-contex…), a piece of software was open-sourced (github.com/anthropics/PyS…) although it's not supported anymore.

@jeremyphoward This is an interesting one by Chen, Olshausen and LeCun arxiv.org/abs/2103.15949

@jeremyphoward arxiv.org/abs/1809.08037 a work lead by @alon_jacovi

@jeremyphoward Hi, we tried to visualise this for QA tasks in 2019. You can find the demo at: visbert.demo.datexis.com

@jeremyphoward Visualizing attention heads for transformers have always been a helpful way for me to reason about the intuition behind attention-based networks. aclanthology.org/P19-3007/

@jeremyphoward If any of recent works in AI can relate itself to (sub-)symbolic inference, that must be anatomy of a learned NLP model.

@jeremyphoward jalammar.github.io/hidden-states/ this might be helpful

@jeremyphoward BertViz

@jeremyphoward RNN based architectures (for language) and layered CNN ones (images) learn differently. Dont know if the learning can be human interpretable when broken down.