Feedback about NLP From Scratch: Generating Names with a Character-Level RNN

[guyhoffman]

There is the following issue on this page: https://docs.pytorch.org/tutorials/intermediate/char_rnn_generation_tutorial.html

I might be missing something, but in the code on this tutorial, there is no nonlinearity in the recurring part of the RNN, degrading all of the internal updates to a single linear function.

[pranay-3108]

Hi, I tried running the tutorial locally to understand the issue better

The code runs correctly on my side and the model trains and generates names as expected. While looking through the RNN code I also noticed that the hidden state update is done with a linear layer:

hidden = self.i2h(input_combined)

So there is no explicit non-linearity applied to the hidden state update I’m not sure if this was intentional to keep the example simple for learning purposes, or if adding something like torch.tanh would make the recurrence closer to a typical RNN cell

Just sharing my observation after running the example If I misunderstood something, please to correct me