首先可以给出a mathematical proof of a sufficient condition for vanishing sensitivity in vanilla RNNs: (link)，总结起来就是权重矩阵的二范数不能太小，由此可以引出一个weight initialization的技术，也是上面link中提到的，用来缓解一下梯度消失的问题。
To ensure the integrity of our messages in the real world, we write them down. Writing is an incremental change that can be additive (pen on paper) or subtractive (carving in rock), and which remains unchanged absent outside interference. In LSTMs, everything is written down and, assuming no interference from other state units or external inputs, carries its prior state forward.
Practically speaking, this means that any state changes are incremental, so that
但是在实际的问题中，仅仅是把所有变化以增量的方式记录也是不够的：
The fundamental challenge: Uncontrolled and uncoordinated writing.
Uncontrolled and uncoordinated writes, particularly at the start of training when writes are completely random, create a chaotic state that leads to bad results and from which it can be difficult to recover.
事实证明效果也确实不好，因为我们想要模型做到对于输入和state传递的信息有特征选择和记忆保持的功能，但实际还是会出现“input weight conflict”, “output weight conflict”, the “abuse problem”, and “internal state drift”这几个问题（这几个术语来自lstm原始论文(link)），下面我搬了原论文的解释。
Input weight conflict: For simplicity, let us focus on a single additional input weight . Assume that the total error can be reduced by switching on unit j in response to a certain input and keeping it active for a long time (until it helps to compute a desired output). Provided i is nonzero, since the same incoming weight has to be used for both storing certain inputs and ignoring others, will often receive conflicting weight update signals during this time (recall that j is linear). These signals will attempt to make participate in (1) storing the input (by switching on j) and (2) protecting the input (by preventing j from being switched off by irrelevant later inputs). This conflict makes learning difficult and calls for a more context-sensitive mechanism for controlling write operations through input weights.
Output weight conflict: Assume j is switched on and currently stores some previous input. For simplicity, let us focus on a single additional outgoing weight . The same has to be used for both retrieving j’s content at certain times and preventing j from disturbing k at other times. As long as unit j is nonzero, will attract conflicting weight update signals generated during sequence processing. These signals will attempt to make participate in accessing the information stored in j and—at different times—protecting unit k from being perturbed by j. For instance, with many tasks there are certain short time-lag errors that can be reduced in early training stages. However,at later training stages, j may suddenly start to cause avoidable errors in situations that already seemed under control by attempting to participate in reducing more difficult long-time-lag errors. Again, this conflict makes learning difficult and calls for a more context-sensitive mechanism for controlling read operations through output weights.
Input and output weight conflicts are not specific for long time lags; they occur for short time lags as well. Their effects, however, become particularly pronounced in the long-time-lag case. As the time lag increases, stored information must be protected against perturbation for longer and longer periods, and, especially in advanced stages of learning, more and more already correct outputs also require protection against perturbation
defforward(self, x, h_prev_t): """GRU cell forward. Input ----- x: (input_dim) observation at current time-step. h_prev_t: (hidden_dim) hidden-state at previous time-step. Returns ------- h_t: (hidden_dim) hidden state at current time-step. """ self.x = x self.hidden = h_prev_t # Add your code here. # Define your variables based on the writeup using the corresponding # names below.
assert self.r.shape == (self.h,) assert self.z.shape == (self.h,) assert self.n.shape == (self.h,) assert h_t.shape == (self.h,) # h_t is the final output of you GRU cell.
return h_t
defbackward(self, delta): """GRU cell backward. This must calculate the gradients wrt the parameters and return the derivative wrt the inputs, xt and ht, to the cell. Input ----- delta: (hidden_dim) summation of derivative wrt loss from next layer at the same time-step and derivative wrt loss from same layer at next time-step. Returns ------- dx: (1, input_dim) derivative of the loss wrt the input x. dh_prev_t: (1, hidden_dim) derivative of the loss wrt the input hidden h. """ # 1) Reshape self.x and self.hidden to (input_dim, 1) and (hidden_dim, 1) respectively # when computing self.dWs... # 2) Transpose all calculated dWs... # 3) Compute all of the derivatives # 4) Know that the autograder grades the gradients in a certain order, and the # local autograder will tell you which gradient you are currently failing.
# ADDITIONAL TIP: # Make sure the shapes of the calcu # lated dWs and dbs match the # initalized shapes accordingly
sys.path.append("mytorch") from gru_cell import * from linear import *
classCharacterPredictor(object): """CharacterPredictor class. This is the neural net that will run one timestep of the input You only need to implement the forward method of this class. This is to test that your GRU Cell implementation is correct when used as a GRU. """
def__init__(self, input_dim, hidden_dim, num_classes): super(CharacterPredictor, self).__init__() """The network consists of a GRU Cell and a linear layer.""" self.rnn = GRUCell(input_dim, hidden_dim) # TODO self.projection = Linear(hidden_dim, num_classes) # TODO self.projection.W = np.random.rand(num_classes, hidden_dim)
defforward(self, x, h): """CharacterPredictor forward. A pass through one time step of the input Input ----- x: (feature_dim) observation at current time-step. h: (hidden_dim) hidden-state at previous time-step. Returns ------- logits: (num_classes) hidden state at current time-step. hnext: (hidden_dim) hidden state at current time-step. """ hnext = self.rnn.forward(x, h) # TODO # self.projection expects input in the form of batch_size * input_dimension # Therefore, reshape the input of self.projection as (1,-1) logits = self.projection.forward(hnext.reshape(1, -1)) # TODO logits = logits.reshape(-1,) # uncomment once code implemented return logits, hnext
definference(net, inputs): """CharacterPredictor inference. An instance of the class defined above runs through a sequence of inputs to generate the logits for all the timesteps. Input ----- net: An instance of CharacterPredictor. inputs: (seq_len, feature_dim) a sequence of inputs of dimensions. Returns ------- logits: (seq_len, num_classes) one per time step of input.. """ # This code should not take more than 10 lines. logits = [] inputs = np.array(inputs) hidden = np.zeros((net.rnn.h)) for i inrange(inputs.shape[0]): logit, hidden = net.forward(inputs[i], hidden) logits.append(logit)
.png)
