Problem about Inference Time

[Hatins]

Hi @NikolaZubic

I have some trouble testing the inference time. I set both sequence length and batch size equal to 1 and tested the inference time on a single RTX4090 GPU. I found the time for lstm is about 4 ms, while the s5 block takes 7 ms.

I tried to exchange the forward to forward_rnn to evaluate the inference time as you mentioned, but I got the same error as @yuyangpoi reported.

Therefore, I wonder how to test the inference time of S5 block is correctly?