training a LLM w/ shifted sparse attention from the scratch?

[we1k]

thanks for the great work. i’m curious about the model performance if replacing GQA with S^2 attention at the beginning, then train the model from the scratch, will there be a degradation in performance?