New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Feature] support sequence parallel in head layer and embedding layer #95

Open

1 task done

sunpengsdu opened this issue Mar 20, 2024 · 1 comment

Open

1 task done

[Feature] support sequence parallel in head layer and embedding layer #95

sunpengsdu opened this issue Mar 20, 2024 · 1 comment

Assignees

Labels

Contributor

sunpengsdu commented Mar 20, 2024

Describe the feature

they should not in separated parameter group

Will you implement it?

I would like to implement this feature and create a PR!

sunpengsdu added the enhancement label

sunpengsdu assigned yingtongxiong and huangting4201

Contributor Author

sunpengsdu commented Mar 20, 2024

or at least they should not impact the value of global norm

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment