New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

custom allreduce cuda kernel #20703

Open

wangyems wants to merge 13 commits into main from wangye/custom_reduce

Member

wangyems commented May 16, 2024 •

edited

Description

Conditionally route to custom AllReduce kernel when buffer size and gpu numbers meet certain requirements. Otherwise, keep using NCCL's AllReduce.

Motivation and Context

wangyems added 6 commits

May 16, 2024 19:57


          checkin custom reduce

be8e676


          suppress Windows nvcc warnings

49461b1


          fix misspell

bace947


          update

e7d37cb


          rocm

c7471b6


          rocm

435f1d3

wangyems marked this pull request as ready for review

May 17, 2024 04:22

github-advanced-security bot found potential problems

View reviewed changes

onnxruntime/test/python/onnxruntime_test_collective.py Fixed Show fixed Hide fixed

wangyems requested a review from a team as a code owner

May 17, 2024 21:32


          update

9ba3637

wangyems force-pushed the wangye/custom_reduce branch from ead5e90 to 9ba3637 Compare

May 21, 2024 04:20

yuslepukhin reviewed

View reviewed changes

onnxruntime/core/providers/js/operators/conv.h Outdated Show resolved Hide resolved

yuslepukhin reviewed

View reviewed changes

orttraining/orttraining/core/optimizer/compute_optimizer/padding_elimination.cc Outdated Show resolved Hide resolved

yuslepukhin reviewed

View reviewed changes

onnxruntime/contrib_ops/cuda/collective/sharded_moe.h Show resolved Hide resolved

yuslepukhin reviewed

View reviewed changes

onnxruntime/contrib_ops/cuda/collective/custom_reduce_impl.h Outdated Show resolved Hide resolved

yuslepukhin requested changes

View reviewed changes

Member

yuslepukhin left a comment

🕐

yuslepukhin reviewed

View reviewed changes

onnxruntime/contrib_ops/cuda/collective/custom_reduce_impl.h Outdated Show resolved Hide resolved

wangyems added 2 commits

May 22, 2024 21:33


          update

46a1eb7


          Merge branch 'main' of github.com:microsoft/onnxruntime into wangye/c…

ec1c605

…ustom_reduce

tianleiwu reviewed

View reviewed changes

onnxruntime/contrib_ops/cuda/collective/custom_reduce_impl.cu Outdated Show resolved Hide resolved

tianleiwu reviewed

View reviewed changes

onnxruntime/contrib_ops/cuda/collective/custom_reduce_impl.cu Outdated Show resolved Hide resolved

tianleiwu reviewed

View reviewed changes

onnxruntime/contrib_ops/cuda/collective/custom_reduce_impl.cu Outdated Show resolved Hide resolved


          review comments

a478025

wangyems requested review from tianleiwu and yuslepukhin

May 28, 2024 16:21

wangyems added the release:1.18.1 label

tianleiwu previously approved these changes

View reviewed changes


          protect rank_to_experts_start_index_

799f58e

wangyems dismissed tianleiwu’s stale review via

799f58e

May 28, 2024 19:13


          Merge branch 'main' of github.com:microsoft/onnxruntime into wangye/c…

7eabb61

…ustom_reduce

wangyems requested a review from tianleiwu

May 29, 2024 04:52

tianleiwu previously approved these changes

View reviewed changes

snnn previously approved these changes

View reviewed changes

yuslepukhin reviewed

View reviewed changes

onnxruntime/contrib_ops/cuda/collective/ipc_utils.cc Outdated Show resolved Hide resolved


          review comments

9eac1a1

wangyems dismissed stale reviews from snnn and tianleiwu via

9eac1a1

May 30, 2024 23:59

wangyems requested a review from yuslepukhin

May 31, 2024 05:54

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment