RuntimeError: No such operator tutel_ops::cumsum #190
Comments
|
What about |
|
Thank you. I have tried this, and the new error is reported: |
|
Gotcha, this problem is not from The root cause could be an improper CUDA SDK configuration in your system that makes Tutel fail to build up some components, like missing Suggestion: |
|
Thank you for your suggestion，I'll have a try. |
|
Hi, I also meet similar issues. I attach my installation log (using source installation method) below, and do not see any clues to facilitate identifying the cause. May I ask some further suggestions to find problems? |
Have you tried: |
|
I get this error after setting [1,319]<stderr>:│ /root/.local/lib/python3.9/site-packages/tutel/impls/fast_dispatch.py:108 in │
[1,319]<stderr>:│ update │
[1,319]<stderr>:│ │
[1,319]<stderr>:│ 105 │ │ if self.is_cuda != indices_[0].is_cuda: │
[1,319]<stderr>:│ 106 │ │ │ self.is_cuda = indices_[0].is_cuda │
[1,319]<stderr>:│ 107 │ │ │ if self.is_cuda not in TutelMoeFastDispatcher.kernel_pool: │
[1,319]<stderr>:│ ❱ 108 │ │ │ │ self.func_fwd = jit_kernel.create_forward(self.dtype, │
[1,319]<stderr>:│ 109 │ │ │ │ self.func_bwd_data = jit_kernel.create_backward_data(s │
[1,319]<stderr>:│ 110 │ │ │ │ self.func_bwd_gate = jit_kernel.create_backward_gate(s │
[1,319]<stderr>:│ 111 │ │ │ │ TutelMoeFastDispatcher.kernel_pool[self.is_cuda] = sel │
[1,319]<stderr>:│ │
[1,319]<stderr>:│ /root/.local/lib/python3.9/site-packages/tutel/jit_kernels/sparse.py:21 in │
[1,319]<stderr>:│ create_forward │
[1,319]<stderr>:│ │
[1,319]<stderr>:│ 18 if not is_cuda: [1,319]<stderr>: │
[1,319]<stderr>:│ 19 │ return JitCompiler.generate_cpu_kernel(kernel_type=0) │
[1,319]<stderr>:│ 20 │
[1,319]<stderr>:│ ❱ 21 return JitCompiler.generate_kernel({'dtype': get_kernel_dtype(param_ │
[1,319]<stderr>:│ 22 │ #define __dtype @dtype@ │
[1,319]<stderr>:│ 23 │ │
[1,319]<stderr>:│ 24 │ extern "C" __global__ __launch_bounds__(1024) void execute(__dtype │
[1,319]<stderr>:│ │
[1,319]<stderr>:│ /root/.local/lib/python3.9/site-packages/tutel/impls/jit_compiler.py:40 in │
[1,319]<stderr>:│ generate_kernel │
[1,319]<stderr>:│ │
[1,319]<stderr>:│ 37 │ def generate_kernel(keyword_dict, template): │
[1,319]<stderr>:│ 38 │ for key in keyword_dict: │
[1,319]<stderr>:│ 39 │ │ template = template.replace('@%s@' % key, str(keyword_dict[key] │
[1,319]<stderr>:│ ❱ 40 │ return JitCompiler.create_raw(template) │
[1,319]<stderr>:│ 41 │ │
[1,319]<stderr>:│ 42 │ @staticmethod │
[1,319]<stderr>:│ 43 │ def generate_cpu_kernel(kernel_type): │
[1,319]<stderr>:│ │
[1,319]<stderr>:│ /root/.local/lib/python3.9/site-packages/tutel/impls/jit_compiler.py:29 in │
[1,319]<stderr>:│ create_raw │
[1,319]<stderr>:│ │
[1,319]<stderr>:│ 26 │ def create_raw(source): │
[1,319]<stderr>:│ 27 │ │ torch.cuda.init() │
[1,319]<stderr>:│ 28 │ │ if not hasattr(tutel_custom_kernel, 'inject_source'): │
[1,319]<stderr>:│ ❱ 29 │ │ │ raise Exception('CUDA support is disabled during Tutel inst │
[1,319]<stderr>:│ 30 │ │ __ctx__ = tutel_custom_kernel.inject_source(source) │
[1,319]<stderr>:│ 31 │ │ │
[1,319]<stderr>:│ 32 │ │ def func(*inputs, extra=[], blocks=[]): │
[1,319]<stderr>:╰──────────────────────────────────────────────────────────────────────────────╯
[1,319]<stderr>:Exception: CUDA support is disabled during Tutel installation. Please configure
[1,319]<stderr>:CUDA correctly and reinstall Tutel to enable CUDA support, or report Tutel
[1,319]<stderr>:installation logs for help.
[1,176]<stderr>:╭───────────────────── Traceback (most recent call last) ──────────────────────╮ |
This is the solution. I was missing the cuda toolkit. |
|
I don't suggest you install cuda toolkit over default Ubuntu repository, as they are too old. You should follow the instruction here: https://developer.nvidia.com/cuda-downloads?target_os=Linux&target_arch=x86_64&Distribution=Ubuntu After CUDA SDK is successfully, please purge previous tutel and do a fresh installation based on those new CUDA SDK libraries. |
|
Thanks for the suggestion! I'm using an older driver && once I added nv repo, I was able to select the version that I needed. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Hello, thanks for providing such a great work. However, I cannot use tutel successfully. I have followed the library installation steps:
But when I try the followed test:
The followed error is reported：
The text was updated successfully, but these errors were encountered: