[WASI-NN] neural_speed: add backend structure #3303

grorge123 · 2024-03-25T07:14:08Z

No description provided.

juntao · 2024-03-25T07:14:11Z

Hello, I am a code review bot on flows.network. Here are my reviews of code commits in this PR.

codecov · 2024-03-25T12:05:37Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 79.84%. Comparing base (0f11476) to head (065fc95).
Report is 19 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #3303      +/-   ##
==========================================
- Coverage   79.84%   79.84%   -0.01%     
==========================================
  Files         253      253              
  Lines       34945    34948       +3     
  Branches     6130     6143      +13     
==========================================
+ Hits        27903    27905       +2     
- Misses       5613     5623      +10     
+ Partials     1429     1420       -9

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

grorge123 · 2024-04-25T06:50:06Z

@hydai
I have finished the basic implement that #3260 mentioned.
And I create a rust example.
second-state/WasmEdge-WASINN-examples#135

plugins/wasi_nn/neuralspeed.cpp

plugins/wasi_nn/CMakeLists.txt

plugins/wasi_nn/neuralspeed.cpp

Signed-off-by: grorge <g97116@yahoo.com>

hydai · 2024-05-31T07:18:30Z

plugins/wasi_nn/CMakeLists.txt

@@ -149,6 +149,64 @@ if(BACKEND STREQUAL "ggml")
  endif()
 endif()

+if(BACKEND STREQUAL "neuralspeed")
+  find_package(simdjson QUIET)


We changed the way to find the simdjson package, please use the new way like this pr: #3426

hydai · 2024-05-31T07:19:43Z

plugins/wasi_nn/neuralspeed.cpp

+#endif
+void printImformation(Graph &GraphRef, Context &CxtRef) {
+  spdlog::info(
+      "[WASI-NN][Info] Neural speed backend: Number of input tokens: {}"sv,


You don't need to add [Info] here, because the spdlog::info will do this for you.

hydai · 2024-05-31T07:20:44Z

plugins/wasi_nn/neuralspeed.cpp

+          "[WASI-NN][Debug] Neural speed: Model path not found in nn-preload, "
+          "write model into a tmpfile."sv);
+    }
+    // TODO: pass the model directly to ggml


I would like to know the details of this TODO item.

It refers to ggml backend. Although current implementation is enough. Should I remove it?

We can remove it.

hydai · 2024-05-31T07:21:37Z

plugins/wasi_nn/neuralspeed.cpp

+  auto &CxtRef = Env.NNContext[ContextId].get<Context>();
+  auto &GraphRef = Env.NNGraph[CxtRef.GraphId].get<Graph>();
+  if (!Py_IsInitialized()) {
+    spdlog::info(


We should use spdlog::error here.

hydai · 2024-05-31T07:22:39Z

plugins/wasi_nn/neuralspeed.cpp

+Expect<WASINN::ErrNo> compute(WasiNNEnvironment &Env,
+                              uint32_t ContextId) noexcept {
+  if (!Py_IsInitialized()) {
+    spdlog::info(


We should use spdlog::error here. This is a fatal error.

hydai · 2024-05-31T07:23:25Z

plugins/wasi_nn/neuralspeed.cpp

+        "[WASI-NN] neural speed backend: Input transfer tensor failed."sv);
+    return WASINN::ErrNo::InvalidArgument;
+  }
+  // PyObject *GenerateArgs = PyTuple_Pack(1, LongTensor);


Why does this line comment out?

hydai · 2024-05-31T07:23:48Z

plugins/wasi_nn/neuralspeed.cpp

+    }
+  }
+  Py_DECREF(Result);
+  // Py_DECREF(GenerateArgs);


hydai · 2024-05-31T07:26:45Z

test/plugins/wasi_nn/CMakeLists.txt

+  elseif(BACKEND STREQUAL "neuralspeed")
+    message( STATUS "Download ML artifacts to ${CMAKE_CURRENT_BINARY_DIR}/wasinn_neural_speed_fixtures")
+    download(
+      https://huggingface.co/TheBloke/Llama-2-7B-Chat-GGUF/resolve/main/llama-2-7b-chat.Q4_0.gguf


Do we really need to download this huge file if we would like to do the test?
Is it possible to have a small one instead?

hydai · 2024-05-31T07:29:57Z

test/plugins/wasi_nn/wasi_nn.cpp

+  uint32_t BuilderPtr = UINT32_C(0);
+  uint32_t LoadEntryPtr = UINT32_C(0);
+  uint32_t SetInputEntryPtr = UINT32_C(0);
+  uint32_t OutBoundPtr = UINT32_C(61000 * 65536);


Should we use UINT32_C(61000) * UINT32_C(65536)?

grorge123 requested a review from ibmibmibm as a code owner March 25, 2024 07:14

github-actions bot added c-Plugin An issue related to WasmEdge Plugin c-WASI-NN c-CMake labels Mar 25, 2024

grorge123 force-pushed the neural_speed branch 4 times, most recently from f916aeb to af7d519 Compare March 25, 2024 11:53

grorge123 requested a review from q82419 as a code owner March 25, 2024 11:53

github-actions bot added the c-Test An issue/PR to enhance the test suite label Mar 25, 2024

grorge123 force-pushed the neural_speed branch from af7d519 to fb9bdc7 Compare March 25, 2024 11:54

github-actions bot removed the c-Test An issue/PR to enhance the test suite label Mar 25, 2024

grorge123 marked this pull request as draft March 25, 2024 12:55

grorge123 marked this pull request as ready for review March 29, 2024 06:22

grorge123 changed the title ~~[WIP] [WASI-NN] neural_speed: add backend struct~~ [WASI-NN] neural_speed: add backend structure Mar 29, 2024

grorge123 requested a review from hydai as a code owner April 8, 2024 06:35

github-actions bot added the c-Test An issue/PR to enhance the test suite label Apr 8, 2024

grorge123 force-pushed the neural_speed branch from 2ad58a0 to 17f0157 Compare April 25, 2024 03:23

github-actions bot added c-CLI An issue related to WasmEdge CLI tools c-Installer c-CI labels Apr 25, 2024

grorge123 force-pushed the neural_speed branch from 17f0157 to 2ad58a0 Compare April 25, 2024 03:34

github-actions bot removed c-CLI An issue related to WasmEdge CLI tools c-Installer c-CI labels Apr 25, 2024

grorge123 force-pushed the neural_speed branch 3 times, most recently from a60159f to 984d13a Compare April 25, 2024 04:06

dm4 requested changes Apr 25, 2024

View reviewed changes

plugins/wasi_nn/neuralspeed.cpp Outdated Show resolved Hide resolved

plugins/wasi_nn/neuralspeed.cpp Outdated Show resolved Hide resolved

plugins/wasi_nn/neuralspeed.cpp Outdated Show resolved Hide resolved

plugins/wasi_nn/CMakeLists.txt Outdated Show resolved Hide resolved

grorge123 force-pushed the neural_speed branch from d121c80 to 84f1240 Compare April 26, 2024 06:43

hydai requested changes May 1, 2024

View reviewed changes

plugins/wasi_nn/neuralspeed.cpp Show resolved Hide resolved

grorge123 added 14 commits May 8, 2024 15:38

[WASI-NN] neural_speed: add backend struct

b070862

Signed-off-by: grorge <g97116@yahoo.com>

test: add successful test

7dc9672

Signed-off-by: grorge <g97116@yahoo.com>

feat: implement neural network backend

ab21835

Signed-off-by: grorge <g97116@yahoo.com>

fix: neural backend and basic test

86460fb

Signed-off-by: grorge <g97116@yahoo.com>

fix: update pythonlib and share library

9ca1be1

Signed-off-by: grorge <g97116@yahoo.com>

feat: add setting model type

c315224

Signed-off-by: grorge <g97116@yahoo.com>

fix: python finalize deadlock

bc41f6d

Signed-off-by: grorge <g97116@yahoo.com>

test: add successful test

36c003b

Signed-off-by: grorge <g97116@yahoo.com>

feat: update build neural speed cmake

285062d

Signed-off-by: grorge <g97116@yahoo.com>

feat: aneural speed backend ad unload function

09881d6

Signed-off-by: grorge <g97116@yahoo.com>

fix: simdjson not fund

ce82d5a

Signed-off-by: grorge <g97116@yahoo.com>

fix: remove finiSingle and set simdjson tags/v3.9.1

e8ad57a

Signed-off-by: grorge <g97116@yahoo.com>

feat: print run time

37dac5f

Signed-off-by: grorge <g97116@yahoo.com>

fix: change time to chrono

ff28898

Signed-off-by: grorge <g97116@yahoo.com>

grorge123 force-pushed the neural_speed branch 4 times, most recently from e136fdd to 1ffd3ac Compare May 8, 2024 09:28

fix: windows dlopen

d9f92c9

Signed-off-by: grorge <g97116@yahoo.com>

grorge123 force-pushed the neural_speed branch 2 times, most recently from 5cc73fc to ef72299 Compare May 13, 2024 06:41

test: add more test

065fc95

Signed-off-by: grorge <g97116@yahoo.com>

grorge123 force-pushed the neural_speed branch from ef72299 to 065fc95 Compare May 13, 2024 07:19

hydai added the v-0.14.1 label May 22, 2024

hydai requested changes May 31, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WASI-NN] neural_speed: add backend structure #3303

[WASI-NN] neural_speed: add backend structure #3303

grorge123 commented Mar 25, 2024

juntao commented Mar 25, 2024 •

edited

codecov bot commented Mar 25, 2024 •

edited

grorge123 commented Apr 25, 2024

hydai May 31, 2024

hydai May 31, 2024

hydai May 31, 2024

grorge123 May 31, 2024

hydai May 31, 2024

hydai May 31, 2024

hydai May 31, 2024

hydai May 31, 2024

hydai May 31, 2024

hydai May 31, 2024

hydai May 31, 2024

[WASI-NN] neural_speed: add backend structure #3303

Are you sure you want to change the base?

[WASI-NN] neural_speed: add backend structure #3303

Conversation

grorge123 commented Mar 25, 2024

juntao commented Mar 25, 2024 • edited

codecov bot commented Mar 25, 2024 • edited

Codecov Report

grorge123 commented Apr 25, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

juntao commented Mar 25, 2024 •

edited

codecov bot commented Mar 25, 2024 •

edited