Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

How to enable OpenCL with llama.cpp in Android App? #3694

Closed
4 tasks done
DavdGao opened this issue Oct 20, 2023 · 7 comments
Closed
4 tasks done

How to enable OpenCL with llama.cpp in Android App? #3694

DavdGao opened this issue Oct 20, 2023 · 7 comments
Labels
stale

Comments

@DavdGao
Copy link

DavdGao commented Oct 20, 2023

Prerequisites

Please answer the following questions for yourself before submitting an issue.

  • I am running the latest code. Development is very rapid so there are no tagged versions as of now.
  • I carefully followed the README.md.
  • I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
  • I reviewed the Discussions, and have a new bug or useful enhancement to share.

Expected Behavior

I have run llama.cpp in an Android APP successfully. Now I want to enable OpenCL in Android APP to speed up the inference of LLM.

Current Behavior

Cross-compile OpenCL-SDK

  • First, following README.md I first cross-compile OpenCL-SDK as follows
#!/bin/bash 

set -e

git clone --recurse-submodules https://github.com/KhronosGroup/OpenCL-SDK.git
mkdir OpenCL-SDK/build
cd OpenCL-SDK/build


# cross-compile for  android
NDK_PATH=/Users/david/Library/Android/sdk/ndk/25.1.8937393/
TOOLCHAIN=${NDK_PATH}/toolchains/llvm/prebuilt/darwin-x86_64

cmake .. \
	-DCMAKE_BUILD_TYPE=Release \
	-DBUILD_DOCS=OFF \
	-DBUILD_EXAMPLES=OFF \
	-DBUILD_TESTING=OFF \
	-DBUILD_TESTS=OFF \
	-DOPENCL_SDK_BUILD_SAMPLES=OFF \
	-DOPENCL_SDK_TEST_SAMPLES=OFF \
	-DCMAKE_TOOLCHAIN_FILE=${NDK_PATH}/build/cmake/android.toolchain.cmake \
	-DCMAKE_C_COMPILER=${TOOLCHAIN}/bin/aarch64-linux-android33-clang \
	-DCMAKE_CXX_COMPILER=${TOOLCHAIN}/bin/aarch64-linux-android-33-clang++ \
	-DBUILD_SHARED_LIBS=OFF \
	-DBUILD_STATIC_LIBS=ON \
	-DANDROID_ABI=arm64-v8a \
	-DANDROID_PLATFORM=android-33

cmake --build . --config Release
cmake --install . --prefix install_android
  • After that, the following files are created in dir install_android
    20231020154708

Cross-compile CLBlast

  • After that, I need to cross-compile CLBlast for android. The following is my scripts
#!/bin/bash

set -e

git clone https://github.com/CNugteren/CLBlast.git
mkdir CLBlast/build
cd CLBlast/build

NDK_PATH=/Users/david/Library/Android/sdk/ndk/25.1.8937393/
TOOLCHAIN=${NDK_PATH}/toolchains/llvm/prebuilt/darwin-x86_64

cmake .. \
	-DCMAKE_BUILD_TYPE=Release \
	-DBUILD_DOCS=OFF \
	-DBUILD_EXAMPLES=OFF \
	-DBUILD_TESTING=OFF \
	-DBUILD_TESTS=OFF \
	-DCMAKE_TOOLCHAIN_FILE=${NDK_PATH}/build/cmake/android.toolchain.cmake \
	-DCMAKE_C_COMPILER=${TOOLCHAIN}/bin/aarch64-linux-android33-clang \
	-DCMAKE_CXX_COMPILER=${TOOLCHAIN}/bin/aarch64-linux-android-33-clang++ \
	-DBUILD_SHARED_LIBS=OFF \
	-DANDROID_ABI=arm64-v8a \
	-DANDROID_PLATFORM=android-33 \
	-DOPENCL_LIBIARIES=/Users/david/Documents/C++/OpenCL-SDK/build_android/install_android 

cmake --build . --config Release
cmake --install . --prefix install_android
  • However, running the above scripts reports Could NOT find OpenCL library. Okay, I think maybe I should set the path of OpenCL somewhere. The detailed logging is as follows
CMake Deprecation Warning at CMakeLists.txt:12 (cmake_minimum_required):
  Compatibility with CMake < 2.8.12 will be removed from a future version of
  CMake.

  Update the VERSION argument <min> value or use a ...<max> suffix to tell
  CMake that the project does not need compatibility with older versions.


-- Building CLBlast with OpenCL API (default)
-- Could NOT find 'OpenCL/cl.h' or 'CL/cl.h', install OpenCL or set OPENCL_ROOT
-- Could NOT find OpenCL library, install it or set OPENCL_ROOT
CMake Error at /opt/homebrew/Cellar/cmake/3.25.2/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:230 (message):
  Could NOT find OpenCL (missing: OPENCL_INCLUDE_DIRS OPENCL_LIBRARIES)
Call Stack (most recent call first):
  /opt/homebrew/Cellar/cmake/3.25.2/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:600 (_FPHSA_FAILURE_MESSAGE)
  cmake/Modules/FindOpenCL.cmake:77 (find_package_handle_standard_args)
  CMakeLists.txt:169 (find_package)


-- Configuring incomplete, errors occurred!
See also "/Users/david/Documents/C++/CLBlast/build/CMakeFiles/CMakeOutput.log".
You have changed variables that require your cache to be deleted.
Configure will be re-run and you may have to reset some variables.
The following variables have changed:
CMAKE_C_COMPILER= /Users/david/Library/Android/sdk/ndk/25.1.8937393//toolchains/llvm/prebuilt/darwin-x86_64/bin/aarch64-linux-android33-clang
CMAKE_CXX_COMPILER= /Users/david/Library/Android/sdk/ndk/25.1.8937393//toolchains/llvm/prebuilt/darwin-x86_64/bin/aarch64-linux-android-33-clang++

-- Generating done
CMake Generate step failed.  Build files cannot be regenerated correctly.

export OPENCL_ROOT=/Users/david/Documents/C++/OpenCL-SDK/build_android/install_android/include/

in script and add cmake option like

DOPENCL_ROOT=/Users/david/Documents/C++/OpenCL-SDK/build_android/install_android/include/

However, they both failed, and it still failed to find OpenCL. I wonder what's the problem and how can I cross-compile CLBlas for android?

Environment and Context

  • hardware: macbook pro, M1 CPU
  • cmake: 3.25.2
  • g++: Apple clang version 14.0.0
  • android ndk version: 25.1.8937393
@freedomtan
Copy link

In case you haven't figure it out. The OPENCL_ROOT doesn't work as expected. You have to set OPENCL_INCLUDE_DIRS andOPENCL_LIBRARIES. And the OPENCL_LIBRARIES should include the libraries you want to link with. E.g.,

-DOPENCL_LIBIARIES=/Users/david/Documents/C++/OpenCL-SDK/build_android/install_android/lib/libOpenCL.so
-DOPENCL_INCLUDE_DIRS=/Users/david/Documents/C++/OpenCL-SDK/build_android/install_android/include

@gpokat
Copy link

gpokat commented Feb 1, 2024

You can try to do cmake with

NDK_PATH=/playground/sdk/ndk/25.2.9519653
TOOLCHAIN=${NDK_PATH}/toolchains/llvm/prebuilt/linux-x86_64

cmake .. \
	-DCMAKE_BUILD_TYPE=Release \
	-DBUILD_DOCS=OFF \
	-DBUILD_EXAMPLES=OFF \
	-DBUILD_TESTING=OFF \
	-DBUILD_TESTS=OFF \
	-DCMAKE_TOOLCHAIN_FILE=${NDK_PATH}/build/cmake/android.toolchain.cmake \
	-DCMAKE_C_COMPILER=${TOOLCHAIN}/bin/aarch64-linux-android33-clang \
	-DCMAKE_CXX_COMPILER=${TOOLCHAIN}/bin/aarch64-linux-android33-clang++ \
	-DBUILD_SHARED_LIBS=OFF \
	-DANDROID_ABI=arm64-v8a \
	-DCMAKE_SYSTEM_NAME=Android \
	-DANDROID_PLATFORM=33 \
	-DCMAKE_ANDROID_STL_TYPE=gnustl_static \
	-DOPENCL_LIBRARIES=/mediatek/libOpenCL.so \
	-DOPENCL_INCLUDE_DIRS=/mediatek/CL \
	-DOPENCL_ROOT=/mediatek/

The libOpenCL.so has been taken from device, the CL headers folder has been taken from https://github.com/KhronosGroup/OpenCL-Headers/tree/main

@github-actions github-actions bot added the stale label Mar 19, 2024
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Apr 4, 2024

This issue was closed because it has been inactive for 14 days since being marked as stale.

@github-actions github-actions bot closed this as completed Apr 4, 2024
@v-prgmr
Copy link

v-prgmr commented Apr 24, 2024

@gpokat cmake finds the library after setting the include, root and libraries flag. But while running
cmake --build . --config Release , the compiler cannot find CL/opencl.h

(base) user@razer-1:~/workspace/PublicRepos/CLBlast/build$ cmake ..    -DCMAKE_BUILD_TYPE=Release      -DBUILD_DOCS=OFF        -DBUILD_EXAMPLES=OFF    -DBUILD_TESTING=OFF     -DBUILD_TESTS=OFF       -DCMAKE_TOOLCHAIN_FILE=${NDK_PATH}/build/cmake/android.toolchain.cmake  -DCMAKE_C_COMPILER=${TOOLCHAIN}/bin/aarch64-linux-android33-clang         -DCMAKE_CXX_COMPILER=${TOOLCHAIN}/bin/aarch64-linux-android33-clang++   -DBUILD_SHARED_LIBS=OFF         -DANDROID_ABI=arm64-v8a         -DCMAKE_SYSTEM_NAME=Android     -DANDROID_PLATFORM=33   -DCMAKE_ANDROID_STL_TYPE=gnustl_static    -DOPENCL_LIBRARIES=~/workspace/libOpenCL.so     -DOPENCL_INCLUDE_DIRS=/home/user/workspace/PublicRepos/OpenCL-SDK/build/install_android/include/CL     -DOPENCL_ROOT=/home/user/workspace/PublicRepos/OpenCL-SDK/build/install_android/include
CMake Deprecation Warning at CMakeLists.txt:12 (cmake_minimum_required):
  Compatibility with CMake < 2.8.12 will be removed from a future version of
  CMake.

  Update the VERSION argument <min> value or use a ...<max> suffix to tell
  CMake that the project does not need compatibility with older versions.


-- The C compiler identification is Clang 14.0.6
-- The CXX compiler identification is Clang 14.0.6
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /home/user/Android/Sdk/ndk/25.1.8937393/toolchains/llvm/prebuilt/linux-x86_64/bin/clang - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /home/user/Android/Sdk/ndk/25.1.8937393/toolchains/llvm/prebuilt/linux-x86_64/bin/clang++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Building CLBlast with OpenCL API (default)
-- Found OpenCL: /home/user/workspace/PublicRepos/OpenCL-SDK/build/install_android/include/CL  
-- Configuring done
-- Generating done
CMake Warning:
  Manually-specified variables were not used by the project:

    BUILD_DOCS
    BUILD_EXAMPLES
    BUILD_TESTING
    BUILD_TESTS


-- Build files have been written to: /home/user/workspace/PublicRepos/CLBlast/build
(base) user@razer-1:~/workspace/PublicRepos/CLBlast/build$ cmake --build . --config Release
[  0%] Building CXX object CMakeFiles/clblast.dir/src/database/database.cpp.o
In file included from /home/user/workspace/PublicRepos/CLBlast/src/database/database.cpp:16:
In file included from /home/user/workspace/PublicRepos/CLBlast/src/utilities/utilities.hpp:27:
/home/user/workspace/PublicRepos/CLBlast/src/clpp11.hpp:57:12: fatal error: 'CL/opencl.h' file not found
  #include <CL/opencl.h>
           ^~~~~~~~~~~~~
1 error generated.
gmake[2]: *** [CMakeFiles/clblast.dir/build.make:76: CMakeFiles/clblast.dir/src/database/database.cpp.o] Error 1
gmake[1]: *** [CMakeFiles/Makefile2:114: CMakeFiles/clblast.dir/all] Error 2
gmake: *** [Makefile:136: all] Error 2

@gpokat
Copy link

gpokat commented Apr 24, 2024

@v-prgmr I just repeated steps, so there is how-to to reproduce:
/playground/mediatek folder contain openCL.so from device and CL folder with content from public repo mentioned above:
ls /playground/mediatek/CL

cl_d3d10.h                    cl_gl.h
cl_d3d11.h                    cl.h
cl_dx9_media_sharing.h        cl_half.h
cl_dx9_media_sharing_intel.h  cl_icd.h
cl_egl.h                      cl_layer.h
cl_ext.h                      cl_platform.h
cl_ext_intel.h                cl_va_api_media_sharing_intel.h
cl_function_types.h           cl_version.h
cl_gl_ext.h                   opencl.h

do cmake as below

export NDK_PATH=/playground/sdk/ndk/25.2.9519653 

cmake .. \
    -DCMAKE_SYSTEM_NAME=Android \
    -DCMAKE_SYSTEM_VERSION=33 \
    -DCMAKE_ANDROID_ARCH_ABI=arm64-v8a \
    -DCMAKE_ANDROID_NDK=$NDK_PATH \
    -DTUNERS=ON \
    -DCMAKE_ANDROID_STL_TYPE=c++_static \
    -DOPENCL_ROOT=/playground/mediatek \
    -DCMAKE_FIND_ROOT_PATH_MODE_LIBRARY=BOTH \
    -DCMAKE_FIND_ROOT_PATH_MODE_INCLUDE=BOTH

CMake Deprecation Warning...

-- Android: Targeting API '33' with architecture 'arm64', ABI 'arm64-v8a', and processor 'aarch64'
-- Android: Selected unified Clang toolchain
-- The C compiler identification is Clang 14.0.7
-- The CXX compiler identification is Clang 14.0.7
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /playground/sdk/ndk/25.2.9519653/toolchains/llvm/prebuilt/linux-x86_64/bin/clang - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /playground/sdk/ndk/25.2.9519653/toolchains/llvm/prebuilt/linux-x86_64/bin/clang++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Building CLBlast with OpenCL API (default)
-- Found OpenCL: /playground/mediatek  
-- Configuring done
-- Generating done
-- Build files have been written to: /playground/clblast/CLBlast/build

do make

/playground/clblast/CLBlast/build$ make

[  0%] Building CXX object CMakeFiles/clblast.dir/src/database/database.cpp.o
[  1%] Building CXX object CMakeFiles/clblast.dir/src/routines/common.cpp.o
...
[100%] Building CXX object CMakeFiles/clblast_tuner_routine
[100%] Linking CXX executable clblast_tuner_routine_xtrsv
[100%] Built target clblast_tuner_routine_xtrsv

Getting libclblast.so
Good luck.

@v-prgmr
Copy link

v-prgmr commented Apr 24, 2024

@gpokat Hope you have a great day there! Thank you for the fix, I can now successfully compile CLBlast for arm64-v8a

@gustrd
Copy link
Contributor

gustrd commented May 12, 2024

Pointing another thread discussing this topic: #7016

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
stale
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@freedomtan @gpokat @v-prgmr @gustrd @DavdGao