為什麼會有這篇筆記

Kuwa GenAI OS從v0.2.0(2024/4/15發布)就開始安裝到目前最新版v0.3.2(2024/7/5發布)，不得不讚賞Kuwa GenAI OS團隊，一鍵就可在Windows上部署生成式AI GUI平台，並立即使用。
不過在不同硬體上，面對不同使用者需求，還需要額外調校。因為怕自己未來重新部署時，也忘記要調整什麼，所以就生出這一篇筆記。
如人飲水冷暖自知。看別人做事很容易，沒有實際自己摸索過，永遠不會知道固中道理。

Ｗindows可攜式版本安裝到：

acer P3-131
acer P259-MG-30ES
Apple MacBook Pro(2020)
文書PC ASUSPRO M840MB
組裝PC i7-14700KF/Z790/GeForce RTX 4090

Docker v0.3.0安裝到：

Apple MacBook Pro M3 Pro

Docker v0.3.2安裝到：

Apple MacBook Pro(2020)

Hello World

一鍵部署完，就可以跟模型說Hello，請他自我介紹驗明正身。

Image Not Showing Possible Reasons

The image was uploaded to a note which you don't have access to
The note which the image was originally uploaded to has been deleted

Learn More →

調校筆記

proxy設定

如果Ｗindows可攜式版本在下載套件的過程有出現失敗、逾時等錯誤訊息，代表所處網路環境不穩或需要設定proxy代理伺服器才能對外連線。
若需使用Ollama或vLLM結合KUWA GenAI OS進行推論，則需設定no_proxy排除localhost和127.0.0.1/32。
v0.3.2以前於下面檔案第3列加入如下指令後，重新執行C:\kuwa\GenAI OS\windows\build.bat，則理論上可以正常下載相關套件。
C:\kuwa\GenAI OS\windows\src\variables.bat










set "PROXY=http://proxy.XXX.com.tw:3128"
set "HTTP_PROXY=http://proxy.XXX.com.tw:3128"
set "HTTPS_PROXY=http://proxy.XXX.com.tw:3128"
set "ALL_PROXY=http://proxy.XXX.com.tw:3128"
set "NO_PROXY=localhost,127.0.0.1/32,192.168.0.0/16"
set "proxy=http://proxy.XXX.com.tw:3128"
set "http_proxy=http://proxy.XXX.com.tw:3128"
set "https_proxy=http://proxy.XXX.com.tw:3128"
set "all_proxy=http://proxy.XXX.com.tw:3128"
set "no_proxy=localhost,127.0.0.1/32,192.168.0.0/16"

但若要使用Ollama或vLLM結合KUWA GenAI OS的RAG進行推論，則需使用以下git指令擷取程式碼並合併，合併後KUWA GenAI OS就會使用Windows的proxy設定和不使用代理伺服器的相關設定。就不用在上述的C:\kuwa\GenAI OS\windows\src\variables.bat加入proxy或no_proxy相關設定。

git cherry-pick 34099e8 39f6d93

Log

C:\kuwa\GenAI OS\src\kernel\logs
C:\kuwa\GenAI OS\src\multi-chat\storage\logs

C:\kuwa\GenAI OS\windows\executors\taide\run.bat
executors下模型的run.bat加入參數"--log" "debug"

Timeout秒數調整(120變成1200)

C:\kuwa\GenAI OS\windows\packages\php\php.ini


max_execution_time = 1200


default_socket_timeout = 1200

C:\kuwa\GenAI OS\windows\packages\nginx\conf\nginx.conf


fastcgi_read_timeout 1200;

C:\kuwa\GenAI OS\src\executor\docqa\src\kuwa_llm_client.py


async def chat_complete(self, auth_token: str = None, messages: list = [], timeout=1200, streaming=True):

C:\kuwa\GenAI OS\src\kernel\src\kuwa\kernel\routes\chat.py


response = requests.post(dest[0], headers=headers, data=form, stream=True, timeout=1200)

C:\Kuwa\genai-os\src\library\client\src\kuwa\client\base.py


async def chat_complete(self, auth_token: str = None, messages: list = [], timeout=1200, streaming=True):

重新install kernel

執行C:\kuwa\GenAI OS\windows\tool.bat，輸入cmd (會幫忙把環境變數設定好，這功能很重要，例如：本身沒裝git就可以拿到相關環境變數)
再執行指令:

cd ..\src\kernel
pip install .

限制IP

C:\kuwa\GenAI OS\windows\packages\nginx\conf\nginx.conf


allow	192.168.0.0/16;
deny    all;

C:\kuwa\GenAI OS\src\multi-chat\.env


ALLOWED_IPS=127.0.0.1/8,192.168.0.0/16#只能輸入網段

網站預設語言

C:\kuwa\GenAI OS\src\multi-chat.env


locale=zh_tw
fallback_locale=zh_tw

網站相關圖片

C:\kuwa\GenAI OS\windows\packages\nginx-1.24.0\html\images
C:\kuwa\GenAI OS\src\multi-chat\public\images

首頁
C:\kuwa\GenAI OS\src\multi-chat\resources\views\components\logo.blade.php





 <img class="w-[150px]" src="{{ asset('images/01-tpclogo-jpg.jpg') }}">
            <!--</a>-->
            <div class="flex flex-col ml-4 text-[50px]">
                <span class="text-[60px] dark:text-green-300">這裡輸入你想要的網站名稱</span>
                <span class="text-[72px]">生成式AI平台</span>

應用程式Logo
C:\kuwa\GenAI OS\src\multi-chat\resources\views\components\application-logo.blade.php


<img src="{{ asset('images/01-tpclogo-jpg.jpg') }}" {{ $attributes }}>

首頁四分之一圖
C:\kuwa\GenAI OS\src\multi-chat\resources\views\components\welcome_body.blade.php







<h2 class="text-xl font-semibold text-center text-gray-900 dark:text-white">開心使用地端生成式AI</h2>
                <div id="chatroom"
                    class="flex-1 p-4 justify-center overflow-hidden flex flex-col scrollbar rounded-lg">
                    <div class="flex justify-center items-center">
                        <img class="w-auto" src="{{ asset('images/S__47554573.jpg') }}">
                    </div>
                </div>

指定模型回答

DB_QA的executor後面加上--model "taide"指定TAIDE回答

置換Construct RAG的embedding-model

C:\kuwa\GenAI OS\src\toolchain\construct_vector_db.py







def construct_db(
    docs_path:str,
    output_path:str,
    chunk_size:int = 512,
    chunk_overlap:int = 128,
    embedding_model:str = 'thenlper/gte-base-zh'
    ):

改成







def construct_db(
    docs_path:str,
    output_path:str,
    chunk_size:int = 512,
    chunk_overlap:int = 128,
    embedding_model:str = 'intfloat/multilingual-e5-large'
    ):


parser.add_argument("--embedding-model", help="the embedding model to use", type=str, default="thenlper/gte-base-zh")

改成


parser.add_argument("--embedding-model", help="the embedding model to use", type=str, default="intfloat/multilingual-e5-large")

RAG後的DB_QA回應的參考資料出現本機絕對路徑

C:\kuwa\GenAI OS\src\executor\docqa\src\docqa.py




title = ref.title if ref.title is not None else src
content = f'\n\n```plaintext\n{ref.content}\n```\n' if self.display_ref_content else ''
link = src if src.startswith("http") else pathlib.Path(src).as_uri()
result += f'{i+1}. [{title}]({link}){content}\n'

改成




title = ref.title if ref.title is not None else pathlib.Path(src).name
content = f'\n\n```plaintext\n{ref.content}\n```\n' if self.display_ref_content else ''
link = src if src.startswith("http") else pathlib.Path(src).name
result += f'{i+1}. [{title}]({link}){content}\n'

上傳檔案大小限制，假設開放200MB的錄音檔上傳。

v0.3.2已參數化至系統設定中。

C:\kuwa\GenAI OS\windows\packages\php\php.ini


upload_max_filesize = 200M


post_max_size = 250M


memory_limit = 512M
;(若錄音檔超過512M則要動到此設定)

C:\kuwa\GenAI OS\src\multi-chat\app\Http\Controllers\RoomController.php
v0.3.1以前版本需手動修改程式碼。


'file' => 'max:20480',

改成


'file' => 'max:204800',

C:\kuwa\GenAI OS\windows\packages\nginx\conf\nginx.conf


client_max_body_size 250m;

C:\kuwa\GenAI OS\src\multi-chat\resources\views\layouts\app.blade.php
v0.3.1以前版本需手動修改程式碼。


20 * 1024 *

改成


200 * 1024 *

上傳檔案儲存路徑

C:\kuwa\GenAI OS\src\multi-chat\public\storage\pdfs

開放多人使用TAIDE

複製C:\kuwa\GenAI OS\windows\executors\chatgpt目錄並重新命名成給vllm用的，例如：taide_vllm。
修改C:\kuwa\GenAI OS\windows\executors\taide_vllm\run.bat如下：

pushd C:\kuwa\GenAI OS\windows\executors\taide_vllm
call ..\..\src\variables.bat
popd
set EXECUTOR_ACCESS_CODE=taide8B-vllm
pushd ..\..\..\src\multi-chat
php artisan model:config "taide8B-vllm" "TAIDE" --image "..\..\windows\executors\taide_vllm\TAIDE.png"
popd
start /b "" "kuwa-executor" "chatgpt" "--access_code" "taide8B-vllm" --base_url "http://127.0.0.1:8000/v1" --model "taide/Llama3-TAIDE-LX-8B-Chat-Alpha1" "--system_prompt" "你是一個來自台灣的AI助理，你的名字是TAIDE，樂於以台灣人的立場幫助使用者，會用繁體中文回答問題。" --log debug "--no_override_api_key" --api_key dummy

將C:\kuwa\GenAI OS\windows\executors\taide_vllm目錄複製多個，只要ACCESS_CODE相同，就可以在KUWA GenAI OS提供同時多人呼叫vLLM API。
連線至http://127.0.0.1:9000/v1.0/worker/debug，就可以看到taide8B_vllm有6條線可以同時提供服務。

Ollama在KUWA GenAI OS則可使用C:\kuwa\GenAI OS\windows\executors\ollama\init.bat先進行設定，再將C:\kuwa\GenAI OS\windows\executors\ollama目錄複製多個，即可於KUWA GenAI OS提供同時多人呼叫Ollama API。

Ollama

v0.2.0後支援Parallel requests及Multiple models。
查看目前安裝Ollama的版本：
ollama -v
從Hugging Face下載TAIDE官方*.gguf檔
https://huggingface.co/taide/Llama3-TAIDE-LX-8B-Chat-Alpha1-4bit

建Modelfile檔：

FROM ./taide-8b-a.3-q4_k_m.gguf
TEMPLATE """{{ if .System }}<|start_header_id|>system<|end_header_id|>

{{ .System }}<|eot_id|>{{ end }}{{ if .Prompt }}<|start_header_id|>user<|end_header_id|>

{{ .Prompt }}<|eot_id|>{{ end }}<|start_header_id|>assistant<|end_header_id|>

{{ .Response }}<|eot_id|>"""
PARAMETER num_ctx 8192
PARAMETER num_keep 24
PARAMETER stop "<|start_header_id|>"
PARAMETER stop "<|end_header_id|>"
PARAMETER stop "<|eot_id|>"

將*.gguf檔轉換成ollama能運作的模型：

ollama create Llama3-TAIDE-LX-8B-Chat-Alpha1-4bit -f Modelfile

列出ollama在地端能用的模型：
ollama list

用命令提示字元和模型聊天：

ollama run Llama3-TAIDE-LX-8B-Chat-Alpha1-4bit

跑API服務的方式：
ollama serve

看哪些模型被載入：
ollama ps

刪除指定的模型：
ollama rm <model name>
執行kuwa\GenAI OS\windows\executors下的init.bat

因為在公司需使用proxy才能上網，而localhost也被導到proxy，導致無法使用KUWA直接呼叫Ollama API。但在自己的Macbook pro環境搭配KUWA docker版使用起來還不錯。

vLLM

docker run --runtime nvidia --gpus all \
-v ~/.cache/huggingface:/root/.cache/huggingface \
--env "HUGGING_FACE_HUB_TOKEN=<secret>" \
-p 127.0.0.1:8000:8000 \
--ipc=host \
vllm/vllm-openai:latest \
--model taide/Llama3-TAIDE-LX-8B-Chat-Alpha1 --dtype half

{
  "builder": {
    "gc": {
      "defaultKeepStorage": "20GB",
      "enabled": true
    }
  },
  "experimental": false,
  "runtimes": {
    "nvidia": {
      "path": "/usr/bin/nvidia-container-runtime",
      "runtimeArgs": []
    }
  }
}

git指令

可從下面這個檔案裡
C:\kuwa\GenAI OS\windows\update.bat
抽絲剝繭，舉一反三成，但不保證成功：

git fetch
git status -uno
git stash
git pull
git stash list
git stash show
git stash pop
git stash clear

本地的版本若有改，在執行git stash pop時，理論上會自行標註並合併程式。可以在git status -uno時，看到那些檔案有異動，屆時可以打開原始碼，自行判斷要留github，還是本地端自己改的程式碼。

v0.3.2升級v0.3.3後可能會出現的問題

如果大家都跟筆者一樣，喜歡沒事就執行C:\kuwa\GenAI OS\windows\update.bat，看看有沒有新功能上線的話，可能會在7月5日升級到v0.3.2，7月17日後更新到新的Commits。在更新完成後，

啟動KUWA GenAI OS登入後會出現nginx的502 Bad Gateway錯誤訊息。

解決方法如下：
複製
C:\kuwa\GenAI OS\windows\src\nginx.conf
取代
C:\kuwa\GenAI OS\windows\packages\nginx\conf\nginx.conf
再重啟KUWA GenAI OS即可解決502 Bad Gateway問題。

無法使用LLaMA3.1 8B Instruct和Gemma2 2B Instruct問題

檢查C:\kuwa\GenAI OS\windows\executors\llama3_1目錄下是否已下載完成llama3_1-8b-q4_k_m.gguf
檢查C:\kuwa\GenAI OS\windows\executors\llama3_1\run.bat裡的gguf是否與C:\kuwa\GenAI OS\windows\executors\llama3_1目錄下檔名一致。
用KUWA OS與LLaMA3.1 8B Instruct聊天，命令提示字元畫面出現如下訊息：

No machine for llama3.1-8b-instruct has founded, returning NOMACHINE code

查看C:\kuwa\GenAI OS\windows\logs\start.log，在gemma2有錯誤訊息：

llama_model_load: error loading model: error loading model architecture: unknown model architecture: 'gemma2'
llama_load_model_from_file: failed to load model
Traceback (most recent call last):
  File "runpy.py", line 196, in _run_module_as_main
  File "runpy.py", line 86, in _run_code
  File "C:\kuwa\GenAI OS\windows\packages\python-3.10.11-embed-amd64\Scripts\kuwa-executor.exe\__main__.py", line 7, in <module>
    sys.exit(main())
  File "C:\kuwa\GenAI OS\windows\packages\python-3.10.11-embed-amd64\lib\site-packages\kuwa\executor\cli.py", line 93, in main
    args, unknown_args = parser.parse_known_args( )
  File "argparse.py", line 1866, in parse_known_args
  File "argparse.py", line 2061, in _parse_known_args
  File "argparse.py", line 2038, in consume_positionals
  File "argparse.py", line 1943, in take_action
  File "C:\kuwa\GenAI OS\windows\packages\python-3.10.11-embed-amd64\lib\site-packages\kuwa\executor\cli.py", line 84, in __call__
    executor = executor_class()
  File "C:\kuwa\GenAI OS\windows\packages\python-3.10.11-embed-amd64\lib\site-packages\kuwa\executor\./example\llamacpp.py", line 147, in __init__
    super().__init__()
  File "C:\kuwa\GenAI OS\windows\packages\python-3.10.11-embed-amd64\lib\site-packages\kuwa\executor\base_executor.py", line 58, in __init__
    self.setup()
  File "C:\kuwa\GenAI OS\windows\packages\python-3.10.11-embed-amd64\lib\site-packages\kuwa\executor\./example\llamacpp.py", line 201, in setup
    self.model = Llama(model_path=self.model_path, n_gpu_layers=self.args.ngl, n_ctx=self.context_window)
  File "C:\kuwa\GenAI OS\windows\packages\python-3.10.11-embed-amd64\lib\site-packages\llama_cpp\llama.py", line 341, in __init__
    self._model = _LlamaModel(
  File "C:\kuwa\GenAI OS\windows\packages\python-3.10.11-embed-amd64\lib\site-packages\llama_cpp\_internals.py", line 57, in __init__
    raise ValueError(f"Failed to load model from file: {path_model}")
ValueError: Failed to load model from file: gemma-2-2b-it-Q8_0.gguf

查看C:\kuwa\GenAI OS\windows\logs\build.log在llama-cpp-python升級到0.2.87，有出現例外錯誤訊息：

Collecting llama-cpp-python==0.2.87,>=0.2.77 (from -r requirements.txt (line 12))
  Using cached llama_cpp_python-0.2.87.tar.gz (80.5 MB)
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
ERROR: Exception:
Traceback (most recent call last):
  File "C:\kuwa\GenAI OS\windows\packages\python-3.10.11-embed-amd64\lib\site-packages\pip\_internal\cli\base_command.py", line 105, in _run_wrapper
    status = _inner_run()
  File "C:\kuwa\GenAI OS\windows\packages\python-3.10.11-embed-amd64\lib\site-packages\pip\_internal\cli\base_command.py", line 96, in _inner_run
    return self.run(options, args)
  File "C:\kuwa\GenAI OS\windows\packages\python-3.10.11-embed-amd64\lib\site-packages\pip\_internal\cli\req_command.py", line 67, in wrapper
    return func(self, options, args)
  File "C:\kuwa\GenAI OS\windows\packages\python-3.10.11-embed-amd64\lib\site-packages\pip\_internal\commands\install.py", line 379, in run
    requirement_set = resolver.resolve(
  File "C:\kuwa\GenAI OS\windows\packages\python-3.10.11-embed-amd64\lib\site-packages\pip\_internal\resolution\resolvelib\resolver.py", line 95, in resolve
    result = self._result = resolver.resolve(
  File "C:\kuwa\GenAI OS\windows\packages\python-3.10.11-embed-amd64\lib\site-packages\pip\_vendor\resolvelib\resolvers.py", line 546, in resolve
    state = resolution.resolve(requirements, max_rounds=max_rounds)
  File "C:\kuwa\GenAI OS\windows\packages\python-3.10.11-embed-amd64\lib\site-packages\pip\_vendor\resolvelib\resolvers.py", line 397, in resolve
    self._add_to_criteria(self.state.criteria, r, parent=None)
  File "C:\kuwa\GenAI OS\windows\packages\python-3.10.11-embed-amd64\lib\site-packages\pip\_vendor\resolvelib\resolvers.py", line 173, in _add_to_criteria
    if not criterion.candidates:
  File "C:\kuwa\GenAI OS\windows\packages\python-3.10.11-embed-amd64\lib\site-packages\pip\_vendor\resolvelib\structs.py", line 156, in __bool__
    return bool(self._sequence)
  File "C:\kuwa\GenAI OS\windows\packages\python-3.10.11-embed-amd64\lib\site-packages\pip\_internal\resolution\resolvelib\found_candidates.py", line 174, in __bool__
    return any(self)
  File "C:\kuwa\GenAI OS\windows\packages\python-3.10.11-embed-amd64\lib\site-packages\pip\_internal\resolution\resolvelib\found_candidates.py", line 162, in <genexpr>
    return (c for c in iterator if id(c) not in self._incompatible_ids)
  File "C:\kuwa\GenAI OS\windows\packages\python-3.10.11-embed-amd64\lib\site-packages\pip\_internal\resolution\resolvelib\found_candidates.py", line 53, in _iter_built
    candidate = func()
  File "C:\kuwa\GenAI OS\windows\packages\python-3.10.11-embed-amd64\lib\site-packages\pip\_internal\resolution\resolvelib\factory.py", line 186, in _make_candidate_from_link
    base: Optional[BaseCandidate] = self._make_base_candidate_from_link(
  File "C:\kuwa\GenAI OS\windows\packages\python-3.10.11-embed-amd64\lib\site-packages\pip\_internal\resolution\resolvelib\factory.py", line 232, in _make_base_candidate_from_link
    self._link_candidate_cache[link] = LinkCandidate(
  File "C:\kuwa\GenAI OS\windows\packages\python-3.10.11-embed-amd64\lib\site-packages\pip\_internal\resolution\resolvelib\candidates.py", line 303, in __init__
    super().__init__(
  File "C:\kuwa\GenAI OS\windows\packages\python-3.10.11-embed-amd64\lib\site-packages\pip\_internal\resolution\resolvelib\candidates.py", line 158, in __init__
    self.dist = self._prepare()
  File "C:\kuwa\GenAI OS\windows\packages\python-3.10.11-embed-amd64\lib\site-packages\pip\_internal\resolution\resolvelib\candidates.py", line 235, in _prepare
    dist = self._prepare_distribution()
  File "C:\kuwa\GenAI OS\windows\packages\python-3.10.11-embed-amd64\lib\site-packages\pip\_internal\resolution\resolvelib\candidates.py", line 314, in _prepare_distribution
    return preparer.prepare_linked_requirement(self._ireq, parallel_builds=True)
  File "C:\kuwa\GenAI OS\windows\packages\python-3.10.11-embed-amd64\lib\site-packages\pip\_internal\operations\prepare.py", line 527, in prepare_linked_requirement
    return self._prepare_linked_requirement(req, parallel_builds)
  File "C:\kuwa\GenAI OS\windows\packages\python-3.10.11-embed-amd64\lib\site-packages\pip\_internal\operations\prepare.py", line 642, in _prepare_linked_requirement
    dist = _get_prepared_distribution(
  File "C:\kuwa\GenAI OS\windows\packages\python-3.10.11-embed-amd64\lib\site-packages\pip\_internal\operations\prepare.py", line 72, in _get_prepared_distribution
    abstract_dist.prepare_distribution_metadata(
  File "C:\kuwa\GenAI OS\windows\packages\python-3.10.11-embed-amd64\lib\site-packages\pip\_internal\distributions\sdist.py", line 56, in prepare_distribution_metadata
    self._install_build_reqs(finder)
  File "C:\kuwa\GenAI OS\windows\packages\python-3.10.11-embed-amd64\lib\site-packages\pip\_internal\distributions\sdist.py", line 126, in _install_build_reqs
    build_reqs = self._get_build_requires_wheel()
  File "C:\kuwa\GenAI OS\windows\packages\python-3.10.11-embed-amd64\lib\site-packages\pip\_internal\distributions\sdist.py", line 103, in _get_build_requires_wheel
    return backend.get_requires_for_build_wheel()
  File "C:\kuwa\GenAI OS\windows\packages\python-3.10.11-embed-amd64\lib\site-packages\pip\_internal\utils\misc.py", line 706, in get_requires_for_build_wheel
    return super().get_requires_for_build_wheel(config_settings=cs)
  File "C:\kuwa\GenAI OS\windows\packages\python-3.10.11-embed-amd64\lib\site-packages\pip\_vendor\pyproject_hooks\_impl.py", line 166, in get_requires_for_build_wheel
    return self._call_hook('get_requires_for_build_wheel', {
  File "C:\kuwa\GenAI OS\windows\packages\python-3.10.11-embed-amd64\lib\site-packages\pip\_vendor\pyproject_hooks\_impl.py", line 321, in _call_hook
    raise BackendUnavailable(data.get('traceback', ''))
pip._vendor.pyproject_hooks._impl.BackendUnavailable: Traceback (most recent call last):
  File "C:\kuwa\GenAI OS\windows\packages\python-3.10.11-embed-amd64\lib\site-packages\pip\_vendor\pyproject_hooks\_in_process\_in_process.py", line 77, in _build_backend
    obj = import_module(mod_path)
  File "importlib\__init__.py", line 126, in import_module
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 992, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1004, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'scikit_build_core'

解決方法如下：

執行C:\kuwa\GenAI OS\windows\tool.bat，輸入cmd (會幫忙把環境變數設定好，這功能很重要，例如：本身沒裝git就可以拿到相關環境變數)
輸入指令pip install scikit-build-core
輸入指令

pip install --upgrade https://github.com/abetlen/llama-cpp-python/releases/download/v0.2.87-cu123/llama_cpp_python-0.2.87-cp310-cp310-win_amd64.whl

4. KUWA OS測試聊天

參考資料

Kuwa AI網站 https://kuwaai.tw/zh-Hant/os/intro
Kuwa GenAI OS https://github.com/kuwaai/genai-os
陳伶志，「Kuwa AI 安裝過程筆記」https://hackmd.io/@cclljj/r1mIc3tNR
使用繁體中文評測各家 Embedding 模型的檢索能力 https://ihower.tw/blog/archives/12167
YWC 科技筆記，抓別人的 GGUF 模型，用 Ollama 在本機執行！ https://ywctech.net/ml-ai/ollama-import-custom-gguf/
TAIDE - 推動臺灣可信任生成式AI發展計畫 https://taide.tw/index
https://github.com/NVIDIA/nvidia-container-toolkit/issues/150
Ollama https://github.com/ollama/ollama/blob/main/docs/modelfile.md#build-from-a-safetensors-model

致謝

Kuwa GenAI OS開發團隊胡詠翔先生回應我的各種問題。
公司同事處理NV4090 PC採購的行政流程。

為什麼會有這篇筆記

Hello World

調校筆記

proxy設定

Log

Timeout秒數調整(120變成1200)

重新install kernel

限制IP

網站預設語言

網站相關圖片

指定模型回答

置換Construct RAG的embedding-model

RAG後的DB_QA回應的參考資料出現本機絕對路徑

上傳檔案大小限制，假設開放200MB的錄音檔上傳。

上傳檔案儲存路徑

開放多人使用TAIDE

Ollama

vLLM

git指令

v0.3.2升級v0.3.3後可能會出現的問題

啟動KUWA GenAI OS登入後會出現nginx的502 Bad Gateway錯誤訊息。

無法使用LLaMA3.1 8B Instruct和Gemma2 2B Instruct問題

參考資料

致謝

Read more

使用llama.cpp將Hugging Face模型權重(safetensors)轉換成GGUF並進行量化

Kuwa GenAI OS建構在MAC OS相關筆記