## call_stream_model
在輔助函數中,若LLM模型為若為***openai***, ***huggingface***, ***remote***類模型,可以使用akasha.call_stream_model()來得到流輸出
``` python
import akasha
prompt = "say something."
model_obj = akasha.handle_model("openai:gpt-3.5-turbo", False, 0.0)
streaming = akasha.call_stream_model(model_obj, prompt)
for s in streaming:
print(s)
```
## Doc_QA stream
Doc_QA class的函式皆可使用參數stream=True來得到流輸出
## Stream Output
要在網頁上或API中使用流輸出(及時一個字一個字輸出語言模型回答)時,若為***openai***, ***huggingface***, ***remote***類模型,可以使用model_obj.stream(prompt),以下為streamlit write_stream在網頁上即時輸出回答為範例
``` python
import streamlit as st
import akasha
import gc, torch
if "pre" not in st.session_state:
st.session_state.pre = ""
if "model_obj" not in st.session_state:
st.session_state.model_obj = None
def clean():
try:
gc.collect()
torch.cuda.ipc_collect()
torch.cuda.empty_cache()
except:
pass
def stream_response(prompt:str, model_name:str="openai:gpt-3.5-turbo"):
# Mistral-7B-Instruct-v0.3 Llama3-8B-Chinese-Chat
mdl_type = model_name.split(':')[0]
streaming = st.session_state.model_obj.stream(prompt)
for s in streaming:
if mdl_type == "openai":
yield s.content
else:
yield s
model = st.selectbox("select model", ["openai:gpt-3.5-turbo","hf:model/Mistral-7B-Instruct-v0.3"])
prompt = st.chat_input("Say something")
if st.session_state.pre != model:
st.session_state.model_obj = None
clean()
st.session_state.model_obj = akasha.helper.handle_model(model, False, 0.0)
st.session_state.pre = model
if prompt:
st.write("question: " + prompt)
st.write_stream(stream_response(prompt, model))
```
使用model_obj = akasha.helper.handle_model(model, False, 0.0)建立模型物件,當要使用推論時,使用model_obj.stream(prompt)進行推論,可使用yield讓stream_response函式回傳generator, 便可即時輸出回答。
### 模型為openai類型時,s.content才是輸出文字,其他類型s即是輸出文字。