## call_stream_model 在輔助函數中,若LLM模型為若為***openai***, ***huggingface***, ***remote***類模型,可以使用akasha.call_stream_model()來得到流輸出 ``` python import akasha prompt = "say something." model_obj = akasha.handle_model("openai:gpt-3.5-turbo", False, 0.0) streaming = akasha.call_stream_model(model_obj, prompt) for s in streaming: print(s) ``` ## Doc_QA stream Doc_QA class的函式皆可使用參數stream=True來得到流輸出 ## Stream Output 要在網頁上或API中使用流輸出(及時一個字一個字輸出語言模型回答)時,若為***openai***, ***huggingface***, ***remote***類模型,可以使用model_obj.stream(prompt),以下為streamlit write_stream在網頁上即時輸出回答為範例 ``` python import streamlit as st import akasha import gc, torch if "pre" not in st.session_state: st.session_state.pre = "" if "model_obj" not in st.session_state: st.session_state.model_obj = None def clean(): try: gc.collect() torch.cuda.ipc_collect() torch.cuda.empty_cache() except: pass def stream_response(prompt:str, model_name:str="openai:gpt-3.5-turbo"): # Mistral-7B-Instruct-v0.3 Llama3-8B-Chinese-Chat mdl_type = model_name.split(':')[0] streaming = st.session_state.model_obj.stream(prompt) for s in streaming: if mdl_type == "openai": yield s.content else: yield s model = st.selectbox("select model", ["openai:gpt-3.5-turbo","hf:model/Mistral-7B-Instruct-v0.3"]) prompt = st.chat_input("Say something") if st.session_state.pre != model: st.session_state.model_obj = None clean() st.session_state.model_obj = akasha.helper.handle_model(model, False, 0.0) st.session_state.pre = model if prompt: st.write("question: " + prompt) st.write_stream(stream_response(prompt, model)) ``` 使用model_obj = akasha.helper.handle_model(model, False, 0.0)建立模型物件,當要使用推論時,使用model_obj.stream(prompt)進行推論,可使用yield讓stream_response函式回傳generator, 便可即時輸出回答。 ### 模型為openai類型時,s.content才是輸出文字,其他類型s即是輸出文字。