解釋概念

在自然語言處理中"前處理"通常包含哪些步驟?

請說明在自然語言處理中"token"的意思。

解說語法與規則

請說明Python變數的設定規則

請說明Python中escape character的用法和列出常見的例子

請說明Python中串列和元組的差異

請說明Python中串列和元組的應用

請簡單說明python中的串列生成式

程式除錯、解釋錯誤原因和修正

設計程式時出現以下錯誤，請協助修正：
AttributeError                            Traceback (most recent call last)
<ipython-input-8-b9d8f89049c4> in <cell line: 1>()
----> 1 a.upper()

AttributeError: 'int' object has no attribute 'upper'

請Debug以下程式:
a = 235 # integer
b = "two three five" # string
c = True # boolean
d = (2, 3, 5) # tuple
e = [2, 3, 5] # list
f = {"two": 2, "three": 3, "five": 5} # dictionary
type(f)
a.upper()

閱讀與解說程式碼

請解釋以下的程式碼：

from math import log2

count_new = 0
count_york = 0
count_new_york = 0

for i,tok in enumerate(tokens):
    if tok == "new":
        count_new += 1
        try:
            next_tok = tokens[i+1]
            if next_tok == "york":
                count_new_york += 1
        except IndexError:
            pass
    if tok == "york":
        count_york += 1

p_new = count_new / len(tokens)
p_work = count_york / len(tokens)
p_new_york = count_new_york / len(tokens)
pmi = log2(p_new_york / (p_new * p_work))
print("p_new =", p_new)
print("p_york =", p_work)
print("p_new_york =", p_new_york)
print("pmi =", pmi)

為程式碼加上註解

請為以下的程式碼加上註解：

from math import log2

count_new = 0
count_york = 0
count_new_york = 0

for i,tok in enumerate(tokens):
    if tok == "new":
        count_new += 1
        try:
            next_tok = tokens[i+1]
            if next_tok == "york":
                count_new_york += 1
        except IndexError:
            pass
    if tok == "york":
        count_york += 1

p_new = count_new / len(tokens)
p_work = count_york / len(tokens)
p_new_york = count_new_york / len(tokens)
pmi = log2(p_new_york / (p_new * p_work))
print("p_new =", p_new)
print("p_york =", p_work)
print("p_new_york =", p_new_york)
print("pmi =", pmi)

依照我們的指示設計各類程式

1. 請用Python畫出log2的圖形，並包含程式坐標軸與x=1

2. Remove the punctuations of a string using Python.

3. 幫我計算'text'欄位的每一筆資料的單字字數並儲存在新欄位'length'

重構與改進程式

請改進以下程式碼：

from math import log2

count_new = 0
count_york = 0
count_new_york = 0

for i,tok in enumerate(tokens):
    if tok == "new":
        count_new += 1
        try:
            next_tok = tokens[i+1]
            if next_tok == "york":
                count_new_york += 1
        except IndexError:
            pass
    if tok == "york":
        count_york += 1

p_new = count_new / len(tokens)
p_work = count_york / len(tokens)
p_new_york = count_new_york / len(tokens)
pmi = log2(p_new_york / (p_new * p_work))
print("p_new =", p_new)
print("p_york =", p_work)
print("p_new_york =", p_new_york)
print("pmi =", pmi)

其他 (早期部份LLM可能需要)

Please use traditional chinese in all the following response