LLMモジュールとM5Stick C Plus2との通信は、/dev/ttyS1で行われている。CPlus2には、スケッチが書き込まれている。ここまでは前にやったところ。
LLM Moduleを普通に使う｜kinneko｜pixivFANBOX
https://kinneko.fanbox.cc/posts/9376067
socatでシリアルポートを仮想化して、中継し、ミラーした結果を取ろうかと思ったけど、なかなかうまくいかない。
基本的な考え方は、/dev/ttyS1を/dev/ttyS1_realにリネームし、socatコマンドを使って/dev/ttyS1という名前の仮想シリアルポートを作成し、それを/dev/ttyS1_realに接続するというもの。realのほうにデータがコピーされて流れてくるので、それを見るという感じ。
でも、うまくいかないので放棄する。
llm_sysの設定で、使用するシリアルポートが指定できれば、もう少し楽なのだけど、残念ながら設定ファイルには、そのための情報がない。telnetのポート設定があるので、シリアルもあるかもしれないのだけど、ハードコードの可能性もないでもない。
さて、あとは出力の解釈が面倒だけど、straceを使って、やりとりをモニタする方法がある。
llm_sysのプロセスを調べる。
root@m5stack-LLM:~# ps ax |grep llm
2643 ? Ssl 0:00 /opt/m5stack/bin/llm_sys
2644 ? Ssl 0:00 /opt/m5stack/bin/llm_asr
2645 ? Ssl 0:07 /opt/m5stack/bin/llm_audio
2646 ? Ssl 0:00 /opt/m5stack/bin/llm_camera
2647 ? Ssl 0:48 /opt/m5stack/bin/llm_kws
2648 ? Ssl 0:02 /opt/m5stack/bin/llm_llm
2649 ? Ssl 0:12 /opt/m5stack/bin/llm_melotts
2650 ? Ss 0:00 /opt/m5stack/bin/llm_skel
2655 ? Ssl 0:00 /opt/m5stack/bin/llm_tts
2656 ? Ssl 0:00 /opt/m5stack/bin/llm_vlm
2661 ? Ssl 0:00 /opt/m5stack/bin/llm_yolo
ポートを指定して、straceする。
root@m5stack-LLM:~# strace -p 2643 -e trace=read,write,ioctl -s 1024 -o serial_trace.log -f
strace: Process 2643 attached with 19 threads
strace: Process 3964 attached
strace: Process 3965 attached
strace: Process 3966 attached
strace: Process 3967 attached
strace: Process 3971 attached
strace: Process 3972 attached
strace: Process 3973 attached
strace: Process 3974 attached
strace: Process 3975 attached
strace: Process 3976 attached
strace: Process 3977 attached
strace: Process 3978 attached
strace: Process 3979 attached
strace: Process 3980 attached
strace: Process 3981 attached
あれ？ 切れた。
straceのログは人間が読めるしろものではないけど、AIであれば文句を言わずに人間にわかるやりとりにしてくれる。それによると、ここまでのログでは以下のようなやりとりがある。
Pingの送信：
{"request_id":"sys_ping","work_id":"sys","action":"ping","object":"None","data":"None"}
Pingの応答：
{"created":1739962203,"data":"None","error":{"code":0,"message":""},"object":"None","request_id":"sys_ping","work_id":"sys"}
Version確認の送信：
{"request_id":"sys_version","work_id":"sys","action":"version","object":"None","data":"None"}
Version確認の応答：
{"created":1739962203,"data":"v1.3","error":{"code":0,"message":""},"object":"sys.utf-8","request_id":"sys_version","work_id":"sys"}
Resetコマンドの送信：
{"request_id":"sys_reset","work_id":"sys","action":"reset","object":"None","data":"None"}
Resetコマンドの応答：
{"created":1739962204,"data":"None","error":{"code":0,"message":"llm server restarting ..."},"object":"None","request_id":"sys_reset","work_id":"sys"}
ログが短いのは、リセットでプロセスが再起動されてしまって、当初のプロセスが終わっているからか...
リセットが起きているのは、Arduinoのスケッチでは、このあたりの処理の後になる。
while (1) {
if (module_llm.checkConnection()) {
break;
}
}
checkConnection()は、M5ModuleLLM.cppにあるけど、リセットはしてない感じ。
bool M5ModuleLLM::checkConnection()
{
const bool result = (sys.ping() == MODULE_LLM_OK);
llm_version = (sys.version() == MODULE_LLM_OK);
return result;
}
応答は正常に返っているので、エラーでリセットが発動しているわけではなさそう。リセットどこでしよるとね？
リセット後にstraceを再度プロセスIDを入れて動かすけれど、やっぱり欠ける部分がある。
たぶん、まずこんな感じでAudioの初期化しているのだろうけど、それがない。
{
"request_id": "1",
"work_id": "audio",
"action": "setup",
"object": "audio.setup",
"data": {
"capcard": 0,
"capdevice": 0,
"capVolume": 0.5,
"playcard": 0,
"playdevice": 1,
"playVolume": 0.5
}
}
Audioの初期化の応答もない。
次にKWSユニットの初期化があるのだろうけど、それもない。たぶん、こんな感じで流れているのだろう。
{
"request_id": "2",
"work_id": "kws",
"action": "setup",
"object": "kws.setup",
"data": {
"model": "sherpa-onnx-kws-zipformer-gigaspeech-3.3M-2024-01-01",
"response_format": "kws.bool",
"input": "sys.pcm",
"enoutput": true,
"kws": "ECHO"
}
}
KWSユニットの初期化の応答からは取れた。
{"created":1739964629,"data":"None","error":{"code":0,"message":""},"object":"None","request_id":"kws_setup","work_id":"kws.1001"}
ASR(音声認識)のセットアップ。応答はないな。
{
"request_id": "asr_setup",
"work_id": "asr",
"action": "setup",
"object": "asr.setup",
"data": {
"model": "sherpa-ncnn-streaming-zipformer-20M-2023-02-17",
"response_format": "asr.utf-8.stream",
"enoutput": true,
"enkws": true,
"rule1": 2.4,
"rule2": 1.2,
"rule3": 30,
"input": ["sys.pcm","kws.1001"]
}
}
LLM(大規模言語モデル)のセットアップ。こちらも応答がない。
{
"request_id": "llm_setup",
"work_id": "llm",
"action": "setup",
"object": "llm.setup",
"data": {
"model": "qwen2.5-0.5B-prefill-20e",
"response_format": "llm.utf-8.stream",
"enoutput": true,
"enkws": true,
"max_token_len": 127,
"prompt": "",
"input": ["asr.1002","kws.1001"]
}
}
TTS(音声合成)のセットアップ。これも応答がない。
{
"request_id": "tts_setup",
"work_id": "melotts",
"action": "setup",
"object": "melotts.setup",
"data": {
"model": "melotts_zh-cn",
"response_format": "sys.pcm",
"input": ["llm.1003","kws.1001"],
"enoutput": false,
"enaudio": true
}
}
telnetから再現できるかどうか調べる。
root@m5stack-LLM:~# telnet localhost 10001
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
{"request_id":"sys_ping","work_id":"sys","action":"ping","object":"None","data":"None"}
{"created":1739966632,"data":"None","error":{"code":0,"message":""},"object":"None","request_id":"sys_ping","work_id":"sys"}
{"request_id":"sys_version","work_id":"sys","action":"version","object":"None","data":"None"}
{"created":1739966664,"data":"v1.3","error":{"code":0,"message":""},"object":"sys.utf-8","request_id":"sys_version","work_id":"sys"}
{"request_id":"sys_reset","work_id":"sys","action":"reset","object":"None","data":"None"}
{"created":1739966698,"data":"None","error":{"code":0,"message":"llm server restarting ..."},"object":"None","request_id":"sys_reset","work_id":"sys"}
Connection closed by foreign host.
リセットすると切れるというのは確認できた。
再接続して続き。一応、送った初期化コマンドには、すべてのサービスが正常応答を返している。
root@m5stack-LLM:~# telnet localhost 10001
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
{
"request_id": "1",
"work_id": "audio",
"action": "setup",
"object": "audio.setup",
"data": {
"capcard": 0,
"capdevice": 0,
"capVolume": 0.5,
"playcard": 0,
"playdevice": 1,
"playVolume": 0.5
}
}
{"created":1739966741,"data":"None","error":{"code":0,"message":""},"object":"None","request_id":"1","work_id":"audio"}
{
"request_id": "2",
"work_id": "kws",
"action": "setup",
"object": "kws.setup",
"data": {
"model": "sherpa-onnx-kws-zipformer-gigaspeech-3.3M-2024-01-01",
"response_format": "kws.bool",
"input": "sys.pcm",
"enoutput": true,
"kws": "ECHO"
}
}
{"created":1739966802,"data":"None","error":{"code":0,"message":""},"object":"None","request_id":"2","work_id":"kws.1001"}
{
"request_id": "asr_setup",
"work_id": "asr",
"action": "setup",
"object": "asr.setup",
"data": {
"model": "sherpa-ncnn-streaming-zipformer-20M-2023-02-17",
"response_format": "asr.utf-8.stream",
"enoutput": true,
"enkws": true,
"rule1": 2.4,
"rule2": 1.2,
"rule3": 30,
"input": ["sys.pcm","kws.1001"]
}
}
{"created":1739966845,"data":"None","error":{"code":0,"message":""},"object":"None","request_id":"asr_setup","work_id":"asr.1002"}
{
"request_id": "llm_setup",
"work_id": "llm",
"action": "setup",
"object": "llm.setup",
"data": {
"model": "qwen2.5-0.5B-prefill-20e",
"response_format": "llm.utf-8.stream",
"enoutput": true,
"enkws": true,
"max_token_len": 127,
"prompt": "",
"input": ["asr.1002","kws.1001"]
}
}
{"created":1739966895,"data":"None","error":{"code":0,"message":""},"object":"None","request_id":"llm_setup","work_id":"llm.1003"}
{
"request_id": "tts_setup",
"work_id": "melotts",
"action": "setup",
"object": "melotts.setup",
"data": {
"model": "melotts_zh-cn",
"response_format": "sys.pcm",
"input": ["llm.1003","kws.1001"],
"enoutput": false,
"enaudio": true
}
}
{"created":1739966941,"data":"None","error":{"code":0,"message":""},"object":"None","request_id":"tts_setup","work_id":"melotts.1004"}
初期化はできたので、音声応答を試す。「エコー」と声をかけてみる。ログが出て、差し替えた応答音がした。
{"created":1739966961,"data":true,"error":{"code":0,"message":""},"object":"kws.bool","request_id":"2","work_id":"kws.1001"}
「are you fine?」と聞いたのだけど、「are you wying」と認識されている。
{"created":1739966986,"data":true,"error":{"code":0,"message":""},"object":"kws.bool","request_id":"2","work_id":"kws.1001"}
{"created":1739966990,"data":{"delta":" are you wy","finish":false,"index":0},"error":{"code":0,"message":""},"object":"asr.utf-8.stream","request_id":"asr_setup","work_id":"asr.1002"}
{"created":1739966991,"data":{"delta":" are you wying","finish":false,"index":1},"error":{"code":0,"message":""},"object":"asr.utf-8.stream","request_id":"asr_setup","work_id":"asr.1002"}
{"created":1739966991,"data":{"delta":" are you wying.","finish":true,"index":2},"error":{"code":0,"message":""},"object":"asr.utf-8.stream","request_id":"asr_setup","work_id":"asr.1002"}
{"created":1739966991,"data":{"delta":"No, I","finish":false,"index":0},"error":{"code":0,"message":""},"object":"llm.utf-8.stream","request_id":"llm_setup","work_id":"llm.1003"}
{"created":1739966992,"data":{"delta":" am here to","finish":false,"index":1},"error":{"code":0,"message":""},"object":"llm.utf-8.stream","request_id":"llm_setup","work_id":"llm.1003"}
{"created":1739966992,"data":{"delta":" assist you with","finish":false,"index":2},"error":{"code":0,"message":""},"object":"llm.utf-8.stream","request_id":"llm_setup","work_id":"llm.1003"}
{"created":1739966993,"data":{"delta":" any questions or","finish":false,"index":3},"error":{"code":0,"message":""},"object":"llm.utf-8.stream","request_id":"llm_setup","work_id":"llm.1003"}
{"created":1739966993,"data":{"delta":" concerns you may","finish":false,"index":4},"error":{"code":0,"message":""},"object":"llm.utf-8.stream","request_id":"llm_setup","work_id":"llm.1003"}
{"created":1739966993,"data":{"delta":" have. How","finish":false,"index":5},"error":{"code":0,"message":""},"object":"llm.utf-8.stream","request_id":"llm_setup","work_id":"llm.1003"}
{"created":1739966993,"data":{"delta":" can I help","finish":false,"index":6},"error":{"code":0,"message":""},"object":"llm.utf-8.stream","request_id":"llm_setup","work_id":"llm.1003"}
{"created":1739966994,"data":{"delta":" you today?","finish":false,"index":7},"error":{"code":0,"message":""},"object":"llm.utf-8.stream","request_id":"llm_setup","work_id":"llm.1003"}
{"created":1739966994,"data":{"delta":"\n","finish":false,"index":8},"error":{"code":0,"message":""},"object":"llm.utf-8.stream","request_id":"llm_setup","work_id":"llm.1003"}
{"created":1739966994,"data":{"delta":"","finish":true,"index":9},"error":{"code":0,"message":""},"object":"llm.utf-8.stream","request_id":"llm_setup","work_id":"llm.1003"}
応答の生成と発音もできている。
あとは、起動時にこのプロセスを内部で呼べれば、LLMモジュール単独で動作できる。
シリアルから通信手順的にやるか、telnetからやるか... 悩ましい。動作プロセスが死ぬと、リセットされちゃうしなぁ...
100円投げ銭 https://kinneko.booth.pm/items/1963720
オリジナル投稿：
#M5StackLLM 通信の内容を調べる｜kinneko｜pixivFANBOX
https://kinneko.fanbox.cc/posts/9410149
