MacでMLXでローカルLLM -1-
macOSMLXLLM2025-01

MacでMLXでローカルLLM -1-

kinneko
kinneko

　GOROmanさんが、macOSでの再現手順を公開してくれたので追試する。

　MLX + MLX_VLM + Qwen2-VL-2B-Instruct-4bit で画像をVLMで解説してもらう

https://gist.github.com/GOROman/8947f32074df2370ea8c4b5877e9632b

　構築するのは、MLX + MLX_VLM + Qwen2-VL-2B-Instruct-4bitという組み合わせの環境。

　環境はM3 MBA 24GB。Sonoma 14.3。

　ストレージは足りると思うけど、メモリが足りるのだろうか？

　% cd Documents

　Pythonのバージョン確認。GOROmanさんの環境よりちょっと新しい。

　% python3 --version

　Python 3.13.1

　uvいない。

　% uv --version

　zsh: command not found: uv

　pipもいない。

　% pip install uv

　zsh: command not found: pip

　pipいたわ。

　% pip3 --version

　pip 24.3.1 from /opt/homebrew/lib/python3.13/site-packages/pip (python 3.13)

　brewはいる。

　% brew --version

　Homebrew 4.4.11

　brewを更新しておく。この環境、結構LLM関連のパッケージが入っているな。

　% brew update

　% brew upgrade

　uv入れる。

　% brew install uv

　==> Downloading https://ghcr.io/v2/homebrew/core/uv/manifests/0.5.18

######################################################################### 100.0%

　==> Fetching uv

　==> Downloading https://ghcr.io/v2/homebrew/core/uv/blobs/sha256:39e0eedf37bbab3

######################################################################### 100.0%

　==> Pouring uv--0.5.18.arm64_sonoma.bottle.tar.gz

　==> Caveats

　zsh completions have been installed to:

　/opt/homebrew/share/zsh/site-functions

　==> Summary

　🍺 /opt/homebrew/Cellar/uv/0.5.18: 17 files, 30.1MB

　==> Running brew cleanup uv...

　Disable this behaviour by setting HOMEBREW_NO_INSTALL_CLEANUP.

　Hide these hints with HOMEBREW_NO_ENV_HINTS (see man brew).

　Python仮想環境を作る。

　% mkdir MLX_VLM

　% cd MLX_VLM

　% uv venv

　Using CPython 3.13.1 interpreter at: /opt/homebrew/opt/python@3.13/bin/python3.13

　Creating virtual environment at: .venv

　Activate with: source .venv/bin/activate

　% source .venv/bin/activate

　72個ほど入る。結構入っているのがあるので、普通はもっと多いかも。

　% uv pip install mlx-vlm

　Resolved 72 packages in 1.96s

　Built markupsafe==2.1.5

　× Failed to build scipy==1.13.1

　├─▶ The build backend returned an error

　╰─▶ Call to mesonpy.build_wheel failed (exit status: 1)

[stdout]
+ meson setup
/Users/kinneko/.cache/uv/sdists-v6/pypi/scipy/1.13.1/xO5yLqJj4XpHsTYsYcAjP/src
/Users/kinneko/.cache/uv/sdists-v6/pypi/scipy/1.13.1/xO5yLqJj4XpHsTYsYcAjP/src/.mesonpy-tx6qh0sn
-Dbuildtype=release -Db_ndebug=if-release -Db_vscrt=md
--native-file=/Users/kinneko/.cache/uv/sdists-v6/pypi/scipy/1.13.1/xO5yLqJj4XpHsTYsYcAjP/src/.mesonpy-tx6qh0sn/meson-python-native-file.ini
The Meson build system
Version: 1.6.1
Source dir:
/Users/kinneko/.cache/uv/sdists-v6/pypi/scipy/1.13.1/xO5yLqJj4XpHsTYsYcAjP/src
Build dir:
/Users/kinneko/.cache/uv/sdists-v6/pypi/scipy/1.13.1/xO5yLqJj4XpHsTYsYcAjP/src/.mesonpy-tx6qh0sn
Build type: native build
Project name: scipy
Project version: 1.13.1
C compiler for the host machine: cc (clang 15.0.0 "Apple clang version
15.0.0 (clang-1500.3.9.4)")
C linker for the host machine: cc ld64 1053.12
C++ compiler for the host machine: c++ (clang 15.0.0 "Apple clang
version 15.0.0 (clang-1500.3.9.4)")
C++ linker for the host machine: c++ ld64 1053.12
Cython compiler for the host machine: cython (cython 3.0.11)
Host machine cpu family: aarch64
Host machine cpu: aarch64
Program python found: YES
(/Users/kinneko/.cache/uv/builds-v0/.tmpv0blOs/bin/python)
Did not find pkg-config by name 'pkg-config'
Found pkg-config: NO
Run-time dependency python found: YES 3.13
Program cython found: YES
(/Users/kinneko/.cache/uv/builds-v0/.tmpv0blOs/bin/cython)
Compiler for C supports arguments -Wno-unused-but-set-variable: YES
Compiler for C supports arguments -Wno-unused-function: YES
Compiler for C supports arguments -Wno-conversion: YES
Compiler for C supports arguments -Wno-misleading-indentation: YES
Library m found: YES
Fortran compiler for the host machine: gfortran (gcc 14.2.0 "GNU Fortran
(Homebrew GCC 14.2.0_1) 14.2.0")
Fortran linker for the host machine: gfortran ld64 1053.12
Compiler for Fortran supports arguments -Wno-conversion: YES
Compiler for C supports link arguments -Wl,-ld_classic: YES
Checking if "-Wl,--version-script" : links: NO
Program pythran found: YES 0.15.0 0.15.0
(/Users/kinneko/.cache/uv/builds-v0/.tmpv0blOs/bin/pythran)
Did not find CMake 'cmake'
Found CMake: NO
Run-time dependency xsimd found: NO (tried pkgconfig, framework and
cmake)
Run-time dependency threads found: YES
Library npymath found: YES
Library npyrandom found: YES
pybind11-config found: YES
(/Users/kinneko/.cache/uv/builds-v0/.tmpv0blOs/bin/pybind11-config)
2.12.1
Run-time dependency pybind11 found: YES 2.12.1
Run-time dependency scipy-openblas found: NO (tried pkgconfig)
Run-time dependency openblas found: NO (tried pkgconfig, framework and
cmake)
Run-time dependency openblas found: NO (tried framework)

　../scipy/meson.build:163:9: ERROR: Dependency lookup for OpenBLAS with

　method 'pkgconfig' failed: Pkg-config for machine host machine not

　found. Giving up.

　A full log can be found at

　/Users/kinneko/.cache/uv/sdists-v6/pypi/scipy/1.13.1/xO5yLqJj4XpHsTYsYcAjP/src/.mesonpy-tx6qh0sn/meson-logs/meson-log.txt

　hint: This usually indicates a problem with the package or the build

　environment.

　help: scipy (v1.13.1) was included because mlx-vlm (v0.1.10) depends

　on scipy

　エラーになった。pkg-config, OpenBLASが足りなかったらしい。

　pkg-config入れる。

　% brew install pkg-config

　==> Downloading https://formulae.brew.sh/api/formula.jws.json

　==> Downloading https://formulae.brew.sh/api/cask.jws.json

　==> Downloading https://ghcr.io/v2/homebrew/core/pkgconf/manifests/2.3.0_1-1

######################################################################### 100.0%

　==> Fetching pkgconf

　==> Downloading https://ghcr.io/v2/homebrew/core/pkgconf/blobs/sha256:bee6257d97

######################################################################### 100.0%

　==> Pouring pkgconf--2.3.0_1.arm64_sonoma.bottle.1.tar.gz

　🍺 /opt/homebrew/Cellar/pkgconf/2.3.0_1: 27 files, 474.3KB

　==> Running brew cleanup pkgconf...

　Disable this behaviour by setting HOMEBREW_NO_INSTALL_CLEANUP.

　Hide these hints with HOMEBREW_NO_ENV_HINTS (see man brew).

　openblas入れる。いるみたいだけど...

　% brew install openblas

　Warning: openblas 0.3.29 is already installed and up-to-date.

　To reinstall 0.3.29, run:

　brew reinstall openblas

　openblasは、仮想環境に環境変数設定する必要があるみたい。

　% brew --prefix openblas

　/opt/homebrew/opt/openblas

　% export OPENBLAS=$(/opt/homebrew/bin/brew --prefix openblas)

　% export LDFLAGS="-L${OPENBLAS}/lib"

　% export CPPFLAGS="-I${OPENBLAS}/include"

　% export PKG_CONFIG_PATH="${OPENBLAS}/lib/pkgconfig"

　再試行。パッケージキャッシュは使わないでやり直し。でも、同じエラー。

　% uv pip install --no-cache-dir mlx-vlm

　hint: This usually indicates a problem with the package or the build

　environment.

　help: scipy (v1.13.1) was included because mlx-vlm (v0.1.10) depends

　on scipy

　エラーじゃないけど、cmakeがないと言っている。

gccのシンボリックリンクになっているclangがOpenMP(並列処理ライブラリ)を有効にするオプションである-fopenmpをサポートしていないので、scipyがビルドできないということかな？

　gccいるやん。

　% brew install gcc

　==> Downloading https://formulae.brew.sh/api/formula.jws.json

######################################################################### 100.0%

　==> Downloading https://formulae.brew.sh/api/cask.jws.json

######################################################################### 100.0%

　Warning: gcc 14.2.0_1 is already installed and up-to-date.

　To reinstall 14.2.0_1, run:

　brew reinstall gcc

　なんかいっぱいある。

　% brew list gcc

　/opt/homebrew/Cellar/gcc/14.2.0_1/bin/aarch64-apple-darwin23-c++-14

　/opt/homebrew/Cellar/gcc/14.2.0_1/bin/aarch64-apple-darwin23-g++-14

　/opt/homebrew/Cellar/gcc/14.2.0_1/bin/aarch64-apple-darwin23-gcc-14

　/opt/homebrew/Cellar/gcc/14.2.0_1/bin/aarch64-apple-darwin23-gcc-ar-14

　/opt/homebrew/Cellar/gcc/14.2.0_1/bin/aarch64-apple-darwin23-gcc-nm-14

　/opt/homebrew/Cellar/gcc/14.2.0_1/bin/aarch64-apple-darwin23-gcc-ranlib-14

　/opt/homebrew/Cellar/gcc/14.2.0_1/bin/aarch64-apple-darwin23-gfortran-14

　/opt/homebrew/Cellar/gcc/14.2.0_1/bin/aarch64-apple-darwin23-gm2-14

　/opt/homebrew/Cellar/gcc/14.2.0_1/bin/c++-14

　/opt/homebrew/Cellar/gcc/14.2.0_1/bin/cpp-14

　/opt/homebrew/Cellar/gcc/14.2.0_1/bin/g++-14

　/opt/homebrew/Cellar/gcc/14.2.0_1/bin/gcc-14

　/opt/homebrew/Cellar/gcc/14.2.0_1/bin/gcc-ar-14

　/opt/homebrew/Cellar/gcc/14.2.0_1/bin/gcc-nm-14

　/opt/homebrew/Cellar/gcc/14.2.0_1/bin/gcc-ranlib-14

　/opt/homebrew/Cellar/gcc/14.2.0_1/bin/gcov-14

　/opt/homebrew/Cellar/gcc/14.2.0_1/bin/gcov-dump-14

　/opt/homebrew/Cellar/gcc/14.2.0_1/bin/gcov-tool-14

　/opt/homebrew/Cellar/gcc/14.2.0_1/bin/gfortran

　/opt/homebrew/Cellar/gcc/14.2.0_1/bin/gfortran-14

　/opt/homebrew/Cellar/gcc/14.2.0_1/bin/gm2

　/opt/homebrew/Cellar/gcc/14.2.0_1/bin/gm2-14

　/opt/homebrew/Cellar/gcc/14.2.0_1/bin/lto-dump-14

　/opt/homebrew/Cellar/gcc/14.2.0_1/include/c++/ (825 files)

　/opt/homebrew/Cellar/gcc/14.2.0_1/lib/gcc/ (1025 files)

　/opt/homebrew/Cellar/gcc/14.2.0_1/libexec/gcc/ (15 files)

　/opt/homebrew/Cellar/gcc/14.2.0_1/sbom.spdx.json

　/opt/homebrew/Cellar/gcc/14.2.0_1/share/gcc-14/ (4 files)

　/opt/homebrew/Cellar/gcc/14.2.0_1/share/man/ (12 files)

　CCまわりの環境変数をgccを使うように整備する。

　% export CC=$(brew --prefix)/bin/gcc-$(brew list gcc | grep -Eo '^[0-9]+$' | tail -n1)

　% echo $CC

　/opt/homebrew/bin/gcc-

　% export CXX=$(brew --prefix)/bin/g++-$(brew list gcc | grep -Eo '^[0-9]+$' | tail -n1)

　% echo $CXX

　/opt/homebrew/bin/g++-

　% export FC=$(brew --prefix)/bin/gfortran

　% echo $FC

　/opt/homebrew/bin/gfortran

　cmakeも入れておく。

　% brew install cmake

　Warning: Treating cmake as a formula. For the cask, use homebrew/cask/cmake or specify the --cask flag. To silence this message, use the --formula flag.

　==> Downloading https://ghcr.io/v2/homebrew/core/cmake/manifests/3.31.4

######################################################################### 100.0%

　==> Fetching cmake

　==> Downloading https://ghcr.io/v2/homebrew/core/cmake/blobs/sha256:082e2ed3a5ac

######################################################################### 100.0%

　==> Pouring cmake--3.31.4.arm64_sonoma.bottle.tar.gz

　==> Caveats

　To install the CMake documentation, run:

　brew install cmake-docs

　Emacs Lisp files have been installed to:

　/opt/homebrew/share/emacs/site-lisp/cmake

　==> Summary

　🍺 /opt/homebrew/Cellar/cmake/3.31.4: 3,767 files, 56.5MB

　==> Running brew cleanup cmake...

　Disable this behaviour by setting HOMEBREW_NO_INSTALL_CLEANUP.

　Hide these hints with HOMEBREW_NO_ENV_HINTS (see man brew).

　再再試行。

　% uv pip install --no-cache-dir mlx-vlm

　Resolved 72 packages in 2.08s

　Built markupsafe==2.1.5

　× Failed to build scipy==1.13.1

　├─▶ The build backend returned an error

　╰─▶ Call to mesonpy.build_wheel failed (exit status: 1)

[stdout]
+ meson setup
/private/var/folders/hm/k81y3ysn3fndfk0syfktk3xr0000gn/T/.tmpZVIH1y/sdists-v6/pypi/scipy/1.13.1/e8VV1m3V2oge10zRL_0Xy/src
/private/var/folders/hm/k81y3ysn3fndfk0syfktk3xr0000gn/T/.tmpZVIH1y/sdists-v6/pypi/scipy/1.13.1/e8VV1m3V2oge10zRL_0Xy/src/.mesonpy-cz5qypmn
-Dbuildtype=release -Db_ndebug=if-release -Db_vscrt=md
--native-file=/private/var/folders/hm/k81y3ysn3fndfk0syfktk3xr0000gn/T/.tmpZVIH1y/sdists-v6/pypi/scipy/1.13.1/e8VV1m3V2oge10zRL_0Xy/src/.mesonpy-cz5qypmn/meson-python-native-file.ini
The Meson build system
Version: 1.6.1
Source dir:
/private/var/folders/hm/k81y3ysn3fndfk0syfktk3xr0000gn/T/.tmpZVIH1y/sdists-v6/pypi/scipy/1.13.1/e8VV1m3V2oge10zRL_0Xy/src
Build dir:
/private/var/folders/hm/k81y3ysn3fndfk0syfktk3xr0000gn/T/.tmpZVIH1y/sdists-v6/pypi/scipy/1.13.1/e8VV1m3V2oge10zRL_0Xy/src/.mesonpy-cz5qypmn
Build type: native build
Project name: scipy
Project version: 1.13.1

　../meson.build:1:0: ERROR: Unknown compiler(s):

[['/opt/homebrew/bin/gcc-']]
The following exception(s) were encountered:
Running `/opt/homebrew/bin/gcc- --version` gave "[Errno 2] No such file
or directory: '/opt/homebrew/bin/gcc-'"

　A full log can be found at

　/private/var/folders/hm/k81y3ysn3fndfk0syfktk3xr0000gn/T/.tmpZVIH1y/sdists-v6/pypi/scipy/1.13.1/e8VV1m3V2oge10zRL_0Xy/src/.mesonpy-cz5qypmn/meson-logs/meson-log.txt

　hint: This usually indicates a problem with the package or the build

　environment.

　help: scipy (v1.13.1) was included because mlx-vlm (v0.1.10) depends

　on scipy

　だいぶ短くなったけど、まだエラー。

　指定したgccまわりの環境変数がおかしいらしい。パスを確認して明示で定義。

　% export CC=/opt/homebrew/bin/gcc-14

　% export CXX=/opt/homebrew/bin/g++-14

　% export FC=/opt/homebrew/bin/gfortran

　もっかい。

　% uv pip install --no-cache-dir mlx-vlm

　Resolved 72 packages in 1.82s

　Built markupsafe==2.1.5

　Built scipy==1.13.1

　Prepared 72 packages in 4m 51s

　Installed 72 packages in 317ms

  • aiofiles==23.2.1

  • aiohappyeyeballs==2.4.4

  • aiohttp==3.11.11

  • aiosignal==1.3.2

  • annotated-types==0.7.0

  • anyio==4.8.0

  • attrs==24.3.0

  • audioop-lts==0.2.1

  • certifi==2024.12.14

  • charset-normalizer==3.4.1

  • click==8.1.8

  • datasets==3.2.0

  • dill==0.3.8

  • fastapi==0.115.6

  • ffmpy==0.5.0

  • filelock==3.16.1

  • frozenlist==1.5.0

  • fsspec==2024.9.0

  • gradio==5.12.0

  • gradio-client==1.5.4

  • h11==0.14.0

  • httpcore==1.0.7

  • httpx==0.28.1

  • huggingface-hub==0.27.1

  • idna==3.10

  • jinja2==3.1.5

  • markdown-it-py==3.0.0

  • markupsafe==2.1.5

  • mdurl==0.1.2

  • mlx==0.22.0

  • mlx-vlm==0.1.10

  • multidict==6.1.0

  • multiprocess==0.70.16

  • numpy==2.2.1

  • orjson==3.10.14

  • packaging==24.2

  • pandas==2.2.3

  • pillow==11.1.0

  • propcache==0.2.1

  • pyarrow==18.1.0

  • pydantic==2.10.5

  • pydantic-core==2.27.2

  • pydub==0.25.1

  • pygments==2.19.1

  • python-dateutil==2.9.0.post0

  • python-multipart==0.0.20

  • pytz==2024.2

  • pyyaml==6.0.2

  • regex==2024.11.6

  • requests==2.32.3

  • rich==13.9.4

  • ruff==0.9.1

  • safehttpx==0.1.6

  • safetensors==0.5.2

  • scipy==1.13.1

  • semantic-version==2.10.0

  • shellingham==1.5.4

  • six==1.17.0

  • sniffio==1.3.1

  • starlette==0.41.3

  • tokenizers==0.21.0

  • tomlkit==0.13.2

  • tqdm==4.67.1

  • transformers==4.48.0

  • typer==0.15.1

  • typing-extensions==4.12.2

  • tzdata==2024.2

  • urllib3==2.3.0

  • uvicorn==0.34.0

  • websockets==14.1

  • xxhash==3.5.0

  • yarl==1.18.3

　ようやく通った。

　んで、実行。え？何を？ 仮想環境のディレクトリには、.venvしかいない。

　MLX_VLMって、Macのハード環境でONNX変換したものを使えるやつだったのでは？

　Qwen2-VL-2B-Instruct-4bitを入れないといけないのかな？

　あ、GOROman Gistにスクリプトも付いてたw

　% vi test.py

　import mlx.core as mx

　import numpy as np

　from mlx_vlm import load, generate

　from mlx_vlm.prompt_utils import apply_chat_template

　from mlx_vlm.utils import load_config

Load the model

model_path = "mlx-community/Qwen2-VL-2B-Instruct-4bit"
model, processor = load(model_path)
config = load_config(model_path)

Prepare input

image = ["yellow-hage.jpg"]
prompt = "Describe this image in detail."

Apply chat template

formatted_prompt = apply_chat_template(
processor, config, prompt, num_images=len(image)
)

Generate output

output = generate(model, processor, formatted_prompt, image, verbose=True, dtype=np.float32)
print(output)

　実行する。

　% python3 test.py

　None of PyTorch, TensorFlow >= 2.0, or Flax have been found. Models won't be available and only tokenizers, configuration and file/data utilities can be used.

　None of PyTorch, TensorFlow >= 2.0, or Flax have been found. Models won't be available and only tokenizers, configuration and file/data utilities can be used.

　chat_template.json: 100%|██████████████████| 1.05k/1.05k [00:00<00:00, 3.09MB/s]

　preprocessor_config.json: 100%|████████████████| 567/567 [00:00<00:00, 3.26MB/s]

　special_tokens_map.json: 100%|██████████████████| 613/613 [00:00<00:00, 455kB/s]

　model.safetensors.index.json: 100%|██████████| 108k/108k [00:00<00:00, 4.26MB/s]

　added_tokens.json: 100%|███████████████████████| 392/392 [00:00<00:00, 1.71MB/s]

　config.json: 100%|█████████████████████████| 1.41k/1.41k [00:00<00:00, 4.32MB/s]

　tokenizer_config.json: 100%|███████████████| 4.30k/4.30k [00:00<00:00, 6.69MB/s]

　merges.txt: 100%|██████████████████████████| 1.67M/1.67M [00:01<00:00, 1.45MB/s]

　vocab.json: 100%|███████████████████████████| 2.78M/2.78M [00:04<00:00, 573kB/s]

　tokenizer.json: 100%|██████████████████████| 11.4M/11.4M [00:10<00:00, 1.09MB/s]

　model.safetensors: 100%|███████████████████| 1.25G/1.25G [05:43<00:00, 3.63MB/s]

　Fetching 11 files: 100%|████████████████████████| 11/11 [05:44<00:00, 31.30s/it]

　Fetching 11 files: 100%|█████████████████████| 11/11 [00:00<00:00, 46045.25it/s]

　==========

　Image: ['yellow-hage.jpg']

　Prompt: <|im_start|>system

　You are a helpful assistant.<|im_end|>

　<|im_start|>user

　Describe this image in detail.<|vision_start|><|image_pad|><|vision_end|><|im_end|>

　<|im_start|>assistant

　Traceback (most recent call last):

　File "/Users/kinneko/Documents/MLX_VLM/test.py", line 22, in 

output = generate(model, processor, formatted_prompt, image, verbose=True, dtype=np.float32)
File "/Users/kinneko/Documents/MLX_VLM/.venv/lib/python3.13/site-packages/mlx_vlm/utils.py", line 1099, in generate
for response in stream_generate(model, processor, prompt, image, **kwargs):
~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/kinneko/Documents/MLX_VLM/.venv/lib/python3.13/site-packages/mlx_vlm/utils.py", line 1013, in stream_generate
inputs = prepare_inputs(
processor, image, prompt, image_token_index, resize_shape
)
File "/Users/kinneko/Documents/MLX_VLM/.venv/lib/python3.13/site-packages/mlx_vlm/utils.py", line 768, in prepare_inputs
images = [process_image(img, resize_shape, image_processor) for img in images]
~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/kinneko/Documents/MLX_VLM/.venv/lib/python3.13/site-packages/mlx_vlm/utils.py", line 753, in process_image
img = load_image(img)
File "/Users/kinneko/Documents/MLX_VLM/.venv/lib/python3.13/site-packages/mlx_vlm/utils.py", line 736, in load_image
raise ValueError(
f"The image {image_source} must be a valid URL or existing file."
)
ValueError: The image yellow-hage.jpg must be a valid URL or existing file.

　あ、指定してあるyellow-hage.jpgを置いてないな。置く。

　もっかい。

　% python3 test.py

　None of PyTorch, TensorFlow >= 2.0, or Flax have been found. Models won't be available and only tokenizers, configuration and file/data utilities can be used.

　None of PyTorch, TensorFlow >= 2.0, or Flax have been found. Models won't be available and only tokenizers, configuration and file/data utilities can be used.

　Fetching 11 files: 100%|████████████████████| 11/11 [00:00<00:00, 155344.59it/s]

　Fetching 11 files: 100%|█████████████████████| 11/11 [00:00<00:00, 19929.74it/s]

　==========

　Image: ['yellow-hage.jpg']

　Prompt: <|im_start|>system

　You are a helpful assistant.<|im_end|>

　<|im_start|>user

　Describe this image in detail.<|vision_start|><|image_pad|><|vision_end|><|im_end|>

　<|im_start|>assistant

　Traceback (most recent call last):

　File "/Users/kinneko/Documents/MLX_VLM/test.py", line 22, in 

output = generate(model, processor, formatted_prompt, image, verbose=True, dtype=np.float32)
File "/Users/kinneko/Documents/MLX_VLM/.venv/lib/python3.13/site-packages/mlx_vlm/utils.py", line 1099, in generate
for response in stream_generate(model, processor, prompt, image, **kwargs):
~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/kinneko/Documents/MLX_VLM/.venv/lib/python3.13/site-packages/mlx_vlm/utils.py", line 1029, in stream_generate
for n, (token, logprobs) in enumerate(
~~~~~~~~~^
generate_step(input_ids, model, pixel_values, mask, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
):
^
File "/Users/kinneko/Documents/MLX_VLM/.venv/lib/python3.13/site-packages/mlx_vlm/utils.py", line 947, in generate_step
outputs = model(input_ids, pixel_values, cache=cache, mask=mask, **kwargs)
File "/Users/kinneko/Documents/MLX_VLM/.venv/lib/python3.13/site-packages/mlx_vlm/models/qwen2_vl/qwen2_vl.py", line 106, in __call__
input_embddings = self.get_input_embeddings(
input_ids, pixel_values, image_grid_thw
)
File "/Users/kinneko/Documents/MLX_VLM/.venv/lib/python3.13/site-packages/mlx_vlm/models/qwen2_vl/qwen2_vl.py", line 69, in get_input_embeddings
hidden_states = self.vision_tower(
pixel_values, image_grid_thw, output_hidden_states=False
)
File "/Users/kinneko/Documents/MLX_VLM/.venv/lib/python3.13/site-packages/mlx_vlm/models/qwen2_vl/vision.py", line 292, in __call__
rotary_pos_emb = self.rot_pos_emb(grid_thw)
File "/Users/kinneko/Documents/MLX_VLM/.venv/lib/python3.13/site-packages/mlx_vlm/models/qwen2_vl/vision.py", line 278, in rot_pos_emb
rotary_pos_emb_full = self.rotary_pos_emb(max_grid_size)
File "/Users/kinneko/Documents/MLX_VLM/.venv/lib/python3.13/site-packages/mlx_vlm/models/qwen2_vl/vision.py", line 91, in __call__
seq = mx.arange(seqlen, dtype=inv_freq.dtype)
TypeError: arange(): incompatible function arguments. The following argument types are supported:
1. arange(start : Union[int, float], stop : Union[int, float], step : Union[None, int, float], dtype: Optional[Dtype] = None, *, stream: Union[None, Stream, Device] = None) -> array
2. arange(stop : Union[int, float], step : Union[None, int, float] = None, dtype: Optional[Dtype] = None, *, stream: Union[None, Stream, Device] = None) -> array

　Invoked with types: mlx.core.array, kwargs = { dtype: mlx.core.Dtype }

　うーん。arange()が期待する引数の型が、mlxライブラリ内で渡された型と一致していないって感じ。mlx.core.array型が想定しているのは、intやfloatだそうだ。

　ダメなら使えと書かれているパッチで解決しそう。

　とりあえずパッチを適用する。

　Fix trainer and Qwen2-VL #179

https://github.com/Blaizzy/mlx-vlm/pull/179/files

　もとのコードをバックアップ。

　% cp .venv/lib/python3.13/site-packages/mlx_vlm/models/qwen2_vl/vision.py .venv/lib/python3.13/site-packages/mlx_vlm/models/qwen2_vl/vision.py.org

　% cp .venv/lib/python3.13/site-packages/mlx_vlm/trainer/trainer.py .venv/lib/python3.13/site-packages/mlx_vlm/trainer/trainer.py.org

　tests/test_trainer.pyとかはどこにもない。

　% find ./.venv | grep test_trainer.py

　手パッチ。

　% vi .venv/lib/python3.13/site-packages/mlx_vlm/models/qwen2_vl/vision.py

　% diff -Naur .venv/lib/python3.13/site-packages/mlx_vlm/models/qwen2_vl/vision.py .venv/lib/python3.13/site-packages/mlx_vlm/models/qwen2_vl/vision.py.org

　--- .venv/lib/python3.13/site-packages/mlx_vlm/models/qwen2_vl/vision.py 2025-01-14 17:35:04

　+++ .venv/lib/python3.13/site-packages/mlx_vlm/models/qwen2_vl/vision.py.org 2025-01-14 17:21:55

　@@ -88,7 +88,7 @@

inv_freq = 1.0 / (
self.theta ** (mx.arange(0, self.dim, 2, dtype=mx.float32) / self.dim)
)
-        seq = mx.arange(seqlen.tolist(), dtype=inv_freq.dtype)
+        seq = mx.arange(seqlen, dtype=inv_freq.dtype)
freqs = mx.outer(seq, inv_freq)
return freqs

　こっちも。

　% vi .venv/lib/python3.13/site-packages/mlx_vlm/trainer/trainer.py

　% diff -Naur .venv/lib/python3.13/site-packages/mlx_vlm/trainer/trainer.py .venv/lib/python3.13/site-packages/mlx_vlm/trainer/trainer.py.org

　--- .venv/lib/python3.13/site-packages/mlx_vlm/trainer/trainer.py 2025-01-14 17:39:05

　+++ .venv/lib/python3.13/site-packages/mlx_vlm/trainer/trainer.py.org 2025-01-14 17:23:22

　@@ -89,15 +89,14 @@

image_token_index = self.config["image_token_index"]

inputs = prepare_inputs(
+            self.image_processor,
self.processor,
images,
prompts,
image_token_index,
self.image_resize_shape,
)
-        input_ids = inputs["input_ids"]
-        pixel_values = inputs["pixel_values"]
-        mask = inputs["attention_mask"]
+        input_ids, pixel_values, mask = inputs[:3]
kwargs = {
k: v
for k, v in zip(
@@ -110,10 +109,7 @@
],
inputs[3:],
)
-            for k, v in inputs.items()
-            if k not in ["input_ids", "pixel_values", "attention_mask"]
}
-
if mask is None:
mask = mx.ones_like(input_ids)

　@@ -230,11 +226,16 @@

input_ids = input_ids[:, :-1]

  •    kwargs = {

  •        k: v

  •        for k, v in batch.items()

  •        if k not in ["input_ids", "pixel_values", "attention_mask"]

  •    }

  •    kwargs = {}

  •    image_keys = [

  •        "image_grid_thw",

  •        "image_sizes",

  •        "aspect_ratio_ids",

  •        "aspect_ratio_mask",

  •        "cross_attention_mask",

  •    ]

  •    if any(key in batch for key in image_keys):

  •        kwargs = {key: batch[key] for key in image_keys if key in batch}

Forward pass

outputs = model(input_ids, pixel_values, attention_mask, **kwargs)

　何度目かの実行。

　% python3 test.py

　None of PyTorch, TensorFlow >= 2.0, or Flax have been found. Models won't be available and only tokenizers, configuration and file/data utilities can be used.

　None of PyTorch, TensorFlow >= 2.0, or Flax have been found. Models won't be available and only tokenizers, configuration and file/data utilities can be used.

　Fetching 11 files: 100%|████████████████████| 11/11 [00:00<00:00, 115922.97it/s]

　Fetching 11 files: 100%|█████████████████████| 11/11 [00:00<00:00, 47613.36it/s]

　==========

　Image: ['yellow-hage.jpg']

　Prompt: <|im_start|>system

　You are a helpful assistant.<|im_end|>

　<|im_start|>user

　Describe this image in detail.<|vision_start|><|image_pad|><|vision_end|><|im_end|>

　<|im_start|>assistant

　The image shows a person wearing a bright yellow shirt with two visible pockets on the front. The background is a solid, bright yellow color, providing a high contrast with the person's clothing. The person has a bald head and is wearing black-framed glasses.

　==========

　Prompt: 1116 tokens, 146.706 tokens-per-sec

　Generation: 53 tokens, 76.397 tokens-per-sec

　Peak memory: 3.080 GB

　The image shows a person wearing a bright yellow shirt with two visible pockets on the front. The background is a solid, bright yellow color, providing a high contrast with the person's clothing. The person has a bald head and is wearing black-framed glasses.

　キタキタ。

　いちいちpyを修正してられないので、FastAPIでブラウザから画像ドロップできるようにやり直す。

　依存パッケージはぜんぶ入っているのね。

　% uv pip install fastapi uvicorn python-multipart

　Audited 3 packages in 28ms

　ChatGPTにFastAPIでWeb画面を作ってもらう。

　FANBOXでは保存エラーになるので、Gistに置いた。

https://gist.github.com/kinneko/b795f3419fac6966ced328909b8045e9

　% python test2.py

　None of PyTorch, TensorFlow >= 2.0, or Flax have been found. Models won't be available and only tokenizers, configuration and file/data utilities can be used.

　None of PyTorch, TensorFlow >= 2.0, or Flax have been found. Models won't be available and only tokenizers, configuration and file/data utilities can be used.

　Fetching 11 files: 100%|████████████████████| 11/11 [00:00<00:00, 124695.52it/s]

　Fetching 11 files: 100%|█████████████████████| 11/11 [00:00<00:00, 57527.86it/s]

　INFO: Started server process [3747]

　INFO: Waiting for application startup.

　INFO: Application startup complete.

　INFO: Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)

　画像を選択して、アップロードすると、応答を表示する。

　INFO: Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)

　==========

　Image: ['uploads/GgKOGJObUAALnIU.jpg']

　Prompt: <|im_start|>system

　You are a helpful assistant.<|im_end|>

　<|im_start|>user

　この画像を詳細に説明してください。日本語で応答してください。<|vision_start|><|image_pad|><|vision_end|><|im_end|>

　<|im_start|>assistant

　この画像は、日本語で「山」を示しています。山は、地質の変化や地殻の動揺によって形成される山の峰や山の斜面を指します。この画像は、山の峰や山の斜面が明確に描かれ、その形状が美しいです。

　==========

　Prompt: 3032 tokens, 88.393 tokens-per-sec

　Generation: 71 tokens, 65.322 tokens-per-sec

　Peak memory: 12.111 GB

　INFO: 127.0.0.1:52981 - "POST /process/ HTTP/1.1" 200 OK

　何度もリクエストを投げると、英語だったのが中国語になってしまった。さすがQwen2だな。というわけで、このサンプルではプロンプトを日本語に強制してやる。なんだ、Qwen2、日本語できるんじゃん...

prompt = "Describe this image in detail. Respond in English only."
prompt = "この画像を詳細に説明してください。日本語で応答してください。"

　日本語への翻訳はしなくてよさそうなので、残りの課題はこんな感じ。やるかどうかはわからない。

　・1.5MBのモナリザをアップロードしたらエラーになった。というか落ちた。1M以下なら大丈夫そう。

　・レスポンスが改行しないので読みにくい

　・macOSのsayコマンドでいいので、結果を読み上げる

　・トークンが足りずに応答が途中で切れることがある

　このページの絵を変換して解説させてみた。Llama 3.2-Visionのほうが優秀だな。プロンプト細かく指定したら違うのだろうか？

　Ollama で Llama 3.2-Vision のテストと翻訳 #Python - Qiita

https://qiita.com/7shi/items/500fc95ecf80866ba83a

{
"description": "この画像は、アニメーションで描かれた女性のイラストです。女性は短い髪を留め、黄色い瞳を特徴とします。彼女は赤いジャケットを着ており、彼女はピザを食べています。ピザは赤いカレーと赤いトマトのシートで、女性の手にはピザを食べている様子が見えます。背景には窓があり、窓の外は明るい光が入っていることが分かります。"
}

　読んでないけど、あとで参考になるかも。

　「Llama-3.2-11B-Vision-Instruct-4bit」と「MLX-VLM」を使ったローカルLLM ＋ 画像入力を M4 Mac mini で試した時のメモ #Python - Qiita

https://qiita.com/youtoy/items/90b80eb1a3b3cac4c184

オリジナル投稿：
MacでMLXでローカルLLM -1-｜kinneko｜pixivFANBOX
https://kinneko.fanbox.cc/posts/9209714

LLMモジュール用に給電HUB買う -2-
Prev post

LLMモジュール用に給電HUB買う -2-
Next post

新年早々Aliexpressにやられた

新年早々Aliexpressにやられた
Copyright 2021,2022,2023,2024,2025,2026 kinneko@gmail.com