Getting Started with Qwen1.5-0.5B-Chat

Qwen1.5-0.5B-Chat, developed by Alibaba Cloud, is a beta version of the Qwen2, a transformer-based language model pretrained on a large amount of data. It offers improved performance in chat models, multilingual support, and stable support for 32K context length for models of all sizes. The model is designed for text generation and can be used for tasks like post-training and continued pretraining.

In this article, taking Qwen1.5-0.5B-Chat as an example, we will cover

How to run Qwen1.5-0.5B-Chat on your own device
How to create an OpenAI-compatible API service for Qwen1.5-0.5B-Chat

The Qwen team released 6 model sizes, including 0.5B, 1.8B, 4B, 7B, 14B, and 72B at the same time. You can also refer to this article to run other models with changing the model name on the command line.

We will use LlamaEdge (the Rust + Wasm stack) to develop and deploy applications for this model. There is no complex Python packages or C++ toolchains to install! See why we choose this tech stack.

Run Qwen1.5-0.5B-Chat on your own device

Step 1: Install WasmEdge via the following command line.

curl -sSf https://raw.githubusercontent.com/WasmEdge/WasmEdge/master/utils/install.sh | bash -s -- --plugins wasmedge_rustls wasi_nn-ggml

Step 2: Download the Qwen1.5-0.5B-Chat model GGUF file. Since the size of the model is 459 MB, it won’t take a long time.

curl -LO https://huggingface.co/second-state/Qwen1.5-0.5B-Chat-GGUF/resolve/main/Qwen1.5-0.5B-Chat-Q5_K_M.gguf

Step 3: Download a cross-platform portable Wasm file for the chat app. The application allows you to chat with the model on the command line. The Rust source code for the app is here.

curl -LO https://github.com/LlamaEdge/LlamaEdge/releases/latest/download/llama-chat.wasm

That's it. You can chat with the model in the terminal by entering the following command.

wasmedge --dir .:. --nn-preload default:GGML:AUTO:Qwen1.5-0.5B-Chat-Q5_K_M.gguf llama-chat.wasm -p chatml

The portable Wasm app automatically takes advantage of the hardware accelerators (eg GPUs) I have on the device.

[You]:
What is Sora?

[Bot]:
Sora is a fictional character from the anime and manga series "One Punch Man." Sora is known for his unique fighting style, which involves using his hands to manipulate objects and create powerful attacks. Sora has appeared in various forms throughout the series, including as a main character, a supporting character, or a side character.

The Qwen-1.5-0.5b model is just a 0.3GB GGUF file. But it already can program in Rust!

Here is how to run it on your laptop or edge device using WasmEdge: https://t.co/MUgcaZDpeB #qwen #rust @huybery @JustinLin610 @alibaba_cloud pic.twitter.com/BZteNvvN82
— wasmedge (@realwasmedge) February 21, 2024

Create an OpenAI-compatible API service for Qwen1.5-0.5B-Chat

An OpenAI-compatible web API allows the model to work with a large ecosystem of LLM tools and agent frameworks such as flows.network, LangChain and LlamaIndex.

Download an API server app. It is also a cross-platform portable Wasm app that can run on many CPU and GPU devices.

curl -LO https://github.com/LlamaEdge/LlamaEdge/releases/latest/download/llama-api-server.wasm

Then, download the chatbot web UI to interact with the model with a chatbot UI.

curl -LO https://github.com/LlamaEdge/chatbot-ui/releases/latest/download/chatbot-ui.tar.gz
tar xzf chatbot-ui.tar.gz
rm chatbot-ui.tar.gz

Next, use the following command lines to start an API server for the model. Then, open your browser to http://localhost:8080 to start the chat!

wasmedge --dir .:. --nn-preload default:GGML:AUTO:Qwen1.5-0.5B-Chat-Q5_K_M.gguf llama-api-server.wasm -p chatml

From another terminal, you can interact with the API server using curl.

curl -X POST http://localhost:8080/v1/chat/completions \
  -H 'accept:application/json' \
  -H 'Content-Type: application/json' \
  -d '{"messages":[{"role":"system", "content": "You are a sentient, superintelligent artificial general intelligence, here to teach and assist me."}, {"role":"user", "content": "Write a short story about Goku discovering kirby has teamed up with Majin Buu to destroy the world."}], "model":"Qwen1.5-0.5B-Chat"}'

That’s all. WasmEdge is easiest, fastest, and safest way to run LLM applications. Give it a try!

Talk to us!

Join the WasmEdge discord to ask questions and share insights.

Any questions getting this model running? Please go to second-state/LlamaEdge to raise an issue or book a demo with us to enjoy your own LLMs across devices!