Google recently released Gemma 2 models in 9B and 27B Sizes, which are the latest models of its Gemma models family. According to its technical report, there will be an open sourced Gemma-2-2b model in the upcoming days. The technical report also demonstrates that the Gemma-2-9B model outperforms the Mistral-7B, Llama-3-8B, and the Gemma 1.5 models in several benchmarks.
In this article, taking Gemma-2-9B as an example, we will cover
- How to run Gemma-2-9B on your own device
- How to create an OpenAI-compatible API service for Gemma-2-9B
We will use LlamaEdge (the Rust + Wasm stack) to develop and deploy applications for this model. There are no complex Python packages or C++ toolchains to install! See why we choose this tech stack.
Run Gemma-2-9B on your own device
Step 1: Install WasmEdge via the following command line.
curl -sSf https://raw.githubusercontent.com/WasmEdge/WasmEdge/master/utils/install_v2.sh | bash -s -- -v 0.13.5 --ggmlbn=b3259
Step 2: Download the Gemma-2-9B-it model GGUF file. Since the size of the model is 6.40G, it could take a while to download.
curl -LO https://huggingface.co/second-state/gemma-2-9b-it-GGUF/resolve/main/gemma-2-9b-it-Q5_K_M.gguf
Step 3: Download a cross-platform portable Wasm file for the chat app. The application allows you to chat with the model on the command line. The Rust source code for the app is here.
curl -LO https://github.com/LlamaEdge/LlamaEdge/releases/latest/download/llama-chat.wasm
That's it. You can chat with the model in the terminal by entering the following command.
wasmedge --dir .:. --nn-preload default:GGML:AUTO:gemma-2-9b-it-Q5_K_M.gguf llama-chat.wasm -p gemma-instruct -c 4096
The portable Wasm app automatically takes advantage of the hardware accelerators (eg GPUs) I have on the device.
[You]:
Tell me what is gemma?
[Bot]:
Gemma is an open-weights AI assistant developed by Google DeepMind.
Here are some key things to know about Gemma:
* **Open Weights:** This means the underlying code and model weights are freely accessible to the public. Anyone can inspect, modify, or build upon Gemma.
* **Text-Only:** Gemma is designed primarily for text-based interactions. It can understand and generate human-like text but doesn't have capabilities like image recognition or voice synthesis.
* **Focus on Research & Development:** Gemma is intended as a tool for researchers and developers to explore and experiment with different AI models and techniques.
**Why is Gemma significant?**
* **Transparency and Collaboration:** Open-weights nature promotes transparency in AI development and encourages collaboration among researchers worldwide.
* **Customization:** Users can adapt Gemma to their specific needs by fine-tuning it on their own datasets.
* **Accessibility:** Makes advanced AI technology accessible to a wider range of individuals and organizations, beyond those with large resources.
**Keep in mind:**
* Gemma is still under development and may have limitations compared to more mature, closed-source models.
* Its capabilities are primarily focused on text processing.
Create an OpenAI-compatible API service for Gemma-2-9b-it
An OpenAI-compatible web API allows the model to work with a large ecosystem of LLM tools and agent frameworks such as flows.network, LangChain, and LlamaIndex.
Download an API server app. It is also a cross-platform portable Wasm app that can run on many CPU and GPU devices.
curl -LO https://github.com/LlamaEdge/LlamaEdge/releases/latest/download/llama-api-server.wasm
Then, download the chatbot web UI to interact with the model with a chatbot UI.
curl -LO https://github.com/LlamaEdge/chatbot-ui/releases/latest/download/chatbot-ui.tar.gz
tar xzf chatbot-ui.tar.gz
rm chatbot-ui.tar.gz
Next, use the following command lines to start an API server for the model. Then, open your browser to http://localhost:8080 to start the chat!
wasmedge --dir .:. --nn-preload default:GGML:AUTO:gemma-2b-it-Q5_K_M.gguf llama-api-server.wasm -p gemma-instruct -c 4096
From another terminal, you can interact with the API server using curl.
curl -X POST http://localhost:8080/v1/chat/completions \
-H 'accept:application/json' \
-H 'Content-Type: application/json' \
-d '{"messages":[{"role":"system", "content": "You are a sentient, superintelligent artificial general intelligence, here to teach and assist me."}, {"role":"user", "content": "Write a short story about Goku discovering kirby has teamed up with Majin Buu to destroy the world."}], "model":"Gemma-2b-it"}'
That’s all. WasmEdge is easiest, fastest, and safest way to run LLM applications. Give it a try!
Talk to us!
Join the WasmEdge discord to ask questions and share insights.
Any questions getting this model running? Please go to second-state/LlamaEdge to raise an issue or book a demo with us to enjoy your own LLMs across devices!