DeepSeek-R1-Distill-Qwen is a series of distilled large language models derived from Qwen 2.5, utilizing outputs from the larger DeepSeek-R1 model. These models are designed to be more efficient and compact while retaining strong performance, especially in reasoning tasks. The distillation process allows them to inherit the knowledge and capabilities of the larger model, making them suitable for resource-constrained environments and easier deployment.
These distilled models have shown impressive results across various benchmarks, often outperforming other models of similar size. For example, the 1.5B parameter version has demonstrated superior performance in math and logical reasoning tasks compared to models like GPT-4o and Claude 3.5. DeepSeek-R1-Distill-Qwen models are available in various sizes, offering a range of options for different needs and applications
- DeepSeek-R1-Distill-Qwen-1.5B: https://huggingface.co/second-state/DeepSeek-R1-Distill-Qwen-1.5B-GGUF
- DeepSeek-R1-Distill-Qwen-7B: https://huggingface.co/second-state/DeepSeek-R1-Distill-Qwen-7B-GGUF
- DeepSeek-R1-Distill-Qwen-14B: https://huggingface.co/second-state/DeepSeek-R1-Distill-Qwen-14B-GGUF
- DeepSeek-R1-Distill-Qwen-32B: https://huggingface.co/second-state/DeepSeek-R1-Distill-Qwen-32B-GGUF
In this article, we will cover how to run and interact with DeepSeek-R1-Distill-Qwen-1.5B on your own edge device.
We will use the Rust + Wasm stack to develop and deploy applications for this model. There are no complex Python packages or C++ toolchains to install! See why we choose this tech stack.
Run DeepSeek-R1-Distill-Qwen-1.5B-GGUF
Step 1: Install WasmEdge via the following command line.
curl -sSf https://raw.githubusercontent.com/WasmEdge/WasmEdge/master/utils/install_v2.sh | bash -s -- -v 0.14.1
Step 2: Download the quantized DeepSeek-R1-Distill-Qwen-1.5B-GGUF model file. The model is 1.29 GB in size and should be ready soon. If you want to run a different model, you will need to change the model download link below.
curl -LO https://huggingface.co/second-state/DeepSeek-R1-Distill-Qwen-1.5B-GGUF/resolve/main/DeepSeek-R1-Distill-Qwen-1.5B-Q5_K_M.gguf
Step 3: Step 3: Download the LlamaEdge API serverapp. It is also a cross-platform portable Wasm app that can run on many CPU and GPU devices.
curl -LO https://github.com/LlamaEdge/LlamaEdge/releases/latest/download/llama-api-server.wasm
Step 4: Download the chatbot UI to interact with the DeepSeek-R1-Distill-Qwen model in the browser.
curl -LO https://github.com/LlamaEdge/chatbot-ui/releases/latest/download/chatbot-ui.tar.gz
tar xzf chatbot-ui.tar.gz
rm chatbot-ui.tar.gz
Next, use the following command lines to start a LlamaEdge API server for the model. LlamaEdge provides an OpenAI compatible API, and you can connect any chatbot client or agent to it!
wasmedge --dir .:. --nn-preload default:GGML:AUTO:DeepSeek-R1-Distill-Qwen-1.5B-Q5_K_M.gguf \
llama-api-server.wasm \
--prompt-template chatml \
--ctx-size 8096 \
--model-name DeepSeek-R1-Distill-Qwen-1.5B-Q5_K_M
Chat
Open your browser and visit http://localhost:8080 to chat with the DeepSeek-R1-Distill-Qwen-1.5B model! Our new version of Chatbot UI supports the recognition of Deepseek's unique
Use the API
The LlamaEdge API server is fully compatible with OpenAI API specs. You can send an API request to the model.
curl -X POST http://localhost:8080/v1/chat/completions \
-H 'accept:application/json' \
-H 'Content-Type: application/json' \
-d '{"messages":[{"role":"system", "content": "You are a helpful assistant."}, {"role":"user", "content": "Can a person be at the North Pole and the South Pole at the same time??"}], "model": "DeepSeek-R1-Distill-Qwen-1.5B-Q5_K_M"}'
{
"id": "chatcmpl-de54d623-d35d-4ef2-b477-e1094bbbb171",
"object": "chat.completion",
"created": 1738924494,
"model": "DeepSeek-R1-Distill-Qwen-1.5B-Q5_K_M",
"choices": [
{
"index": 0,
"message": {
"content": "Okay, so the user is asking if it's possible for a person to be at both the North Pole and the South Pole at the same time. Hmm, let me think about this.\n\nFirst, I know that the Earth is a sphere, right? And on this sphere, there are two main points of interest: the North Pole and the South Pole. These are antipodal points, meaning they're directly opposite each other across the Earth's surface. So if you were standing at one pole, the exact opposite point would be where you'd be facing when you turned halfway around the globe.\n\nNow, thinking about people, we live on the surface of the Earth, which is roughly a sphere. So physically, someone can only be in one place at a time unless they change their location suddenly. But even then, how exactly could that happen? Like, if I were to move from my home to another city, I would pass through most of the cities along my route.\n\nBut wait, in this case, both poles are points on the Earth's surface. So can someone exist at both places simultaneously? Well, logically, yes, mathematically, because they occupy different locations but lie on the same sphere. But practically, physically, that seems impossible because you can't occupy two distinct points in space unless your location changes instantaneously.\n\nBut hold on, maybe we're talking about something else here. The question doesn't specify whether it's talking about a person moving or being at those poles simultaneously without changing locations. If someone is at both poles at the same time, that would imply they are in two different places at once, which isn't possible because you can only be in one place at any given moment.\n\nAnother angle: maybe we're considering whether these points exist as separate entities rather than specific physical locations. But even if we think of them as mathematical points on a sphere, someone can still only be at one of those points or none of them simultaneously unless they move into both.\n\nI should also consider the Earth's shape. The Earth is slightly oblate due to its rotation, meaning it's not a perfect sphere but rather an ellipsoid. In that case, even though there are two poles, the exact antipodal point might be slightly different or perhaps non-existent because of this shape difference. But I think for the purpose of this question, we can assume a more idealized Earth without considering its oblateness.\n\nSo to sum up, unless someone changes their location instantaneously, which isn't practical on a human scale, they cannot be at both poles simultaneously. The North Pole and South Pole are distinct points on the Earth's surface that are exactly opposite each other. Therefore, in terms of physical existence, only one of them can be occupied at any given time.\n\nWait, but what about someone who is alive? Could their body somehow occupy both poles at once? That seems impossible because a person is a living entity occupying a three-dimensional space. The North Pole and South Pole are points on the Earth's surface, which is a two-dimensional manifold embedded in three-dimensional space. So, in topology, they are separate entities.\n\nBut perhaps someone can be in both places through their body somehow? Hmm, that doesn't make much sense because each part of the body occupies specific physical locations. For instance, the head would have to be at one point, and the body parts spread out around it. It seems implausible for any single person to be simultaneously at both poles.\n\nMoreover, in terms of human anatomy, there's no structure that allows a person to switch places instantaneously or occupy two points at once without changing their location over time. So even if we consider the body as a whole occupying space, each part must have its own distinct position.\n\nTherefore, after thinking through all these possibilities, it seems clear that only one of the North Pole and South Pole can be occupied by a person unless they change their location during a transition period, which would require some instantaneous movement or shifting in their position.\n</think>\n\nNo, it is not possible for a person to exist simultaneously at both the North Pole and the South Pole. These are two distinct points on Earth that are antipodal to each other, lying on opposite sides of the Earth. A person can only occupy one location at any given time unless they change their position, which would involve some sort of movement or shifting in their physical presence.\n\nAnswer: No, a person cannot be simultaneously at both the North Pole and the South Pole.<｜end▁of▁sentence｜>",
"role": "assistant"
},
"finish_reason": "stop",
"logprobs": **null**
}
],
"usage": {
"prompt_tokens": 51,
"completion_tokens": 518,
"total_tokens": 569
}
}
RAG and embeddings
Finally, if you are using this model to create agentic or RAG applications, you will likely need an API to compute vector embeddings for the user request text. That can be done by adding an embedding model to the LlamaEdge API server. Learn how this is done.
Gaia
Alternatively, the Gaia network software allows you to stand up the Mistral LLM, embedding model, and a vector knowledge base in a single command. Try it with DeepSeek-R1!
Join the WasmEdge discord to share insights. Any questions about getting this model running? Please go to second-state/LlamaEdge to raise an issue or book a demo with us to enjoy your own LLMs across devices!