NVIDIA released “Chat with RTX,” an AI chatbox accelerated using TensorRT-LLM and supported on RTX 30 and 40 series GPUs last month. It uses generative AI to answer questions using a localized dataset on your PC or a web address. Such a model is highly efficient for generating results from a set of data points available offline, significantly improving accessibility and privacy while reducing processing costs and time. As usual, AMD has followed up with a solution of its own that’s compatible across a wider range of devices. We’ll call it “Chat with Ryzen AI.”
“Chat with Ryzen AI” is supported on the Ryzen 7000 and Ryzen 8000 processors with the “XDNA” NPU and the “Ryzen AI” branding. This includes the Ryzen 7040 “Phoenix Point” and the Ryzen 8040 “Hawk Point” processors and the Radeon RX 7000 “RDNA 3” series GPUs. AMD has released a guide on how to run an LLM-based chatbot on your PC. The process for the GPU and CPUs is slightly different, but shouldn’t take long to initialize.
3. In the search tab copy and paste the following search term depending on what you want to run:
a. If you would like to run Mistral 7b, search for: “TheBloke/OpenHermes-2.5-Mistral-7B-GGUF” and select it from the results on the left. It will typically be the first result. We are going with Mistral in this example.
b. If you would like to run LLAMA v2 7b, search for: “TheBloke/Llama-2-7B-Chat-GGUF” and select it from the results on the left. It will typically be the first result.
c. You can also experiment with other models here.
4. On the right-hand panel, scroll down till you see the Q4 K M model file. Click download.
a. We recommend Q4 K M for most models on Ryzen AI. Wait for it to finish downloading.
5. Go to the chat tab. Select the model from the central, drop-down menu in the top center and wait for it to finish loading up.
6. If you have an AMDRyzen AI PC you can start chatting!
a. If you have an AMDRadeon graphics card, please:
i. Check “GPU Offload” on the right-hand side panel.
ii. Move the slider all the way to “Max”.
iii. Make sure AMD ROCm is being shown as the detected GPU type.
iv. Start chatting!
Unfortunately, the process is more complex than “Chat with RTX,” requiring manual selection of the model and processor in LM Studio. Hopefully, AMD will streamline the process in future releases.
Processors, PC gaming, and the past. I have written about computer hardware for over seven years with over 5000 published articles. I started during engineering college and haven't stopped since. On the side, I play RPGs like Baldur's Gate, Dragon Age, Mass Effect, Divinity, and Fallout. Contact: areejs12@hardwaretimes.com.
