NVIDIA has unveiled Chat with RTX, showcasing the capability of AI chatbots to operate locally on Windows PCs utilizing its RTX GPUs. Traditionally, AI chatbots rely on web platforms like ChatGPT or API queries, which entail drawbacks such as costs, latency, and privacy concerns associated with data transfer over cloud computing servers.Now, with NVIDIA's RTX range of GPUs, running a Language Model (LLM) locally on a Windows PC is feasible even without internet connectivity. Chat with RTX empowers users to create personalized chatbots using Mistral or Llama 2, incorporating retrieval-augmented generation (RAG) and NVIDIA's inference-optimizing TensorRT-LLM software.

Users can direct Chat with RTX to a local folder on their PC, prompting it with questions related to the files stored therein, supporting various formats including .txt, .pdf, .doc/.docx, and .xml. This local analysis ensures swift processing with no data shared over potentially vulnerable networks.Additionally, users can utilize Chat with RTX to query YouTube video URLs, though internet access is required. However, this feature offers an efficient means to obtain information without the need to watch lengthy videos. To access Chat with RTX, users must download the application, compatible with Windows 10 or 11, alongside a GeForce RTX 30 Series GPU or higher, boasting a minimum 8GB of VRAM.

Despite being in demo phase, Chat with RTX exhibits the future potential of LLMs, albeit with some bugs and limited contextual memory. This advancement highlights the evolving landscape of LLM interaction, emphasizing on-device AI usage and open-source models' adoption over proprietary alternatives. However, widespread integration of RTX GPU computing power into smaller devices like mobiles and laptops may require further technological advancements.