Chat RTX AI: NVIDIA launches its own AI chatbot

Artificial intelligence has marked a before and after in the history of technology since it really makes our lives easier. Recently the renowned company NVIDIA announced the launch of its local chatbot Chat RTX AI. It is expected that security and privacy will stand out among its features and this is something that has caught the public’s attention.

What is Chat RTX AI?

This is a completely free tech demo currently available for download combined with a customizable chatbot. The system is compatible with two prominent AI language models:

1. Llama2 13B INT4

2. Mistral 7B INT4

These models represent some of the most widely used and advanced AI language processing systems available today. The INT4 designation indicates that both models use 4-bit integer quantization, a technique that reduces model size and increases inference speed while maintaining performance.

The first uses 13 billion parameters and the second 7 billion parameters (remember that one billion Anglo-Saxons are equivalent to one billion Europeans).

Chat RTX AI employs Recall Augmented Generation (RAG) to enhance the precision and dependability of its generative AI models. This method incorporates information from external databases to supplement the model’s outputs.

Additionally, the system utilizes NVIDIA TensorRT-LLM software, a user-friendly library that maximizes the AI processing capabilities of Tensor cores, found in devices like GeForce RTX graphics cards. This integration can boost performance up to fourfold, optimizing AI-related computations.

Also Read: LuzIA: The multi-channel chatbot arrives on WhatsApp and Telegram

How does RTX AI chat work?

The process is very simple and this is exactly one of the great values ​​that this technical demonstration brings. Bringing the chatbot’s AI model closer to even the least experienced users, it applies automatic settings and configuration processes.

To begin, simply download the software using the provided link and initiate the installation. The process will complete automatically. Once finished, you’ll find a shortcut icon on your desktop. Double-clicking this icon will launch a command console, which will then open your default browser to the appropriate tab.

The user interface is straightforward. At the bottom, you’ll find an input area where you can type your questions or select from suggested prompts based on the pre-installed database.

The interface allows for effortless toggling between AI models:

1. Llama2: Recommended for powerful computer systems

2. Mistral: Suitable for less robust hardware configurations

Switching between these models is accomplished with a simple click.

Additionally, the upper-right section of the interface provides options for selecting the AI model’s data source. This feature allows you to customize the information base the AI uses for generating responses.

Options include a pre-installed RTX database chat, your own collection of text, PDF, and document files, or even YouTube videos and playlists.

RTX AI Chat Features

Chat RTX AI is a demo, as NVIDIA explains. This means that the software is not 100% complete and may have some bugs. However, this is a very interesting alternative that can help speed up tasks, such as searching for specific data in documents stored on a PC or creating a summary of videos uploaded to YouTube.

This app has a minimalist interface and its operation is not complicated at all.

After installation, Chat with RTX offers users a choice between two primary large language models (LLMs):

1. Mistral, developed by the eponymous French startup

2. LLaMA 2, created by Meta

Users can select which of these two AI models they prefer to power their Chat with RTX experience. Then they must select a folder that contains the documents they want to scan with the NVIDIA software and that’s it. Now they will be able to ask questions to chatbots, just like when using applications like ChatGPT, Gemini or Copilot.

The AI system processes and extracts information from the files you provide. For instance, if you allow Chat RTX to scan a folder containing documents about your upcoming trip, you can then query it for specific details.

As an example, you could ask, “When is my flight scheduled?” and the AI would retrieve and present the relevant date and time information from the scanned files.

The NVIDIA chatbot will not only provide you with informative comments in text format, but will also link the documents it extracts data from so you can quickly open them.

Using the same example, if you have defined an activity schedule for your trip and it is in PDF format, you can have it extract this information and summarize it in seconds. However, Chat RTX is also capable of working with items that are not stored locally, such as YouTube videos.

How can you use Chat RTX?

Chat with RTX can also be used to scan PDF files and verify data. Chatbot responses are delivered without the latency that typically occurs with cloud-based chatbots. As it stands, Chat with RTX is one of the first demos for developers.

The application installs a web server and a Python version on the local machine and uses the Mistral or Llama 2 model to process data. To speed up query processing, the chatbot uses Tensor cores from Nvidia GPUs. The command line displays processing information and error codes.

Also Read: LaMDA AI: What is it and how does this Google chatbot work?

NVIDIA lets you create your own ChatGPT with ChatRTX

If instead of using local data you want to extract information from videos hosted on YouTube, Chat RTX allows you to do it without problem. Just copy the link and paste it into the NVIDIA app. The AI ​​will be in charge of processing the content so you can ask questions about it. Additionally, like documents, you can create a summary and provide it in text format.

In the promotional ad for this software, NVIDIA mentions that this app works with “featured video.” Although it does not delve deeper into this topic, it is reasonable to think that the results will vary depending on the content. To put it more simply: don’t be surprised if the app crashes quite often when you use this feature.

Chat RTX has specific hardware and software requirements:

1. Graphics Card: An NVIDIA GPU is essential. Compatible models include:

   • RTX 30 series

   • RTX 40 series Both require a minimum of 8 GB of VRAM.

2. Operating System: Windows 11

3. System Memory: At least 16 GB of RAM

4. Graphics Driver: Version 535.11 or newer

These specifications are necessary to install and run Chat RTX effectively on your computer.

While it is possible to get the most out of the app, you will need a fairly powerful PC. As The Verge’s Tom Warren mentioned, the app takes up around 40GB of storage and uses a version of Python that consumes around 3GB of RAM when running. Finally, it is worth clarifying that, if you use the application with local data, this information will not be uploaded to the cloud and will not leave your computer.

ChatRTX Limitations

Although it is a promising app, Chat with RTX still has its limitations. For example, the tool does not remember previous conversations, which can limit their continuity.

The accuracy of the AI’s responses can vary due to multiple factors:

Question formulation: How the query is phrased can affect the answer’s precision.

Model selection: The performance of the chosen AI model plays a role in response quality.

Dataset scope: The breadth and depth of the information provided to the AI influence its ability to give accurate answers.

These elements collectively impact the reliability of the information provided by the system.

However, Nvidia’s initiative stands out for making it easier to run AI models locally, a growing trend according to the World Economic Forum, which predicts a “significant” increase in the number of affordable devices capable of running GenAI models offline.

The advantages are clear: greater security because the processed data never leaves the device; Reduces latency and is more cost-effective than cloud-hosted models.