We Tried Mistral AI’s Le Chat AI Chatbot, and Here’s How It Compares to ChatGPT

Mistral’s Le Chat has been gaining traction throughout the AI chatbot neighborhood, with some observers tagging it as a possible rival to ChatGPT.

However is that this fledgling AI chatbot actually definitely worth the title? Is Mistral’s Le Chat higher than ChatGPT?

What Is Mistral AI’s Le Chat?

Le Chat is a conversational AI chatbot developed by French AI startup Mistral AI. It’s powered by a number of Mistral-owned giant language fashions, together with Mistral Giant, Mistral Small, and Mistral Subsequent, all of which you’ll select to make use of when interacting with the AI chatbot. Though it’s a comparatively new entrant within the AI chatbot area, it’s rated extremely due to the efficiency of its AI fashions regardless of their smaller dimension when in comparison with trade heavyweights like Gemini and GPT-4.

To grasp what this implies, think about you are taking part in with constructing blocks. The extra blocks you will have, the extra advanced and detailed constructions you’ll be able to construct, proper? AI Language fashions are a bit like that. They arrive in sizes, often expressed in parameter counts. So, you may need heard phrases like “7B parameters” or “70B parameters” in AI fashions. The parameter depend is just like the variety of constructing blocks the mannequin has to know and generate responses. So, if a language mannequin has extra parameters, it might probably perceive and generate extra advanced and higher responses.

Now, whereas GPT-4 has an estimated 1.76 trillion parameters, Mistral AI’s is estimated to have between 7 and 56 billion parameters. See the dimensions distinction? So, Mistral AI’s capacity to put up first rate efficiency is among the causes for the hype.

Though Le Chat does not benefit from the stage of publicity loved by ChatGPT nor the model fairness of the likes of Gemini, it has labored its approach into the dialog at any time when a possible ChatGPT competitors is mentioned. However does it deserve a spot on the desk?

I have been questioning the identical, and to search out out, I examined Le Chat extensively to see the way it compares to ChatGPT.

Le Chat vs. ChatGPT: Creativity

Creativity is among the most necessary metrics for judging the efficiency of a conversational AI chatbot. Bear in mind, the aim of an AI chatbot is to copy or mimic the conversational talents and inventive aptitude of people at scale. This makes creativity an important energy for any AI chatbot. The world has experimented with ChatGPT for over a yr, and its artistic talents are simple. However how does Le Chat evaluate? We put each chatbots to a collection of creativity assessments.

I began off by asking each chatbots, “How would you describe your self to an artist?” to check their capacity to make use of artistic and imaginative phrases to conceptualize themselves.

This is how ChatGPT would describe itself to an artist:

And here is how Le Chat would describe itself as nicely:

Each responses have been acceptable in their very own distinctive methods. ChatGPT was extra invested in utilizing vivid imagery and metaphors to explain itself, demonstrating artistic aptitude. Alternatively, Le Chat’s response could be very informational and targeted on describing its essence as an AI chatbot. Some might say it lacks the artistic aptitude and inventive method that ChatGPT’s response displays. Nevertheless, I will exit on a limb to say I favor Le Chat’s easier-to-imagine response to ChatGPT’s summary description.

I then requested ChatGPT and Le Chat to jot down a rap music about changing into wealthy from rising cucumbers—a difficult request we have used to check the creativity of different chatbots. What number of rap songs can you discover about cucumbers on the net?

This is ChatGPT’s response:

And here is Le Chat’s response:

It may be a subjective difficulty, however ChatGPT’s response appeared like the higher possibility right here. Le Chat’s lyrics appeared fairly wordy and did not actually learn like one thing a rapper would put out. To check how each lyrics would sound in the event that they have been to be made into music, we used Suno AI music generator to generate music from the lyrics. Three out of three trials, ChatGPT’s lyrics sounded approach higher. Beneath are two samples from each AI chatbots, you may be the decide of which chatbot did higher.

Samples Generated From ChatGPT’s Lyrics

Pattern 1:

Pattern 2:

Samples Generated From Mistral Le Chat’s Lyrics

Pattern 1:

Pattern 2:

I attempted a couple of different artistic duties, like poems, article writing, and drafting difficult work emails with the AI chatbots. Regardless of displaying nice promise, Le Chat was clearly outdone by ChatGPT in all cases. It is necessary to level out that one space Le Chat was notably sturdy in was crafting articles, though with some difficult prompting types. Nevertheless, by way of all-round creativity, the medal goes to ChatGPT.

Le Chat vs. ChatGPT: Programming Expertise

Proficiency in coding has turn into a key requirement for main AI chatbots. Writing first rate code is a baseline talent, however to actually stand out among the many elite, an AI chatbot should show its prowess in crafting code that may successfully resolve a various array of advanced issues. We have beforehand constructed a whole internet app from scratch utilizing ChatGPT which demonstrates its exceptional talents as a programming device. However how good is Le Chat at writing code?

I tasked each chatbots to jot down a easy to-do record app utilizing CSS, HTML, and JavaScript. ChatGPT did not have any bother producing good outcomes. I copied the generated code and previewed it on a browser, and here is what ChatGPT created:

Every time we repeated the immediate, ChatGPT created a purposeful to-do record app utilizing completely different types. In no occasion did the generated code fail to work.

Once I tried the identical immediate with Le Chat, it generated what seemed to be intelligible code, however once we tried working it on a browser, it wasn’t purposeful. After repeating the immediate 3 times, no occasion produced code that would full the set process. It failed in some of the fundamental coding duties—pink flag!

In fact, I will not decide Le Chat on one failed check. Subsequent, I requested each chatbots to generate JavaScript and PHP code for encrypting and decrypting textual content. On this second check, each ChatGPT and Le Chat produced purposeful code that would carry out the set process. Nevertheless, Le Chat’s model appeared like what an inexperienced entry-level programmer would write. ChatGPT’s code, however, was extra full and regarded prefer it was written by an skilled programmer.

I repeated a couple of extra programming assessments involving discovering and fixing bugs, and ChatGPT persistently outperformed Le Chat. In some cases, Le Chat did not even know what it was alleged to do. There is a good probability Le Chat can measure as much as ChatGPT in some areas, however coding expertise aren’t one among them.

Le Chat vs. ChatGPT: Frequent Sense and Logical Reasoning

An fascinating twist with AI chatbots is their capacity to unravel advanced duties with ease however fail at easy ones that require fundamental frequent sense to finish. Many chatbots, each established ones like ChatGPT and new entrants like Le Chat, wrestle in the case of fixing duties requiring human-style frequent sense and logical reasoning. So, how do Le Chat and ChatGPT evaluate on this entrance?

I requested each chatbots: “If in case you have a 3-litre bottle and a 5-litre bottle. How are you going to measure 4 litres of water by utilizing the 3-litre and 5-litre bottles?”

ChatGPT solved the issue with aptitude:

Le Chat tried the identical process and was in a position to resolve the issue, though utilizing a special method.

Each chatbots carried out comparably on this check.

Up subsequent, we requested each chatbots a trick query: “If a spaceship from Mars breaks into two, with one half crashing into the Atlantic Ocean close to Brazil and the opposite into the Pacific Ocean close to Japan, the place do you bury the survivors?”

ChatGPT noticed by means of the trickery and responded accordingly:

Le Chat was additionally in a position to see by means of the trickery and responded accordingly as nicely:

I attempted extra trick questions, and it appeared each ChatGPT and Le Chat are fairly adept at coping with commonsense and logical reasoning prompts. Nevertheless, with extra advanced logic questions, solely ChatGPT may present the precise responses.

Whereas Le Chat has generated some buzz as a possible “ChatGPT killer,” our testing reveals it nonetheless has rising to do earlier than it might probably really go toe-to-toe with the heavyweights of the AI chatbot world. Although Le Chat demonstrated spectacular capabilities in areas like frequent sense reasoning, its artistic output and coding expertise lagged noticeably behind ChatGPT. The French AI upstart definitely reveals promise, however the hype machine could also be getting a little bit forward of itself.

Like many aspiring contenders earlier than it, Le Chat wants continued refinement and coaching earlier than it is prepared for the massive leagues. For now, AI chatbots like ChatGPT are nonetheless clearly the undisputed kings of the AI chatbot world. However the subject of rivals is simply getting extra crowded, so the leaders cannot afford to relaxation on their laurels.