Behind the ability of AI to understand our language lies a feat that is as fascinating as it is energy-consuming, which questions the place we want to give them in our society.
Behind this question lies what is undoubtedly the most fascinating discovery in the development of current artificial intelligence. Answering it is an excellent way to explain why the energy requirements of LLMs like ChatGPT are so titanic. Rare are the opportunities to combine intellectual wonder and ecological awareness, so let’s seize it.
The answer can be summed up in one word: “embedding”. To quickly understand this notion, let’s step into the shoes of researchers who managed to invent it. To succeed in this transmutation, our brave people had to wander for a long time in the darkness, before finding the solution: the vectors. Explanation: let’s take a word, “tree”. So let’s create a digital space and draw a vector, called a “tree”. For the moment it does not yet have the meaning of this word, this action is purely arbitrary. Let’s then ensure that the computer manages to draw another vector, “leaf”, in relation to the first.
Let us set arbitrary rules for this: if two vectors go in the same direction, they have the same direction, if they are perpendicular, their relationship has no meaning. Let’s make sure that the machine can draw the second vector the right way. This is where statistics and data come in, but let’s skip this topic to keep this explanation clear. Once this problem is solved, the calculation is launched and, miraculously, the “leaf” vector appears, almost parallel to “tree”.
Let’s take a moment to celebrate, then program this same process for all existing concepts and press the enter button. Hundreds, then billions of vectors appear. Ultimately, the amount of information contained in this single “tree” vector is equivalent to the relationship between this vector, and all of the billions of other traced vectors. And there, magic: the vector acquires the meaning of “tree”.
But then, for the LLM to be able to “read” a sentence, it is necessary to supply energy to this gigantic vector space for each word? Yes, and more precisely for each token, which is more or less a division of these words, a bit like you who truly understand a term by isolating its root, suffix and prefix. At this point in the explanation, we can begin to intuitively understand the amount of energy required for the chatbot to read a sentence, so let’s take the time to imagine the consumption required for a response.
Well we are still well below reality. You have undoubtedly drawn in your imagination a vector space in two or three dimensions, that is to say flat or in volume. Our brave researchers have gone much further: the vector spaces of current LLMs vary on average between 700 and 4000 dimensions. At this stage the vectors are no longer represented by lines, but by clouds of points.
So now, when we hear on the news that ChatGPT consumes the electricity equivalent of a metropolis, we can now be amazed by this information and by the intellectual treasure that causes it. This is a good basis for meditating on the place that artificial intelligence should have in our society.
