The Importance of Tesla’s FSD Dojo Supercomputer

The rally in Tesla’s share price in recent days has been fueled by a buy recommendation from Morgan Stanley. The analyst firm sees the incremental value of Tesla’s proprietary chip design and new supercomputer at up to $500 billion. In just a few hours, the company’s market value rose by $80 billion. First, here’s an analysis on Bloomberg Television:

But the question is, what exactly makes Tesla’s Dojo supercomputer and chip design so valuable? And to do that, we need to understand exactly how neural networks work as the backbone of artificial intelligence – and thus of Tesla’s Full Self Driving (FSD).

Cookbook Versus Artificial Intelligence

I want to start with a comparison to creating and using a cookbook. If we want to write a cookbook, we buy the ingredients, find out old recipes from grandma and mom, make sure we have a stove, oven, blender, and other cooking and baking utensils, and start cooking and baking the food. We cook and bake the same recipe so many times until we have the perfect recipe. Finally, we take an appetizing photo of the finished deliciousness and write down the final recipe for the cookbook.

Our final product is then a cookbook and that is what we then sell. Our readers then only buy the cookbook, but they don’t get any ingredients or even finished food with it. They have to do that themselves.

An AI – for example, a large language model (LLM) or the FSD AI model downloaded to the Tesla – is trained with data, such as billions of images, hundreds of billions of pages of text, or tens of millions of hours of video of car trips. In the process, the neural network, which can consist of a few to millions of nodes, and in which matrix calculations – i.e. conversions of input from several other nodes – are performed and parameters are stored.

The model is then perfected and corrected with subsequent fine-tuning and improvements. Just as the ingredients and baking times are changed and adjusted in a recipe.

Such an AI model with the stored parameters (=cookbook with recipes) can then be downloaded, but it comes without the videos, images or texts, just as a cookbook comes without the ingredients or finished dishes.

Improving the AI Model

As it turns out, the generally AI models work better the more data they are trained with. Before GPT-4, there was GPT-1/2/3/3.5, and each had more training data, and the tweaks and fine-tuning became more sophisticated.

Tesla can theoretically draw on more than five million cars delivered now that have the autonomous driving hardware kit installed, and get the data – if owners have allowed it – and train its own supercomputer with it. Back in 2017, one user noted that Tesla was downloading about 1.8 gigabytes per month from his Tesla.

Super Kitchen Versus Super Computer

Nathan Myhrvold, former CTO of Microsoft, invested several million (reportedly beyond $10 million) dollars in a “super kitchen” a few years ago, equipping it with equipment normally found only in chemistry labs, such as a vacuum distiller, among other things. He began creating Modernist Cuisine, a series of cookbooks unlike any other. Not only are the photos like something from another world, so are the recipes. A recipe for French fries in the vacuum chamber takes 48 hours to arrive at the perfect fries. He also invited a number of top chefs from around the world to help make his project a reality.

The top bosses, the top AI experts, are now employed by the big American companies. And they, too, have access to the best chip clusters to build their AI models The bigger (and thus hopefully better) the model is to be, the better the kitchen equipment, i.e. the computing hardware, has to be. And the more recipes you want to have and the better they should be, the more ovens and stoves they need to be able to try out a corresponding number of variants. And it’s similar with AI models. We need the appropriate computing power, especially when processing billions of images and tens of millions of hours of video.

Now, here with AI, we mainly use GPUs and TPUs. We certainly know CPUs (Central Processing Units). They’re in all PCs, often with the Intel logo. But GPUs (Graphical Processing Units) are better suited for the many parallel calculations that AIs need (we are talking about millions of nodes in which matrix calculations are performed). Nvidia had originally developed these for the graphics representations on their graphics cards, but now it turns out they are great for both AIs and perceptual systems like autonomous cars have. Their cameras, lidars, or radars take in a lot of graphical and vector data that needs to be computed in parallel on a massive scale. TPUs (Tensor Processing Units), on the other hand, go one step further and are even more specifically conditioned to the matrix computations of AI models.

So just as a stove is good for baking cutlets but not good for baking cakes, CPUs are good for scalar multiplications but not for the massive parallel computations.

With the Tesla supercomputer, which consists of 10,000 of Nvidia’s H100 GPUs (each one currently costs $30,000, by the way), Tesla can create, fine-tune, and tweak the models for the FSD much more quickly, and put the resulting models back on the cars with the parameters more quickly. Models as plural because there are actually multiple models. For example, Tesla also has the Deep Rain model, among others, which performs correction calculations to the raindrops that act like lenses on the cameras and distort the image, ensuring that a Tesla can safely see through even in the rain.

The changes with the FSD 12

There’s also a difference in approach between how Tesla has approached FSD so far and how they’ve now trained the system with FSD 12. And for that, we can compare it to other examples where there has been a similar paradigm shift.

Translation tools like Google Translate were trained primarily by linguists until a few years ago. Among other things, they specified precise grammar rules or which words had positive or negative connotations. This resulted in a very complex set of rules, and the appropriate connections had to be made between the languages. But there were limits, because the translations improved only slowly and less and less, and the effort grew with each new language.

When a neural network was added in 2016, Google was not only able to reduce the number of lines of code by a factor of 1,000 (from 500,000 to 500), but also massively improve the results. Instead of prescribing rules to the system, the AI now picks them out of the training data itself and creates probability graphs.

The same happens with the generative AIs that are currently attracting so much interest. Thus, in the billions of lines of text, a sentence like “The cat is sitting on …” certainly occurs several times and can be supplemented with table/tree/window board/airplane wing/steam jet. But not all are equally likely. Some more, some not at all.

Incidentally, Google’s Deepmind took a similar approach with AlphaGo, AlphaGo Zero and AlphaZero. AlphaGo was taught the rules and trained with 30 million games played by people. AlphaGo Zero was no longer trained with games, just taught the rules and then had it play against itself. AlphaGo Zero won 100-0 against AlphaGo (AlphaGo had defeated Korean Go world champion Lee Sedol 4-1 in 2016). AlphaZero was not trained with games, nor was it told the (entire) rulebook. So it could play Go and chess, among other things. And it won against AlphaGo Zero with 60:40.

The previous versions of FSD were given the rules. “Stop at red”, “Turn when there is no car/pedestrian/dog/duck/construction site cone” and so on. In version 12, however, Tesla moved away from that and let the system derive the rules itself from the video data – which comes from the millions of Teslas around the world. Elon Musk pointed this out several times in his demo of FSD 12.

Of course, Tesla also has to fine-tune and tweak this model, but Tesla doesn’t have to painstakingly teach the system all the rules beforehand.

Impact

With its own supercomputer, Tesla can thus tackle a whole host of sub-problems more quickly. For example, not only autonomous driving itself (which is already complex enough), but also the correction calculation of camera data in the rain (Deep Rain). You can train the model specifically for certain things (dogs, wheelchair users, tunnel driving, special light/shadow conditions, rare objects on the road).

This will probably also allow FSD 12 to migrate more quickly to other types of vehicles. From Model 3 to Semi Truck, which has a completely different weight, driving behavior or camera position. And possibly also let other manufacturers install the FSD on their cars. There have already been initial reports on this. Or, you know, putting it on a Tesla Bot.

It is therefore easy to see that this could open up new revenue potential for Tesla. The in-house expertise in AI, the development of its own AI chip, which is already installed in millions of Teslas and its use, as well as that of Nvidia chips in a Tesla Dojo with probably tens of thousands of processors in the future, makes Tesla an AI powerhouse. And that perhaps brings a clue to the future potentials opening up for Tesla and its investors.

More AI Knowledge

I talk about this and similar breakdowns on (generative) artificial intelligence in my second book on AI (in German), coming out in November 2023. Creative Intelligence: How ChatGPT and Co. will change the world. It can already be pre-ordered.

This article was also published in German.

Read More