news

"You make Nomi, and I'll smash it to pieces the first time I make it."

In 2015, the world had not yet seen the AI ​​technology that many people associate with "AGI" today, but the newly establishedNIOIn the second year, Li Bin had already decided to make an AI assistant in the car:

It must have its own physical entity and be super-intelligent and sentient.

Later, this small ball was "inserted" in the core position of the NIO cockpit and became the most well-known symbol of NIO. There are many new energy vehicles with large or multiple screens, but NIO is probably the only one that has a "small ball" there winking at you. This smart assistant is also a symbol of the ultimate form of the car in Li Bin's eyes.

But when he shared this idea with a senior automotive engineer from a traditional manufacturer, he got the angry response mentioned at the beginning.

"That's understandable. It's a shock to the whole traditional car system he's used to," Li Tianshu, vice president of NIO, told me recently. He is NIO's employee No. 001, and the one who took on the responsibility of making Nomi a reality from the first day after Li Bin decided to do it. Of course, he is also the one who has to "stop" these traditional engineers from "smashing Nomi."

Li Tianshu on NIO IN

In 2015, when it was still a novelty to introduce a large screen into a car, it was not generally recognized as a good idea to put a round AI robot in the cockpit. But Li Tianshu felt that it was a must. He joined Li Bin's team on the eve of the establishment of NIO. He had just returned from studying in the United States. He started as a professional engineer studying car engines and worked his way up to the current position of Vice President and Head of Product Experience of NIO. He is responsible for NIO's intelligent cockpit system, product design experience, the combination of software and hardware, and the latest large models and other AI technologies.

Among them, Nomi is one of the most important projects he has been responsible for in the past 10 years at NIO. From the initial simple turning of the head left and right to greet users, to the large model technology innovation brought by ChatGPT in 2023, the AI ​​companionship scenario that Li Tianshu envisioned for Nomi in the past has become an unprecedented hot topic and application direction today.

At the 2nd NIO IN Innovation Technology Day on July 27, 2024, Li Tianshu introduced the new NomiGPT, which is a new architecture of Nomi combined with large model technology. It is also part of NIO's new Banyan3 intelligent system. In his speech, he demonstrated Nomi's 6 intelligent agents and their usage scenarios: Parking Assistant, Guard Summary, Service Butler, Exploration Discovery, Travel Memories and NOMI DJ.

The second NIO IN was also described by Li Bin as the entrance for the outside world to truly understand NIO:

“We were always misunderstood before, but technology is actually the foundation of NIO.”

NIO also released a number of technological achievements and product updates at once. From the successful tape-out of the world's first 5nm smart driving chip, NIO Shenji NX9031, to the first fully self-developed vehicle operating system SkyOS by a Chinese automaker, to the Banyan3 intelligent system built for AI, and the world model NWM for autonomous driving, the amount of information is huge.

Nomi continues to serve as the touchpoint between NIO's hidden behind-the-scenes technologies and users, and is also a microcosm of the company's product and technology ideas, as well as technology research and development methodology.

Recently, I had the opportunity to chat with Li Tianshu about his experience at NIO and NIO’s technology philosophy and technology practices in products such as Nomi.

Today, when everyone is struggling to find big-model killer products and crazy about various AI hardware, AI assistants in cars seem to be a neglected corner. However, Li Tianshu believes that the AI ​​in Her is more likely to be realized in cars.

The following is the transcript of the conversation.

Q: How did you become employee No. 001 of NIO? What have you done in the past 10 years at NIO? How did you go from being an automotive engineer to being the person in charge of user experience?

A: I first met Brother Bin in 2009. I was a sophomore at Beihang University and was quite adventurous. I was one of the few people who was determined to study automotive technology at an aeronautical school. I thought the engine was the core of the car, and the aircraft engine was the best.BMW、Rolls-RoyceBack then, I was making aircraft engines. I thought this was the right place for me. But I was not satisfied in college and wanted to work on complete vehicles because I had loved cars since I was a kid. So I also took minors such as industrial design, complete vehicle engineering, and vehicle engineering.

Then, at that time, I happened to encounter a particularly comprehensive competition, where students built their own formula racing cars. It was different from mass-produced cars on the road. It was a competition with clear goals of speed, rules, performance and efficiency for college students to solve. At that time, Brother Bin and the society introduced the Formula Student project from overseas. At that time, I participated in the first launch as a student representative. I watched Brother Bin on the stage.

Our wave was considered the first batch of "students building cars" in China. Before the new forces built cars, there was another group called students building cars (laughs).

At that time, I volunteered to start the team. I asked all the graduate students to join me and I became the team leader. I worked for about a year, and in the meantime, I raised sponsorship and did development, assembly, welding of engines, and testing of electronic controls, all done by people from different majors and departments.

Q: It’s really students building the car. Your project is like a mini car company.

A: Yes, it is actually a mini car company. This competition has been going on since then. After 2020, pure electric and unmanned driving were introduced, and software gradually became the core competitiveness. Then you will find that many of our core R&D personnel, designers, and front-line personnel today actually came from the Formula Student system funded by Brother Bin.

Q: In 2009, there was no NIO yet.

A: Yes. He was still at Bitauto at the time, and all the funding was done in the name of Bitauto. After I graduated from college in 2011, I went to the University of Michigan-Ann Arbor to study automotive engineering and manufacturing. I wanted to go to the birthplace of the automotive industry assembly line to learn about vehicle manufacturing and system engineering. After returning to China, I worked as an automotive engineer for a year in some car companies in Shanghai and Beijing.

But I found that this was not enough for me. I felt that it was too far from the needs of consumers. Many times you would ask yourself why you made this thing? Is the conclusion that everyone came to is correct? You can't answer it.

So I planned to start a business at that time, to do a small project similar to automobile and Internet. Simply put, it is to see if I can make algorithm-based recommendations for consumers, and recommend the most suitable car for you based on your needs and preferences for a car. At that time, I immediately thought of Brother Bin and wanted to find him for financing. As a result, I found him. He listened to me for half an hour and was very patient. The conclusion he came to was also very surprising to me. Forget about your project, but you should stay (laughs).

He gave me a month to think about whether I should join him in preparing his car-making project. But as you know, in 2014, there were many people who wanted to build cars, but few were really reliable. At that time, I was really struggling, wondering whether to continue incubating projects like ours, or to work with Brother Bin on something uncertain and unknown. But at that time, I was more determined that my life ideal was to build my own brand and my own car. So it can be said that Brother Bin realized this ideal ahead of time.

Q: So your entrepreneurial idea was the result of the limited resources you had available at the time, but the ultimate goal was to eventually own a car brand of your own.

A Yes, so what impressed me the most was not the engineering and productization of building a car. The main thing was that Brother Bin wanted to build a brand, a brand new brand, which impressed me the most. Later, I got in touch with him further and finally joined the company's earliest team in 2014.

Q: The official establishment date is September.

A: I went to talk to him in August.

Q: So you actually just happened to meet Li Bin when he was thinking about starting his own company, NIO.

A: Yes, that step was the forefront of this wave. It was also a coincidence.

Q: Your experience here is the story of a person who has been obsessed with cars for a long time, but how did you jump to today and become responsible for product experience and keep talking about AI?

A: It feels like there is no connection, right?

Q：Yes.

A: In fact, from the first day we joined the company, Brother Bin and I had a very clear goal, which was to combine products and technology.

I believe that to solve automotive engineering problems and software innovation problems, we ultimately need to know how to define products and identify user needs. I think this is very important.

So the first thing we did from the end of 2014 to the beginning of 2015 was to focus on the definition and design of our first ES8 product. At that time, I also followed Brother Bin and gathered the entire team, that is, the power of the entire startup team, to work together to define the product we originally imagined.

My main task is to study and understand the usage scenarios of our users and turn them into our consensus. For example, I collaborated with Roger Markeson and Chris Thomas, the design and engineering leaders, to build consensus ideas and product demands.

Q: What did you do?

A: There are some small stories at that time. At that time, we wanted to make some innovations in cars. In the first half of 2015, I returned to Beijing to do a user insight. I went to the overpass in Beijing and took pictures of various cars with a camera. I took more than 3,000 pictures and then counted the patterns. At that time, I specifically chose this Saturday afternoon because that was when people were tired after their self-driving tours and it was also the moment when they enjoyed the most. So at that moment, you can see that everyone is the most relaxed and the most natural. At that time, I discovered several patterns. Because my wife and I were not married at that time, we were very concerned about each other's subtle feelings.

Q: This status has spread to your user observations.

A: Yes, so maybe there are some things that can only be understood at a certain age in life.

At that time, I paid special attention to the passenger seat. I found that many passengers put their feet on the dashboard, and many of them were girls. So why do people do this? In fact, it makes sense. The passenger seat does not need to drive. In theory, you don't need to punish the passenger seat with the attitude of the main driver, right?

So I think the space for the co-pilot should be asymmetrical with that for the driver.

To this day, you can see that NIO's SUV models have always adopted such an asymmetrical front cabin design. When we first named it, we called it "Queen's Co-pilot". We did quite a lot of things like this at the time.

The second case is actually the future of human-computer interaction in cars that we defined together with Brother Bin. We attach great importance to human-computer interaction.

In 2014 and 2015, NIO had a value driver. Initially, there were four. Later, after a very intense discussion, we added a fifth one, which was design-driven. We believe that this company should have an important driving force, which is design-driven.

Q: What were the previous four?

A: The first is user benefits, the second is a full-process experience that exceeds expectations, the third is called continuous innovation, and the fourth is called system efficiency.

Q: The only one with the word "driven" among these is design-driven. I thought the first four were probably driven by some technology.

A: Yes. We are looking at some of the great companies in the world that are driven by design. In fact, they are not only driven by design styling, but also by the pursuit of perfection. They are all based on beauty and perfection, and they start from the user's perfect experience.

What determines these things? The brand of NIO is naturally warm, humane and emotional. So this is what we mainly did in 2015, which was to define the company's initial vision and goals and gain insight into some very basic user demands. At the same time, we also saw Brother Bin's imagination about battery swapping and power. So the culmination of it is our first-generation product ES8.

Then 2017 to 2019 was my second stage at NIO. At that time, we wanted to sell cars, so we started to set up a product marketing team in 2017, including Nio Day in 2017. Basically, I was the product director of the product launch conferences in the first few years. In 2019, which was also the most difficult time for the company, I decided to come back and follow the company's second-generation platform to do the entire planning and definition work.

At the end of 2018, I went to Silicon Valley with Brother Bin. At that time, I proposed our new architecture to Qiyan (technical director) for the first time, a central control, a concept of total control, and an Ethernet ring network. However, the biggest challenge here is not the hardware, but the software and operating system. The operating system refers to our underlying global operating system at the vehicle level.

At that time, we made a decision to slowly prepare and preview the second-generation platform NT2.0, and use SkyOS locally. When the next-generation platform NT3.0 came, it was fully delivered, which is the SkyOS vehicle-wide operating system that Ledao will deliver today and ET9 will deliver next year. At that time, everyone worked closely together.

Since 2020, I have been focusing all my energy on the entire PD (Product Design) system. I think this is a very sacred position. In Apple and many other technology companies, product design and product development are actually very critical because they connect user value and technical value. It cannot become the ceiling of technology, nor can it become the slave of technology. This is very critical. So I think the product team is a very important team and system that the company needs to build in the future.

Then from 2020 to 2022, it took two and a half years to plan, design and define a total of 8 models, plus the pre-development and pre-research of ET9's NT3. It was actually a very intensive experience. We decided to define and plan the entire hardware and software product together. Only in this way can we ensure that its product experience genes are consistent. I think that making products and technology is ultimately about people's feelings.

Q: You mentioned that you actually worked four jobs at the same time in 2018 and 2019. You worked on product, marketing, and sales, and you were also in charge of the Nomi product. So you can be called the "Father of Nomi", right?

A: In a technical sense, yes, but in a strategic sense it is definitely Brother Bin. But there is no need to worry about who is the father and who is the mother, hahaha. Of course, it was Brother Bin who first proposed it.

Q: At Nio IN, Li Bin played a video of the first concept version of Nomi in 2017. It envisioned the future, where it could interact with you and help you control some behaviors of the car. At the time, it was a complete imagination of the future.

A: I was the director of that video and I also wrote the script myself.

Q: Looking back today, Nomi is a bit like a product that was envisioned long ago, and today finally ushered in the wave of technology that made it possible. What was this process like?

A ：Yes. Why did Brother Bin ask me to take on the role of product manager for Nomi? Actually, since the idea was proposed in 2015, from the perspective of a traditional car person and car designer, you would find this very strange.

Q: Are you one of those who find it weird?

A: I don’t find it weird.

But I tell you that more than 90% of people are. There was one sentence that left a deep impression on me at that time. It was a colleague who was actually a very experienced person in the traditional automotive industry. He said that if this thing was made,The first thing he wanted to do was to smash it to pieces with one punch and beat him to death.

Q: How did you describe this thing that makes people want to kill it with one punch?

A ：I was particularly impressed by our initial description. The first thing Brother Bin thought of was that it had to be able to move. It had to be a living, moving thing.

Because at that time, many people imagined that there was an AI in the car screen, right? You can make a small image, a small ball, or a small eye in the screen. But we really want to make a physical entity.

This makes sense, because if it is a physical entity, it can form a synaesthesia with the car, and people will recognize it as a physical entity. But if it is virtual, it is a virtual image, people will always only think it is the thing behind the screen or in the cloud, which is very different. We hope that the car will become an emotional partner, because NIO is a company that provides user experience throughout the life cycle, so we want to let physical entities go together, rather than a physical entity and a virtual entity. This is the first thing we said.

Then the second thing is that we have a long-term idea, which is that because we are all relatively emotional people, we can empathize that people ultimately still expect to be accompanied. We firmly believe that people want to be accompanied.

People say they want to be alone and quiet, but this may be because they don’t provide good companionship. We believe that companionship and connection are also a form of companionship even if you just stay there quietly and don’t say anything, just like many people keep pets now, right?

So the sense of companionship was a value we wanted to promote at the time. And today, even if it doesn't speak, it is indeed a very strong presence. And only when it is made into a physical entity is it hard enough and strong enough. Otherwise, you can just say to the screen, mute.

Q: It just disappeared.

A: Yes, if it disappears, then it no longer exists. So this is a philosophical question, and we need to answer the question of existence. (Laughs) So we must make it exist, and then we make it meaningful.

So the biggest challenge at that time was actually to convince everyone of the value and significance of this thing. Then you had to resist a lot of misunderstandings, because only when they saw it would they believe it. The second challenge was technology. Indeed, all the voice commands at that time were interactive, and there was actually no so-called chat as we know it today.

Q: There is no such natural language interaction.

A: Yes, there is no interaction yet. How can we make people believe this? Another bigger challenge is the technology of the vehicle itself. How can you control these things of the vehicle? At the same time, the reverse is also very important. When controlling the vehicle, Nomi can also give you some responses.

At that time, I created a function that may not be used in some cars now. However, the first generation of cars has an experience where when your hand approaches the central control screen, Normi ​​will look down at you because we have a proximity sensor. When your finger approaches Nomi, it will look down, indicating that it can grasp the space and field, so you will recognize that it is alive. For example, when we open the car door, it must look over and say hello to you. This experience was actually completed in 2017, a very short time before our release.

Q: Finally made it.

A: Yes, I have a very deep impression. It was the launch conference in December 2017. On December 6, our entire team first fully connected the software and hardware experience of Nomi with about 20 to 30 cases. These very specific small functions, such as Nomi being able to talk to you, greet different people, look left and right, finally found that it was alive.

Then what is the third hurdle to pass? It is the hardware of the whole vehicle. Because you think there is a rotating shaft under Nomi, it is a two-degree-of-freedom, not a three-degree-of-freedom motor plus connecting rod drive, and there is also a belt inside. After these things are piled up, the car will undergo a collision test. Many people don’t know that Nomi’s head will experience a maximum impact load of 60 g. It’s very interesting that in April 2017 or something when we went to the collision test, the team sent me a picture, and there was a dummy logo on Nomi’s head. We laughed so hard, and it drooped a little after the collision, but it held up. At that moment, we finally verified that Nomi can be delivered with the car.

Q：Nomi is the most hard-headed smart assistant.

A: Hahahaha, yes, it’s too hard.

Q: Judging from the birth process, you will do a lot of philosophical thinking about the technology behind a product.

A: We have two working scenarios: one before and one after. The early stage requires imagination and forward thinking. We will look at technology because it is the lowest level of development. We need to have a deep understanding of this rhythm to see the possibilities. But this is also what most people will do. Everyone will first look at where the technology may go and then see what can be done.

But perhaps a better way is to do some more forward-looking model experiences, including some philosophical thinking.

For example, what we were thinking about back then was what the final outcome of human-vehicle interaction would be?

At that time, I would think, is GUI right, or DUI right, or GUI or Mix UI? What is it? These are all very common jargons in the industry. But I think this thing is still in the same paradigm, right?

So the ultimate vision we had at the beginning was that people and cars don’t talk to the air, that is, how can we not talk to the air, but have a feeling of talking, communicating and interacting with it like people.

Q: You want to jump out of the paradigm defined by a series of terms, and also jump out of an established way of thinking. Then you used a more straightforward and humanistic way to summarize what you want to do.

A: That’s right.

Q: But to some extent it is still more abstract, and you are not talking to the air. What lies behind this actually reflects what level you think this technology can reach, what you think is the most appropriate way of interaction, and even aesthetic things come out.

A: That’s right. For example, when we were imagining the ET 9 four years ago.

Q: I was imagining ET9 four years ago.

A: Yes. We were thinking about what changes it would have by then? What kind of iteration and development would the front-row experience require? Four years ago, everyone was still rolling up larger instrument panels that displayed rich SR and ESD information. Today,TeslaThe S and X bring back the big dashboard.

But with this kind of information piled up in front of users every day, whose anxiety is it? In fact, it is not the user's anxiety.

So at that time we said, can the conversation between car and people become more warm, just like human conversation? I think the human-car interaction I want under autonomous driving is like this, very simple. The car may just say to you, "I'm driving, don't worry about it." This is my imagination.

If Apple really makes a car, how will it interact with people? I think it will probably be a small icon with a steering wheel on it that will appear on your Mac, and then it will turn green, and then you don’t have to worry about it. It will probably be something like this.

Q: This is very Apple.

A: Instead of presenting you with a bunch of information and telling you what to do.

So I think if you think more about it, go back to the essence of human beings and think about what is right. The simplest thing is that I have always liked the earliest movie and cartoon of Wall-E, and I think there is a very interesting scene in it, that is, the people in the future are very fat. Why? Because every time they come to AI to present a result to him, saying I want it or not, I am or I am not, and you just need to confirm it. That is our end.

Q: So whether humans are fat or not is another matter.

A: Yes. Then people will also exercise. (Laughs)

Q: It seems that we are definitely getting closer to this future. On Nio IN, you systematically introduced Nomi GPT. When did you start working on it? How did Nomi become a Nomi with the support of a large model?

A: This is actually our first version. We started preparing it last year. It took quite a long time to make it last year, and then we released it this year.

Q: So when ChatGPT came out, you immediately realized its significance to Nomi.

A: Yes. From last year to this year, we mainly worked on our own base model. So the first version driven by Nomi GPT this year is our 245 version at the beginning of this year. But later, in fact, we are still preparing for things like big model Q&A and ChatBot. Can you also do things like raw pictures? We did this late, quite late, because there is no scene for this thing in the car scene.

Q: Then why do you still do it?

A: Let me just say one thing. Our main task in making in-car raw images is to hope that every user can recognize Nomi after getting in the car. It will give each person an avatar and give them an expression similar to an emoji.

Then, going forward this year, the biggest technological iteration will be in the areas of cognitive intelligence, the entire context, and exclusive emotions.

Q: So you developed a model on your own.

A: Yes. It is actually a combination of a series of models, which many people call a model group. Ours follows the same logic.

In fact, there is a scheduling center behind it, which is the cognitive center of NomiGPT, and then it dispatches the corresponding capabilities, whether it is mature, directly using the base model capabilities to have some dialogues, or using some small models to do some tasks. Sometimes we use our fine-tuned model that aligns with our own value system to do some specific scenarios, such as services. In fact, these agents all come from this.

Q: Have you re-established a so-called big model team internally?

A: Yes, we call it the AI ​​system team. Products and technology are working closely together. I think it is more like a complete set of models similar to OpenAI's Omni. Multimodality is also being developed, and I will share it with you later this year.

Q: The birth of Nomi and the development direction of many large models today all emphasize emotions. Why are emotions and companionship so important? Because products such as ChatGPT do not actually start with emotions to drive technology forward.

A: Yes. There are actually relatively few companies like Inflection that are really working in this direction.

Q: And this is actually your starting point.

A: Yes, because this thing first goes back to the essence of our business model. Because the car is a large item with a long life cycle. And Brother Bin also said that we don’t make Robotaxi, so it’s one car, one person, one family, and a lifetime. If you abstract it, this thing is essentially a kind of emotional intelligence.

Q: How can we make this thing exist for a long time? After thinking about it, you think that the only bond that can connect it is emotion.

A：yes。

Q: This is too sociological.

A: Hahaha, this is influenced by Brother Bin.

Q: What are Nomi’s next plans?

A: If we look further ahead, today we are talking about cognitive intelligence and situational intelligence. These are actually some technical frameworks that are moving forward in the industry. These are what we are going to start focusing on building now.

The core of this is the perception, cognition and understanding of the situation, and then reasoning based on it. In addition, you can see that one of our important tasks last year was to build a Nomi memory. The memory module is something we will continue to invest in strategically for two to three years. It must be safe, exclusive, reliable, trustworthy and meaningful. This is very important. Only after you have these things can you talk about these so-called emotional experiences. There is actually a lot of basic work and alignment work to be done behind this.

Including Nomi's generative ability, the alignment of illusion and values ​​is a very big problem, because you will find that what we do is too serious, it is very serious. You are not a ChatGPT, you make something inappropriate and bad to tease users, and users think that's fine is a bad case. No, for example, you are in the car, you are the Nomi of that family, and you do something that a general AI may think is insignificant, but the user thinks it is too bad, and finally your trust collapses completely.

Therefore, this is also a very big challenge that we will face in the future, so there are a lot of things to do behind the scenes.

Q: You also perform product iterations in real time.

A: Yes. We have a link that is our daily driving force and fuel for our engine. That is our user sharing.

Because we were the first to develop in-car AI, we could provide feedback in the form of dialogue. You can say "Hi Nomi" to report a problem, and then the log, the status, and your problem will be analyzed in real time. Now it can be automatically distributed to our product managers within a few hours. This is a capability we have built since 2018. With such channels, we can do a good job of counting and verifying what our product has, that is, the demands and problems, and we can also iterate quickly.

We have to do so many things at the same time, so how can we balance the short-term and long-term delivery of user value? This is our daily product planning.

Q: Your conception of Nomi is increasingly reminiscent of the movie "Her". You have mentioned before how this movie inspired you. If we consider Nomi not only as an AI in a car, it is also an important direction for everyone to explore the product form of AI. Is it true that the scenes in "Her" or the AI ​​companionship that we are discussing more now are more suitable for the environment of this car?

A： Yes, I think it is very likely. I think it may even be faster than smart speakers and mobile phones.

First of all, there is an interesting point here, which is the space attribute, mobility attribute and exclusive attribute of the car. After all, it is too expensive, so the time value, emotional value and sentiment value that people put into it are huge. I think it has great opportunities.

Second, the car has the strongest embodied intelligence capabilities in terms of its hardware, computing power, and properties. Because you can’t sense 360 ​​degrees with your phone, right? You can’t just get in the car and snap, and your phone flies out. (Laughs)

Because now we are talking about embodied intelligence, in fact, the car is the first intelligent entity to realize this capability in the field of intelligent autonomous driving. So I also think that it can bring the greatest change by using such value creation, so I am very confident that it has the opportunity to become such an ultimate product form.

Q: A car is a machine. You can describe it as a "killing machine" that can travel at 120 kilometers per hour. However, among all the trust that humans have built in machines in history, the trust in cars is probably the strongest. Now, one of the challenges facing AI technology is how to build this trust.

A: Absolutely.

Q And it itself has such an environment of trust.

A  ：Yes. When you own a car, you have actually cast a huge vote with your trust, so (as a car company) you should not disappoint the expectations of users.