The triumph of the assistance systems.
' Autonomous driving will gradually become reality. '
RALF GUIDO HERRTWICH
From here the S-Class knows the way home because it has travelled it many times already. It effortlessly finds its way through the dense traffic of the megalopolis, where cars, trucks, buses, bicyclists and pedestrians are all intent on moving at their own pace. In the traffic-reduced area the vehicle adheres to the prescribed walking pace because it can read traffic signs and thanks to radar sensors and stereo cameras it also keeps a keen eye on pedestrians at all times. After a few minutes the destination is reached and there’s even a parking space to be found. The vehicle stops to let its passengers get out. A push on the electronic key and the vehicle parks itself in a space whose dimensions would provide a challenge even for experienced drivers.
Until a few years ago engineers and computer scientists developed such science fiction scenarios to provide a visionary outlook of the mobility of the 21st century. Now reality has caught up with them, because all the manoeuvres described above are already possible or are being tested under real-life conditions with the help of the latest assistance systems from Mercedes-Benz.
Consequently, everyday life will face a profound revolution. Even though the vision of autonomous driving goes back many decades, it is only now that the combination of steadily increasing computing power, innovations in the area of sensor technology and the scanning of a vehicle’s surroundings paired with the rapid digitisation and networking of everyday life makes driverless mobility attainable. There are many possibilities to enhance traffic safety, to make mobility more efficient and environmentally compatible and to create unimaginable freedoms for all road users. However, before the goal of highly or even fully autonomous driving is reached, several development obstacles must be overcome in order to make the hardware and software faster, more intelligent and more affordable. In addition to Mercedes-Benz, researchers and development engineers of electronics companies, automotive suppliers and universities are working on intelligent hardware and software intended to gradually make the vehicles autonomous. At the same time the infrastructure, legislative bodies and society will have to prepare for this new dimension of motoring.
“Autonomous driving will gradually become reality”, states Ralf Guido Herrtwich, Head of Driving Assistance and Chassis Systems in Group Research and Advanced Engineering at Daimler. “Initially we will drive autonomously on certain classes of roads, starting with the motorway and maybe only under certain weather or lighting conditions. In the beginning the system will also have to be monitored rather than grabbing a book and tuning out completely.”
Accordingly the researcher also warns of too quickly placing too high an expectation on autonomous vehicles that manage without any human intervention. “At low speeds, in stop-and-go traffic or when parking driverless mobility is a matter of years away. At high speeds and in complex situations the driver will be involved at least for the next ten years”, says Herrtwich. There are several reasons for this realistic assessment. Assistance systems already on the market have shown that partly autonomous vehicles can lower the accident figures because they compensate for human errors and react precisely within milliseconds – in many cases even more precisely than most people. According to the latest traffic safety study by the World Health Organisation more than 1.2 million people die each year on the roads. By 2020 the European Commission (EU) wants to cut the number of traffic deaths in the 27 EU member states – 35,000 in 2010 – by half.
Driving assistance systems like the ones that are to some extent already standard equipment at Mercedes-Benz play a crucial role in this context. Today these technologies are already capable of merging comfort and safety. They include DISTRONIC PLUS proximity control, which keeps the desired distance from the vehicle travelling ahead. In addition, the STEER CONTROL steering assistance system, for example in the new Mercedes-Benz E-Class (Fuel consumption of 10.3-4.1 l/ 100 km (NEDC combined), CO2 emissions, combined 242-107 g/km, efficiency class: F-A+)* and S-Class (Fuel consumption, combined cycle: 10.3-5.5 l/100km, CO2 emissions, combined: 242-146 g/km, energy efficiency class: F-A)*, keeps the vehicle in the centre of the lane. However, drivers need to keep their hands on the steering wheel at all times. Active Lane Keeping Assist can intervene when the driver unintentionally crosses a dotted line and the adjacent lane is occupied. The previous generation of the lane-keeping assistance system was already capable of detecting when a solid line was crossed. BAS PLUS Brake Assist with Cross-Traffic Assist cannot only prevent rear-end collisions, but can also intervene in the event of impending collisions with crossing traffic at junctions, if need be even including a full emergency stop. The latest version can now recognise pedestrians walking in front of the vehicle, warn the driver visually and audibly or in emergency situations even initiate autonomous braking.
* The figures are provided in accordance with the German PKW-EnVKV and apply only to the German market. They do not concern an individual vehicle and are not part of the offer; they are provided solely for the purposes of comparison between different types of vehicles.
These intelligent systems are made possible by an array of sensors that provide the vehicle with a 360-degree view of what is going on. Radar sensors of different ranges can “see” for a distance of up to 200 metres. Their input is complemented by a stereo camera behind the windscreen. Thanks to its two eyes, the camera can see a three-dimensional image of the area up to about 50 metres in front of the car and from there on – similar to human eyes to infinity – two-dimensionally.
All the data constantly streaming in are processed by various on-board systems, for instance, to calculate the trajectory of crossing vehicles or a pedestrian in anticipatory fashion, “read” traffic signs and issue appropriate warnings or initiate reactions. This makes it possible, for example, to let a vehicle drive or even autonomously overtake other vehicles safely at high speeds with the Mercedes-Benz Motorway Pilot system that has already undergone successful testing under real-life conditions.
Ideally autonomous automobiles equipped with the necessary sensor package, detailed map data and sufficient computing power can travel virtually any arbitrary route. One of the milestones for autonomous driving was the DARPA Grand Challenge, which was organised by the research and development branch of the US Department of Defense in the desert of Nevada in 2004 and 2005. Only on the second try did some of the expensive and hair-raisingly retrofitted vehicles manage to complete the route that stretched over 240 kilometres of very rough terrain.
' It is astonishing how far we have come this past decade. '
WILLIAM 'RED' WHITTAKER
“Many of the hardware and software components are still too expensive. They are plainly and simply unaffordable for normal consumers. If I had that much money, I’d buy a great sports car and drive myself”, jokes Emilio Frazzoli, professor of aerospace engineering at the Massachusetts Institute of Technology (MIT), who normally deals with autonomous vehicles travelling by land or air.
This is why Daimler researchers like Ralf Guido Herrtwich are trying to offer an intelligently assembled array of radar sensors and cameras that collects the required information even without costly lasers in order to travel safely, efficiently and comfortably. “This technology ultimately mustn’t cost any more than today’s driving assistance systems, that is to say, a couple of thousand euros”, Herrtwich stresses. This also includes a continuously updated digital map that provides significantly more detail and may also remain more up to date than those of conventional navigation systems. Otherwise an autonomous vehicle will flounder if it encounters a new, non-registered construction zone or a recorded bend that deviates from the values measured by the on-board sensors. However, vehicles can assist each other in creating such new real-time maps because theoretically every car is able to record the route it travels and to feed the route data into databases.
Experts like CMU professor Whittaker expect autonomous vehicles to see the world differently. Their navigation aids have little in common with the combination of conventional maps and superimposed images we know from today’s assistance systems. “We are already able to create three-dimensional models of our environment that are better and more detailed than the human eye would ever be able to perceive”, says Whittaker of the initial prototypes. Such super-realistic models of the environment are generated partly on board and – thanks to mobile broadband access to the internet in future vehicles – partly in the Cloud.
Not only the vehicles need to evolve, the surrounding infrastructure does as well. Companies like Daimler have long researched so-called car-to-x communications that allow vehicles to exchange data with each other and their surroundings, including road signs and traffic cameras mounted above the road.
In April the Los Angeles metropolitan area became the first city in the world to synchronise all its 4500 traffic lights. Magnetic sensors in the road and hundreds of cameras feed their data into a central computer that dynamically controls all traffic lights to speed up the traffic flow of seven million daily commuters. During rush hour the system can phase the traffic lights only for bus lanes while other vehicles have to wait. “Especially for driving in an urban area surrounded by hundreds of thousands of other vehicles we already have a wealth of information as well as the infrastructure for lowering the costs and complexity of autonomous driving”, MIT researcher Frazzoli reflects. “A car can use its surroundings and other vehicles for its eyes and ears”.
Besides all the technical advances that are happening rapidly this also requires another change that has already begun. Society at large and legislative bodies have to rethink what constitutes the nature of a vehicle and of the modern transportation system overall. Because what would be possible technically is frequently legally impermissible. The Vienna Convention on Road Traffic from 1968 determines who may steer a car: “Every driver must have control of his vehicle at all times […]”. Nobody thought of a computer of whatever kind at the wheel 45 years ago. And thus questions about certification and insurance as well as liability in the event of accidents are still a grey area.
Some legislators have tackled the issue. The US states of Nevada, California and Florida were the first to pass laws that govern the certification and operation of autonomous automobiles. This provides an incentive to companies to test their prototypes there and serves as a role model for one of the world’s largest automobile markets. Should the US establish national rules for autonomous vehicles, the EU and China would soon follow suit. Until then, autonomous driving will continue to be relegated to narrowly defined areas of application where people may never really take their hand and eyes off the steering wheel.
“We build all the systems in a way that ensure the driver regains full control the moment he or she wants to take over. Our systems are fully dedicated to providing support and relief”, says Daimler researcher Herrtwich. In his mind the transition from partly to fully autonomous systems is not only a matter of the technical capabilities of the systems, but goes hand in hand with the driver’s growing trust. “Once you personally experience that such a system works, then you trust it in more and more situations.“
That is precisely what people seem to do if they are members of the group called ‘digital natives’, that is to say, all those who grew up surrounded by digital devices and services and in many cases willingly and completely count on technology. They hope that autonomous vehicles will relieve them of performing tedious routine tasks, such as commuting to and from work. People who try to talk on the phone, write something on their smartphone or even read their emails while driving will mostly be thrilled by the prospect of soon leaving the driving largely to the vehicle. The designers are already sketching driver’s seats for concept vehicles that swivel to let drivers direct their attention to a tablet computer or the newspaper instead of watching traffic.
Many senior citizens will also put their hope in the next vehicle generation or the one after that because their sensor systems and algorithms can make up for their own declining abilities. This promises to increase the mobility radius for millions of people who previously had been severely limited by old age, illness or disability.
Against this background it is understandable that Google promotes the prototypes of its autonomous vehicles with a video showing a blind man regaining his mobility thought to have been long lost. “For people with disabilities and senior citizens autonomous driving is a question of human dignity”, robotics researcher Whittaker believes. “For that we by no means need vehicles that drive autonomously under all conditions”. He envisions fully automatic people mover systems for public transportation, such as are already in existence at many airports. And some local authorities are considering their use in inner cities.
' The ‘sharing economy’ and autonomous driving are two sides of the same coin. '
Instead of waiting for all-capable fully autonomous vehicles to arrive, says Frazzoli, carsharing services should be outfitted with vehicles that have a limited list of capabilities, such as finding the way to the nearest filling or charging station, picking up a waiting customer at a specific address, or if needed moving to another location. Such cars would solve several problems of autonomous driving at once, the scientist argues: “Since they’re driving without human passengers, they can always take the easiest route, for example, like a municipal commercial vehicle they could initially drive slowly at the edge of the road, and even if their steering and braking manoeuvres were a bit jerky, it wouldn’t bother any occupants. “In this way it is possible to lower the requirements on autonomous vehicles and at the same time expand their fields of application”. With growing experience the autonomous carsharing fleet could increase its effective radius.
There is still the question of how people at the wheel will come to terms with vehicles that act ever more independently. Experts agree that for the foreseeable future there will be mixed operations: some of the vehicles will be steered by people, while others drive partly or highly autonomously. Vehicles will enter and exit parking spaces at the push of a button. Or learn an oft-travelled route to derive independent actions therefrom. The urban infrastructure will increasingly exchange data with the road users. But at the same time there will be older vehicles on the road that have a lot less electronics and intelligence.
To William Whittaker this interaction of man and machine is not a problem. “When we drive on the motorway, we don’t have any direct contact with other drivers even at high speeds. You observe and interpret the behaviour of other road users. This works for all kinds of driving situations without having to draw a distinction between man and machine. Only one thing is for sure: autonomous driving is already a done deal today and will continue to advance steadily”.
THE HISTORY OF AUTONOMOUS DRIVING AT MERCEDES-BENZ
At Mercedes-Benz the research and development work for realising the vision of autonomous driving and thereby offering motorists a maximum of safety and comfort goes back more than half a century. It all started with cruise control to keep a preset speed. This became standard equipment in all Mercedes-Benz vehicles in Europe in the 1960s.
The next milestone was the Prometheus (short for “Programme for European Traffic with Highest Efficiency and Unprecedented Safety”) research project initiated by Daimler-Benz that was intended to demonstrate new perspectives for the traffic of the future. The project was part of the European “Eureka” research initiative from 1986 to 1994 and involved unprecedented cooperation between all major European automobile manufacturers, suppliers and numerous science institutes. The results were presented to the public in October 1994. Prometheus addressed the key questions of tomorrow’s mobility, such as how to enhance safety despite the growing number of vehicles and how to harmonise traffic flow without building new roads and causing more environmental stress. Researchers from Mercedes-Benz soon realised that this would require the integration of new technologies – from microelectronics and sensor systems to telecommunications and data processing – in as comprehensive a fashion as possible.
In close cooperation with a team headed by Ernst Dickmann, professor at the Bundeswehruniversität (University of the German Armed Forces) in Munich and a pioneer in the areas of computerised image recognition and autonomous driving, the company presented several autonomous test vehicles in the 1980s and 1990s. They included the “VaMP” (“Versuchsfahrzeug für autonome Mobilität und Rechnersehen” or test vehicle for autonomous mobility and computer vision”) based on a Mercedes-Benz 500 SEL, as well as the VITA and VITA II. The test vehicles made headlines in 1994 when they drove some 1000 kilometres in the Paris metro area and in 1995 when they drove from Munich to Copenhagen and back. VITA, an abbreviation for “Vision Technology Application”, was an autopilot capable of braking, accelerating and steering. The intelligent test vehicle has small video cameras behind the front and rear windscreen that provided the on-board computer with an overview of the situation surrounding the vehicle. The computer was for the first time able to discern the course of the road and register whether VITA was on a collision course. The Prometheus sub-project demonstrated that an automatic collision prevention system was technically feasible.
Another milestone was the Electronic Stability Program (ESP), whose concept had been patented way back in 1959 as a “regulating device” and which celebrated its world début in the S-Class in 1995. ESP recognises critical handling situations as they develop, brakes one or several wheels as required and adjusts the engine torque to keep the vehicle on track. Another result of Prometheus is also well familiar to motorists today: the intelligent cruise control that always keeps the required safety distance. It was introduced in the S-Class in the mid-90s and today is standard specification at Mercedes-Benz under the name DISTRONIC and DISTRONIC PLUS. Other innovations such as a lane-change assistance system, the automatic PRE-SAFE Brake or an electronic parking aid stem from this research programme. Work on implementing the visions put forward some 30 years ago continues to today – on the road to the stated goal of accident-free driving.
Since around the year 2000 the Mercedes-Benz experts have also been working on communication between vehicles and their surroundings and among each other, referred to as “car-to-x communication”. If vehicles communicate with each other, they can warn each other of obstacles on the road or a bend covered in black ice. This evolved into the “simTD” project, which stands for “safe, intelligent mobility – test site Germany”, which has been undergoing trials in the Rhine-Main area around Frankfurt since 2007. This projects involves companies from the automotive and telecommunications sectors, the government of the German state of Hesse as well as renowned universities and research institutes. simTD is also sponsored by the German Federal Ministry of Economics and Technology, the Ministry of Education and Research as well as the Ministry of Transport, Building and Urban Development.
As early as 2009 Mercedes-Benz presented ATTENTION ASSIST, which can recognise driver inattention and drowsiness based on changes in the steering behaviour and a multitude of other parameters. Today the system cannot only remain vigilant at speeds between 60 and 200 km/h, but can be calibrated to individual drivers. Observing drivers will play an even more important role in autonomous driving, as some stages of the automation will depend on monitoring driver attentiveness. Many of the most recent safety features were presented to the public in the ESF 2009 experimental safety vehicle, followed by the F125! research vehicle two years later. It was equipped with numerous functions for partly autonomous driving, such as the fully automatic lane change at the push of a button.
Since 2012 the entire range of intelligent assistance systems from Mercedes-Benz has been summed up under the term “Intelligent Drive”. The basis is provided by the innovative 6D vision technology, which processes and analyses the images from the stereo cameras on board. The vehicle sensors register what is going on around them within fractions of a second. Stereo cameras paired with radar sensors recognise moving objects such as vehicles or pedestrians, measure their position and direction of travel including their speed and compute how they will behave in traffic. In critical situations the assistance systems installed in the car then react appropriately and at lightning speed.
Initial applications of “Intelligent Drive” in production cars are DISTRONIC PLUS proximity control with STEER CONTROL as the Stop & Go Pilot in the new Mercedes-Benz S- and E-Class as well as the BAS PLUS Brake Assist with Cross-Traffic Assist. 360-degree vision makes partly autonomous driving in dense traffic possible. Mercedes-Benz provided a more concrete vision of the future of autonomous driving in early 2013 with an E-Class that for testing purposes was fitted with the new “Motorway Pilot” system that allows fully autonomous overtaking manoeuvres. The system independently searches for a sufficiently large gap in the left lane, changes lanes, overtakes slower vehicles and automatically moves back to the right lane in front of them.