Sensors4Rail: A successful project comes to an end

On April 6 the Sensors4Rail test vehicle "Santana" was successfully transferred from Hamburg to Berlin Spandau to Havelländische Eisenbahn (HVLE) and subsequently decommissioned. This marked the successful completion of the 15-month test and data collection in the Hamburg S-Bahn network and thus the final phase of the project.

The first phase of the project ran from 2019 to the end of 2021 and included system development with industry partners, vehicle conversion and demonstration of live functionality at the 2021 ITS World Congress in Hamburg. In the second phase, which lasted from January 2022 to June 2023, the "Santana" test vehicle was optimized for continuous test operation. Subsequently, a total of over 500 hours of test runs were carried out in the S-Bahn network of the city of Hamburg. By extending the test area to other lines in the Hamburg S-Bahn network, valuable insights were gained into the performance of the "object and rail detection" functions in different scenarios and line sections. In addition, the focus was on increasing the diversity of the data. In addition to new test sections, this also included the recording of raw data under a wide range of weather conditions and at different times. On June 14, the joint project closing meeting of all project partners and participants took place at HVLE area in Berlin Spandau. In the future, the train "Santana" will be used as a demonstration object for presentation purposes. On "Tag der Schiene" on September 15 and 16, 2023, it will be presented to the public in Berlin Spandau for the first time since 2021.

One of the goals of Digitale Schiene Deutschland is fully automated driving, i. e. automated driving in Grade of Automation 4 (ATO GoA4) in the open railroad system. It enables a flexible deployment of available personnel and traction units. Through this new, digital technology the full potential of the digitization of rail operations can be exploited. However, the development of approval-ready products in the rail sector for those at ATO GoA4 is still at an early stage. This requires, among other things, environment perception, a more precise localization and digital maps – technologies that are not yet available for a roll-out. For this reason, Digitale Schiene Deutschland, together with industrial partners, has carried out a prototyping of the required hardware and software components as part of the Sensors4Rail development project. In the process, a vehicle of the 472 series of the S-Bahn Hamburg was equipped with numerous sensors for environment perception and localization, which are explained in detail in the video embedded above. Based on this, industry partners Siemens Mobility, Bosch Engineering, MicroVision (formerly Ibeo Automotive System GmbH) and HERE Technologies prototyped and demonstrated six fundamental functions of the future GoA4 operation.

The technological challenges, such as range of the required sensor technology and robustness against environmental influences, differ significantly in the railroad system from those in other industries. In the automotive sector, for example, a visibility range of 150 m is typically required for functions in an urban environment. In the case of trains, the braking distance is many times longer – therefore, in the rail sector, higher demands must be placed on the visibility of the sensor technology at the front of the train to ensure at least the same level of safety as in today's operations. For future, it is therefore necessary to quantify these requirements more precisely.

In the second phase of the project, a total of over 500 hours of time-synchronous multi-sensor data and functional data of the Sensors4Rail system were recorded from the trial operation in the Hamburg S-Bahn network. This corresponds to over 450 terabytes of data. This includes the raw data from 14 sensors installed at the front of the vehicle (cameras, lidars, radars) as well as the course of the track detected by the system, detected landmarks, people, trains and potential obstacles in and at the edge of the clearance profile at any given time. The recording period covered a little less than a year (from May 2022 to the end of March 2023). Thus, it was possible to record data at any time of the year. The recording times had to be closely coordinated with the S-Bahn Hamburg so as not to influence the ongoing S-Bahn operations there. The test runs on the so-called off-peak times between 9 a.m. and 2 p.m. as well as 9 p.m. and 5 a.m. The Sensors4Rail vehicle drove along during ongoing operations, but without passengers. At the same time, the specially trained driver had full responsibility for the vehicle at all times. The Sensors4Rail system was monitored remotely by the experts of the project team.

After recording, the data was transferred to the new data center of Digitale Schiene Deutschland, the so-called Data Factory, where it was subdivided into so-called "data sets". A data set represents one trip from terminal station to terminal station of a line in the Hamburg S-Bahn network, e. g., on the S21 line from Elbgaustraße to Aumühle. Since all data sets can be identified by a time stamp, filtering by time of year or time of day is easily possible. The data sets were additionally marked with "labels" that characterize the route traveled, the state of the system, or the software version of the system used during recording. Thus, for example, all data sets recorded on the S21 line can be filtered during the evaluation in order to analyze the influence of the respective weather and lighting conditions on the performance of the mentioned use cases.

The primary goal in the second phase was to take advantage of the unique opportunity to record data from a real operating environment on different lines using the Sensors4Rail system. Especially for the development of sensor-based environmental perception, data with a high diversity is needed. The reason for this: The future, train-side environment perception is to be optimized for operation. The AI working behind it must therefore learn as many different, realistic situations as possible during the development phases. These include, for example, crowded platforms, objects in the track bed or workers in the track area – regardless of the time of day or season. Data sets from a test track that is always the same, recorded only in the afternoon, for example, would not be as diverse.

The data will be used in a variety of ways, and the identification of use cases has only just begun:

• In the area of environment perception, a catalog of scenarios relevant to railroad operations is being compiled, which will be incorporated into the subsequent specification of the series systems.
• In the area of localization, detailed evaluation of the various absolute and relative localization data recorded, such as GNSS, landmark-based localization and the IMU (Inertial Measurement Unit), as well as odometry data, is currently taking place. Performance in tunnels without continuous GNSS reception plays a particularly important role.

The development partners are using the raw sensor data collected in the project as a basis for further developing the functions, which have so far only been implemented as prototypes, for the next development steps towards series production.

In the next step, the Data Factory team at Digitale Schiene Deutschland selects particularly interesting data sets from the entirety of the 500 hours of test data and then has these raw data "labeled" (annotated) by partners. This means that a so-called "ground truth" (reference) is added manually to each image. People appearing in a camera image and the course of the track are marked. This ground truth forms the basis for comparing the performance of different algorithms. Once this complex process has been completed, Digitale Schiene Deutschland will publish some of the most relevant data sets. As was the case with the first open data set, OSDaR23, it will once again collaborate with the “Deutschen Zentrum für Schienenverkehrsforschung” and make use of international standards. After all, data is the most important resource in the development of automated systems. Digitale Schiene Deutschland wants to share this resource with the sector and make it freely available to industry, operators and research institutions – for a more robust and modern railroad of the future.

Note:  AutomatedTrain - the follow-up project after Sensors4Rail - has started.