Skip to main content

SMIRK Dataset

The dataset contains 4,928 scenarios of pedestrians crossing, or moving close to, a straight road at different speeds and trajectories in relation to the camera. Moreover, analogous scenarios of basic geometrical shapes crossing the road are included. All data has been generated using ESI Pro-SiVIC. The dataset has been used in the development of the experimental pedestrian automatic emergency braking system SMIRK led by RISE.

SMIRK dataset

In short

4,928 scenarios of pedestrians and objects sampled at 10 FPS. 

SMILE III research project led by RISE.

Data Type
The dataset is synthetic, and the images were collected from a forward-facing camera together with a bounding box and segmentation maps for the moving objects. Data quality assurance was provided as part of SMIRK’s safety case.

Camera images: png
Segmentation maps: png
Labels: csv

The dataset is synthetic therefore no anonymization is needed.

Ground truth semantic segmentation of pedestrians provided by ESI Pro-SiVIC. All crossing objects are annotated with bounding box and segmentation map (100%). No other parts of the scene are annotated.

The total size of the dataset is 120 GB and the size of individual files within the dataset is 6 x ~20 GB zip files + 1 x ~1GB zip file. Each image is ~300KB and each segmentation map is ~6KB.

The dataset is available for all AI Sweden partners.

Terms and Conditions
To use this dataset, you must comply with the SMIRK Terms and Conditions available below

SMIRK Terms and Conditions

RISE and ESI Group have agreed to share the data under a Creative Commons license: Attribution-NonCommercial 4.0 International (CC BY-NC 4.0). Please check out this explanatory file for a better understanding of what the Creative Commons license implies.

Guide for Creative Commons licens

Dataset specifics

Synthetic images collected from a forward-facing camera together with bounding box and segmentation maps for the moving objects. The images correspond to scenarios simulated in ESI Pro-SiVIC of single pedestrians and basic geometric shapes crossing a country road. The dataset corresponds to SMIRK’s deliberately restricted operational design domain, i.e., a straight rural road, in an open landscape, in clear weather conditions, with no other traffic agents.

Eight different pedestrian models and five gray basic shapes are represented.

  • P1 Casual female pedestrian
  • P2 Casual male pedestrian
  • P3 Business casual female pedestrian
  • P4 Business casual male pedestrian
  • P5 Business female pedestrian
  • P6 Business male pedestrian
  • P7 Child
  • P8 Male construction worker
  • N1 Sphere
  • N2 Cube
  • N3 Cone
  • N4 Pyramid
  • N5 Cylinder

Each pedestrian scenario is represented by 616 configurations organized into four groups

  • A. Crossing the road from left to right (280 scenario configurations)
  • B. Crossing the road from right to left (280 scenario configurations)
  • C. Movement parallel to the road toward ego car (28 scenario configurations)
  • D. Movement parallel to the road away from ego car (28 scenario configurations)

For groups A and B, combinations of the following parameter values represent a scenario:

  • Speed (m/s): [1, 2, 3, 4]
  • Angle (degree): [30, 50, 70, 90, 110, 130, 150]
  • Longitudinal distance (m): [10, 20, 30, 40, 50, 60, 70, 80, 90, 100]

For groups C and D, combinations of the following parameter values represent a scenario:

  • Speed (m/s): [1, 2, 3, 4]
  • Lateral offset (m): [-3, -2, -1, 0, 1, 2, 3]

More specifications can be found here

Use cases to date

1. SMIRK, an ML-based pedestrian automatic emergency braking system

The dataset has been used in the development of the experimental pedestrian automatic emergency braking system SMIRK led by RISE. ML components are increasingly being integrated into cyber-physical systems such as cars. Unfortunately, there are few completely open ML-based demonstrator systems for the research community to study. While systems such as OpenPilot and Baidu Apollo are available under open-source software licenses, their underlying ML models and their datasets are not available for study. As ML-based systems interweave data and source code, this is a research impediment.

The dataset synthesizes scenarios (pedestrians and geometric shapes) moving towards and away from a mono camera. These scenarios were used to train models for object detection/recognition and anomaly detection in the SMIRK system. The system is implemented for the simulator ESI Pro-SiVIC with core components including a radar sensor, a mono camera, a YOLOv5 model, and an ML-based anomaly detector.

The team successfully developed the SMIRK system and its complete development documentation is available on GitHub under a GPL license. As a fully transparent driver-assistance system, SMIRK can support future research on trustworthy AI systems, such as verification and validation, requirements engineering, and AI testing.

Further reading
→ SMIRK GitHub repository
→ Original Software Publication (OSP) has been accepted by Elsevier Software Impacts

2. Safety assurance of the ML component in SMIRK

How to best integrate ML components in safety-critical systems is an open challenge in both research and practice. ML-based computer vision is considered a key enabler for cyber-physical systems that rely on environmental perception. However, the step from demonstrating impressive results on computer vision benchmarks to deploying systems that rely on ML for safety-critical functionalities is substantial. An ML model can be considered an unreliable function that sometimes will fail to generalize new input to its learned representations. How should safety assurance for ML-based components be approached?

It was decided to develop the safety case in accordance with the emerging standard ISO 21448 SOTIF and the methodology Assurance of Machine Learning for use in Autonomous Systems (AMLAS).

Based on a highly restricted operational design domain for SMIRK, we present a complete safety case for its ML component, i.e., a structured safety argumentation supported by evidence. A safety argumentation using goal structuring notation is available in the SMIRK GitHub repository. Together with 34 individual artifacts, prescribed by AMLAS, this constitutes proof that SMIRK’s ML component has achieved its safety goals.

Further reading
→ SMIRK GitHub repository
→ arXiv preprint of the paper describing the safety case 



The dataset is available for all AI Sweden partners. Contact Beatrice Comoli and she will give you further instructions on how to access the data. Become a partner and engage in the Data Factory. If you are interested in becoming a partner of AI Sweden, getting access to the partner benefits, including the Data Factory and datasets, or in sharing a dataset or a model, please feel free to reach out.

Beatrice Comoli
Beatrice Comoli
Administrative Lead Data Factory
+46 (0)70-146 09 64