Fraunhofer-Publica

The Fraunhofer-Publica has been successfully documenting the research results of the Fraunhofer-Gesellschaft for over 30 years. The platform enables the collaborative linking of research-relevant objects and disseminates within the international scientific community.

The Fraunhofer-Publica thus fulfils its responsibility to promote the transfer of knowledge and know-how to industry and society.

Categories

267555

Research outputs

As an application-oriented research organisation, Fraunhofer aims to conduct highly innovative and solution-oriented research - for the benefit of society and to strengthen the German and European economy.

13974

Projects

Fraunhofer is tackling the current challenges facing industry head on. By pooling their expertise and involving industrial partners at an early stage, the Fraunhofer Institutes involved in the projects aim to turn original scientific ideas into marketable products as quickly as possible.

7681

Researchers

Scientific achievement and practical relevance are not opposites - at Fraunhofer they are mutually dependent. Thanks to the close organisational links between Fraunhofer Institutes and universities, science at Fraunhofer is conducted at an internationally first-class level.

77

Institutes

The Fraunhofer-Gesellschaft is the leading organisation for applied research in Europe. Institutes and research facilities work under its umbrella at various locations throughout Germany.

Recent Additions

  • Publication
    Can you trust your ML metrics? Using Subjective Logic to determine the true contribution of ML metrics for safety
    ( 2024)
    Herd, Benjamin orcid-logo
    ;
    Burton, Simon
    Metrics such as accuracy, precision, recall, F1 score, etc. are generally used to assess the performance of machine learning (ML) models. From a safety perspective, relying on such single point estimates to evaluate safety requirements is problematic since they only provide a partial and indirect evaluation of the true safety risk associated with the model and its potential errors. In order to obtain a better understanding of the performance insufficiencies in the model, factors that could influence the quantitative evaluation of safety requirements such as test sample size, dataset size and model calibration need to be taken into account. In safety assurance, arguments typically combine complementary and diverse evidence to strengthen confidence in the safety claims. In this paper, we make a first step towards a more formal treatment of uncertainty in ML metrics by proposing a framework based on Subjective Logic that allows for modelling the relationship between primary and secondary pieces of evidence and the quantification of resulting uncertainty. Based on experiments, we show that single point estimates for common ML metrics tend to overestimate model performance and that a probabilistic treatment using the proposed framework can help to evaluate the probable bounds of the actual performance.
  • Publication
    Human and Machine Performance in Counting Sound Classes in Single-Channel Soundscapes
    ( 2023)
    Abeßer, Jakob
    ;
    Ullah, Asad
    ;
    Ziegler, Sebastian
    ;
    Grollmisch, Sascha
    Individual sounds are difficult to detect in complex soundscapes because of a strong overlap. This article explores the task of estimating sound polyphony, which is defined here as the number of audible sound classes. Sound polyphony measures the complexity of a soundscape and can be used to inform sound classification algorithms. First, a listening test is performed to assess the difficulty of the task. The results show that humans are only able to reliably count up to three simultaneous sound sources and that they underestimate the degree of polyphony for more complex soundscapes. Human performance depends mainly on the spectral characteristics of the sounds and, in particular, on the number of overlapping noise-like and transient sounds. In a second step, four deep neural network architectures, including an object detection approach for natural images, are compared to contrast human performance with machine learning–based approaches. The results show that machine listening systems can outperform human listeners for the task at hand. Based on these results, an implicit modeling of the sound polyphony based on the number of previously detected sound classes seems less promising than the explicit modeling strategy.
  • Publication
    Can you trust your Agent? The Effect of Out-of-Distribution Detection on the Safety of Reinforcement Learning Systems
    ( 2024)
    Haider, Tom
    ;
    Roscher, Karsten
    ;
    Herd, Benjamin orcid-logo
    ;
    Schmoeller da Roza, Felippe
    ;
    Burton, Simon
    Deep Reinforcement Learning (RL) has the potential to revolutionize the automation of complex sequential decision-making problems. Although it has been successfully applied to a wide range of tasks, deployment to real-world settings remains challenging and is often limited. One of the main reasons for this is the lack of safety guarantees for conventional RL algorithms, especially in situations that substantially differ from the learning environment. In such situations, state-of-the-art systems will fail silently, producing action sequences without signalizing any uncertainty regarding the current input. Recent works have suggested Out-of-Distribution (OOD) detection as an additional reliability measure when deploying RL in the real world. How these mechanisms benefit the safety of the entire system, however, is not yet fully understood. In this work, we study how OOD detection contributes to the safety of RL systems by describing the challenges involved with detecting unknown situations. We derive several definitions for unknown events and explore potential avenues for a successful safety argumentation, building on recent work for safety assurance of Machine Learning components. In a series of experiments, we compare different OOD detectors and show how difficult it is to distinguish harmless from potentially unsafe OOD events in practice, and how standard evaluation schemes can lead to deceptive conclusions, depending on which definition of unknown is applied.

Most viewed