Integration between different data formats, and between data belonging to different collections, is an ongoing challenge in the MIR field. Semantic Web tools have proved to be promising resources for making different types of music information interoperable. However, the use of these technologies has so far been limited and scattered in the field. To address this, the Polifonia project 1 is developing an ontological ecosystem that can cover a wide variety of musical aspects (musical features, instruments, emotions, performances). In this paper, we present the Polifonia Ontology Network, an ecosystem that enables and fosters the transition towards Semantic MIR.

ISMIR late breaking demo

Semantic integration of MIR datasets with the polifonia ontology network

In this paper we propose the Music Note Ontology, an ontology for modelling music notes and their realisation. The ontology addresses the relation between a note represented in a symbolic representation system, and its realisation, ie a musical performance. This work therefore aims to solve the modelling and representation issues that arise when analysing the relationships between abstract symbolic features and the corresponding physical features of an audio signal. The ontology is composed of three different Ontology Design Patterns (ODP), which model the structure of the score (Score Part Pattern), the note in the symbolic notation (Music Note Pattern) and its realisation (Musical Object Pattern).

Proceedings of the 12th workshop on ontology design and patterns (WOP 2021), online, october 24, 2021.

The music note ontology

The use of Semantic Technologies - in particular the Semantic Web - has revealed to be a great tool for describing the cultural heritage domain and artistic practices. However, the panorama of ontologies for musicological applications seems to be limited and restricted to specific applications. In this research, we propose HaMSE, an ontology capable of describing musical features that can assist musicological research. More specifically, HaMSE proposes to address sues that have been affecting musicological research for decades: the representation of music and the relationship between quantitative and qualitative data. To do this, HaMSE allows the alignment between different music representation systems and describes a set of musicological features that can allow the music analysis at different granularity levels.

Proceedings of the 1st workshop on multisensory data and knowledge (MDK 2021), Zaragoza, Spain.

The HaMSE ontology: Using semantic technologies to support music representation interoperability and musicological analysis

The HaMSE ontology

This deliverable describes the results of the WP2 Musical Heritage Knowledge Graphs in its first 12 months with respect to the task T2. 1: Ontology-based knowledge graphs for music objects. The project, and specifically WP2 and T2. 1, uses knowledge engineering as a process and methodology to build ontologies and knowledge graphs that facilitate large-scale integration of various sources of musical heritage data. In this deliverable, we focus on music objects broadly understood as artefacts carrying musical content and meaning (recordings, music notation, musical theory, musical features) rather than contextual descriptions (eg composer, year of performance). New conceptualisations, in the form of ontologies, are needed to formalise the semantics of this domain and allow the creation of interoperable knowledge graphs from datasets. To achieve this, we apply a well-known ontology design methodology, eXtreme Design (XD), which emphasises:(i) ontology reuse and modularity through the design and use of Ontology Design Patterns (ODPs); and (ii) gathering a comprehensive inventory of ontological requirements through competency questions (CQs). To implement (i), we survey existing ontologies and ODP that are relevant to the domain, and existing datasets from Polifonia’s WP1 Pilots–building on the dataset survey work from WP1 [1]–and from the Music Information Retrieval (MIR) community; to implement (ii) we follow a thorough process of collecting and normalising CQs from the stories and personas that stem from the Pilots of WP1.

D2.1: Ontology-based knowledge graphs for music objects (V1. 0)

D2.1

Structure perception is a fundamental aspect of music cognition in humans. Historically, the hierarchical organization of music into structures served as a narrative device for conveying meaning, creating expectancy, and evoking emotions in the listener. Thereby, musical structures play an essential role in music composition, as they shape the musical discourse through which the composer organises his ideas. In this paper, we present a novel music segmentation method, pitchclass2vec, based on symbolic chord annotations, which are embedded into continuous vector representations using both natural language processing techniques and custom-made encodings. Our algorithm is based on long-short term memory (LSTM) neural network and outperforms the state-of-the-art techniques based on symbolic chord annotations in the field.

Proceedings of the 1st workshop on artificial intelligence and creativity

Pitchclass2vec: Symbolic music structure segmentation with chord embeddings

Pitchclass2vec

The annotation of music content is a complex process to represent due to its inherent multifaceted, subjectivity, and interdisciplinary nature. Numerous systems and conventions for annotating music have been developed as independent standards over the past decades. Little has been done to make them interoperable, which jeopardises cross-corpora studies as it requires users to familiarise with a multitude of conventions. Most of these systems lack the semantic expressiveness needed to represent the complexity of the musical language and cannot model multi-modal annotations originating from audio and symbolic sources. In this article, we introduce the Music Annotation Pattern, an Ontology Design Pattern (ODP) to homogenise different annotation systems and to represent several types of musical objects (eg chords, patterns, structures). This ODP preserves the semantics of the object’s content at different levels and temporal granularity. Moreover, our ODP accounts for multi-modality upfront, to describe annotations derived from different sources, and it is the first to enable the integration of music datasets at a large scale.

Proceedings of the 13th workshop on ontology design and patterns - WOP 2023

The music annotation pattern

Various disconnected chord datasets are currently available for music analysis and information retrieval, but these are limited by either their size, non-openness, lack of timed information, and interoperability. Together with the lack of overlapping repertoire coverage, this limits cross-corpus studies on harmony over time and across genres, while hampering research in computational music analysis (chord recognition, pattern mining, computational creativity), where machine learning methods are notoriously eager for data. To address this gap, we propose the Chord Corpus (ChoCo), a large-scale dataset that semantically integrates harmonic data from 18 different sources in various representations and formats (Harte, Leadsheet, Roman numerals, ABC, etc.). We leverage JAMS (JSON Annotated Music Specification), a popular data structure for annotations in Music Information Retrieval, to effectively represent and enrich a variety of chord-related information (chord, key, mode, etc.) in a uniform way. On top of it, we propose a novel ontology modelling music annotations and involved entities (artists, scores, etc.), build a 30M triple knowledge graph, and find 4K+ links to other datasets (MIDI-LD, LED).

Scientific Data, Nature

ChoCo: A chord corpus and a data transformation workflow for musical harmony knowledge graphs

ChoCo

Computationally creative systems for music have recently achieved impressive results, fuelled by progress in generative machine learning. However, black-box approaches have raised fundamental concerns for ethics, accountability, explainability, and musical plausibility. To enable trustworthy machine creativity, we introduce the Harmonic Memory, a Knowledge Graph (KG) of harmonic patterns extracted from a large and heterogeneous musical corpus. By leveraging a cognitive model of tonal harmony, chord progressions are segmented into meaningful structures, and patterns emerge from their comparison via harmonic similarity. Akin to a music memory, the KG holds temporal connections between consecutive patterns, as well as salient similarity relationships. After demonstrating the validity of our choices, we provide examples of how this design enables novel pathways for combinational creativity. The memory provides a fully accountable and explainable framework to inspire and support creative professionals -- allowing for the discovery of progressions consistent with given criteria, the recomposition of harmonic sections, but also the co-creation of new progressions.

The Web Conference 2023

The Harmonic Memory: a Knowledge Graph of harmonic patterns as a trustworthy framework for computational creativity

Harmory

This paper introduces a pattern to formalise context-free grammars in OWL and how to use it for sequence classification. The proposed approach and existing methods are compared and discussed based on their complexity as well as their pragmatic applicability with examples in the music domain. 

European Semantic Web Conference 2023

Classifying sequences by combining context-free grammars and OWL ontologies

Grammar

Music similarity is an essential aspect of music retrieval, recommendation systems, and music analysis. Moreover, similarity is of vital interest for music experts, as it allows studying analogies and influences among composers and historical periods. Current approaches to musical similarity rely mainly on symbolic content, which can be expensive to produce and is not always readily available. Conversely, approaches using audio signals typically fail to provide any insight about the reasons behind the observed similarity. This research addresses the limitations of current approaches by focusing on the study of musical similarity using both symbolic and audio content. The aim of this research is to develop a fully explainable and interpretable system that can provide end-users with more control and understanding of music similarity and classification systems.

Knowledge-Based Multimodal Music Similarity

The semantic description of music metadata is a key requirement for the creation of music datasets that can be aligned, integrated, and accessed for information retrieval and knowledge discovery. It is nonetheless an open challenge due to the complexity of musical concepts arising from different genres, styles, and periods – standing to benefit from a lingua franca to accommodate various stakeholders (musicologists, librarians, data engineers, etc.). To initiate this transition, we introduce the Music Meta ontology, a rich and flexible semantic model to describe music metadata related to artists, compositions, performances, recordings, and links. We follow eXtreme Design methodologies and best practices for data engineering, to reflect the perspectives and the requirements of various stakeholders into the design of the model, while leveraging ontology design patterns and accounting for provenance at different levels (claims, links). After presenting the main features of Music Meta, we provide a first evaluation of the model, alignments to other schema (Music Ontology, DOREMUS, Wikidata), and support for data transformation.

International Society for Music Information Retrieval Conference - ISMIR 2023

The Music Meta Ontology: A Flexible Semantic Model for the Interoperability of Music Metadata

Music Meta Ontology

In the music domain, several ontologies have been proposed to annotate musical data, in both symbolic and audio form, and generate semantically rich Music Knowledge Graphs. However, current models lack interoperability and are insufficient for representing music history and the cultural heritage context in which it was generated; risking the propagation of recency and cultural biases to downstream applications. In this article, we propose the Polifonia Ontology Network (PON) for music cultural heritage, centred around four modules: Music Meta (metadata), Representation (content), Source (provenance) and Instrument (cultural objects). We design PON with a strong accent on cultural stakeholder requirements and competency questions (CQs), contributing an NLP-based toolkit to support knowledge engineers in generating, validating, and analysing them; and a novel, high-quality CQ dataset produced as a result. We show current and future use of these resources by internal project pilots, early adopters in the music industry, and opportunities for the Semantic Web and Music Information Retrieval communities.

International Semantic Web Conference - ISWC 2023

The Polifonia Ontology Network: Building a Semantic Backbone for Musical Heritage

Polifonia Ontology Network

In the Western music tradition, chords are the main constituent components of harmony, a fundamental dimension of music. Despite its relevance for several Music Information Retrieval (MIR) tasks, chord-annotated audio datasets are limited and need more diversity. One way to improve those resources is by leveraging the large number of chord annotations available online, but it requires aligning them with music audio. However, existing audio-to-score alignment techniques, which typically rely on Dynamic Time Warping (DTW), fail to address this challenge, as they require weak aligned data for precise synchronisation. In this paper, we introduce ChordSync, a novel conformer-based model designed to seamlessly align chord annotations with audio, eliminating the need for weak alignment. We also provide a pretrained model and a user-friendly library, enabling users to synchronise chord annotations with audio tracks effortlessly. In this way, ChordSync creates opportunities for harnessing crowd-sourced chord data for MIR, especially in audio chord estimation, thereby facilitating the generation of novel datasets. Additionally, our system extends its utility to music education, enhancing music learning experiences by providing accurately aligned annotations, thus enabling learners to engage in synchronised musical practices.

Sound and Music Computing Conference - SMC 2024

ChordSync: Conformer-Based Alignment of Chord Annotations to Music Audio

ChordSync

We address the task of ontology learning by combining the structured NeOn methodology framework with Large Language Models (LLMs) for translating natural language domain descriptions into Turtle syntax ontologies. The main contribution of the paper is a prompt pipeline tailored for domain-agnostic modeling, exemplified through the application to a domain-specific case study: the wine ontology. The resulting pipeline is used to develop NeOn-GPT, a workflow for automatic ontology modeling, and its proof of concept implementation, integrated on top of the metaphactory platform. NeOn-GPT leverages the systematic approach of the NeOn methodology and LLMs’ generative capabilities to facilitate a more efficient ontology development process. We evaluate the proposed approach by conducting comprehensive evaluations using the Stanford wine ontology as the gold standard. The obtained results show, that LLMs are not fully equipped to perform procedural tasks required for ontology development, and lack the reasoning skills and domain expertise needed. Overall, LLMs require integration with the workflow or trajectory tools for continuous knowledge engineering tasks. Nevertheless, LLMs can significantly alleviate the time and expertise needed. Our code base is publicly available for research and development purposes, accessible at: https://github.com/andreamust/NEON-GPT.

International Semantic Web Conference - ISWC 2024

NeOn-GPT: A Large Language Model-Powered Pipeline for Ontology Learning

NeOn-GPT

The digital humanities is a diverse area of research, encompassing just as diverse data as disciplines, thus posing particular challenges for ontology engineering. This paper pushes the application of ontology design patterns to solve these recurring challenges. It proposes a set of potential generalised patterns identified in various DH-related case studies, which address recurring modelling issues in the DH domain: annotations, knowledge provenance, or the interpretation of cultural artefacts. Functioning as a proofof-concept, the work showcases how patterns could improve interoperability and knowledge capturing in ontologies for the DH.

15th Workshop on Ontology Design and Patterns - WOP 2024

One pattern to express them all? Towards Generalised Patterns for Ontology Design in the Digital Humanities

In the era of big data and machine learning, the fragmentation of musical datasets and the lack of standardised representations continue to hinder advancements in Music Information Retrieval (MIR). The multifaceted nature of music complicates both the representation of content - with small, task-specific datasets scattered across various formats, and context (metadata), where there is a lack of a consistent, structured terminology. These challenges increase the effort required for data collection and pre-processing, reduce reproducibility, and limit the scalability of MIR models. To address these issues, this thesis proposes a unified semantic model to foster interoperability and advance MIR tasks.
A specific instance of this fragmentation can be found in harmonic annotations, where harmony - an essential musical dimension - is inconsistently represented across datasets, formats, and notational systems. Taking harmony as a use case and leveraging the proposed semantic model, this thesis develops a standardised workflow to harmonise previously disconnected datasets, enabling the creation of new, large-scale unified corpora. Building on these harmonised datasets, a key contribution of the thesis is the exploration of harmonic similarity, which is used to organise, explore, and reveal connections across diverse tracks, historical periods, and genres. To this end, we implement novel state-of-the-art harmonic similarity functions, advancing current research in MIR. Moreover, by utilising the created large, varied data collections, we uncover deeper relationships within music and explore how diverse stylistic excerpts can be used for generative tasks.
While integrating symbolic data offers significant advantages, certain limitations persist. Primary challenges include the limited diversity of annotated data, often biased toward a narrow range of musical genres, and the inherent ambiguity and subjectivity in harmonic annotations. Such challenges have led MIR tasks like Audio Chord Estimation (ACE) to hit a "glass ceiling," where neither increasing computational power nor the volume of data has led to improved results.
To address these issues, this thesis explores a multimodal approach aimed at enhancing the analysis and understanding of harmonic data by jointly leveraging audio and chord annotations. We first propose a novel method for enriching the dataset with audio annotations aligned to the existing symbolic data. Building on this foundation, we introduce a new model for ACE that embeds formalised music theory concepts such as consonance and dissonance, addressing both training and evaluation challenges in the field. This model aims to mitigate the limitations of chord vocabulary imbalance and annotation subjectivity, ultimately improving the state-of-the-art in audio-based harmonic analysis.

PhD Thesis, University of Bologna

Harmonising Music Information Retrieval with Semantics: from Data Integration to Multimodality

Developing open-source foundation models is essential for advancing research in music audio understanding and ensuring access to powerful, multipurpose representations for music information retrieval. We present OMAR-RQ, a model trained with self-supervision via masked token classification methodologies using a large-scale dataset with over 330,000 hours of music audio. We experiment with different input features and quantization options, and achieve state-of-the-art performance in music tagging, pitch estimation, chord recognition, beat tracking, segmentation, and difficulty estimation among open self-supervised models. We open-source our training and evaluation pipelines and model weights.

Proceedings of the 32nd ACM International Conference on Multimedia (MM 2024)

OMAR-RQ: Open Music Audio Representation Model Trained with Multi-Feature Masked Token Prediction

Audio Chord Estimation (ACE) holds a pivotal role in music information research, having garnered attention for over two decades due to its relevance for music transcription and analysis. Despite notable advancements, challenges persist in the task, particularly concerning unique characteristics of harmonic content, which have resulted in existing systems' performances reaching a glass ceiling. These challenges include annotator subjectivity, where varying interpretations among annotators lead to inconsistencies, and class imbalance within chord datasets, where certain chord classes are over-represented compared to others, posing difficulties in model training and evaluation. As a first contribution, this paper presents an evaluation of inter-annotator agreement in chord annotations, using metrics that extend beyond traditional binary measures. In addition, we propose a consonance-informed distance metric that reflects the perceptual similarity between harmonic annotations. Our analysis suggests that consonance-based distance metrics more effectively capture musically meaningful agreement between annotations. Expanding on these findings, we introduce a novel ACE conformer-based model that integrates consonance concepts into the model through consonance-based label smoothing. The proposed model also addresses class imbalance by separately estimating root, bass, and all note activations, enabling the reconstruction of chord labels from decomposed outputs.

Proceedings of the 26th International Society for Music Information Retrieval Conference (ISMIR 2025)

From Discord to Harmony: Decomposed Consonance Based Training for Improved Audio Chord Estimation

The evaluation of music understanding in Large Audio-Language Models (LALMs) requires a rigorously defined benchmark that truly tests whether models can perceive and interpret music, a standard that current data methodologies frequently fail to meet.This paper introduces a meticulously structured approach to music evaluation, proposing a new dataset of 320 hand-written questions curated and validated by experts with musical training, arguing that such focused, manual curation is superior for probing complex audio comprehension.To demonstrate the use of the dataset, we benchmark six state-of-the-art LALMs and additionally test their robustness to uni-modal shortcuts.

Proceedings of the 4th Workshop on NLP for Music and Audio (NLP4MusA 2026)

HumMusQA: A Human-written Music Understanding QA Benchmark Dataset

Recent approaches in music generation rely on disentangled representations, often labeled as structure and timbre or local and global, to enable controllable synthesis. Yet the underlying properties of these embeddings remain underexplored. In this work, we evaluate such disentangled representations in a set of music audio models for controllable generation using a probing-based framework that goes beyond standard downstream tasks. The selected models reflect diverse unsupervised disentanglement strategies, including inductive biases, data augmentations, adversarial objectives, and staged training procedures. Our analysis spans four key axes: informativeness, equivariance, invariance, and disentanglement, which are assessed across datasets, tasks, and controlled transformations. Our findings reveal inconsistencies between intended and actual semantics of the embeddings, suggesting that current strategies fall short of producing truly disentangled representations, and prompting a re-examination of how controllability is approached in music generation.

Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP 2026)

Evaluating Disentangled Representations for Controllable Music Generation

We present the extended NeOn-GPT pipeline, an LLM-powered ontology learning pipeline grounded in the NeOn methodology. NeOn-GPT is a domain-agnostic ontology learning pipeline that comprises two components: (i) ontology draft generation, a multi-step prompting pipeline following the NeOn methodology; and (ii) automated ontology verification and resolution achieved through orchestrated calls to third-party tools complemented by LLM-suggested repairs. We evaluate NeOn-GPT across four distinct domains (Wine, Cheminformatics, Environmental Microbiology, and Sewer Networks) using both proprietary (GPT-4o) and open-source (Mistral, Llama-4, DeepSeek) models. Results demonstrate that LLMs can produce ontologies with high relational expressivity and partial conceptual alignment, though performance varies by domain and model.

Publications