CCB Magazine

How far should technology go in reshaping our daily life? Is the blind trust in algorithms a thread to our autonomy? A new tech-company based in Berlin promises breathtaking advances in future music creation: “Melodrive uses Artificial Intelligence to create infinite, unique music in real-time that adapts to the emotional journey in digital experiences like virtual reality, games, and videos”, the strategic White paper outlines. Is this really useful? We asked one of its founders, Valerio Velardo, a pianist, composer, former conductor, and PhD in music and Artificial Intelligence (AI).  
 

INTERVIEW Boris Messing
 

CCB Magazin: Valerio, you are one of the three founders of the company. You claim that through Melodrive the user engagement in games and virtual reality (VR) will increase by 40%. According to a study of ARD and ZDF people in the age between 14 and 29 spend in average 4,5 hours per day online, time on TV and game consoles not included. From 2000 to 2017, the user engagement on the internet octuplicated. Is that not enough? Do we really need to spend more time playing games and lingering online? 

Valerio Velardo (smiling): I get your point, but I think you are conflating things. It all depends on how technology is used. I give you an example. A couple days ago, I have been on the Oculus Store in which they offered a new program called “I am a man”. It was an incredible experience! Through a head-mounted display you were teleported in a virtual reality of the 1960s in the USA, when Martin Luther King was assassinated. You were living as one of these poor black men experiencing his daily life of being downgraded and disdained. You could almost touch and feel what these people were going through. It was a powerful experience and a piece of art somehow. Through VR you could learn about another time in a very immediate way and enhance your knowledge. For those purposes a deeper immersion through music is very useful. It’s more than just entertainment. 

CCB Magazin: You founded the company in 2016. Melodrive is based on the usage of AI. How would you define AI? 

Valerio Velardo: AI is the umbrella term for algorithms that are created in order to solve specific tasks. I give you an example. Let’s say the task is to win a chess game. In this task, the goal is clear and objective. The algorithms are built to analyze every single move of the chess game in order to calculate any possible outcome. The conclusions, finally, result in the decision to play this or that move. With Melodrive it’s different. We don’t have an objective goal when we create music. 

CCB Magazin: So, how exactly does it work?

Valerio Velardo:Melodrive is the first AI music system that composes an infinite stream of original, emotionally variable music in realtime. Melodrive’s music continuously adapts to user interaction and emotion, as well as the affective conditions of an experience. You might know the „Painting Fool“, an AI based program that is able to paint creatively. It’s a little bit like that but just with music. The music in a game or VR provided by Melodrive is varying and individual for each player or user. It’s like a real composer is sitting next to you while you’re online and creating music for you. This is called computational creativity.  

People in general tend to be very conservative about new things, but once a technology becomes familiar and mainstream the skepticism disappears

Valerio Velardo with some of his empolyees from Melodrive (Photo: Jens Thomas )
 

CCB Magazin: Melodrive is created for digital platforms of interactive, non-linear content like video games or VR. What do you mean by interactive and non-linear?

Valerio Velardo: Any game or VR where people can connect with each other is interactive. Let’s take, for example, World of Warcraft. Here the gaming experience is non-linear, because every single action depends on the behavior of the other players online. The general outcome of the game is unclear and, therefore, exciting. 

CCB Magazin: I used to play Final Fantasy VII on PlayStation 1. How does the music work in that game? Would you say it has a non-linear setting?

Valerio Velardo:This is a good example. Final Fantasy VII is an early version of a non-linear game. The map allows the player to make a decision what he or she wants to do next within the context of the game. But Final Fantasy is accompanied only by adaptive music. That means that the music changes with the action. For example, if the health of your character in the game worsens, the music may change from calm to tense in order to adapt his emotional stage. The music’s purpose is always to intensify the emotional presence of the character or scene and, therefore, of the player. But it’s pre-composed and comes from a library. Deep adaptive music, in contrast, offered by Melodrive for games like Final Fantasy or VR experiences, brings it to the next level. Imagine, the music in a game is not fixed but created instantly in real-time instead. 

CCB Magazin: So Melodrive will be used more for entertainment than for learning in the end. And even in VR like in your example of the 1960s in the States the learning process is underwent without any effort and, therefore, less storable in the brain. Reading books is much more sustainable as far as gaining knowledge is concerned.  

Valerio Velardo: Nobody keeps you from reading books! See the usage of AI as an access to options. VR is an extension of possibilities to learn. Also, there is quite extensive research in psychology that suggest that learning with interactive content such as video games and VR may be more effective than learning through traditional media such as books. People remember information better in interactive settings, because they can perform tasks and interact with the virtual world, instead of only passively reading words in a book or watching a documentary. But let me give you another example for the positive side of VR. A few days ago, I got into this VR called Rec Room, a virtual world where you, embodied as an avatar, can hang out with friends and do a lot of different things together, play paint ball, charade, darts, etc. It was a cool experience! Where else can you do such things with people from all over the world at a touch of button? You can’t do that in Berlin.

CCB Magazin: For me, the main difference between the digital world and the “real” one is that in the latter the encounter of people sets off an immediate emotional reaction. It’s immediate, because you perceive the other person with all your senses, you see, smell, hear, and feel him or her in a way that enables you to respond in an empathetic way. Some scientists assume that acting socially online or playing games makes us less empathetic. Others claim the opposite. Does hanging out online lead to social alienation?

Valerio Velardo: Yes and no. People in general tend to be very conservative about new things, but once a technology becomes familiar and mainstream the skepticism disappears. It’s a broke world. First-person shooter games and blockbusters usually don’t enhance our knowledge about things. They’re often a cheap and dull distraction. But technology itself is neither positive nor negative, it always has been neutral. Technology is actually giving people the opportunity to be a creator once again. User generated content on digital platforms is only one example for this. People writing blog posts, making videos, building experiences online, etc. 

Boris Messing and Valerio Velardo: talking about AI-based music compositions (Photo: Jens Thomas) 
 

CCB Magazin: How can composers use Melodrive? Are they able to choose different instruments to make up music themes, for instance?

Valerio Velardo: Yes, of course. The AI analyzes any music input you give and shapes a new, original output from this. For composers it’s like exploring self-made music themes and extending them. How would this song sound like and evolve if I pursue this direction? What happens if I add this or that instrument? Etc. 

CCB Magazin: From the perspective of a composer, isn’t there a danger that I lose my skills, when relying too much on AI? In the sense that I only have to give the ingredients in the mixer and it blends it to something big and beautiful that I could never have achieved alone? 

Valerio Velardo: Did we lose our music skills when the piano was invented? Or when Ableton came on the market? No! At the end of the day AI is a tool, a powerful tool, to be sure. If used the right way, it may enrich our music creative potential and let us do things that are not possible today. Nobody, for example, can write music in real-time that matches the emotional setting of a video game. With AI, composers could arrive at that. In the end, it’s not about replacing composers; it’s about providing more creative possibilities. 

Our vision is to develop Melodrive to the stage where it composes and performs music in real-time with the quality of a Hollywood movie soundtrack

CCB Magazin: How do you charge people using Melodrive?

Valerio Velardo: We just finished off the early access version. We have test users that integrate Melodrive in their games and give us feedback in order to improve the software. We want to sell it, finally, to digital platforms where users are able to create and share content. Digital platforms will pay for integrating Melodrive based on the number of users they have. We also plan to allow composers to sell their own virtual instruments, styles, and music packages on a digital marketplace.

CCB Magazin: The outcome of Melodrives instant compositions is still very granular. To me, it sounds a little bit like the early years of techno music. How will the future of Melodrive sound like?   

Valerio Velardo: Our vision is to develop Melodrive to the stage where it composes and performs music in real-time with the quality of a Hollywood movie soundtrack. The compositional outcome will be infinite. The composer of a Hollywood movie knows exactly what’s going to happen in the next one, two, or five minutes of the film. It takes him months to create the soundtrack. Melodrive, instead, creates the music instantly. But it is not made to replace the composer but to extend his skills, to explore musical themes in a much faster and flexible way. Music software like Amper and Aiva Technologies go in the direction of replacing composers. Melodrive tries to help them, as well as people with no musical skills, to improve their music creative potential. 

An early version of Melodrives instant compositions
 

CCB Magazin: According to Bitkom many business are afraid of the possibly negative effects of digitalization. Millions of jobs could be at stake - or not. Will software like Melodrive create new kinds of jobs, in your opinion, or lead to losses?

Valerio Velardo: Digitalization as a whole could lead to mass unemployment; there is definitely a danger looming from afar. However, if well regulated, the advent of AI could mean the liberation of people from the need to work to survive. We could have machines taking care of “basic” jobs and have people focusing on more creative endeavors. Concerning Melodrive, I don’t think it will have a big impact on unemployment.  

CCB Magazin: Valerio, in three years from now if Google offered you a billion dollars for your company, would you sell it?

Valerio Velardo (laughing): No.