When So-Hee Woo builds sets for artists like The Kid Laroi or Trippie Redd, his tools aren’t a hammer and saw, but 3D computer graphics programs and game engines to build virtual worlds.
Ms. Woo, head of XR design at Encore, creates computer-generated sets for augmented reality (AR) shows, which fans watch in the Encore app, paying 10 euro cents for each clap.
If that sounds like a lot to digest, relax: even people who work at the interface of music and online virtual spaces, or what’s loosely called the metaverse, find the pace of change dizzying.
“At first, not all artists really know what it’s about,” says Woo.
AR performances in the app are just one of many approaches to the idea of bringing music into the metaverse, combining the excitement of crowded live events with the massive potential audiences of online platforms.
The forced isolation of the pandemic has given rise to many attempts to bring the concerts to living rooms, from shows on YouTube to virtual festivals complete with twirling avatars and sound and light shows.
Often these attempts have not quite achieved their goal. It turned out that the multi-sensory excitement of a crowded room was difficult to replicate.
But the success was enough to prove the potential of virtual concerts.
Today, as the necessary technology continues to improve, millions of dollars are poured into vastly different formats of live music in the metaverse, in a race to find which will be the most popular.
These experiences might change the way you experience music, but probably not in the way you expect.
Wait… what is the metaverse again?
Basically, the metaverse is a collection of interconnected spaces – some entirely virtual, some physical but with virtual elements – in which you or your avatar can have a number of immersive, 3D, real-time experiences. .
The idea is that one day there would be some level of “interoperability”. That is to say that these spaces could talk to each other.
If your avatar buys a t-shirt or wears their hair a certain way, those characteristics could be transferred from one virtual world (for example, Meta’s) to another (that of a game developer like Epic). The same goes for any data you generate, such as preferences.
As you might have guessed, these immersive virtual worlds don’t exist yet, despite all the talk and Facebook’s decision last year to change its name to Meta, a metaverse company.
But there’s more to the Metaverse than avatars with long hair and tight T-shirts. It can also be a more subtle mix of virtual and physical worlds.
This is where Augmented Reality (AR) comes in. For example, have you ever used an app to test the look of a table or sofa in your home? Or point your phone at the sky to discover different constellations? This is AR in action.
“You probably already know a lot of different apps and experiences you can have that point to the metaverse,” says Sofie Hvitved, media officer at the Copenhagen Institute for Future Studies.
“This layering, this interaction between our virtual and physical worlds, is the [chose] the most interesting about the metaverse. »
Where music and metaverse meet
You might be wondering: When will I be able to attend a concert at Madison Square Garden in New York (or elsewhere) from my living room in Australia?
And if I can’t do it, what’s the point anyway?
It would be difficult to recreate the feeling of attending a live event. At the very least, it would require many technological advances, including photorealistic 3D imaging and precise hand and eye tracking.
Additionally, “all the senses play an important role,” says Sydney-based metaverse and blockchain consultant Patricia Haueiss. This includes touch, smell and even taste.
“The multi-sensory integration of spatial audio, smells and the connection of haptic combinations with experiences would take things to the next level. »
Instead of these totally immersive experiences, we currently have an array of different approaches.
On the one hand, there are those that essentially attempt to recreate a physical performance but in an entirely virtual space, using existing (limited) technology. Think crowds of avatars moving and moving.
And at the other end are those that offer an overlay of online interactions on physical performance.
Last year, for example, Justin Bieber donned a motion-capture suit that translated his movements onto a digital avatar and danced to the soundtrack of a few of his hits at a concert he attended in the metaverse.
Last week, English hyperpop queen Charli XCX performed her songs in a “galaxy of superstars” released on the Roblox gaming platform.
These and other metaverse concerts have attracted millions of viewers, but they have a downside: expensive production and an audience consisting mostly of young players already present on the platforms.
As a result, metaverse apps that are more accessible and require less production cost from artists have recently been released.
Democratize the production of shows on the metaverse
Performance app company Encore, launched earlier this year, is trying to “democratize” metaverse shows, according to co-founder and chief executive Jonathan Gray.
“Only the top 0.00001% of artists today get what you might call a metaverse show.”
The Kid Laroi show in May, for example, lacked the fancy avatars and relatively simple 3D graphics that Ms. Woo completed in a matter of hours.
The Aussie rapper sang and danced on green screen, looking like he was in a 90s video game.
Encore splits the profits with performers, who “typically earn” at least $2,000 per show from these 10-cent claps, Gray says. Crowd sizes varied from less than a hundred to 14,000 people.
This week, the company launched a new app allowing artists to create their own augmented reality environments, including 3D objects and visual effects, so they don’t have to rely on designers like Ms. Woo.
“And that brings you closer to what can be done for a six- or seven-figure amount by these great global recording artists,” Gray says.
Entertainment company Live Nation and social media platform Snapchat have taken a similar route to Encore, with AR content that works on mobile phones. In April, they co-launched AR filters that festival patrons could use during shows.
“When you look at the Live Nation experience, they’re trying to see how they can engage people in new ways…in addition to their experiences in the physical world,” says Ms Hvitved of the Copenhagen Institute for Future Studies.
Ms Haueiss, the metaverse consultant from Sydney, explains that AR at festivals could provide additional information, such as “animations, instructions such as the nearest camp sides, the location of your friends, and updates. day on the line-up in real time”.
“Companies are testing and playing around, hosting festivals, galleries, parties, concerts… the sky is the limit. »
Another hint of where entertainment is headed came in March, when Epic Games, the maker of hit video games like Fortnite, acquired independent online music platform Bandcamp.
Epic hasn’t revealed its plans for Bandcamp, but it’s clear it sees an opportunity in the music and metaverse combination.
In 2020, a Travis Scott concert in Fortnite was watched by 12 million players and led to a record number of online players.
The Australian start-up Splash, based in Brisbane, is also experimenting in this area.
She created a game in Roblox (a gaming platform) where players have access to a series of loops (guitars, synthesizers, voices, etc.) that they can use to write music, which they can then perform as an avatar.
Fans can rate their performance, which affects their level of in-game fame. Splash also has a currency that players can earn and use to buy clothes for their avatar.
Most Roblox users are teenagers or children and are too young to perform in clubs, says Thomas Stenning, who works in machine learning at Splash.
“The person who is going to be a DJ in a club is most likely over 18 or has a whole bunch of benefits [comme] have expensive DJ decks and probably spent several years honing his skills,” he says.
For him, the potential of the game is to unleash the creativity of people who otherwise would not consider themselves musicians.
The open, immersive spaces available on Roblox can be a sign of what’s to come, much the same way MySpace blazed a trail in social media that Facebook later followed, Stenning said.
“Some people call what Roblox is the start of the metaverse.”
What holds us back?
Besides the technical challenges of creating interactive virtual worlds frequented by millions of users at the same time, there is the simple problem that most people don’t own VR headsets.
“Hardware, like mixed reality headsets, needs to become less bulky, more affordable, with long battery life, a wide field of view, and stellar quality even in challenging situations like direct sunlight,” explains Ms. Haueiss.
And even if we had the headsets and suits, could Australia’s currently underperforming internet support the massive amounts of data needed to render 3D spaces in real time?
Perhaps it’s for these reasons that some of the most ambitious and well-funded recent metaverse gigs have been marred by technical issues and frustrations.
At a concert hosted by Meta in February with the Foo Fighters, fans were unable to enter the venue after the digital lobby crashed, while the immersive 3D experience was spoiled by the presence of human cameramen.
In addition to technical issues, there are social issues. Who will own the metaverse? Will it be open, like the World Wide Web, or proprietary? Will it be accessible to everyone or will it be a niche and a portal? These questions remain unanswered.
Despite the potential issues, Hvitved believes we should focus on the direction of technology and the behaviors of young audiences, rather than the vision of a specific company or technology.
“The famous way of looking at things is this: if [Henry] Ford had asked people what they wanted, they would have wanted faster horses,” she says.
“It is sometimes difficult to see these solutions”.