3D Virtual Cyber Band in Unity3D

dietilaminometilcumarin

Hi all!

In general, I’m sure no one has raised such a topic on this forum yet, although I could be wrong, but I’ll continue anyway. I am a musician, in music for the third decade. He started as a drummer in a youth group at the city's House of Pioneers (now the House of Youth Creativity). For many years he has been professionally engaged in programming in the financial field on the 1C platform. Recently, the craving for music has returned, but in a new incarnation. I started writing my own instrumental music. Fortunately, many different goodies have been created for this, such as all kinds of digital studios, streaming platforms, and so on. But all this "junk" is clearly not enough if the main component of a good band is missing and this is talent. I won't say that I don't have it. Each person is talented, but each in his own way. So, after writing two albums already, and this is about two dozen tracks, I began to notice that there are a lot of creators like me in all our vast mother earth. Each in his own way writes not bad music, but not all of them have a group with which they perform in front of the public. It turns out that while sitting at home a person is engaged in absolute creativity, but in no way can this creativity be revived, that is, go on stage and do the most ordinary concert.

I myself, having spent a lot of time looking for live musicians, set out to perform in front of the public. But either the high demands of the musicians themselves, or their employment in other more successful projects, or my lazy mother did not allow us to combine our efforts. Time passed, and with it the hope that someday I would go on stage and the audience would again applaud the glory of our national Rock flowed away. After a little whimpering, I somehow moved away from this topic, well, that is, the topic of music, and began to engage in game development. I got carried away with 3DMax, Unreal and Unity. I learned to do all sorts of different things by posting them on special sites like free3d and others. A little later, having become acquainted with the daz3d program, I enthusiastically discovered the world of characters (avatars) with quite a decent appearance and endless possibilities for modeling, animation and other design. Suddenly, I was visited by another crazy idea, but what if?

What if you take these vavtars and make friends with my musical creativity, and not just make friends, but make a musical group out of them, or in English BAND. To pick up clothes for them, musical instruments and teach them to play my own music. After all, as I already wrote above it, as much as two albums. And months of painstaking work flew by. Choosing avatars and equipping them with all the necessary equipment was not difficult, but teaching them how to play, here I had to reinvent my wheel. How I did it, read below...

In order to teach a 3d character to play a musical instrument, the main thing is to be able to play it yourself. Otherwise, how can he be shown which button to press and which drum to beat. Fortunately, my experience and knowledge in this matter helped me. But in order to create the desired animation, at first I had to put each finger on the right key and record the drum parts by making magical passes over the drums with the hands of the 3d drummer himself. This is a very long and boring business. It was necessary to look for other ways, and I found it. The MIDI (Digital Language of Musical Instruments) technology helped me in this. The fact is that in MIDI each note has its own number, and so, if you read these numbers and associate them with the movements of the limbs of the avatar, you can make him play as they say according to the notes. For a drummer, it's simple. It is necessary to record the necessary hand coordinates for each note of the drum, and when reading the MIDI file, an animation for the drummer will be automatically created.

Fortunately, daz3d has a built-in language and the ability to work with COM technology, which means you can write your own script that will do the whole routine - read files, write the score and give the finished result directly to the 3d editor. No sooner said than done. The first attempts have already allowed me to release my first video where the percussionist grandfather plays the djembe, see the photo below. Writing animation for other musicians was not an easy task. For example, the bass player's game still had to be done manually. Considering that it was necessary to play the SLAP! it was important to show the viewer a certain manner of performance, since slapping is a complex mechanical process. I have not yet brought my keyboard playing to the required level, so even MIDI in this regard does not solve the whole problem yet, and I have to do a lot by hand. But I'm working on it, I hope that a solution will be found. There is really one such miracle method, when using which animation can be written on the fly, using a special suit with built-in gyroscopes on each limb. The cost of such a device is about 3K bucks.

And so, having put everything together, I managed to record my first music video, where all my new avatars (musicians) play my music. Wow! what could be worse. A band that doesn't exist in the real world now plays my music digitally. As you can see from the screenshot, the group has been created and endowed with everything necessary, but another problem has arisen. The fact is that each of the members of my ensemble (band) uses a lot of computing resources. Putting them all in one scene and rendering it is simply impossible. My vidyuha goes into endless idle time and rendering of one frame takes up to an hour. It turns out that to get HD video lasting only one minute with 30 frames, it will take more than a day for continuous rendering. If an average music track lasts about 3-4 minutes, then one clip needs to be rendered for a week. Only a film studio can afford this ;-(

It turns out that creativity, whether you like it or not, stupidly rests on money, that is, on the power of a computer system. You need to have a render farm to write 3D clips. Remote rendering rates are also not encouraging. In general, creating videos for Youtube is too expensive. It turns out that it's cheaper to play with live musicians than to build your own digital cyber band. But what if you can't do it but really want to? And I went back to thinking. After moving away from this idea for a while, I returned to game dev again. After a while, it suddenly dawned on me. Damn, I thought, but if the average computer cannot render my video, then why not do it the way it is done in games, let this clip play in real time, that is, the outline and all the calculations are done on the client side but not on mine . It turns out that the cooler the user's computer, the better the quality of my video.

TIME TO OPEN CHAMPAGNE

In general, friends, I came up with the concept of a music store in a non-standard manner. I now record music videos with 3d musicians not in the form of a video clip, but in the form of a special application for a smartphone where the viewer sees the plot being generated right now. The video clip is born before the very eyes of the viewer, of course, according to a pre-formed scenario, as it happens at a live concert. Now any composer who writes his own music, even without the presence of a live group of musicians, can release his creation into the world, even in the form of animated characters, but his music can no longer only be heard but also seen. Plus, the performance can be filled with light and video effects. For this purpose, I came up with a special mechanism that controls the lighting fixtures placed on the stage. Now you can even create a light show with reference to the timeline.

I'll probably surprise you, but it seemed to me too little. Watching a clip even in real time with a good render is good, but somewhat banal. This is the same as clips on Youtub, only a different streaming technology and nothing more. I put my musical group in virtual reality and now, wearing 3d glasses or a VR helmet, you find yourself right among 3d musicians and listen to music in complete immersion. You can also move among the musicians, which allows you not only to watch their play, but even learn something from them. I am even thinking about creating a virtual school for playing musical instruments, where the avatars themselves and not people will teach. Ask why? Yes, because the 21st century is in the yard. The future is here friends!

Now I am closely engaged in this project and if you are interested in this topic, I will be glad to cooperate. It doesn't matter if you are a musician or a game master. The main thing is to join forces and create together. At the next stage of my plan, the creation of a virtual catalog of performers similar to the stars of the world stage. I want to see Queen or Metallica in tridesh performance. I think you can even make very good money on this by offering musicians the development of virtual space. But that is another story.

my Patreon: patreon.com/CYBERBAND

All the best!

charles

I'm best friends with one of the world's most natorious DJ's ok. He is friends with Jakel, and was the original creator for Ozzy's Boneyard.  We've worked on deep fakes of musicians and now using AI voices to make WHAT IF's.. Like what IF Jimmy Hendrix replaced Randy Rhodes. The current state of the music industry is SH*T. And everyone involved in it knows it. You want to talk shop, and ideas, I'm game. PM me for a private channel invitiation. MAYBE we can do something..if not BIG..at least FUN.

 

 

dietilaminometilcumarin

charles said:

I'm best friends with one of the world's most natorious DJ's ok. He is friends with Jakel, and was the original creator for Ozzy's Boneyard.  We've worked on deep fakes of musicians and now using AI voices to make WHAT IF's.. Like what IF Jimmy Hendrix replaced Randy Rhodes. The current state of the music industry is SH*T. And everyone involved in it knows it. You want to talk shop, and ideas, I'm game. PM me for a private channel invitiation. MAYBE we can do something..if not BIG..at least FUN.

 

Thanks