Audio, Text-to-speech and Scripts in the VR TimeTRAIN.

Audio is one of the most critical elements in every VR experience. It has several roles in creating an immersive production a successful one. 

Since VR is still developing its own techniques and can’t really borrow tricks used in the cinematography, the sound can help guide the player and convey the mood of the story being told in VR.

The roles of audio in VR could be listed us: 

• Immersion – 360 soundscape helps create immersion
• Presence – background sounds and effects can enhance the presence
• Guidance – rules and directions – helps the visitor to navigate through VR
• Realism – helps to create a realistic and believable experience
• Accessibility – can be an alternative way to interact with the environment.

Considering all the roles of audio in VR and its crucial role, I implemented various audio effects in the TimeTRAIN.

Intro Scene

• Ambient background sound – an instrumental melody called “Joyful Countryside Loop” sourced from the Unity Asset Store (UAS). It’s royalty-free audio. This sounds plays through the entire experience continuously, thanks to the script containing the “don’t destroy on load” function. It is an uplifting, happy tune. This sound creates a relaxing atmosphere in the VR experience but also plays a role in enhancing immersion and presence.

• Waves ambient sound, from Uppbeat called “Waves rolling onto a beach – Volo – long” (“Music from #Uppbeat (free for Creators!)”) . This is also a royalty-free sound I sourced through my free Uppbeat account and fits nicely in the scenery I created. This ambient sound adds to the immersion, presence and realism but also would potentially help to make my experience more accessible for people with sight impairment. 

• Welcome audio – plays on awake – its role is mainly related to guidance and accessibility. I wrote a short text welcoming the visitor in the Time TRAIN, and explaining how to navigate through this world. I then used an IA text-to-speech generator called “play.ht”. 

“Welcome to the Time Train.

In this experience, you don’t need to use your hands. 

You navigate through the scenes with your head movements. Look at certain objects for long enough to activate them and explore the story. 

I hope you enjoy it!”

• The penguin welcomes the player and invites them to join the train journey. I wrote a short script for the penguin and used the Amazon Web Services tool called Polly to convert my text to speech. 

“Hey, you! 

Welcome to the Time Train!

I hope you are not lost.

My fellow penguins and I are about to escape the winter and find summer. We travel to Sunny City. 

We swim there every year but this time we are going to get there differently.

Do you want to come with us?

Let’s look around and see if we can find a train we can take!”

To create the audio I decided to use the American accent of the voice called Ivy. The outcome is a rather funny, peachy voice, that wouldn’t be suitable for creating a human voice but it works well for the speaking penguin. 

Train Scene

• The same “Joyful Countryside” music continues from the intro scene. It has the same volume level for the purpose of not breaking the immersion.

• Train ambient noise – from the Uppbeat royalty-free music website, called “inside-train-rolling-along-railway-fascinated sound” for the purpose of creating a realistic experience of travelling on the train. This sound is muted and subtle but easily recognizable.

• Passengers’ voices and noises – I recorded myself and my close family members speaking and signing in Polish and English languages. I attached those audio files to various passengers on the train. This again enhances the presence, immersion and realistic experience of travelling on the train but also makes the experience more attractive with extra audio effects.

• Penguin’s narration (voice-over) – I wrote a script the penguin is saying. It is an exciting story of commenting on going to the beach on the train with penguins friends and other passengers, for example, the giants (= people). The penguin refers to what is happening on the train and outside of it, what stations are there, how they look and also how the penguin is feeling. 

• Waves sound triggered by the train approaching the beach.

“Oh wow, this is exciting! Travelling on a train with our penguin friends but also with all those giants. They must be very friendly creatures as they don’t seem bothered with us on this train. Perhaps it is pretty common to have penguins and other animals travelling on public transport here……. I wonder who else is on that train?… No no no…. what is going on here? Why are we stopping? 

This is not a Sunny city!

But it looks beautiful! Look at all those sweets, ice cream cones…and magical stars!… but it must be pretty cold with all that ice around… Oh great – the door is closing, and we are on the move again. Lucky us!…

hmm…I hope that all passengers aren’t going to our SECRET place. …But… what if they do? Will it be enough space and, most importantly, enough fish for all of us? 

I’m sure they are going somewhere else…

Why would people want to go to the hot, sunny beach? 

Hopefully, they don’t know about all the seafood and all the yummy fish you can get fresh straight from the water! 

I wish these friendly giants were going somewhere else where they are busy doing their human things. 

Anyway, I wonder how far it is now? How many stops until we get there? A million-and-five? 

Oh! Another stop… this looks like a big city… but not like our Sunny city … I can’t see any sand or palm trees… and it looks like it is raining here… It can’t be the last stop on our journey… Yes! I was right – the train is on the move again… it is so slow that we can look at the city. Wow – blue houses look pretty. Do blue people live in them? How do they look? 

Woohoo, we are going fast now. I wonder what our speed is? It must be a light speed.

What can we do now? Let’s look around and check how many passengers are left. 

There are penguins here, and some penguins there, and there is this blue and yellow guy copying penguins..- how funny! And there is a fox on the chair. It doesn’t move… is that a toy? 

I also noticed small giants, they must be… children… oh no… children like beaches and sand… do you think that they know about Sunny City? 

What is that now? There must be a station here… This place looks magical… look at the green trees and flowers… and even some strawberries there too! Is that a forest? I have never seen a forest before… 

But we are not getting off here.

The doors are closed and the train moves again…

I wish I knew how far from Sunny City we are…

We went through sweet Icy land, a rainy town and the enchanted forest… we must be getting pretty close now…What else would it be if not for our sandy beach… I am so excited and can’t wait to play in the lovely, warm sand and jump straight into the water then. We can race under the water…I can be really fast, you would be surprised! I am also getting hungry… Perhaps… I will go fishing first.

Hiphip Hurray! … we are here…! We have reached our destination! Let’s leave the train and have fun… but let’s not forget to take all our belongings with us!”

The above penguin’s script was rewritten several times while I was converting it to speech in the AWS Polly tool. I was testing it through in small parts before having a complete story that when converted to audio fitted nicely within the action of the Time TRAIN. 

The presented version of the text is suitable for the AI-generated voice but it is slightly too long for being ready by a human being. I tested it by recording my own voice on a mobile phone recorder and then applying it to the unity scene. When I notice that it wouldn’t work with the actions in the experience, I improvised and recorded a slightly shorter version of the story. It was a difficult task to read it out loud nicely and without mistakes in one go but I managed to do it more or less successfully.

For now, I am, however, going to use the AI-generated audio for the speaking penguin. 

I truly enjoyed working on audio from my VR project. It was great to see how it complements the other elements of the Time TRAIN. It was at times complicated and overwhelming but therefore, more satisfying seeing it work.