SuperCollider and Foley: Execute Order 035

Operation: Swordsmanship

As a movie buff, making movie SFX was the first lightbulb that ticked in my head when I saw “Final Project” pop up in the canvas assignments. My plan was to replicate the fight sounds from the iconic duel in Star Wars Episode V: The Empire Strikes Back (1980). Sound designer Ben Burtt originally used a projector motor hum with the buzz of an old television. Since my name is not Ben, nor is it Burtt, I do not have access to that retro equipment. I did not think it would be a good idea to devote my time trying to find electronic devices to replicate those two sounds. Instead, I dove into SuperCollider! Creating the buzz was actually pretty easy. I just generated a low-frequency sawtooth wave:

The hum required a little more experimenting. In a video by the channel ShanksFX (where he attempts to replicate the Star Wars SFX as well), he uses a square wave generator as a substitute. I tried a square wave in SuperCollider by way of a pulse wave, but I did not like the way it sounded. Instead, after searching around I used Impulse.ar as the hum:

Operation: Analog

After SuperCollider my original plan paralleled that of Burtt and Shanks. They played the hum and buzz together through a speaker. They then waved a shotgun mic in front of the speaker to create the famous doppler swing (vwoom vwoom).

Polar pattern of my Snowball iCE (top) and a standard shotgun mic (bottom)

As you can see a shotgun mic’s polar pattern is narrower, which is an advantage for movies and TV since it can isolate specific sounds like dialogue. Using my Blue Snowball iCE was a disadvantage. No matter how I moved it, the saber hum still sounded the same. The Snowball is just too good for lightsaber foley.

Though I did have a cheap shotgun mic (Rode VideoMicro), I discovered to my horror I had no adapter to plug it into my phone!!!! >:(

I considered recording the sound with the shotgun mic to a camera, yet I figured it would be quite a stretch to extract the audio from the video and retain my samples in pristine condition. I resorted to record with the TwistWave app on my iPhone 11 mic. Still horrible. You know the rumble striking your ears when watching a video from a windy day? The wind pickup was all I could hear, even with a gentle swish and flick across my speaker. I tried using different phones and different speakers to no avail. I had to shift gears. “Operation: Analog” was a failure.

Operation: Digital

Luckily, I did have a fallback plan. The doppler swing could be generated in SuperCollider. I already had the hum and buzz generated, so I just needed the full power to control the synth parameters in real time.

I remembered having too much fun with the cursor control (MouseX, MouseY) UGens, blowing my laptop speakers one too many times.

MouseX controls any parameter when the cursor moves left and right. The first assigned value within the UGen is attained when the cursor is at the left, and the second assigned one is attained at the right. MouseY is the same but for moving the cursor up and down.

For the hum synth, my MouseX controlled a multiplier from 1 to 1.1 for the frequency level. The frequency was also controlled with the linearly varying LFNoise1 UGen at a control rate of 1 from 90 to 91 Hertz. To create the sudden change in db levels for swings, I utilized the MouseY parameter on a multiplier for amplitude. Amplitude also had LFNoise1 but with a higher control rate of 19 to create a “flickering” effect. I added a MouseX control over a bandpass filter’s cutoff frequency to remove some of the “digitized nasal” sound.

In the original films I could hear subtle reverberations in the saber swings, so I added a MouseY to the wet mix level of a FreeVerb UGen.

For the buzz synth (sawtooth), I set a MouseX to control the cutoff frequency for a resonant lowpass filter, and a MouseY on an amplitude multiplier as well.

Video demo of mouse for doppler control

Operation: Sizzle

Even though I didn’t get to immerse myself in the hands-on process of the doppler swing, I was not 100% deprived of physical foley. While brainstorming, I was curious about how to recreate contact or clash sounds during lightsaber duels. Ben Burtt rarely speaks of this achievement in interviews. ShanksFX touches metal to dry ice to achieve contact sizzles, but I did not have ownership over dry ice of any sort. I actually spent an entire day trying to replicate the clash sounds in SuperCollider, which I would control with WhiteNoise and the MouseButton UGen. Thinking about stuff that “sizzles”, I considered placing water in a pan of hot oil. Due to fear of burning my house down, I instead used water with an empty, stove-heated pan. At first, I felt that the sizzles were not percussive enough, so I dumped water in the pan in one fell swoop:

Video demo of my foley

My methods were successful! I did keep some of the less percussive sounds for sustained contact between lightsabers. Other experiments with clashes include throwing an ice cube in the pan, and wiping the pan’s hot bottom over water droplets. By dumb luck I found two of my samples to be strikingly similar to the Jedi and Sith saber ignition sounds!

For retraction sounds I just reversed the ignition samples.

Next up, it was time to finally record my saber swings. I searched around and discovered that a general rule of thumb for recording most foley is with mono, so I recorded my saber sample wav files into one channel. I planned to record my samples in time with the finale duel between Luke Skywalker and Darth Vader in Episode V: The Empire Strikes Back (1980). After getting in the groove of keeping in sync with the film’s swings, I recorded my first SuperCollider sample, combining the buzz and hum.  Lord love a duck, it was no fun to listen to. When I was recording whilst watching the scene, I had a lot of fun. After solely listening to it, I found that referencing the movie was a not a good idea. Through the sample alone, I could hear myself constrained by the movie scene. The swinging I heard just felt awfully repetitive, restrained, and boring, and I spaced out halfway through listening. Not a good sign. I had to shift gears again!

Operation: Digital 2.0

In a quick brainstorming session, I tried to stage a fight scene of my own. The two fighters would start far apart (panned left & right), come to the center, and duel. I thought of what else would happen during a lightsaber duel. They would move around during the fight, then somebody might lose their lightsaber. I typed up shorthand “stage directions”  and I have the blocking of my scene here:

After staging the fight scene, I recorded a new sample whilst imagining my own staged scene. Listening to it I did get a little bored, but unleashing some more of my creative liberty gave me more optimism. After this successful (cough cough) recording, I recorded another hum/buzz sample to portray the opponent fighter. “Unfortunately, one week younger Michael Lee was not aware that he would have to shift gears yet again. When playing the two together, he heard the most dreadful thing in his ears.” Long story short, I could hear the phase cancellation! Both fighters shared the same humming frequencies. At some points my mouse wasn’t completely to the left of my screen, and that slight varying frequency made a big difference. The fighters’ hums kept cutting out. Separating the two with panning did not help. In addition, I recorded buzz for both fighters, and when played together they were awfully disruptive.

My solution? I created an option with Select.ar for a second doppler multiplier with a range below 1. The Jedi would solely retain the veteran hum, and now the Sith would have a lower frequency range hum combined with the buzz to represent the gnarly, darker side of The Force. Now it is easier to tell who the heck is who. After finally recording some more takes for Jedi and Sith, I could get right down to mixing and editing.

SynthDefs for saber sounds in the later stage of the production

Operation: “Fix It In Post”

Of course, Adobe Audition CC is a popular choice for mixing and editing. Tis’ quite versatile, as I may bestow upon it movie sound, or recorded musical performance. However, Waveform 11 Pro had something special that Adobe Audition CC did not: Multi Sampler. Like John, I took advantage of the MIDI-emulating resources of Waveform 11 did to compile my clash SFX library. That way, I could add in my clash sounds with the press of keys and the click of the pen tool, as opposed to sluggishly dragging them into the DAW and cumbersomely editing.

Multi Sampler with my clash SFX

I felt like I had unlimited power, but there was an unforeseen cost. I did have lossless audio, but I also suffered the loss of space. With almost twenty 32-bit clash sounds imported in the multi sampler, along with all my 32-bit SuperCollider recordings, there was definitely not enough space to fit my samples into a galaxy far, far away. I did not fly into any glaring struggles with the gargantuan-sized files until the end. Exporting, compressing, and uploading were the big troubles.

Anyways, after writing in most of my clash sounds, I still couldn’t hear a fight scene. The swings of the fighters’ weapons did not sound in unison despite my best efforts. Everything felt scattered and random. I had no choice but to spam the slash key and scramble my spliced saber clips. I really hoped to keep my samples fresh in one take. The process felt like performing excessive surgery on my samples but it is still a component that makes sound editing so essential. After some more scrambling, I finally got a decent sounding lightsaber fight.

Like a song, I have different sections of the duel. I want my sounds to tell a story, but I can only do so much with the samples I had. My course of action was better than recording alongside the movie scene, but in over three minutes, “vwoom” and “pssshhh” sounds get boring. My cat agreed when I played it to her. She fell asleep.

Therefore, I had to prioritize the movement of the fight throughout the space. Soundscape and spatialization are extremely crucial for movies. Mix mode activated! I automated panning to give the fight some realistic motion. I have the fighters start on opposite sides, panning one hum left and the other to the right. By use of automation I move them to the center. To give the fight more unity, I fed the hums and clashes to another track (a substitute for aux send/return) with more automated panning to indicate movement of both of them on the battleground fighting. For transitions in the fight sections, I recorded some footstep sounds, a punching sound for when the Jedi is punched by the Sith, and the flapping of a shirt to convey the Jedi subsequently flying through the air.

Jedi track with automated panning
Saber hums and clashes are fed into this track for more automated panning

Some filters were added on individual tracks. The buzz from the Sith saber overpowered the Jedi saber, so I added some EQ to boost the Jedi track. The clashes sounded too crisp, so I threw on a bandpass filter. Since a few clashes “clashed” with my ears while a few others were at reasonable amplitude, I placed a compressor to ease the sharp difference.

The final step was to create the distance (close/far) impression of the soundscape. I looked back at Huber and echoed this diagram.

From the Huber reading on mixing basics

Since I had to fiddle with volume, I exported my project with normalized audio and imported it into a completely new Waveform project for mastering purposes.  My changes in depth of field are only at the end of the duel. I automated some volume reduction,  EQ frequency decrease,  reverb wet mix increase, and lastly some delay mix for an improved difference.

I was but the learner. Now I am the master.

In Short?

As you can see from my process I had to change my plans several times. Similar to Josh’s project, my original plan was to use purely physical foley to replicate a scene from The Empire Strikes Back (1980). The final plan was to use a mix of SC3 and foley to construct a fight scene envisioned in my head. Petersen forewarned that sound for movies can require a lot of trial and error. A sound “scene” can be much harder to design for when there isn’t even a scene at all! Oddly enough, I found some of the miscellaneous foley to be the most difficult. I never realized how exact the sounds had to be for realism. When listening to the starting footsteps I imagined two people with extremely stiff legs taking baby steps towards each other instead of two trained fighters. I kept this in because I felt it was satisfactory, and it is also a good way of noting how unnecessarily difficult foley can be.

It would be an overstatement to call the designed duel product below “the tip of the iceberg”. It can be difficult to enjoy the final result by itself, so I am contextualizing my piece here as much as I can in this blog. Working on SFX put a lot on my plate, but it is the “fun” kind of work. Once the pandemic is over I will be having more fun with foley in the campus sound studios!

Use headphones (NOT AIRPODS) for best experience. Also close your eyes if you want to imagine the duel in your head.

Tribute to David Prowse, the OG Darth Vader.

Additional Sources

Xfo. (n.d.). Ben Burtt – Sound Designer of Star Wars. Retrieved December 13, 2020, from http://filmsound.org/starwars/burtt-interview.htm

 

Waveform 11 The Sequel: Where the Struggle Begins

Phase 1: Troubleshooting Technical Issues

I had trouble finding the subtractive synth, and I realized it came with the pro version. For some reason I had the free version and when I tried signing in with Petersen’s account the registration process could not complete. I removed all the plists from my libraries and preferences to solve this. However, I still had no luck. Over the weekend I decided to get a free trial of the Pro version with a personal account, but I was eventually successful accessing pro with Petersen’s account.

Phase 2: MIDI Fiddling and Recording

The primary synth was a 40SC preset to the stab synth. I fiddled with EQ and compressor. This way, the synth wouldn’t “stab” my ears. For this lead I just used the pen tool to write in my notes. At first, I tried playing the melodies on my laptop keyboard, but my hands kept fumbling and hitting too many wrong notes.

The stab synth’s notes at the piece’s intro, written with the pen tool

The bass synth was a subtractive synth with a preset to the Bernie’s Mini Bs. I activated the legato button on the bottom left of the virtual instrument. I felt the attack on the notes was too sharp at default, so I set legato on which takes much less time than automating pitch to different notes. Most of the notes for this bass were recorded using my keyboard, but I had to quantize and sometimes manually adjust the notes not only to stay in time, but also to stay in time the way I wanted them.

Subtractive Synth: Legato Mode Activated

I wanted to fiddle with velocity for the all synth notes to give a sense that they were being played by a person and not a robot. However, tinkering with velocity made the DAW crash, so I had to go on without velocity manipulation for this project.

Both the 40SC and Subtractive synths were looped for the first half because I wanted to start with a clarinet solo. I sidechained the stab synth with the clarinet so the two wouldn’t compete.

My percussion was the Micro 808 Drum sampler. I decided to use step clips since I wouldn’t have to manually adjust or quantize notes. I wanted to create a “swing” beat for this track. I disliked the overtones the sample sounds made, especially the snare. I did not use a notch filter and instead just used a pitch shifter to remove the ugly “different key” overtones. This fixed my problem and was left here for the rest of the project, though next time I am aware the notch filter might be better at the job. Throughout the song, more percussion notes were added in addition to the continuous swing beat brought by the kick and the snare. Notes were added in off beats as the song progresses. A side stick, a kick beat, an open hi-hat, all added to fill in the blank spaces from the start. Of course the hi-hat’s release lingered for too long, so I was happy I could shorten the duration of the sound in the drum sampler.

Hi-Hat Shortened

For recording samples I used my clarinet again. I followed the same format to improvise, playing whatever came into my head with multiple takes. The clarinet was positioned the same way, above my microphone. Anjali asked me previously what were the added benefits of my Blue Snowball microphone. One pro I found was that the FFT was much more flat than my iPhone 11, so the frequency pickup would improve the sound of my clarinet. However, I realized how airy my clarinet sound was in my previous project. This airiness was not as noticeable in the few iPhone recordings I did, which shows how sensitive my Snowball mic is. Part of it was because of my bad embouchure, but switching reeds made for some notable improvement. The sound of my clarinet was richer here. Like Ethan, I usually have too much fun with reverb, but I also took a step back to try other plugins. Instead of using reverb, I used an EQ plugin and raised the treble and mid boosts to thicken my instrument’s timbre and improve its space in the soundscape. Last time a lot of tracks sounded narrow and isolated on different areas of the channels when I panned them, but with EQ I was able to widen the space for each synth and instrument. Reverb also made the airy sounds from my clarinet audible. The reverb plugin is only used in the song’s final clarinet clip.

This picture was taken from my previous project, but my setup was more or less the same for this one.

For the second half I also recorded myself whistling faintly to create the sound of wind. The wind sounded very pleasant but also very unsettling to me with the whistling, and I found this juxtaposition fascinating. The air hitting the mic directly created some wind noise, creating the faint sound of thunder. 

I followed through with the call/answer technique from last time. Instead of using my clarinet, I used 40SC stab and string synth, panning them to opposite levels. The strings would play for two measures and the stab would answer. I played with various release parameters on the string synth which added a nice flicker effect.

Above is one of many takes of my faint whistling. I automated the panning to oscillate slowly between left and right levels later on in the final product.

Another track is an AUMidi synth. I was fiddling with the line tool earlier and I liked the sound the written patterns made. I used these patterns to transition my piece. The addition gave some variation. 

Above is the result of me messing around with the line tool

The ending string synth was a preset on the subtractive synth. I was having trouble figuring out how to end my piece, and I thought it would be interesting to finish on the sadder side. I played a few notes using the keyboard, adjusted them to stay on time, and added a tremolo, since I thought it would better accompany my wind whistling. I also automated the tempo from the starting 133 BPM to a slower 88.08 BPM to signify a conclusion. The tempo was again slowed to 64.34 BPM so the clarinet had time to shine and end the piece.

Tempo automation curve

I had a lot of difficulty adjusting attack and release for the subtractive synths. The strings have a very gradual attack and the bass has too quick of a release. During playback, random notes of the subtractive synth would sustain even if I did not write them in. Sometimes I would pause and I would still hear those notes. In my limited experience playing with MIDI, I have dealt with this problem in the past. At the moment this is a mere annoyance but I am worried the problem might escalate. What if I am doing a project and the random note just sustains nonstop, even if I relaunch the software? 

Phase 3: A Conglomeration of Plugins

Matters got very messy here. There were so many different combinations of plugins to achieve different sounds. At the start of the project, many of my plugins were adjusted by ear. Even being aware of what each one did, I still kept trying to fiddle with different combinations. For example, I thought to myself, “What does a treble level of 6dB combined with a compression ratio of 3:1 do?”  Another time I wondered, “Should I insert distortion with a phaser?” Too many questions went through my head.

I ended up having two types of EQ plugins on my percussion track. The AUparametric EQ had automations while the regular EQ improved the timbre of the drums. At some points the drums were overpowering everything else. Removing the AUParametric EQ plugin made the drums faint but evenly balanced.

The same problem happened with my stab synth, but with compressors. One (Compressor/Limiter) played the role of side-chaining with the clarinet while the other (AU Multi-band Compressor) supported the EQ easing the sharpness of the sound. I kept both dynamics filters because of these important filters each one played.

All the plugins for my stab synth

On the bright side, I realize that the EQ plugin is another great tool to improve the soundscape of my music.

Many of the dynamics filters on my tracks were compressors. I used presets for respective MIDI tracks, for example, setting my percussion to the Tracktion Compressor: Drum. I did want my clarinet to lead, but I was worried that it would overpower everything else. The clarinet sounded like it was trying to compete, so I added a compressor and adjusted threshold and ratio by ear. The visual parameter adjustment was extremely helpful for me.

Always nice when I can see what I am doing

Several songs I know create a buildup by automating a lowpass filter, increasing its cutoff frequency and resonance. I tried doing this with my bass synth track, also automating the volume and pitch of a single, sustained string synth note. My clarinet also holds a sustained note with automated volume crescendo. I added a phaser plugin on the clarinet clip since I liked the oscillating push the plugin added for the buildup.

Once again the phaser plugin

All these clips mixed to make a rich build in my piece. Following the buildup, I automated the bandpass on the stab synth back to default to restore the established “sharp” noise it originally came with. 

However, after the buildup, everything was significantly louder than before due to earlier plugin automations. I had to automate the master volume to keep the volume levels stable. I did not trust normalizing the audio to do that job. I did not normalize when exporting this time, and it sounds much more three-dimensional. The volume, however, is very imbalanced in some tracks so I had to go back and manually adjust as much as I could.

In addition, I realized that my audio reached the 0dB level several times. I was worried about clipping so I reduced the master volume, but then there was some static in playback. I was unable to fix this with whatever volume I adjusted, so I will have to find a solution in the future.

The mess got even messier with the aux bus. I just used an FX Chorus plugin with an aux output from the lead, string, and bass synths to reinforce the conclusion of the song, and to vary the MIDI tracks. However, after the class on October 13th, I should have known I could just create a new track as the primary output for the other tracks and automate my plugins there.

Phase 4: Mixing 

After installing the Voxengo Span plugin, I monitored the frequency spectrum and it seemed that there was a great absence between 10kHz to 20kHz. There was a spike around 200Hz to 300Hz with my clarinet, so I adjusted EQ and volume as best I could. Halfway through the piece when the 40SC string synth came in, those frequencies began to fill, but they became absent again following the buildup, despite having the string synths playing. I added AUMidi synth samples with a high pass filter to fill in the missing frequencies for the transition into the conclusion. 

The kick and bass synth occupy from 20Hz to 200Hz. The clarinet and stab synth take up 200HZ to 8kHZ, where all the spikes are. There was an unfortunate emptiness at 10kHz to 20kHz 🙁

Final Thoughts

The piece achieved the goal I set out, but it was much more experimental than my last. 

In experimenting with plugins, it was fun to play around with them but it is hard to keep them organized. Having two different EQ plugins on one track is not ideal, but this experiment helped me appreciate the power of EQ plugins. While I got some great experience with MIDI, I obviously still skimmed the surface with this piece. I did check out as many 40SC and subtractive synth presets as I could, but for this piece most of them just did not match the idea I was looking for. In particular, subtractive synths were difficult to use, since there were so many adjustable parameters on the synth itself. I agree with Petersen’s description that the Joker vomited on the interface. In addition, I think I will need to vary the kinds of plugins I use. This piece mostly had EQ and compression, and I took a step back on FX plugins, but next time I will be more adventuresome.

I DEFINITELY need to work on mixing. After the extension I wanted to improve upon this, but a big barrier was my disorganized plugins. Next time I should use the single-output send method that Petersen shared instead of individually trying to adjust my tracks.  That is not to say I will eliminate individual plugin adjustment methods, but the single-output will be something new to learn, and give my next piece some coherence. Now that I have the Voxengo Span monitor by my side, in Voxengo’s own words, I can “mix not just with my ears, but with my eyes.”

I call this piece “Standing up the Summit”

Summary of Features 

Design Idea: Using percussion to create a swing beat, simultaneously adding a bleak atmosphere.

MIDI: 40SC Stab Synth WMF, Subtractive Synth Bernie Mini Bs, Micro 808 Drum Sampler, 40SC String Synth, Subtractive Synth Cinematic Strings, AUMidi Synth

Plugins: EQ for most tracks, with the lowpass parameter automated. Compressors were also used for several tracks, each one set to a different preset or customized. Tremolo was used on subtractive strings. FX plugins include distortion, phaser, and chorus (aux bus).

Automation: These automation tracks were created separately unlike before, when automation curves were layered on top of my tracks which was very inefficient. Included this time was pitch shift, volume (both individual tracks and master bus), lowpass filter (from EQ), Aux bus automation with Chorus, and Tempo.

My First Attempt at Waveform 11

Blog Entries

The room I was in:

I was in my suite while recording. I intended to record in my own bedroom, but the radiators went off and I had no control over the noise. The practice rooms in Silliman were closed so that was not an option. I felt it was satisfactory to record in my suite. Petersen’s suggestion about recording in a larger space also encouraged me to choose my suite. 

The Mic I used:

I used a Blue Snowball iCE, with a cardioid polar pattern. The Blue Snowball iCE is a large diaphragm condenser mic with a USB cable. I also wanted to experiment with percussion recording, since Petersen mentioned how large diaphragm mics are useful for that. Since I did not have any percussion instruments, I just recorded my fingers tapping a table and placed them at random measures where I saw fit.

I am still unsure as to whether this mic has multiple directional settings. I am hoping not, since Petersen mentioned that sometimes multiple settings can inhibit optimal pickup. I positioned the mic about 6 to 8 inches away from the bell of my clarinet (vertical distance).

Source from this data sheet

The Samples:

I searched FreeSound.org for some clips, which is where I got the night time wildlife ambience that opens and closes the song, as well as the kick beat heard in the intro. Thank goodness Petersen shared SSLIB. I initially wanted the piece to give “the experience of being in outer space and on land”, which is why I added the wildlife ambience with the 80s classic synth samples. However, upon discovering a good number of the samples from the SSLIB were at around 125bpm, I decided to create a tempo change to keep up with that. Hence, the music now turned from a soothing song to a faster-paced song one may hear at a nightclub. I did maintain the ambience chord change motifs for the first half, but many of the samples were in A, so I decided to put the song in A minor in the following measures.

Several samples were borrowed from the Aalto Sequences folder in SSLIB.

Right before the F choir samples (modified with pitch shift to C), you can hear a chord progression inspired by The Second Storm by Robert W. Smith. This snippet helped me transition the song. The next part contained three choir tracks to represent a C-major interval, accompanied by a bell tree (wind chime) sample, recordings of me tapping the table, and my clarinet. This section was inspired by the lullaby from the 2006 film Pan’s Labyrinth. The choir samples sounded awful when looped. At every end I could hear a sudden discontinuity in the playback, since the tracks were sustained notes that did not maintain a consistent wave flatness (visually). To combat this, I used a reverb plugin. I cranked up the damping, room size, and wet mix. This helped mask the looped tracks, but not the discontinuity to the chord changes since I split (slash key) the sample at those points. 

The interface displaying the bell tree sample and the three choir tracks. Discontinuities in looping appear as white lines while discontinuities in chord/pitch changes are shown as spaces.

After this I placed some rests with the wildlife ambience sample. I broke that near-silence with a bass sample. This is where I wanted to create the buildup that we heard in the songs we listened to in class. I automated a rise in volume (crescendo) and cued in more samples in subsequent measures to create this rise. The last sample was a C6 Cognosphere (I guess one could call it a suspenseful glissando) to finish off the buildup.

I followed with a fully established groove sample (the previous percussion samples were individual kicks and hi-hats). For all my percussion samples, I increased the gain since they were overpowered by the lead synths. More Aalto Sequences samples were filling in the leads, with plenty more plugins. 

Saturday was my brightest and darkest day making this song. While I was able to make plentiful loads of progress and experiment with different plugins that I would use, I accidentally deleted the entire beginning chunk of the 80s synth tracks. I wanted to undo this, but I realized I deleted it long before as the undo cues were only changing what I was working on in the later sections. This tells me that though it may take more time, it is important to review the song every few minutes. I am hoping there is a way I can track revision history, but I am not sure. This is why only one 80s sample is heard in the final product’s intro.

On Sunday I was able to finish the song off. I recorded the last of my clarinet and table-tapping samples. I used the 80s synth and wildlife ambience samples to create both the intro and outro of the song.

When reviewing the song, I discovered that the gauge for the decibels was reaching the max of +5db. I knew this was a red flag, but I decided to document it as something to improve on as opposed to fixing it. 

Recording:

Recordings were of me tapping a table and playing my clarinet. I used the 80s synth samples to create chord progressions and then I improvised along with them (yay jazz). I first recorded my playing on my iPhone for convenience. After reviewing my iPhone recordings I played the improv melodies again and recorded with the Blue Snowball directly into Waveform 11.

Initially the recorded clips were very faint. I was using a well-praised microphone, and it was facing directly at my sources so I did not understand what on earth was happening. I then manually raised the mic’s gain on Waveform 11, which improved the sound quality of my recordings. This step also allowed the mic to pick up the acoustics of my suite, but it also picked up some background noise that can be heard if listening very carefully. 

An unseen risk I forgot to account for was the phase cancellation that could have been caused by the sound bouncing off the floor. Some clarinet audio clips fade in and out, so that unfortunately affected my recording. I tried propping up my mic with a calculator, but I am unsure whether this made a big difference. It is possible that the microphone does not have perfect spectral flatness but it does seem satisfactory based on the Blue Snowball data sheet.

Microphone is on the floor below the bell of my clarinet. The mic is propped up with a calculator. 

Automations:

The pitch shifter plugin was my “knight in shining armor” for this project. Since I was not to use MIDI, the pitch shifter allowed me to practice working with not only pitch changing, but also automation in general. Pitch shifting is heard in the beginning. You can hear that ambience rise and fall to keep up with chord changes. The pitch is measured in cents and semitones. I used semitones as my measurement unit. One semitone is equivalent to one half step on a keyboard. While this did give me experience working with automation, I found it to be very tedious. Trying to modify the pitch analog at the bottom of the interface was difficult. Whenever I typed my desired semitone, it would automatically revert to the original pitch  if I did not completely delete text beforehand. I had spent too long on the intro and I decided that for the subsequent chord changes, I would just split my clips and change the pitches without using the shifter plugin. 

Automation steps for the pitch shifter plugin

I did do some panning and automated panning to add more depth. The 80s synth tracks were panned at different levels left and right to add some space. The first notes heard by the clarinet were intended to recreate a “call/answer”. The clarinet is heard on one side and then answers from the other. This is also heard at the brief choir bridge.

Plugins I like (love actually):

I had so much fun fiddling with the phaser plugin that I spent a good 30 minutes playing with it while looping a single audio clip. For lack of better description, the phaser creates the impression that I am covering and uncovering my ears as I listen to the sample. Speed, depth and feedback were my two controls. Raising the feedback, I could hear the sound of a laser whistling in my ears. One of the greatest plugins ever brought to concrete music indeed! Phaser was another way to create a sense of space besides stereo panning and reverb. Sadly I could not figure out how to automate the plugin. If anybody does know, please share! I did use the Phaser plugin to add variation to repetitive melodies. The laser-like plugin also added some spice to the song in my opinion.

Another interesting plugin I found was the “HiPass Filter”. We learned that the low-pass filter filters the frequencies above the selected cutoff. I am guessing a high-pass filter does the opposite! This filter allows frequencies ABOVE the cutoff to pass through while ones below do not. Fiddling with the filter the spectral graph is a mirror of the low-pass filter. The roll-off is at the left of the graph instead of the right. When I drag the cutoff towards the right the sound gets narrower, and the higher frequencies are sharpened. This was used for a lead synth sample towards the end of the song. 

Lowpass Filter (Top) and Highpass Filter (Bottom) plugins

Chorus was another favorite of mine. It added a sense of space to the samples. I could hear the sound resonating at both ears instead of being narrowly focused at one position in the stereo. I have been told that this plugin copies the original sound and plays them back slightly delayed from the original sample.

Of course, I also used reverb for my clarinet. Raising the gain to the max for audio input definitely helped retain the fullness of my clarinet’s sound, but the reverb created a profound sense of space, like a concert hall, that I could not say no to. 

The last plugin that I found worth noting, was the AUDistortion. On my first blog I discussed the genius of Koji Kondo’s 8-bit music. I felt that I could add some “8-bit” spice to a few audio clips. This plugin literally “crushes the bits” of my audio, creating the retro sound heard in the arcades. Upon looking at the “virtual sound board” (plugin interface), I found that manipulating the decimation makes the waves more square. For repetitive melodies, I used the distortion plugin to add some variation in case listeners get bored. 

Top shows a wave with slight distortion. After raising the decimation (first knob in the third row from the top), the waves become jagged.

Project Write-up:

In my experiments, I used two types of ambient samples. One is wildlife, and the other is the 80s synth samples. I placed one to two of these samples at different semitone levels to represent intervals. Almost all of my samples are looped. The most useful plugin in these experiments was the pitch-shifter. It helped me create the different key and chord changes.

Even though I do not enjoy borrowing samples, they helped me explore techniques for audio plugins.

My clarinet performs different notes with different resonances. Below the register, I can hear the air can be heard through my clarinet. Going above the register for my clarinet, the sound fills up the entire room. This is an interesting natural phenomenon, but it gives the impression that I spliced multiple takes in what is a single clip. It appears my suite size is such that created the resonance beyond the clarinet register.

One challenge I was not able to overcome was crashing. Whenever I would delete a larger amount of clips, the interface would just quit. The software fortunately saves my progress, but the crashing was very annoying to deal with. A second challenge arose with live input monitoring. I could sense a slight delay from the live playback when recording clarinet and table-tapping. The instructions issued a warning about this lag, so I sadly disabled the LIM feature, though it would help me hear my clarinet. However, there was still a slight delay in my clarinet tracks. This may either be from some delay in processing. Petersen mentioned that the USB mic has to handle all sound-processing responsibilities. These would ideally be handled by three to four machines. However, I am quite sure that the problem arose from my hesitance to play. I was a little worried about disturbing my floor mates. My best guess is the latter, but I will not rule out the former just yet. On the bright side, my floor mates said that they do not mind my playing, so I just need to get used to this way of practicing. My residential college PHC is figuring out logistics of reopening the Silliman practice rooms. I am keeping my fingers crossed that those rooms will be an option.

The last two problems have to do with exporting. I normalized the audio to maintain a consistent volume of my piece, but the file came out surprisingly much more faint than when played back in Waveform 11. I recall that normalizing audio is supposed to bring all samples to the peak amplitude. Perhaps that peak was not measured in decibels? The final problem was the discontinuities during looping and cut clips. This also resulted in some audio hiccups in the exported product. Hopefully there are ways to ease the transition between loops and two or more samples. 

In my experience editing videos using DaVinci Resolve, I learned that audio quality is perhaps more important than video quality. I guess some video editing skills are transferrable to audio software, but definitely far from most. I merely skimmed the surface with the sound editing in my videos. I think I underestimated the differences and weightiness that Waveform 11 had. I could manipulate the sound of my clarinet to create an “undiscovered” instrument. I had very little experience elaborately recording audio for my clarinet. I had access to different plugins that I could play with to please (or annoy) my ears. The only thing that remains the same was the annoying crashing. Even with this song, I have merely skimmed the surface of the software. I am excited to try out MIDI features once we get there.

Below is the final product. I am so excited to listen to everyone else’s!

 

iPhone 11 Recording Tests

Research on Tech Specifications

My smartphone model is an iPhone 11. On researching the iPhone 11’s microphone specifications, I had to go onto several forums to gather pieces of information in addition to the official Apple site. Diagrams on the Apple site indicate three locations for the microphones. One is at the top-front of the smartphone, used for recording sound with video in real time. The other is at the earpiece, which is used for noise cancellation when making phone calls. The last two, (most important for the research) are at the bottom of the phone next to the charging port.

Figure 1. Layout of the iPhone 11 (from Apple)

Other specifications found on the Apple site include the following:

  • Audio formats supported: AAC‑LC, HE‑AAC, HE‑AAC v2, Protected AAC, MP3, Linear PCM, Apple Lossless, FLAC, Dolby Digital (AC‑3), Dolby Digital Plus (E‑AC‑3), Dolby Atmos, and Audible (formats 2, 3, 4, Audible Enhanced Audio, AAX, and AAX+)
  • Spatial audio playback
  • User‑configurable maximum volume limit

The iPhone’s standard Voice Memos is the default app for audio recording. It does not use wav files, since high quality audio is deemed unnecessary for ordinary users. Through arduous searches on the web and the App Store, I finally found an app called TwistWave Recorder, which allows for recording in wav. This app also allows the option to disable iOS control gain, which I did. 

Information on sampling rate is not available on the Apple site. When making a new file with TW Recorder, I had the option to choose my sampling rate. It was through trial and error that I found that my phone’s sampling rate is 48 kHz. I also learned this through several forums, and this app confirms it. Anything above 48 kHz and the app would alert me with a warning. 

Figure 2. TW Recorder’s sampling rate options, and warning alert if any option above 48 kHz was selected

Since about 20 khz is the max for human hearing, anything drastically higher than 40 for a sampling rate obviously would not make sense since our ears would not be able to detect it. For this reason, many iPhones have 48 kHz as the max sampling rate.

Information about bit depth is not available either. TW recorder has an option for 32-bit temporary files, with better quality but larger size (twice as much). According to the app, the default is 16-bit. I am still not certain that the bit-depth is 16. Some indicate that newer iPhones has 24-bit depth capability, but the 32-bit depth FFT graphs showed little effect in my results, so I decided to stick with 16-bit depth.

The phone also has both stereo and mono options, but I kept the option set as mono. The sweep and noise from Audacity were also generated at mono setting.

Recording Procedure

I had two variables when recording the wave and white noise with my phone. The first was the mic distance from the phone. I placed the bottom mics on the laptop speaker for the first recording, then I flipped the phone upside down for a greater distance (the top mic is not used for recording), then held the phone on top of the open laptop screen (perpendicular to the webcam) for the maximum distance. The second variable was the bit depths. Since I activated the 32-bit temporary file option, I was able to export my audio files in 32, 16 and 8 bits. For sine wave recordings I had 9 files and for white noise recordings I also had 9 files.

Constants for the recordings were the speaker used (laptop), the sampling rate (kept at 48 kHz), and the mono option as said before. Audacity was used to generate the sine sweep, white noise, and analyze my recordings.

Results and Analysis

Figure 3. Input sine sweep (top) compared with 16-bit (center) and 8-bit (bottom) recorded at the laptop speaker

The spectral flatness for these sine sweep graphs are not even close compared to the input graph in Figure 3. While this drastic difference could indicate that the microphones of the iphone may not be the best, I speculate that this problem is a result of the imperfection in the laptop speakers and environmental interference. I was in my dorm bedroom for my recordings. The room is far from being a good recording studio. There are windows, cracks in the doorframe leading to the common room, all those things come into play. 

Figure 4. White noise input from the computer (top) recorded next to the speaker at 16-bit (center) and 8-bit (bottom)

The spectral shape for these white noise graphs are relatively the same in the lower frequency range (Figure 4). The sound was much more “crunched” in playback for 8-bit. However, in both sine sweep and white noise cases, bit depth does not matter too much. 

Figure 5. Sine sweep recorded at 16-bit at distances of one phone length away (top) and one arm’s length (bottom) away from the speaker

Figure 6. White noise recorded at 16-bit at distances of one phone length away (top) and one arm’s length (bottom) away from the speaker

Distance seems to be more important than bit depth in determining recording quality.The further the distance from the speaker, the more likely the recording would be interfered by environmental noise. In both Figures 5 and 6, there is more distortion at lower frequencies. 

In summary, at lower frequencies from 50 to 1000 Hz, mainly when the mic is at the speaker, the FFT plots do not show severe fluctuations, indicating reasonable spectral flatness. However, all the plots fluctuate aggressively compared to the computer input when going beyond 1 kHz. This odd, “shaped” response may not be a result of the iPhone 11’s mic sensitivity, however.

Areas of Improvement

Given the environmental interference, the spectral graphs of my recordings are not a true reflection of my iPhone 11’s capabilities. Ideally, this test should be done in a soundproof room. There are also issues with distortion of the laptop speakers. That distortion would be carried into the iPhone and unfairly create the impression that the iPhone has poor performance. Even if Automatic Gain Control can be disabled, the mic remains especially sensitive to the outside environment. The likelihood of background noise interfering with my recordings is very high. If I were to do this experiment again, I would search around for a place that is much more soundproof. 

Information For iPhone Specs

Kunesh, Andrew. “My IPhone Microphone Is Not Working! Here’s The Fix.” Payette Forward, 27 Nov. 2017, www.payetteforward.com/iphone-microphone-not-working-heres-fix/.

“IPhone 11 – Technical Specifications.” Apple, www.apple.com/iphone-11/specs/.

“Smartphone Recording Tips for Recording Birds in the Field.” Help Center, 9 Apr. 2020, support.ebird.org/en/support/solutions/articles/48001064305-smartphone-recording-tips.

What Audio Quality Is Available on Newer IPhones? 2017, apple.stackexchange.com/questions/276905/what-audio-quality-is-available-on-newer-iphones.

 

 

Computer Music in the Common Household: Koji Kondo

Do you remember the 8-bit tune of Super Mario Bros.? How could a few geometric waves chant such a unique melody that we still hum to ourselves after 35 years? We can look at the composer of the theme, Koji Kondo, as a pioneer in not only video game music, but also the art of computer music.

Koji Kondo, Composer of several Nintendo games, notably Mario and Zelda (Image source here)

Before the (now-retro) video games began to lurk the grounds in the late 70s to early 80s, computer music was indeed an emerging field. However, this new music form was not much entertainment as it was a research field. One could see figures like John Larry Kelly, Jr. and Max Matthews as the Einsteins of computer music. In 1924, the famous theremin was invented. This instrument is known for delivering the sonorous pitches that could be changed by the movement of your hand. Composers often used it with orchestras and concert bands. The instrument was featured notably in the sci-fi film The Day The Earth Stood Still by Bernard Hermann in 1951. In the 1970s, artists like Herbie Hancock and Stevie Wonder incorporated other rudimentary electronic instruments into their songs, such as Chameleon and Superstition, respectively. However, at least in the 70s, this tech was usually performed alongside real instruments, or with singing. Other famous musicians used the innovative electronic instruments only for backing tracks. So how did pure computer music migrate from less prominent use in the 60s to the forefront of today’s entertainment?

Video game music contributed to the rise of computer music popularity. In the late 1970s to the 1980s, the video game industry took shape. Games of Nintendo were inspired by someone fiddling with a calculator at a train station. From this basis for their games, how were the “Nintenders” going to create visuals and sounds with an 8-bit microprocessor, limited picture-processing, and two kilobytes of RAM?

Koji Kondo in his office during the early days of Nintendo (Image source here)

Along came Koji Kondo. He had experience playing on a CS-30 synthesizer, and these skills were transferable into the realm of Nintendo. He explains in a 2001 interview, “Back then, there was no MIDI or other protocol that could directly convert your keyboard playing to data. Everything was composed directly on the computer, entering notes manually.” Kondo follows that he played melodies at home on an Electone organ and entered his arrangements on a computer in the Nintendo quarters. He introduced his Overworld Theme of Super Mario Bros. in 1985.

For Mario, Kondo initially focused on the pixelated but vibrant scenery of the levels. However, once he saw the game being played in real time, the fledgling composer felt the music should focus on the pacing of the game itself. He wanted it to match with our protagonist plumber’s actions, like his running, jumping, and his interactions with enemies and items. Many times Kondo wrote a piece, many times he would scrap it. He eventually pumped out six themes, including the famous signature theme, the waltz of the water levels, and the ominous Jaws-like jingle of the castle levels.

His next game was The Legend Of Zelda, released in 1986. For this game, Kondo originally wanted to use the classical piece Bolero by Maurice Ravel as the main theme. However, he had to scrap that idea since the piece wasn’t in the public domain. Instead, he created the majestic, otherworldly theme for the game that would spawn another popular fantasy series. (He did all this in under a day!) Modern composer and music theorist Andrew Schartmann praised the score of Zelda for how it blends elements of “Gregorian chant, rustic folk, and Hollywood fantasy.”

The Legend of Zelda theme illustrated with geometric waves

Kondo says he aims to emphasize “the experience for the player”, and this stays true in his work. The games, along with their 8-bit themes, made their way into common households. Players of the 1980s were exposed to a lively, virtual world. The digital display of the minimalist setting on the Famicom, the unique tune produced from square waves (“richer in harmonic content”, as mentioned by Kondo), all immersing the player in the game. Kondo is not hailed as a direct influence on computer music as a research field, but thanks to his iconic contributions, computer music in games has become part of everyday life for many. 

The scores of Mario and Zelda indicate how versatile Kondo is with a keyboard the size of a hand. To us, such a little keyboard may look like an impediment. Perhaps by not focusing on instruments themselves, Kondo was able to channel energy to melody. Perhaps this is a new form of musical expression. Was the lack of technology for Nintendo really a restriction, or was it a blessing in disguise? Should we work with limitations without knowing how limitations may help us? 

Mario theme adapted into orchestral arrangement

Information Sources:

Electronic Gaming Monthly. “Interview with Koji Kondo (Electronic Gaming Monthly – December 2005).” Square Enix Music Online, Dec. 2005, www.squareenixmusic.com/composers/kondo/dec05interview.shtml.

Hsu, Hua, et al. “How Video Games Changed Popular Music.” The New Yorker, 30 June 2015, www.newyorker.com/culture/cultural-comment/how-video-games-changed-popular-music.

Jobst, Merlin. It’s the Music That Makes The Legend of Zelda So Extraordinary, 11 Nov. 2015, 2:10pm, www.vice.com/en_us/article/jmakxy/its-music-that-makes-nintendos-the-legend-of-zelda-series-so-extraordinary-330.

Kohler, Chris. “VGL: Koji Kondo Interview.” Wired, Conde Nast, 5 June 2017, www.wired.com/2007/03/vgl-koji-kondo-/.

“Koji Kondo – 2001 Composer Interview.” Shmuplations.com, Game Maestro Vol. 3, 1 May 2001, shmuplations.com/kojikondo/.

Otero, Jose. “A Music Trivia Tour with Nintendo’s Koji Kondo.” IGN, IGN, 24 Oct. 2017, www.ign.com/articles/2014/12/10/a-music-trivia-tour-with-nintendos-koji-kondo.