Piano Capture, MIDI Capture, Video, Muse Sounds... and more!

Piano Capture

Piano Capture is our term for our audio recognition and transcription technology, designed to transcribe real, acoustic piano recordings. We've been working on this technology for a while, and it opens up a whole new way of putting your musical ideas into StaffPad.

This technology uses your device's microphones and on-device machine learning to listen to your piano performance and transcribe it into notation. You can do this in near real-time by enabling record on any instrument staff, pressing record, and then simply playing your (real) piano. As you may expect, this works best in a quiet room, and record with the metronome to ensure the transcription is accurate. On that tip, if you long-press the metronome icon, you'll get some options, including a new click sound - the Digital Click. If you're overdubbing parts (that is, recording on top of existing music), then make sure you connect headphones - otherwise the sound from the speakers will interfere with the transcription!

When recording onto a staff, you'll see that a second or so later, some notes pop on to the staff. The duration of these notes will refine as you continue to record, and may continue to adjust themselves after you've pressed stop. The AI listens as you go, but then has another pass at the full recording once you've pressed stop. On older devices, you may find that the staff lines pulse red for a little while after recording; this is to indicate the AI is still working.

If you'd rather not record in real-time, there is an alternative. You can record your piano performance onto a piano audio staff, tempo map it with the tempo staff, and then enable the Show Notation toggle, found in the Staff Control.

This record-it-first approach lets you refine your performance, either by editing or doing multiple takes, before converting it into notation. Alternatively, you can import an audio file of a piano performance, ensure the tempo Iines up, then toggle the Show Notation view.

Keep in mind the Show Notation toggle lets you flip the audio staff between a waveform and notation view of the audio clip. In reality, it's still an audio staff. You'll hear the audio during playback, and if you edit the audio clips in the waveform view,  the notation view will update accordingly.

This is important: if you wish to edit the notation itself, you need to copy and paste it onto an instrument staff first.

Piano Capture is a magical technology, and we are hopeful to release more types of capture technology in the future to go beyond the piano.

MIDI Capture

Piano Capture is amazing if you have access to a real piano. If you don't, we still have you covered. You can connect a USB or Bluetooth MIDI keyboard and record your performance into instrument staffs.

Be sure to enable your MIDI device in the new MIDI Devices settings page, found in the app settings.

Enabling a MIDI device in the app settings automatically turns off the Piano Capture, i.e., audio transcription features. Therefore, remember to deactivate any MIDI devices in the MIDI Devices panel of the app settings if you want to use Piano Capture. Otherwise, the app will continue listening for MIDI messages rather than audio signals.

Once you've set up your MIDI device, you can record enable any instrument staff from the staff control. The staff will turn red, and you can immediately start playing notes on your MIDI keyboard to hear a preview sound. However, remember that this is genuinely a preview sound, designed to be responsive and useful just for input purposes, and may not reflect the full playback sound.

When you're ready, tap the main record button and start playing! Again, use a metronome for best results.

We've optimized certain elements to make drum input a bit easier — you can play the included drum kit sounds with a "GM" mapping style, and it will automatically assign the notes to the appropriate note-head styles, voice layers, etc.

If you have keyboard skills, MIDI Capture can be a fantastic way to experiment and try ideas quickly.

Video Staff

The Video Staff is a new staff type, found within the Utility category in the Staffs control.

Tapping the staff name brings up the staff control, where you can import a video file, adjust the volume, pan, reverb of the video's audio track, or even add effects to it.

Once you've imported a video, a filmstrip of thumbnails is rendered across the score. You can double-tap this filmstrip to hide or show the video preview window. You can also double-tap on the video window itself to hide it. On iOS, pinch-to-zoom to resize the video preview. On Windows, drag the scale handle in the bottom right corner to resize the video preview.

Imported video files appear as video clips. You can only have one video staff in a score, but you can import multiple video clips onto the same video staff. Just like with audio clips, you can drag the video around using the Pencil or Pen. You can also trim the start and end times by dragging the start and end handles on the video clip with the Pencil or Pen - same as with audio clips.

The Video Staff supports all major video formats. You can even export the video with your music included from the Share screen.

The Video Staff works best in conjunction with the timecode staff and tempo mapping features. It's easy to work out exact cue points, add markers, and make fine adjustments. Together, they provide a formidable set of tools for scoring to picture.

Muse Sounds

We're proud to add the incredible Muse Sounds to the StaffPad Store, available completely free of charge.

The core ethos and mission of Muse Group is to amplify musical talent. As part of this mission, we've been diligently working with our colleagues and friends at MuseScore to bring excellent playback to as many people as possible. The result is Muse Sounds, a new imprint and a collection of playback instruments covering orchestra and choir that can add incredible depth and realism to your score playback. You'll find the Muse Sounds packs available for download in the StaffPad Store; they're free - and I highly recommend that everyone checks them out. 

If you're a MuseScore user and would like to use them for use within MuseScore on Desktop, be sure to grab the Muse Hub and download them from there.

Harmoniser

StaffPad is an incredible collection of unique technologies, and sometimes we just can't resist... the new Harmoniser (or Harmonizer, perhaps!) uses information it gathers from the chord staff, or the harmonic content of the score, and applies a vocoder style effect to your vocal audio. You can find the new effect in the Effects section of a vocal audio staff. It's a lot of fun, and definitely a creative and interesting effect! 

All the little things

Alongside the major features, there are - as usual - a host of smaller improvements. We've improved the MIDI import engine, so that even unquantised, messy MIDI files give better results. On Windows, the responsiveness of the Welcome Page has been significantly improved. We've heard the feedback about the ordering of instruments in the Add Staffs control, and re-ordered the Staff Category column alphabetically, rather than in score order, so that e.g staffs from specific Sound Partners are easier to find. The new metronome sound, the digital click, is designed to work with our new "Metronome Remover" mixer effect. The Metronome Remover effect suppresses the metronome signal in audio recordings, effectively allowing you to record without headphones whilst also using the click.There's a new Dynamic Mastering slider, which replaces the compression slider in earlier versions. This slider acts as an automatic mastering engineer, applying loudness, spectral enhancement, compression, expansion, limiting, and a few other studio tricks to bring loudness and sparkle to your composition's playback.

As usual, there are also many fixes and enhancements included in this update.

Whew! That's a big update.Until next time, I hope you like the new features, and continue to enjoy composing music with StaffPad!

Best wishes,

DWH & The StaffPad Team