AUGMENTED REALITY SIMULATION OF THE VIRTUOSO ASSISTANT (STUDIO 1, NOVARS)
Hands custom rigging in Blender by MANUSAMO&BZIKA STUDIO
Hand animation video demo
Hand animation ported onto unreal engine 4 (aerial view)
sample of virtuoso benchmarking exercises
multimodal capturing session: 2 X myo armband, fisherman violin contact mic, dpa mic, goPro videocam, Studio 1 novars research centre
benchmarking spectral tests by project assistant Chris rhodes
emg first analysis (principal contribution of each emg sensor) to position. analysis team at AMBS, by RICHARD ALLMENDIGER
Myo armband correction factor
raw data from EMG sensor
sample code for analysis
FFT additional analysis for non-emg captured data
violin rig ported onto ue4 for the oculus rift (early test)
maxmsp patch to read and store real-time emg data onto a spreadsheet (every 10 ms)
FPS view of the AR avatar
rigged model in ue4, (It reads the recorded data onto a spreadsheet, for playback and rewind purposes)
vr mock for prototype project report)
cameron, linda and chris in studio 1, novars (Summer 2017)
violinist ignacio lara with the myo armbands (third multimodal recording session)
data visualisation tests in unreal engine 4 using widgets and blueprints
another test of in-game data visualisation of raw data (not of the comparative analysis)
blueprint screenshot which reads data from .cvs spreadsheet file and maps onto scaled 3d cubes
resume of summer sessions (four multimodal captures at the novars research centre)
Virtuoso - a violin musical trainer in VR: This HSIF-funded project Virtuoso, is a collaborative effort between Manchester University academics from Music (NOVARS Research Centre), the Alliance Manchester Business School and EON Reality, a company specialised in Augmented Reality (AR) and Virtual Reality (VR) software for Education and Training.
Technical goals: We set a 2-month task to build a Prof of Concept (POC) by developing a multimodal capturing platform able to collect a range of real-time bio-related data, as well as other data, such as pitch data and live movement.
We wanted to see whether it was technically possible to spot performative differences between professional and an intermediate-level violin players, based on a set of prepared musical exercises. Also we wanted to recreate the player in 3D and VR to potentially connect extracted data with virtual behaviour and visualise those differences.
Feasibility study: another important goal was to run a feasibility study, including audiences marketing and research and business impact, in relation to Virtuoso’s potential market.
Implementation and context: Virtuoso’s first implementation is a biofeedback-based training system for violinists to understand how to reach the highest-levels of musical performance learning from a VR and potentially AR training system. Virtuoso’s context contemplates a musician learning a practice-led discipline at a high level, via technology enhancement, aiming to drive conclusions that, may apply to other disciplines.
Aims: Virtuoso aims to demonstrate ways to create increased value for learners using bio- feedback tools. More in particular, to see whether our system could provide valuable knowledge to understand what distinguishes a very good instrumental player from a real virtuoso performer. It also aims to complement musical instrument’s tutoring in the traditional one-to-one / one-to-few way and not to replace it, while cultivating a culture that supports change in its learning methods and attracts talent. In the long run, the system aims to maximise the potential of acquiring knowledge facilitated by syndication, using digital tools and decentralized data.
Virtuoso Team