I have just managed to project several holograms of Leo as the refugee into my very messy studio/office along with the binaural recording of the actors’ narrative (the same as in the video). It is best listened to with headphones.
I have spent several days in the Advanced Digital Projects space at Central St Martins trying to do this in Unity (with a lot of help). But managed do produce this on my own this afternoon at home. So am very pleased with the outcome. The holograms can be positioned anywhere in the exhibition space and seen through the HoloLens.
I have now finished making the life-size 3D sculptures of Theresa, the wife of the Optician of Lampedusa, and Leo the refugee rescued from the sea.
I have also managed to download the BBC recording of the news broadcast by Emma-Jane Kirby, the author of the Optician of Lampedusa, convert it to MP3 format, and load it into my Fiil Diva Pro headset. This headset has its own inbuilt MP3 player and does not need to be connected to a computer or mobile device either wired or wirelessly in order to play the track. The track continuously loops, and if the headset’s battery is charged overnight the track will play all day. It’s a tricky set up, as at the start of each day the headset needs to be connected to an App on my iPhone in order to initiate play. It can then operate independently. Another feature is that whenever the headset is taken off play will automatically stop (and save battery). However, when the headset is put on again the track starts from where it left off. This feature is obviously designed for a single user. I would prefer it if it started playing the track from the beginning as most likely it will be for a different user. I have contacted Fiil in China to see whether these issues can be overcome. If not I will have to live with them. Currently it works well with these limitations.
I am now ready to film my narrative video of the two actors who posed for the sculptures. Suzy, my wife is playing Theresa, and will speak with an English accent, and Leo is playing the refugee, who will speak in English with an African accent (his family are from Jamaica). I am indebted to film maker, actor and scriptwriter Michael Mueller for the following script. The plan is that it will be five minutes long or at least no longer than seven minutes. I am sure a listener will not stay with it for longer.
We are rehearsing at the green screen studio that Alasdair and I set up at the bottom of his garden. I have had to abandon the idea of recording with my recently acquired 3D 360 degree Vuze camera, as it is currently not possible to show Stereoscopic video in any Augmented Reality device such as the HoloLens. I will now shoot using two camera set ups, one with a binaural head audio recording to my Canon 80D DSLR, and the other with sound using a camera mounted mixer, two Lavalier clip on microphones, with recording to Alasdair’s Canon 60D DSLR. We may need another stand alone microphone but will determine that during rehearsal.
Binaural Head with camera mount
As I cannot play my video in 3D in the HoloLens, only in 2D, I have decided to enhance the listening experience to something equally immersive, using a binaural head. Here is an example of binaural recording called the Virtual Haircut. It is essential that you listen to this using headphones.
If you want to understand more about binaural and other sound recording methods for VR, here are some good places to start.
I will produce two binaural recordings one with and one without video. The sound only recording will be for the main show exhibit, and will be heard in a second Fiil headset, set up as previously described.
The video recording will be seen during timetabled performances with the Microsoft HoloLens Mixed Reality headset, similar to that illustrated below except that the bacjkground will be the real live setting of my MA show main exhibit, and both HoloLens video screens will be of the actors against a transparent background. I am planning to add several holograms of the head of a refugee, placed on the floor around the exhibition space, representing the plight of the refugees in the sea. I am meeting with Sion Fletcher, at CSM 4D next week to determine whether this is technically possible.
I have checked out possible exhibition spaces in Wilson Road, and the projectors in use there during the current BA show.
Ground Floor Room at Wilson Road
There are possibilities for hanging an acrylic screen in both places, using the existing structure and possibly the length of wood resting on the top of two adjoining walls in our B18 Studio. My issue now is that I have a rear screen projector film for my A0 size acrylic sheet, so have to work out where I can work with that, or whether to stick with a straightforward projection on to a wall using our studio short throw projector. The existing projectors in the room and my own are not short throw and need the projector lens to be 2.25m to 2.6m from my intended rear projected acrylic screen. My ideal is to place the sculptures either side of the acrylic screen to represent the void/distance between the rescuers and the refugees in the sea. I will have to assemble the screen with its film and try it out to see if this works and what setting works best. Theresa would be on a white plinth representing the boat, and the refugee would be lower on a clear acrylic plinth representing the sea. The sculptures will face each other.
And finally…. I have contributed to the Zine being prepared as advertising for our MA show by Leonie. Here is my effort.
I am getting closer to understanding what I can do for my MA show. It still needs to be checked for feasibility but here’s the plan so far.
There will be a physical exhibit with video, sculptures, and sound through headphones. Additionally I propose a few separate performances in a different space (probably the Lecture Theatre) using the HoloLens and a single smaller sculpture.
The reasons for separating the HoloLens experience from the main exhibit, is that it necessarily has to be a performance as it cannot be left unattended, and participants need to be instructed and supervised during its use. It also needs entirely different lighting conditions.
The main exhibit will be two spotlit life size sculptures on plinths looking at each other on either side of a transparent projection of the sea and screeching seagulls.Theresa, will be on a taller white plinth, looking down at the refugee on a smaller transparent plinth. The actors’ narratives will be heard through headphones. The projector will be hung from the ceiling, as will the transparent projection screen. The environment needs to be fairly dark, so the ideal location will be in the basement.
I already have an A0 size acrylic sheet and will be coating this with self adhesive rear projection film. I need to experiment with this in the proposed exhibition space with various projectors including my own to see what works best, and to determine screen and projector placement.
The performance will use the HoloLens to show videos of the actors against a transparent background, with the same audio narrative as the main exhibit, overlaid with the sound of the sea and seagulls. Holograms of the refugee’s head will be scattered around the space representing heads in the sea. A smaller physical sculpture of Theresa, the optician’s wife will be on a white plinth.
To accomplish what I have described the film of the two actors together must be mid-shot against a black screen with clothes mics in a recording studio. Hopefully this can be done at either the film studio at CSM or Wimbledon. Otherwise it will be offsite in a private studio in Camberwell (not UAL). The video, sound recordings and holograms will be input to and organised in Unity and then input to the Hololens in Microsoft Visual Studio. Video will be 2D, and sound will be spatial.
I need to measure the room/exhibition spaces showing where the exhibit items are positioned. This is necessary to enable spatial sound to be correctly positioned in Unity. As there will be two exhibition spaces there will need to be two set ups within the Unity project.
Headphones used in the main exhibit have an inbuilt audio recorder. So recordings can be heard wire free, without wi-fi or bluetooth, and each set of headphones can operate independently.
I am getting closer to understanding how best to present my exhibit based on ‘The Optician of Lampedusa’:
Main Exhibition piece – ground floor room Wilson Road with blinds down
Stock film and audio to provide a background of seagulls diving and screeching in open sea, possibly calming to no seagulls, and a short clip of refugees in the water. This is in effect three videos and one audio clip edited together, looping continuously. I have yet to decide how this film will be projected, but hopefully using the existing ceiling projector or the fitting it uses with my own projector. The audio will be coming from the whole of the wall on which the film is projected using my Feonic invisible speaker device attached to the wall and wired to my digital media player and amplifier.
Alternatively, I can blue-tooth from my projector to my sound bar. I have to consider how these would be hidden from view, and how I can easily adjust the sound level. If I use my projector, I have a remote, so in that case, problem solved. I also have to see how the projection maps to the wall. I would like to project to where the wall meets the floor, representing the sea, and possibly partially overlaying on the floor itself. If the latter I could cover the floor with the grey lino I lent Leonie for her installation last year. I may need to paint the wall light grey to match.
Two life sized sculptures on transparent acrylic plinths. One of Theresa, the optician’s wife and the other of Leo as a refugee. Theresa would look down towards the refugee, representing her in the boat and him in the sea. I need to consider lighting, possibly diffused spots from above, or from below inside the plinths. If possible I will 3D print the head section of Leo’s bust of the refugee in PLA (again – it was too small last time in spite of taking 7 days to print!) with runners and riders attached, so that this section can be remade in unfinished bronze, and attached to the rest of his sculpture in white plaster.
The actors narrating their experience. Whilst the sound of the sea can be heard in the exhibition room (hopefully, if others sharing the room do not mind), I will provide two unattached headphones for the audience to listen to this approximately 5 minute track. My FiiO Bravo headphones store the tracks in the headset itself and thus do not need to be attached or linked by blue-tooth or wi-fi to an audio player. They operate independently, and will need to be recharged every night.
There will be a ‘performance’ for a couple of hours each day using the HoloLens. The actors’ performance will have previously been video recorded with spatial sound against a green screen using my Vuze 3D 360 degree camera, edited in 2D (sadly, the HoloLens uses a different holographic technology so cannot view Stereoscopic 3D). A viewer using the HoloLens will see the background video and sculptures in the real world view of the room, and two green screened actors performing in the HoloLens headset, while listening to the audio using an attached B&W P-7 set of headphones. The audio in this case will be 3D sound which will be precisely located where the actors performed, with seagulls flying over their heads and the sea lapping around them. A TV monitor will show the rest of the audience what can be seen in the HoloLens.
Performance in the Lecture Theatre
If possible there will be additional performances whenever the Lecture Theatre in Wilson Road is free and for about 2 hours a day. This will demonstrate the use of the HoloLens for digital art ‘conservation’, which was the theme of my MA dissertation. This will be the same performance as shown in my earlier blog for my wife’s MA show, using same sculptures as the main exhibition piece, but smaller and alongside their holographic versions.
I hope to also provide the audience with the opportunity of viewing my main exhibition entirely in Virtual Reality using the HTC Vive. When filming the actors with the Vuze camera I will also undertake a second edit in Stereoscopic 3D which can be viewed stereoscopically in Virtual Reality. I aim to also include the 3D images of the sculptures, and the background video in this Virtual Reality experience. We will see if I have enough time to do this.
Well that’s it for now. Must get back to completing the life size bust of Theresa, which should be finished today. Hoorah.
Happy Easter – a welcome couple of days off with my family.
This time it’s my wife Suzy’s MA show entitled ‘Curating the Future: Preserving the Past’ at the Museum of Futures, London. I was the contributing artist and the work was curated by Suzy to demonstrate what could be done when Museum exhibits are away on loan, or have been moved to the Museum’s archive or for conservation. The object can continue to be seen holographically.
Here is a picture of Suzy alongside her holographic image entitled ‘The Scream 2030’.
Here is a slideshow I created in Facebook, showing progress making the sculpture both 20cm tall showing and how it was used in Suzy’s MA show, and life size which I will use in mine.
Click the ‘Watch on Facebook’ link in the image above to view the slideshow.