This week at (paid) work I have been mostly grappling with what sounded like a simple task - displaying a 3d image and recording the reaction time for a keypress - that turned out to be both more complex and simpler than at first it seemed.

The usual sort of Psychology Experiment application - display a stimulus, record a response, repeat until all stimulli displayed - with the usual sort of conditions - randomised (or pseudo randomised) order of stimulii presentation and arranged in blocks of trials.

The quirk this time was that the stimulii are 3D Stereo still images recorded with a Fuji camera and the experiment consists of two blocks of trials with the same stimulii, once in 3D and once in 2D. Images to be displayed with a 2 second black screen between them. So far so simple. There was a choice of two possible 3D capable displays - a Samsung 3D HDTV which has its own shutter glasses and built in IR transmitter to synchroinise them, or an LG Flatron W2363D which is 120Hz capable so can display alternate frame left-right images using an external USB driven synchronising IR transmitter all under the control of an NVIDIA driver.

The original proposal by the researcher had been to use the NVIDIA slide show app to display the 3D images from one computer and then have a gubbins box that we would build to detect the image appearing on the screen and fire a trigger to a second computer running an ePrime [spit] script to accept the keyboard response and measure the reaction time. He couldn't simply use ePrime [spit] because it didn't know how to display 3D images. Then run the experiment again usng 2D versions of the image saved by the camera and a different slide-show app for the 2D block.

At first we played around with the idea of building a gubbins box (containing a photo-diode or transistor, and opamp driving a relay) but ran into a number of problems - not least of which were the fact that the images didn't really have a clear white area consistently to give a good disctinction from black to white (although he did propose we edited all the 3D images to put a white patch in the bottom left corner) and the two screens being evaluated gave very different responses (the big TV was giving off a lot of heat (ie infra-red) which tended to confuse the sensor). Also the NVIDIA slide show app doesn't provide a facility to show a blank screen between slides , plus of course the whole setup was pretty Heath-Robinson.

So, we said, lets do it properly using our standard tools - the framework is bog-standard stuff, the only quirk is displaying the 3D, and by the way then we can get rid of ePrime [spit] from this particular researcher's armoury. (Some researchers like it, some prefer to get us to do a proper job for them - we're always in the market for weaning ePrime [spit] users away from their misguided ways)

First problem then is all these .MPO files that the camera is creating - WTF are they?

This turns out to be very simple - an MPO file (standins for multiple-picture-object) is just a wrapper around a number of JPEGs. For stereo use obviously there are two JPEGS in the wrapper, first the left eye image, then a four byte flag to indicate the next image, then the right eye image. In fact if you change the extension on a .MPO file to .JPG then anything that can handle JPEGs will see it as a single jpeg image of just the left eye view and ignore everything else in the file.

Needless to say the examples were all in C++ and I wanted to use VB for the project since it is our standard and I have most of the rest of the experiment framework already available in VB (most Psychology experiments are variations on a theme).

 Public Function OpenMPO(ByVal fname As String) As List(Of Image)
        Dim imgs As New List(Of Image)
        Try
            Dim tempBytes() As Byte = {0, 0, 0, 0}
            Using f As New FileStream(fname, FileMode.Open, FileAccess.Read)
                Array.Resize(tempBytes, f.Length)
                f.Read(tempBytes, 0, f.Length)
            End Using
            Dim imageOffsets As New List(Of Integer)
            Dim offset As Integer = 0
            Dim tempOffset = 0
            Dim keyBytes() As Byte = {&HFF, &HD8, &HFF, &HE1}
            Dim keyBytes2() As Byte = {&HFF, &HD8, &HFF, &HE0}
            While tempOffset <> -1
                tempOffset = SearchBytes(tempBytes, keyBytes, offset, tempBytes.Length)
                If (tempOffset = -1) Then
                    tempOffset = SearchBytes(tempBytes, keyBytes2, offset, tempBytes.Length)
                Else
                    offset = tempOffset
                    imageOffsets.Add(offset)
                    offset += 4
                End If

            End While
            For i = 0 To imageOffsets.Count - 1
                Dim length As Integer
                If i < imageOffsets.Count - 1 Then
                    length = imageOffsets(i + 1) - imageOffsets(i)
                Else
                    length = tempBytes.Length - imageOffsets(i)
                End If
                Dim stream As New MemoryStream(tempBytes, imageOffsets(i), length)
                imgs.Add(New Bitmap(stream))
            Next
        Catch ex As Exception
            MsgBox(ex.Message)
        End Try
        Return imgs
    End Function

OK, so far so easy. Next problem to actually display the images. Obvious first step google around for some sample code - nada. All the examples seem to be focused on creating 3D graphics from scratch on the fly using OpenGL or DirectX to manipulate the buffers on an NVIDIA Quadro graphics card.

Turns out that the machine designated for the experiment only has a near bottom of range GeForce card which although it can display 3D does not give access to the NVIDIA API like the Quadro cards - and no, there isn't a grand available to upgrade the card.

Plan B - use the 3D TV which has its own internal processor and will accept a squeezed left-right (or top-bottom) input with both images squeezed into the same image area. It'll then encode them into alternate frame shitter 3D itself. Easy peasey. The TV will also accept a 120Hz alternate frame 3D image and display that once we can sort out the graphics card issue.

So everything falls into place with one nagging worry - how much delay is there in the processing in the TV - and is it consistent? It looks like I'm going to have to make a gubbins box after all in order to test how long it takes from the application rendering the image to it appearing on the screen. Could become a nice little Arduino project.