System for acquisition and processing of the lightfield representation

Mucha system for acquisition and processing of the lightfield representation

Today, there is a great interest in fuller recording of the surrounding world than a flat 2D image. Many leading research centres and industrial companies are developing techniques of recording, processing and displaying of three-dimensional image. 3D cameras and 3D monitors have been successfully deployed on the market.

However, the ubiquitous 3D technology is based on the registration, processing and displaying of stereoscopic images. The stereoscopic pair allows to convey depth perception to the viewer, but only from one, very specific viewpoint, in which a pair of “digital eyes” of the camera were located. Stereoscopic technique is just a substitute for full registration of 3D information (3D light field) about the scene.

Companies are already developing more advanced technology for displaying 3D/holographic images, Companies such as Holografika, Ostendo or Dimenco [1, 2] developed and introduced to the market a new generation of ultra-realistic Super Multi View (SMV) 3D monitors, that allow observation of the scene like “through the window”. The complex system of lenticular lenses or mirrors and projectors reconstructs all the information about the captures light wave. This allows the viewer to freely moves in front of the monitor and enjoying a ultra-realistic three dimensional image from any place in the room without the need to wear any special glasses or other devices. In addition, when moving, the viewer observes the image with changing perspective, which gives the impression of a scene “suspended” at front of and in the monitor. Such a monitors reconstruct the full parallax of the scene, which allow observation of objects almost from both sides.

Great progress in the techniques of ultra-realistic 3D video displaying has been also recognized by the biggest standards organizations – the International Organization for Standardization (ISO) and International Telecommunication Union (ITU), and they began the works on the development of appropriate standards. For this purpose, a Motion Picture Expert Group (MPEG) of ISO has established a new working group named MPEG-I (MPEG Immersive), whose task is to develop the standards for the registration, processing and displaying of 3D light field video in the 3-4 years’.

Nevertheless, all the novel ultra-realistic 3D monitors that are present on the market today are experiencing the same one problem – the lack of appropriate content for displaying. Almost all currently available content for these commercially available monitors has been artificially computer generated. Other words, there is lack of appropriate 3D light field cameras that enable registration of high quality, wide angle, 3D light field video.
These observations and facts led to develop a system called MUCHA (FLY in English) for 3D light filed video capturing and processing.

3D light filed video camera allows for fuller, more complete, registration of the surrounding world than the classic (2D) camera. Due to it’s unique construction, 3D light filed video camera registers not only intensity of the incoming light in three subbands (RGB), but also its direction. This enables fuller registration of the surrounding world with the use of four dimensional space (position and direction of the incoming light ray) by capturing all information about light waves that reaches the camera. Such cameras are sometimes called holographic cameras (but to be able given light field camera a holographic camera it must be able to record a light filed with wide viewing angle).

Except typical parameters used to describe a classical cameras (image resolution – commonly expressed in MegaPixels and a frame rate – commonly expressed in frames per second, fps) light field cameras are also characterized by a number of registered simultaneously light rays (commonly in MegaRays).

Construction of the Mucha system

To minimize the cost of production of a prototype unit, the system have a modular structure and consist of a set of 3 main modules:

• acquisition module,
• processing module,
• communication module.

Technical parameters

The proposed MUCHA system is characterized by the following parameters:

• number of registered light rays: 192 Mega Rays,

• angular resolution, number of captured light rays per pixel: 100 light rays per pixel,

• resolution of the reconstructed 2D image: 20 Mega Pixels with at least 300 depth layers,

• frame rate: 25 frames per second (fps),

• focus range 10 times wider in comparison with conventional camera with the same effective aperture size,

• 3D scene reconstruction accuracy: 1mm x 1mm x 1mm within the range from 10cm to 2m and 1cm x 1cm x 1cm within the range from 2m to 20m.

• the perspective change: 10 degrees for objects at 1 meter distance.

Construction of the Mucha system

Developed the original software consist of three main parts:

• the software placed in the programmable device of the camera,
• PC software that allows for conversion of light field data,
• specialized software that extends range of application of the proposed system.

The software included in programmable devices of cameras will be written in the Verilog language. The use of Verilog and FPGA allow for very easy exchange of FPGAs for ASIC chips dedicated to the production version of the camera. The application of the hardware implementation of the processing algorithms can process massive data stream recorded in real time, which is not available for other solutions such as signal processors.

The software included in the camera will implement data compression algorithm together with algorithm of data conversion (data registered by optical sensors) to the format that is acceptable by spatial image monitors of the newest generation.

In addition, the algorithms of data conversion to LDV and MVD formats will be implemented to create the possibility of usage of older generation monitors.

The software will implement a set of advanced algorithms for spatial reconstruction of the observed scene:

• conversion and storage of stereoscopic images with very high spatial resolution and length of the base line selected arbitrary (distance between stereoscopic pair),

• displaying of spatial images on a traditional monitors and stereoscopic displays (3D), where using eye tracking techniques the effect of spatial images will be simulated,

• displaying of spatial images, in which the viewer alone will be able to change the perspective of observation of recorded scenes having the effect of „free navigation in the registered scene”,

• conversion and storage of the recorded spatial images in format of image with microlenses,

• conversion and storage of the registered spatial images in format that enables displaying images on the lenticular monitors,

• recording of individual images from a sequence that allows printing of images in lenticular technology,

• creation of a video sequence in which the viewer can freely change the focal plane of image, solely on the basis of the recorded light field data,

• reconstruction of the video sequence in such a way that the images of all the objects are “all-in-focus”.

Meet the Team →

Check our publications →

  • Mucha sp. z o.o. ul. Kręta 35 / 9B, 62-095 Murowana Goślina
  • +48 xxx-xxx-xxx