Entertainment Technology Center @ USC

Session 7: Stereography and Storytelling

Moderator(s):

Rob Engle, 3-D Visual Effects Supervisor, Sony Pictures Imageworks

Panelist(s):

Bernard Mendiburu, Analyst, Author of “3D Movie Making”

Chuck Comisky

Eric Kurland, Independent 3-DIY filmmaker, 3-DIY.com

Phil Streather, Stereo 3D Producer and Consultant, Principal Large Format

Rob Engle

“All 3D is not created equal.  It is first and foremost a very, very powerful creative storytelling tool” Jeffrey Katzenberg

3D Storytelling Choices

- Overall depth (interaxial spacing)

- Subject placement in depth (convergence)

- Roundness

- Where the screen edges are (floating windows)

- Traditional 2D composition

- Atmosphere (ex. smoke, clarity, etc.)

- Editorial pacing

- Depth transitions

Bernard Mendiburu

(Bernard worked on Meet the Robinsons, Monsters vs Aliens)

We are trying to “overcoming Millenniums of flatness” (Ray Zone).  We are where color movies were in the late 1930s.

What have we found:

- There is no screen.  We see through the screen

- Depth does not need to be realistic

- Big challenge is ‘snake oil vendors’, 3D experts since 2010

- Depth treatment must be integrated into the story

Chuck Comisky

It is really important to determine genre and style early in the process.  With Avatar the goals early on were 1) not to tire the audience’s eyes out, and 2) to within 10 minutes have the audience settle in and forget that they are watching a 3D movie.

Journey to the Center of the Earth had a lot of fun with 3D.  Early on they chose to play with the 3D in an obvious manner.

Make sure that the content is shot well; either performance capture or real capture.

In his remote stereography suite he relied on controlling knobs for three things; interoccular, convergence, and focus for the stereo pair.

They did use depth-of-field for creative purposes – soft focus in front and behind the point of interest.

3D @ Home Consortium has 10 rules for 3D that were developed by Chuck and James Cameron (http://www.3dathome.org/webpage.aspx?webpage=1952 ) (Phil Lelyveld note: there is more advice at http://www.3dathome.org/webpage.aspx?webpage=1946 )

Phil Streather

The two really interesting areas of z-space are 1) just in front of the screen to give a sense of intimacy and 2) positive parallax with parallel interoccular going to infinity rather than keystoning to infinity.

What is it about good 3D that makes it good?  Getting the math right.  He likes the idea of a depth budget.   Keep it to 1 ½ to 2% for the bulk of the feature (which works for all genres) and 5-10% for the special effects.  Ask yourself ‘who do I want in my personal space and who do I want in the behind-the-screen space?’

Eric Kurland

(Eric is a microbudget 3D filmmaker and a leader in the Los Angeles Do It Yourself 3D moviemaking community.)  There is a lot we still have to learn, but I have definitely learned that if you want to tell a good story in 3D, first and foremost you have to tell a good story.  The 3D must support the story.  3D does have a language that is different from 2D.  We all need education and practice.  I’m a big proponent of always practicing my art.  Try things and make mistakes on your own small pieces, so you don’t make mistakes on the big features.  YouTube has a 3D player now that works great.

Q&A

Do we want subtle or in-your-face 3D?  (Phil) When the seeds of life appeared around Jake in Avatar, they stayed behind the screen. If they came into the audience, the scene would have been about the seeds instead of about Jake.  (Bernard) A 3D gimmick is like a piece of candy.  Everyone loves a piece of candy, but a full bag will make you sick.

Does the up-charge for 3D theatre tickets mean that we have to give them ‘their money’s worth?’  (Chuck) We are giving them their money’s worth with the story! (Eric)  Outer space in Avatar was shot to look reel, but Hubble 3D ( now at IMAX) had a light-year interaxial distance and it completely engaged the audience.  (Bernard) Outside of Hollywood it is hard to see a 3D movie properly in the theatre.  He would like to see a trailer/intro with the image of a rainbow spanning the screen and the message ‘if you don’t see the full rainbow, including the ends, and you don’t see the full colors of the rainbow, walk out.’ (Eric) It is very important to education the exhibitors.  He went to a theatre where the left/right eyes were reversed.  He spoke to the projectionist, who didn’t consider it to be a problem.

Session 6: After the Capture: What Other Tools Exist?

Moderator(s):

Jim Whittlesey, Sr. Vice President, Technology, Deluxe Digital Media

Speaker(s):

Ray Harrisian, Stereographer, 3ality

Matthew DeJohn, VP/VFX Producer, In-Three Inc.

Peter Postma, FilmLight

Steve Owen, Director of Worldwide Marketing, Quantel

Peter Postma

Baselight is a software color correction product.  It is capable of multiple streams of 4K for color correction.  I’ll talk about what we added for 3D work.

Why would you perform stereo adjustments in a color corrector?  Because you have high performance tools (interactive), a good environment (big screen), the right talent for it, and the right time (at least in part) in the process for color and convergence issues to be addressed.

Stereographers will get eye fatigue if they wear any type of 3D glasses all day.  They can work in side-by-side or simple wipes between the two eyes.  Anaglyph is good for a quick check.  Checkerboard is useful for color adjustments – errors jump out when the content is viewed in checkerboard mode.  Color difference view and interleaved views are also useful.

Stereo tools include a multilayer multitrack timeline, so you can ‘gang color corrections.’  You can also synchronize the color grades, transferring information from one eye to the other.

Baseline is capable of adjusting for keystone, rotation, translation, and scale, and has floating window adjustment tools.

Steve Owen

The challenges of 3D post

What’s the problem?  The data needs to be perfectly synchronized, of the highest quality, comfortable to watch, meet the creative brief, and cost not much more than 2D content.

New issues for Post;

- Where does the image sit in relation to the screen? Creative choice and delivery method / screen size issues.

- Do your edits work in Stereo 3D? What are you asking the viewer’s eyes to do?

What are 5 good questions for any Post house to ask when preparing a quote for a 3D job, especially if your goal is to stay in business and make money.

1 What do the rushes look like?  ‘Fix it in post’ is the wrong attitude for 3D!  Quality control on-set is vital.  Don’t quote a job until you’ve seen the rushes.

2 What does the customer want?  For reference, Sky has posted their requirements for stereoscopic alignment on their website.

3 How much client interaction will there be?  Reliable viewing and play-out is critical to a good client experience.

4 Can your pipeline cope? (Will we make money on the job?)  3D is twice the data to manage, process, move, and store.  Quality matters like never before.  It will stress your network infrastructure, storage, and management systems.  A good 2D pipeline doesn’t guarantee success in 3D.

5 How can your technology help?  Can you do more in one suite?  Do you have real-time tools.  Stereo 3D time-saving tools: stereo color balance, geometry correction, image analysis, and tools to report divergence.

Ray Harrisian

3ality’s stereo image processing box analyzes the left/right image down to 1/100^th of a degree (or whatever the unit is) for accurate alignment.

Where to place things in space may not be obvious when it is shot, but is revealed during editing.  In U2 3D every transition (e.g. many of the cuts) is a visual effect.  They did dynamic depth balancing.  For example, they created the effect in Post of Edge and Bono looking at each other across a cut.

Matthew DeJohn

Dimensionalization as a tool after capture (“Dimensionalization” is a trademarked term of In-Three!) is useful to create and alter content, ensure viewer comfort, and enhance artist control.

Three basic stages for dimensionalization:

- isolation of the elements

- depth generation

- paint process

- (sidebar: transparencies, particles, motion blur, etc. are handled both here and elsewhere)

Doing dimensionalization wrong results in:

- rubber sheet effect – foreground objects wrap around into background objects

- cardboard cutout effect – insufficient roundness

- inaccurate depth layout – conflicting depth cues

- lack of depth continuity

- bad compositing (ex. hair, hard edges that conflict with motion blur, transparencies (foregrounds sticking to background), no paint / auto paint)

Doing dimensionalization right results in:

- good depth, distinct separation, nuanced / detailed, natural fall-off, matched 2D & 3D depth cues, solid depth continuity, and depth that is adjustable to the client’s desire

- compositing is VFX-caliber, hair as good as green screen, stereo-accurate transparency, real image data for occlusions, high quality matters.

When to use Dimensionalization:

- Difficult to capture shots

- Prohibitive environments

- Prohibitive production schedule

- Benefits of traditional 2D shoot

- Flexibility in artistic decisions

- Failed stereo capture

- Alternative stereo capture

- Catalogue titles

Q & A

Regarding catching errors, how much is manual versus software-based detection.  (In-Three) human intervention is a necessary part of the process.

When dimensionalizing, how do you know what is behind the object being dimensionalized. (In-Three) You follow standard visual effects techniques and processes.

Session 5: A Case for Quality in Production and Post-Production

Speaker(s):

Buzz Hays, Executive Stereoscopic 3D Producer, 3D Technology Center, Sony Corporation of America

(Buzz produced the 3D version of G-Force and Monster House)

What constitutes ‘high quality?’

- technical considerations; resolution, artifacts, (mis)alignment (can damage the people working in Post!)

- aesthetic values; the artistry must be very high-quality.  He has sent effects back to be improved.  It can have a lot to do with understanding parallax and stereography.

- effect on the viewers; some people are seeing it for the first time and don’t yet understand what they are looking at.  The audience will become more critical over time.  There is also concern over fatigue and eyestrain.  This is especially important now that it is coming to TV and people will be watching more 3D for longer periods.

Buzz received the completed version of Open Season and was asked to convert it to IMAX.  It had scenes that didn’t work well in 3D.  He used this to make the point that 3D must be considered in the pipeline regardless of plans to make it in 3D or not.

With Beowulf, Zemekis had a lot of experience in 3D, but was now telling a 2 hr. story to an older audience.  How to sustain 3D moments without causing eye fatigue was a key concern.  Phil McNally says that we’ve spent the last 200 years trying to convert the world to 2D.  It has now become its own art form.  We need to discover the fundamental language of 3D.  Motion may tell the story much better than cutting does in 3D.  Perhaps in 3D every shot is a point-of-view shot.

At the Sony 3D Technology Center they’ve started an educational program.  Working with the Local 600 Guild they are focused on the Cinematographers.  They will soon offer the program to Film and TV Directors as well.  They are working with Live Events people to retrain them to instinctively work in 3D.  In addition, they are reaching out to Game Developers to provide them with the education they need to optimize 3D game play experiences.  Later in the year they will be producing an educational program for Editors.

Stereoscopic 3D Terminology and Techniques

- Basic Terms, physiology, good vs bad, examples of 3D content, 3D camera systems, storytelling in 3D, lighting (back to the notion that lighting is used for sculpting), shooting 2D for 3D, production and post-production, practical shooting experience (Sony Pictures sound stage with a 3ality camera where they offer a 1 day class and 2 days of hands-on shooting).

Terms

- interocular distance – distance between the eye centers, about 2.5”, dictates the scale at which we see the world.  Our eyes don’t work like cameras.  We usually shoot at a 1” or less interaxial distance.

- convergence – rotate the cameras inward, but not so much that you produce keystoning on the chip.  It helps push infinity to the right distance.  The keystoning produces vertical misalignment.  Shooting 720p using a 1080p gives you enough information to fix the vertical misalignment in post.

- vergence accommodation – this is a key issue.

- negative parallax / positive parallax – negative is in front of the screen,(right -ye image is to the left of the left-eye image), positive is behind the screen (right eye image is on the right of the left-eye image)

- divergence – eyes point away from each other to fuse the object. At 1920 pixels on a 40’ screen, a 2.5” interocular distance means that more than10 pixels will cause divergence.  Viewing the content improperly on a small monitor will produce massive divergence when the content is projected onto a big screen.

- orthostereoscopy – we now have a chance to create a condition that we couldn’t any other way.  We can create a life-size experience with the audience.  It can simulate sitting in the front row of a theatre because we know something about where people sit when they watch TV.

Techniques that can work differently in 2D and 3D: focal length, framing, blocking action, camera motion (it may be a better way to tell 3D stories), depth of field

Q&A

What one rule would you recommend?  For home viewing, respect the personal space and push the 3D into and behind the screen plain.

Session 4: Keynote Speaker: Mark Schubin, Technology Consultant

What is 3D?

Jan 14, ITU-R SG6 defined 3 generations of 3D; first generation is plano- stereoscopic (with 4 sub-levels, 1 = anaglyph), second generation is multiview, and third generation is object-wave profile (holography), which is 15-20 yrs away.  RabbitHole is delivering film-based holography (we have examples in the ETC Consumer 3D Experience Lab).  It is limited to 1280 frames.

Less than full stereoscopic is enhanced chromostereopsis and pulfrich effect.  Chromostereopsis works as long as you control the colors in the scene.  View-shifting works by jiggling, and microstereopsis (Trioscopics) doesn’t cause vision problems but doesn’t work well either.

The term ‘3D graphics’ is used for both CGI and stereoscopic 3D, which can make conversations confusing.

POOT is plain old ordinary TV.  If you close one eye, you might sense depth.

He reviewed the visual cues mentioned by Pete Lude in a previous talk.  Papers by S. Nagata, 1991 and J. Cutting & P. Vishton, 1995 contain graphs illustrating the influence of the various cues.

Ames rooms confuse perspective vs. size cues.  (Ames rooms are the distorted, optical-illusion rooms that make the same object appear huge in one corner and tiny in the other.)  Five Ames rooms were used in Lord of the Rings to make Gandalf appear much taller than the Hobbits.  If Lord of the Rings was shot in 3D, the effect would not work.

Why does 3D matter?

Things to consider when creating  3D:

- The placement of the convergence point when shooting

- Pupillary distance can be 40-80 mm for children

- Screen size

- Viewing distance (if your pupillary distance matches the negative parallax on the screen, the object will come out 3’ if you are 6’ away, and 25’ if you are 50’ away, which is less credible.)

The movie theatre is not the home.  In a theatre, the audience is most likely entirely in the zone of comfort.  At home, the zone of comfort is only a small portion of the available seating locations around the TV.

Conflict /             Effect

- perspective-size /             incorrect relative size

- occlusion – stereopsis /             graphics difficult to view

- vergence-accommodation /             possible discomfort

- stereovisual – vestibular /             possible discomfort

- stereopsis – vergence /             incorrect depth

- impairment, choice  /             might training help the muscular areas? We don’t know.

Why are so many TVs coming out with active shutter technology?  Because polarization is hard to implement.  With active shutter, you don’t have to do anything to the screen.  You just add the emitter, so it ends up about the same price.  This is good for the CE manufacturer, because they are not necessarily concerned about the cost of the glasses.  Also, the battery life of the battery in the glasses is not their concern.

While people who are sensitive to 3D can watch polarized in 2D by putting the same lens in both eyes, isolating one image for both eyes in active shutter glasses may produce flicker.

BBC R&D White Paper 180, posted on the web, discusses how to synthesize 3D.

Steve Scklair said 3D requires fewer camera positions.”  The same was said at the birth of HDTV.  Audience expectations evolved, so expect the gradual adoption of faster cuts and shorter scenes.

Terms that are often used in marketing and press releases but rarely credible include: “the first”, “successful”, and “good enough.”

A paper by I. Howard & B. Rogers, 1996, discussed microstereopsis.  Human perception of stereo is greater than human perception of luminance.

Digital Optical Technology Systems, The Netherlands, is a glasses-free display that uses a slit iris to induce color fringes that creates a 3D effect.  The patents for this technology have expired.  It works with one ordinary lens per camera position, as long as it is outfitted with a slit iris.  This solution, which has a nearly zero interocular distance, has no ghosting, but it is incapable of a WOW effect.  It is a ‘kinder gentler 3D.’

(See Gary Shapiro’s editorial on 3D in the current issue of Vision magazine.)

Q&A

What types of content are particularly suited for 3D?  From a physiological standpoint, talking heads and children’s shows are well suited for 3D because of narrow depth range and comfort.  Getting people to buy things, though, will take sports and movies.

How can you shoot the Grand Canyon in 3D?  Make the camera spacing very large.  Hyperstereo reduces the 3D at a distance and heightens it up close.

Do you want to make any points about Avatar?  We need to interview everyone who had a problem viewing Avatar.  That is the epitome of 3D, so we need to understand why those people had problems.

Why is this rebirth of 3D different?  We have the digital technology, management of the convergence plains (keystoning), multiple camera sizes are available, and other factors, so from a technical perspective we are in a new place.  I cannot speak to the market question of whether this is the future.

Session 3: 3D Conversion

Moderator(s):

Brad Collar, Vice President, Technology, Warner Bros.

Panelist(s):

Barry Sandrew, Ph.D, Founder and President/COO, Legend3-D, A Legend Films Company

Chris Bond, President – View D, Prime Focus

Chris Yewdall, Executive Director and Chief Executive Officer, DDD USA, Inc.

Warren Littlefield, President, The Littlefield Company

Brad Collar

There are often two controversies when the entertainment industry looks at a new technology; should we do it, and what is the best way to do it?  The list of options in 3D keeps growing.  This panel will focus on 2D-to-3D conversion.

Warren Littlefield

Home 3D is the next breakthrough.  In 1996, he did a 3D episode of 3^rd Rock from the Sun.  It doubled their audience, and helped them win the May sweep.  The lesson learned was that the audience was looking for something new.

There are three categories of 3D content to the home; filmed entertainment, live content, and games.

Four business models (Warren’s term):

- Home video: over 600 episodes of StarTrek are available for a 3D sale.  He thinks 3D conversion will be the killer app

- 3D networks – expect more this year

- Worldwide program distribution

- Digital distribution, including iTunes, Amazon, etc.

The number of sets that will be sold is unknown.  With great cost comes great risk.  The revenue models will depend on how many 3D sets are sold.  There is no format war, just a need for content.  This is an opportunity to up-sell the library.  The box office for 3D indicates a sustained appetite, dispelling the idea of ‘stunt’ appeal.  Building the market will be about volume – volume of set sales and volume of content (live events, movies, and series) together (the chicken plus the egg).

Chris Yewdall

(Chris calls himself the self-proclaimed minister of propaganda for 3D)

Conversion can be done at one of three quality levels: good (embedded in 3D device, automated 3D conversion), better (3D conversion with automated depth recover, manual focal point/depth effect decisions), and best (original 3D content creation, or 3D conversion with manual / semi automated depth recover).  The cost goes up accordingly.

Carl Franklin wrote a great book, “Why Technology Fails.”  Any 3D conversion must pass the “so what” test.

Since 1993 DDD has been developing a process called Depth-Image Based Rendering (DIBR).  A single 2D image is depth mapped using monocular queues to create the conversion.   They have been working to make the manual aspects more efficient by continually improving the computational rules.  Cost went from $100k/minute in 1999 to $1,500/min in 2007 by relying on the automation more and the human intervention much less.

3D conversion consists of two stages: depth recovery from 2D content, and 3D scene reconstruction from depth and source image.  Auto 3D conversion will not deliver the “better” category (first paragraph).  Human intervention is required, particularly for 3D focal point.  At the end of the day, the source content is still 2D, but filming in 2D with the scene set-ups designed for 3D improves the results.

Barry Sandrew, PhD

(He invented the first auto-colorization process, and he contributed to the Alice in Wonderland conversion)

(He thanked Pierre de Lespinois (from the previous panel.  Pierre was very critical of conversion.) for the kind words. (sarcasm))  70% of the over $80M of Alice opening weekend was from 3D screens.  Rather than go through his slides, he showed recent work to present a more accurate sense of the current state of conversion.

Christopher Bond

Christopher oversaw conversion of Clash of the Titans, and presented the chronology of the process.  Prime Focus received the call in mid-January to review the film.  They had the kick-off meeting with the Director, Editorial, and VFX supervisors, who were skeptical that it could be done in 8 weeks, in London on Jan. 19^th.  The delivery date for the final version of the conversion was fixed at March 19^th.  The details of what they were to be given when work started included:

- not a locked cut / expect editorial changes / scene additions

- 90-100 minutes run time, 1000-200 shots

- No graded material, begin work on raw scenes ASAP

- Less than 1/8 of the VFX shots were final at the start of conversion

Having to deal with working with an unlocked, ungraded cut meant redefining their work process:

- Focus efforts on the “most locked” reels

- Work with raw scans (so they were prepped when the graded material arrived) / 8 frame handles head and tail

- 16 frames – 1935 shots = 21.5 additional minutes

- Conceive and develop tools to reprocess graded material once delivered

- Automate as much of the pipeline as possible

- Everything went into a database, which was created for this process

The creative elements came into play by this process:

- Define “Keystone” shots throughout the film

- Bring “Keystone” shots to final ASAP (approximately 5-10 days) and show them to WB

- Apply notes / feedback from WB and propagate to shots within the same sequence

- The dailies went through many rapid-fire iterations.  Dailies were being worked on in 3 rooms, 1 Dolby and 2 RealD

- Because of the quantity of dailies, they came up with a grading system – A (perfect), B (some artifacts), C (first-pass)

- Twice weekly formal client reviews, progressing from scattered shots, to sequences, to reels, to, near the end, the entire movie

- Daily reviews in the last few days covered 15-17 minutes of reviews each day

- Toward the very end they applied convergence and watched the shots in ‘cut’

Lessons learned

- Do not underestimate editorial and ‘conform’ needs and reviews.  This is a massive amount of data.

- Working with an unlocked cut meant that lots of content ended up on the cutting room floor.

- Clients change their minds, even on fundamental issues like how much stereo

- We can convert a movie in 8-10 weeks.  Just stay calm.

Q&A

(Brad) Why is there so much controversy about conversion? (Warren) Consumers will vote with their wallets.  (Barry) Something this new will attract both criticism and praise, but it will make money if done right.  (Chris Bond) Sound, Panavision, TV all met the same criticisms, which faded as they improved and gained market acceptance.

(Brad) Will costs come down?  (Chris Yewdall) TVs can do the ‘good.’  Avatar shows the ‘best.’  We will fill in the middle over time, and costs will come down.

(Brad) Do you see a head-end broadcast converter box?  (Warren) Live conversion will not be as good as human intervention in the near term.  The automated process in the TV cuts out the content maker from the revenue and control, so it is critical that they get involved in the process somehow.

(Brad) Chris (Yewdall), you brought up good, better, best.  Will consumers see the difference?   Should they be branded differently?  (Chris Yewdall) No, they should not be branded differently.  There are already variations; 720p v 1080p v 1080i.  Over time, the conversion technologies will improve to the point where consumers won’t be able to tell the difference because there may not be a difference.

(Brad) What are the technical advantages of converting a new feature versus an old feature.  (Barry)  The degree of creative input is much less for older features.  There is a great deal of creative input for new films, which adds time an cost. (Warren) We’ll have a new generation of filmmakers shooting in 2D, knowing it will be converted.  Their shooting process will be changed by that knowledge, and the viewer will benefit from that.  It is just starting to happen.

Session 2: Programming: Lessons Learned

Moderator(s):

Al Barton, Consultant, Freelance Digital

Panelist(s):

Jason Goodman, CEO, 21st Century 3D

Pierre de Lespinois, Co-Founder, Evergreen Films

Thomas Edwards, VP, Digital Television Testing & Evaluation, FOXPanel

Pierre de Lespinois

Evergreen is working to integrate the storytelling with the engineering.  Technology advances fuel revenue growth.  After conversion to HD had been done, integrating 3D into that workflow process for live events was fairly easy.   They have an “interoccular” crew member, someone “pulling convergence” during the shoot.  The camera operator controls the zoom and focus.  The stereographer in the truck communicates with the “interoccular”, who makes sure that the cameras are balanced.  He has found that most of the time is spent getting the lenses centered so they zoom properly.  They spend a full day tracking and calibrating the lenses before the shoot.  Once the lenses are tracked, they keep the lenses with the camera for the duration.   (He showed a beautify Dave Matthews Band concert clip and a clip from a feature called Totem.)  The cost difference for shooting 3D, due to 2 cameras, rig, stereographer, and other, amounts to an additional cost of 10-15% on production and 10-15% on post.

Thomas Edwards

(He started with an extended Fox Sports clip, highlighting football.)  The trucks use micropol displays because it is hard to synchronize a bank of shutter displays.  Wide/high shots tell the story but produce a toy soldier effect.  Tight/low shots “get into the action,” especially if you zoom in.  Tasteful, occasional use of extreme negative parallax can be good, but you must avoid objects that are too close to the viewer such as a foul net or a foul pole.  Score box placement is an open question.  You want the scores/stats to be in front of the closest image.  Placing the score graphic at the bottom of the screen puts it in front of the grass (sharp foreground), but putting it at the top puts it in the sky (near screen plane).  One bonus benefit of putting the score graphic in the sky/screen plane is that it lets people without glasses read the score.

Challenges

• Equipment is tough to obtain
• Equipment is fragile
• Equipment is large and heavy
• Stereography training – “convergence pullers”, and others
• Discovering what works for 3D sports direction
• Challenges of dual 2D/3D production – more seat kills (e.g. seats lost to cameras in the stadium)
• Challenges of backhaul
• Budget?  Is this ever going to make money?  HD did not make us any money.  3D must make money for us.
• Small number of distribution channels.

Jason Goodman

(Jason is the first person to be recognized by the DGA as a Stereographer.)

21^st Century 3D developed a ground-breaking 3D camera; compact, light-weight, progressive scan 24fps, with the look and feel of a normal camera, binocular viewfinder, and purely digital workflow.   He discussed the evolution of their cameras.  They are announcing their next gen camera, available for purchase, this week at NAB.  (He showed a clip from the Black Eyed Peas movie that they are working on, plus other footage.)

Q&A

Why do you need two cameras? Panasonic is showing a single body camera for the prosumer market.  The Panasonic has 2 lenses, one chip.  (Pierre – going back to an earlier discussion) Dimensionalizing 2D is fooling the public.  Don’t call it 3D.  Certain shots in 2D don’t work in 3D.  Films need to be shot for 3D.  $4.5M to dimensionalize is less than $30M to shoot 3D, but it isn’t real 3D.  Call it dimensionalizing.

Thoughts on edge violations? (Pierre) We make sure that things on the edges of the frames are non-intrusive.

(Al) The hardest thing right now is learning what terms to use when discussing 3D production with someone else.  The actual language used to describe 3D issues and processes is in flux.

Session 1: Understanding Stereopsis and 3D Image Capture

Speaker(s):

Peter Lude, Senior Technology Executive, Sony Electronics

Steve Schklair, CEO, 3ality Digital Systems LLC

Peter Lude

Monocular depth cues (such as motion parallax, depth from motion, and perspective) contribute to our mind’s interpretation of 3D in the real world as well as stereoscopic 3D content. In a given image, something out of focus is perceived as being behind something that is in focus.  Vanishing point, or the convergence of lines as they approach the horizon, also provide visual depth queues.    Color intensity and contrast contribute; things that are far away are duller.  If you look at a 2D image with only one eye before you have seen it with both eyes, you will perceive it as 3D.  Only when binocular vision kicks in do you instantly snap into perceiving it as only 2D.

Mean interocular distance is about 65mm, with a wide variation.  Children start off with a distance 10-15 mm smaller on overage.

Positive parallax corresponds to seeing the right image on the right, and the left image on the left.  This places the object behind the screen plain.  Negative parallax crosses the eyes by positioning the right-eye’s image to the left of the left-eye’s image.  This places the virtual object in front of the screen, in ‘negative space.’

Mistakes in shooting 3D include; vertical misalignment of the image, non-synchronous lens zooms, mismatched focus, color mismatch, and keystoning.  The content should be authored for the largest expected display size.  If you author for a small screen and display it on a large screen, you will produce disparity – you are forcing the audiences’ eyes to turn outward in opposite directions.

3D camera rigs can either be two physically separate cameras locked on a bar, or a camera-pair looking through a silvered mirror beam splitter.  The beam splitter  creates a more ‘human’ interoccular distance.

Steve Schklair

Steve Scklair used a live feed during his presentation to illustrate shooting points and errors.  Production challenges include; developing the pool of trained crew, choosing the appropriate technology, revising the production pipeline, understanding the production budget and logistics, and developing the new language.

Two basic rig types.  The beam splitter simulates the interoccular distance.  It allows you to bring objects right up close to the camera.  Side-by-side rigs match the functionality of the beam splitter rig at a lower cost because no beam-splitter optics.  It is often used in sports because there is no reason to bring anything close to the lens.

Steve conducted a live demonstration of vertical misalignment.  Everything about camera-pair positioning must be remotely controlled, because you cannot have people running up to the rig with wrenches during a shoot.  The idea of keeping vertical locked in a zoom was considered critical from day one, because you have to zoom in when shooting sports.  Focus mismatch and zoom mismatch can occur even when you turn the zoom rigs identically, because the mechanics and lens characteristics aren’t identical.  This can be fixable in post, but it can cause discomfort during live events.

Too narrow an interaxial distance reduces the 3D to 2D.  It is ok to have sustained images 1-2% of the screen width in front of the screen, but holding images in front of the screen further and longer would be problematic.  Keeping the depth fairly consistent among the cameras makes cutting more comfortable, both for the audience and the editor.

A fix for edge violations is to focus on the closest object. Or you can just eliminate the edge violation by reconverging your cameras and putting the object completely in the frame.

3DIQ: Sky paid 3ality to put rigs into Telegenic trucks.  The lessons 3ality and Sky learned from the experience include:

• • Editorial pace is slower from shot to shot (because there is more info in the shot)
• • Staying a bit wider works
• • It is important to be consistent with the depth
• • It is important to level the depth across the edits
• • For live broadcasting, fewer camera positions are needed in 3D than in 2D
• • The story is more important than the WOW factor

On set, the monitors are good enough to view the shots as long as you are positioned properly.  Realignment in post will kill your budget.

Session 9: 3D 10 years Hence: 20/20 in 2020?

Hans Hoffman, Programe Manager EBUI Technical

Panel

Tibar Balogh, cEO/Founder Holografika Kft

Dr. Aljoscha Smolic, Disney Research

Dr. Takayki Ito, NHK

Hans Hofman intro: There will be no interoperability issue in the future, because there will be no glasses.

Tibor Balogh, Holografika Kft

Home TV is in many ways a social experience.  You watch with your family or friends with the lights on.  You may talk about the show or something else.  You are not just staring at the screen.

To see the perfect 3D TV, look out the window.  When you cannot tell if it is a window or a TV then we have the perfect 3D TV.

Pure 3D will reconstruct the light field.  The light field is a function of position and direction.  Variants are X, Y, and angle for field of depth, plus vertical and horizontal parallax.  Reducing the vertical parallax can simplify the solution.

HoloVizio system – Optical modules project light beams to hit the points of the screen with multiple beams under various angles of incidence.  Holographic screen has a direction selective property.  It will be comfortable, glasses-free, no viewing restrictions, 2D/3D compatibility, and won’t cause any visual discomfort or restrict viewers.

The smallest practical pixel size is 0.1mm or 1 minute of arc, at 25cm optimal viewing distance.

There is a difference between trying to construct the 3D scene (light field) and tryig to mimic the human experience (view field).

Aljoscha Smolic (Disney Research, Zurich)

He described basic concepts of acquisition, transmission encoding formats, compression, delivery formats (BD, internet file), decode and regeneration/rendering, display.

With advanced 3D displays, Holografika today shows 60 or more views.  Other autostereo displays use 8,9,16 views without glasses.  But transmission of multiview data is very inefficient.  A spatio-temporal video continuum may be a better way to reduce/transmit/reconstruct the data.  He proposed reducing X views down to a few views accompanied by the data to reconstruct the X views.  He showed graphics imaging how this could be done.  MPEG will issue a Call for Propose in Oct. 2010 for specifications.

He is also involved in the MUSCADE initiative (www.muscade.com ?), which is a large European consortium.

Depth estimation of the 3D scene reconstruction must be accurate, robust, automatic, and real-time.  The form factor and handling of the acquisition systems is a problem.  The rigs must be manageable.

Stereoscopic Warping: Disney has developed a technique to manipulate the 3D without manipulating the image itself.  Good for corrections and conversions.  (His description was not clear.)  He will be presenting a paper on this at Siggraph.

smolic@disneyresearch.com

Takayki Ito (NHK)

Phases of NHK research

- Phase 1: basic research

- Phase 2: 3D TV

- Phase 3: visual comfort / discomfort, visual fatigue

- Phase 4: holographic

The four key natural visual cues are vergence, binocular disparity, motion parallax, accommodation.  The more natural cues you use, the better the 3D effect will be.

The cause of visual fatigue is inconsistent vergence and accommodation (i.e. the vergence accommodation conflict).

Test: He studied subject responses to: 2D still image, 3D still image with large disparity, 3D image with time varying small disparity, 3D image with time varying large disparity.  People experienced more fatigue when viewing time varying disparity than when viewing a 3D still image for a long time.

NHK has produced an integral 3D image which causes focus to vary with viewing z position!   The resolution of their prototype, shown at 2009 NAB, is too low.  Their research and development effort is dependent on technology advances.

Session 8: 3D Consumer Experience in the Home: The Interoperability Challenges

Brian Markwalter, VP Technology and Standards, CEA

Panel

Ami Dror, Chief Strategy officer, XpanD

Steve Venuti, Pres HDMI

Mark Stockfisch, VP and CTO, Quantum Data

Brad Hunt, Pres., Digital Media Directions, LLC

Ami Dror, xPanD

People are buying 3D TVs now, and glasses now.  We need standards for the emitters and the glasses.  We/xPanD are working on universal-format glasses.  Vote with your wallet.

Steve Venuti, HDMI

When we started the HDMI standards effort we knew that we’d need to ensure that devices work when consumers connected them.  We needed to identify the minimum requirements.  Our focus was movies and games because we understood them.  We didn’t understand broadcast.  HDMI v1.4 was released last year with the idea that we’d spend the next year gaining an understanding of broadcast.  Now we have four formats.  The display must be able to handle all four, and the source must be able to handle at least one of them.

Mark Stockfisch, Quantum Data

HDMI is not alone.  There are many other standards; DVI, Diiva, whdi, wifi, dlna, moca, homePNA, Ethernet,…  There are HDMI handshake issues (ex. when you place a legacy device between two HDMI v1.4 compliant devices, you can have problems.)

Issues:

- Will the quality of the in-home experience match theatrical 3D?

- eyewear (weight, cost, compatibility, other)

- compression artifacts and liquid crystal artifacts

Brad Hunt, Digital Media Directions

Is the consumer going to face interoperability issues in the home from a) Blu-ray and b) broadcast system?

a) The Blu-ray Alliance decided on a standard that requires an upgrade (PS3) or a new player.  Today, stay with one manufacturer for the display, blu-ray player, and the glasses to avoid interoperability problems.  ‘The bundled packages are your friends.’  Also, make sure you hook up the devices to your home network, because they need to be able to receive firmware updates.

b) For broadcast, hook up your STB directly to the 3D display, rather than thru an intermediate device.  That way, you’ll get the benefits of the firmware updates and avoid the problems currently associated with the audio signal.

Q&A

Comment on the evolution of HDMI? (Brad) While the Blu-ray spec was being developed and HDMI v1.4 formulated, the broadcasters were already implementing their frame compatible format solutions.  Top/bottom, which many see as preferable over side-by-side, was not in the original HDMI spec. (Steve) We have an ideal world with HDMI v1.4a.  We addressed broadcast’s needs rapidly by quickly modifying v1.4 to v1.4a, which does include top/bottom.  (Ami) 3D TV is developing contemporaneously with IPTV.  At least HDMI did respond rapidly.  One of the biggest problems is the STBs, which are evolving very slowly.  (Mark) There is a trend of not having a STB at all, and streaming directly to the set.  There is a lot of convenience to it when moving content around the home.  (Brian) The nice thing about HDMI is the compatibility between sources and displays in the home.  CEA has formed another group to discuss the business side of 3D interoperability to address things like glossaries and terms for the consumer.  For example, what does ‘3D ready’ mean?

What are your thoughts on the timing of standards?  Would we get universal interoperability without them?  (Mark) CEA 861-2 may be where standard metadata and interface will be addressed.  Eyewear standards for IR glasses may come out in June.  (Brad) How do you allow a legacy 3D source device to do format signaling so the display knows what format to display?  Updating the legacy STBs via firmware updates is a major issue, and we need a commitment from HDMI that there will be a way to handle this signaling.  (Ami) Beyond IR, we at xPanD are already developing Bluetooth active shutter glasses.  I’m pretty sure there will be a standard, if not in June than in August.  By the time it arises we will already be working on the Bluetooth standard.  Also, a huge problem is consumer and salesforce EDUCATION EDUCATION EDUCATION.

What are your thoughts on closed captioning and graphics standards?  (Brad)  This is critically important and being worked on in CEA.  The question is how do you send metadata so the captions/graphics display properly.  There are many proposals currently being discussed related to how much data, processing power, functionality options, etc. are needed.  There must be cooperation between SMPTE and CEA.  (Ami) Subtitling must be standardized in 3D TV.  There are many other parameters that have not been discussed yet, like how do you use your remote when the glasses are on.  We are just in the first pass of discussions.

HDMI decided not to put version numbers on products.  Discuss the issue of interoperability, consumer info, and labeling?  (Steve) Starting with v1.0, each version increased the options.  The version refers to the document.  Because some many things above the basic spec are optional, the version number did not correspond to full interoperability and therefore using it on a product could mislead the consumer.  So HDMI banned the use of version numbers on products unless you specify the exact features that the device supports.  Over time the version number will be completely banned on products, and manufacturers will simply list supported features.

(Pat Griffis question) The USC ETC 720p24 and 1080p24 demo was great.  Top/bottom didn’t look so good.  We’re saying side-by-side for interlace, top/bottom for progressive, which means we aren’t going toward a single standard.  The mandatory formats do not include side-by-side! Respond? (Steve) Four months ago no one knew that.  The topic is being discussed by the technical and management team at HDMI.  (Pat) I applaud the work.  We’re still iterating.  Perhaps you jumped the gun and may need to go back and re-examine base issues.

Session 7: Consumer 3D TV Displays: What are the Technical Differences?

Peter Putman, Contributing Editor, Pro AV

Emissive Displays: CRT Imaging, Plasma, OLED

Pulse-width modulation for plasma displays;  Because plasmas are either on or off, the must be pulsed for grey scale.  Panasonic 3D plasma screens sizes vary between 42” to 103”.

OLED is proving very hard to manufacture in quantity.  Brightness and color uniformity is a problem.  Epson and Samsung have shown 40” OLED displays.  They are super thin at under ½”.

Disadvantages of emissive displays include; color aging (blue channel goes the fastest), glare from ambient light, and they tend to be heavier.

Transmisive displays; LCD, HTPS

LCD displays, 120hz or faster for fast motion, bright saturated color, very high contrast and rightness, and the front LCD can be polarized for 3D.

LCD HD and 3D TVs are moving to backlit rather than reflective displays.  The leading manufacturers are moving to 240hz and 480hz.  The IR emitters are embedded in the front borders.

High Temp Poly Silicon (HTPS) projectors are just getting started in the marketplace.  Typically they use short arc lamps or LEDs for the light source.  LEDs last longer – 30,000 hours.

Disadvantages of transmissive displays include; narrow viewing angles, color shift, black level, contrast, cross-polarization can cause 3D images to black out.  LCDs do poorly under high ambient light (washout), and motion blur issues (requires partial black frame insertion / LED backlights can help).

Reflective displays; DLP, LCoS, D-ILA

They can give you bright, saturated colors.  LCoS can handle 120 hz, and DLP can produce extremely high switching rates.  DLP for 3D home projectors is quite possible.  There are two versions of DLP; RGB and RGB+white.  The white can be used for flashes that synchronize the active shutter glasses.

Disadvantages of reflective displays include narrow viewing angles, color breakup artifacts, and lower contrast and black levels in D-ILA.

Finally, he reviewed the HDMI mandatory 3D formats.  The info is available on the HDMI website.

Conclusion: the best 3D TVs so far are

- Plasma TVs,

- OLEDS,

- 3 chip DLP projection, and

- LED-backlit LCD TVs.