Super8 signal processing

[carllooper]

This is an extension of a discussion started at cinematography.com, (across a number of threads) and is about digital enhancement of Super 8 film.

The first part of such a process is to scan Super8 at the highest defintion possible. My own experimental rig is a 4K stills camera, with bellows and microscope objective. In my setup, due to overshoot for registration purposes, the image area is only 3K.

The second part of the process is to acquire, or write, and apply an image processing algorithm that computes the optical flow from one frame to the next (in the same way that compression algorithms work, but using more sophisticated techniques).

The optical flow needs to be carefully calibrated at the threshold between the signal and the noise of the raw film, otherwise you can end up computing the flow of the noise - which is what you definitely do not want to do.

The optical flow establishes the motion of the signal between one frame and the next - and you integrate the signal along this axis (temporal integration). The result is that grain is suppressed without having to sacrifice spatial resolution.

Suppression of the noise (in the temporal domain) increases the resolution of the signal.

Notional limits on what is achievable are:

1. the resolution at which the film is scanned
2. the resolution limit of the lens used during original acquistion of the filmic image.
3. how good your algorithm is.


Carl

[Nicholas Kovats]

Hi Carl.

Boy...this sounds very interesting!

Can this process potentially make Super 8 look as good as 16mm or even 35mm?

[carllooper]

This is a good question - despite many who think it's a silly question.

Keep in mind that the same process applied to Super 8 can also be applied to 16mm amd 35mm.

So while one might obtain something that looks like 16mm it will only be because the 16mm wasn't itself enhanced. Its a different ball game if trying to look as good as enhanced 16mm. Because, of course, there is more information in the 16mm.

The way I'd pose the question is whether one can get Super 8 to look as good as, or better than 16mm or 35mm scanned at 2K.

The reason for choosing 2K as a benchmark is simply because that is where 16mm/35mm DI has settled as a standard. But it's just an arbitrary reference point.

Now I don't know yet if achieving 35mm quality (as described) is actually possible. There are a number of problems that need to be worked through before I can demonstrate a working implementation.

What I do know is that the result of digitally processing the Super 8 signal is significantly better than not doing so.

Significantly better.

Carl

[Nicholas Kovats]

Could you detail a little bit more your experimental 4k setup, Carl?

[carllooper]

Certainly.

The experimental setup currently involves a Canon EOS 1000 digital stills camera (no HD capability). I chose this camera for a number of reasons. One is that it was one of the cheapest with a removable lens. Second was that Canon provides a reasonably good Software Development kit (SDK) for the camera. And thirdly, it provides a 4K sensor - which is about the minimum I've deemed necessary to assess the Super 8 in terms of 35mm DI standards. In fact is is better. The setup is built in such a way that other cameras (with better sensors) can be interchanged.

A second hand bellows (for $20) with a so called "universal" ring is attached to the camera body via a readily available adapter ring. At the other end of the bellows is a 4X planar microscope objective, with another readily available adapter ring.

When I originally spoke to the camera shop about adapter rings (for bellows) they said (typically) - "oh that's old technology. You won't find it." But sure enough, after research and talking to experts, rather than "professionals", there was indeed readiliy available rings.

Anyway, the microscope objective simply projects an image of an object (such as Super8 film) directly onto the sensor. There is the bare minimum glass involved. Indeed, explaining this to so called "microscope professionals" would have you think you need to adapt your camera to see an object via a microscope eyepiece. But I guess they are inthe business of selling microscopes so that would be what they would recommend, wouldn't they. To be fair, they were very forthcoming once I explained (and that they knew I undersatood) that the microscope body (and eyepiece) was not only redundant but a worse solution than projecting an image directly onto the sensor.

The film is held in place by an old projector gate removed from a projector that had seen better days (and was othewise dead anyway).

The next part of the setup is still under construction and concerns physcial transport of the film. The design is done but the implementation remains to be completed. I've dismantled and removed some useful parts from a Super8 editor/viewer. In particular are the rollers, sprocket drives, and reel spindles. Especially the sprocket drives because machining new ones would not be my cup of tea.

In addition to these parts is a USB stepper motor kit. This consists of a stepper motor, circuit, usb port and ActiveX software dev kit (SDK). The SDK makes the stepper motor programmable. The stepper motor drives the sprocket drive which drives the film through the gate.

The light source is simply a bright LED.

The film is advanced in increments that are actually smaller than the pitch of the film allowing multiple exposures of the film as it passes in front of the microscope objective. This is good because it elliminates sensor noise,

Now all of this mechancial work is simply a prelude to the main task - and that is the algorithms which process the scanned film. This is where the real work lies. If some one could scan Suoer 8 for me at 4K I'd probably pay them to do it and concentrate on the software. But in the absence of 4K scans of Super 8 I'm doing it myself.

The reason I'm doing it is simply because I want to get a better signal from the Super 8. But not just a better signal - I want the best possible signal. I don't care - at this stage - if the algorithms take all week to process a single shot. The purpose is first and foremost to extract the best possible signal. Later on, the algorithms can be optimised to improve workflow and turn around times for the dictates of particular films and schedules.

For now it's just purely research and development.

Carl

[Nicholas Kovats]

Thanks for sharing the detail, Carl.

I have a few comments/questions/info/links I wish to share. I apologize for the detail but I must transfer my knowledge to potentially assist you in your endeavors, i.e.

1. What is the specific microscope objective utilized?

On this point I wish to draw your attention to the excellent work of DR. Richard J. Kinch at www.http://www.truetex.com. He is a CNC, microscope, software/optical engineer who details all of his work on his web site.

This mini project of mine never reached CNC fruition but is a good example of the man's willingness to engage the S8 engineering "fringe". This was my modest sketch for an S8 anamorphic adapter. I am in awe of his modest charges for his quality work, i.e.

http://picasaweb.google.com/nkovats/Ana ... 3088741858

Which he translated into a CAD "sketch" ($50 US), i.e. http://www.truetex.com/kovats2.

Check out the detailed work regarding his non-optical and optical CNC adapters for microscopes, microscope objectives and Canon EF/EOS mounts! i.e.

Making Digital Camera Microscope Adapters, i.e.
http://www.truetex.com/micad.htm

2. I admire your efforts at constructing a film transport.

Check out Dr. Finch's "thought" process and ideas regarding this, i.e.

Making a Movie Film Transport Mechanism
http://www.truetex.com/sprock.htm

-this is amazing! He knows his DC stepping motors.

3. In regards to your software efforts check out his

Flatbed Scanner Digital Telecine (FSDT) Project
http://www.truetex.com/telecine.htm

...where he provides detailed analysis very relevant to your efforts I suspect, i.e.

1. Digital Registration -- Y-Axis Case
2. Sprocket Hole Edge Detection Sample Data Pattern Analysis

I feel this man's knowledge must be released from html captivity.

4. Canon actually has an SDK kit for the Canon EOS 1000? Do they have same for their other models? I am quite surprised by this.

[Nicholas Kovats]

I forgot to mention another valuable resource, i.e. Dr. Klaus D. Schmitt's "The Macrolens Collection Database, i.e.

http://www.macrolenses.de/contact.php

[carllooper]

Thanks for that information. It was a delight to read. I particularly enjoyed Richard's film transport work.

Richard's work on adapting a camera to a microscope eyepiece is useful when you can't remove the lens of the camera, ie. if you are forced to shoot through the camera's lens. In this situation the microscope's eyepiece is therefore necessary.

This differs from what I've done in that I chose a camera in which the lens could be removed. By removing the lens you also remove the need for the microscope eyepiece - in fact you have to remove the eyepiece. The microscope objective effectively becomes the camera's lens.

This is technically a much better result because there is far less glass through which the signal has to travel - and therefore a cleaner signal.

The microscope lens I purchased cost $139. I basically went into a microsope shop and ordered a "4X planar microscope objective". In fact I purchased this before I purchased anything else.

Well, that's not quite correct. I purchased the bellows first.

The "4X" specification for the microscope objective was worked out following discussion with the microscope people and research into how microscopes are made. It was really quite simple. One gets the dimensions of the Super 8 frame, and the dimensions of the sensor to be used, and from such one works out how much bigger the sensor is with respect to the film fame. The sensor was roughly 4X the size of the Super8 frame - so the required lens was a "4X" lens.

The word "planar" refers to the type of lens it is. The lens produces an image without spherical aberration - a flat or 'planar' image, rather than one exhibiting curvature. This wasn't strictly necessary as spherical aberration can be undone in the digital stage.

One thing to keep in mind is that while a microscope lens yields good results they don't have apertures, of rather they have just one big aperture - the diameter of the lens itself. This has implications for depth of field. I found that the image went slightly soft at the edges due to the narrow depth of field that a large aperture introduces.

The solution there was to insert a small aperture in front or behind the lens (eg. some tin foil with a pinhole in it). Of couse, if your light source isn't very bright it means you could be in for a 30 second exposure. So the answer there isjust to increase the brightness of the light source.

[carllooper]

In Australia (where I live) the Canon EOS SDK is available via this link.

http://www.canon.co.nz/en-nz/Support-Se ... /Canon-SDK

But note that it is only of use to software developers (computer programmers). You'll have difficulty trying to use it if you don't know how to program a computer. And there is no assistance provided by Canon for those who might get stuck trying to use it.

[peaceman]

Regarding maximum resolution reproduction of single Super-8 frames, I am using two different techniques:

- a "Duplikin II" 16 to 35mm Printer from Century Optics, USA. That piece is a solidd 4 inch tube with EF mount on one end and a Bausch & Lomb 25mm f/2.7 Animar lens in reverse position on the other end. On top of that is a very precise 16mm film gate and a diffusor. I used to use this on a crop sensor camera where it was perfect for S8. I now own a full frame 5Dmkii where its more fitting for 16mm, but still gives amazing results for S8.

- the other, more flexible approach is a bellow, too (cheapo GDR version for $20) with M42-to-EF adaptor. on the other end, I mouted a M42 to M39 adaptor ring and put a Rodenstock Apo-Rodagon enlarger lens on. This lens is incredibly sharp and known for its very high resolution. Stopping it down to f/5.6 or 8 provides awesome results in 5616x3744 and 14 bit per channel.

I am not totally convinced about LEDs as light source since they do not have a linear color spectrum.

i don't have an automatic advance mechanism for either yet (and would not know how to handle the amount of data) but can highly recommend either way for a minimum-glass/maximum-resolution approach.

I can't wait to see what you get out of this, Carl

Oh, I forgot: It helps big time to shoot the emulsion side and flip the image afterwards. The acetat base of the film does not help sharpness.

[carllooper]

Yes - the size of the RAW files from the camera are indeed large and you will want plenty of storage media.

The RAW files are intermediates. You could discard these files once the target signal has been extracted. My target signal is - for the time being at least - an uncompressed 24bit 2K signal for printing back to 35mm.

Of course, if you intend remastering the work, you might not want go all the way back to the original film to do this. For example you might want to remaster a work because you've updated the signal enhancement algorithms (rather than the scan rig). In this situation it would be a good idea to hold on to the RAW files.

Until I have my film transport component completed I'm only working with a handful of test frames.

Most of the work I'm doing at the moment doesn't actually involve the rig. I'm writing and testing the required algorithms. This involves using computer synthesised images that mimic the structure of film images. This is useful because on the computer you can generate a clean signal (eg. an 3D MTF chart animation) plus a simulation of how the film would encode that signal. You can then compare enhancement of the simulated film signal against the original signal.

[aj]

film gate and a diffusor. I used to use this on a crop sensor camera where it was perfect for S8. I now own a full frame 5Dmkii where its more fitting for 16mm, but still gives amazing results for S8.



I am not totally convinced about LEDs as light source since they do not have a linear color spectrum.

This S8 scanning is a bit of an eternal topic here. Check on the works of member videofred http://www.super-8.be before reinventing the wheel. It has all be done. This is for software, mechanics and camera choice.

There was guy from Chile who was really into HD scanning and processing.

Using a DSLR in still mode may be nice for experiments. But these cameras aren't durable enough for 10.000s of frames with their mirror and shutter going off so many times per day. One super-8 film of 15 meter has almost 4000 frames.

Do these DSLR things nowadays have HD filming with a intervalometer? That might come in handy if the shutter stays open while doing that.

[Nicholas Kovats]

Carl,

1. Do you foresee a Windows binary or Linux CLI interface regarding your efforts?
2. A media file 101 question. Is an AVI file "just" a container/wrapper for the embedded codec? i.e. could it "wrap" a RAW file? Would you want to?
3. What is the proposed native file system? Or will you architect options?

Andre,

Pessimism and righteousness aside I believe there are upper limits to Fred's development of the Avisynth script plus it will not work with interlaced frames, i.e. only non-pull down scanned footage whereby one S8 frame corresponds to AVI frame.

Parallel research and development can only be a good thing. No one approach is definitive nor absolute.

I found 2x examples of mirrorless HDSLR (APS-C sensor), i.e.
1. Samsung NX100, i.e.
http://gizmodo.com/5637263/samsung-nx10 ... era-smooth
2. Panasonic GH2, i.e.
http://www.dpreview.com/previews/panasonicdmcgh2/

Shutterless designs are only a matter of time. This industry perspective believes the Micro 4/3s manufactures will be first out of the gate, i.e.
http://zone-10.com/cmsm/index.php?optio ... mitstart=3

Peaceman,

As a real world Avisynth user and in the absence of Fred's input regarding his Avisynth script what are your experiences regarding non-interlaced AVI output? More specifically what could be improved from a user perspective? Codec perspective? RAW files?

[Nicholas Kovats]

Though I suspect the following shutterless C-Mount machine vision camera is prohibitively expensive I am posting this link as a reference point illustrating current electronic shutter manufacture, i.e.

http://sales.hamamatsu.com/index.php?id ... &undefined

..and it's replacement, i.e.

http://sales.hamamatsu.com/index.php?id ... &undefined

Fred and others in the DIY transfer area also utilize machine vision cameras, i.e.
http://www.super-8.be/s8_Eindex.htm.

Its a matter of time before electronic shutters supersede to some qualified degree mechanical.

The irony is not lost regarding our analog (S8)/digital hybrid processing efforts.

[peaceman]

Check on the works of member videofred http://www.super-8.be before reinventing the wheel. It has all be done.

@aj: Not "all" has been done yet, not even by Freddy
The subject here is obtaining a few (not 3600) 4K (or higher) optimal frame scans to feed Carl's potentail new SR algorithms. This has not been done before. And even if, why not improving?

As a real world Avisynth user and in the absence of Fred's input regarding his Avisynth script what are your experiences regarding non-interlaced AVI output? More specifically what could be improved from a user perspective? Codec perspective? RAW files?

Since I prefer frame-scanners over devices generating (interlaced) video output, I only worked with progressive "one frame per frame" material as input source. I think source material to any postpro engine (like avisynth) should always be as close to the original media as possible, which single frame shots are. The biggest weakness I see in avisynth would thus be that it only handles 8 bit per color channel, at least as far as I know. The color dynamics of e.g. Ektachrome 100D are often challenging CMOS sensors, even the 5Dmkii sometimes runs into clipping with that stock (and its dynamic range is enormous).