3D Reconstruction of William's face.
If you have Acrobat Reader 7 or later installed then the above should be displaying a 3D PDF of William's face (508KB).
- To rotate the model: left drag.
- To zoom: right drag up/down.
- To pan: left+right drag.
- For menu: right click.
Index
181 photos for a hologram
Liti3D will produce a bespoke hologram from 181 images taken at 1 degree intervals around a subject. Now that we have the model in Meshlab the question is how to automate the screen capture, there is no current interface in Meshlab to accommodate this, so I hacked one. At some point I might generalise it to a plug-in that will capture 181 images from left to right of the current view, but at the moment it suffices to capture said images for a hardcoded view rotating about one axis. Add the code below to glarea.cpp and recompile Meshlab to get 181 screenshots when you click on the menu option for Reset Trackball.
//in file meshlab/src/meshlab/glarea.cpp
void GLArea::resetTrackBall()
{
char s[2000];
float thetaRad;
float thetaDeg;
HWND hWnd;
//to display current view in order to work out the values to use below,
//use the following code instead:
// trackball.ToAscii(s);
// TCHAR wstring[2000];
// MultiByteToWideChar(CP_ACP, 0, s, -1, wstring, 2000);
// MessageBox(NULL,wstring,TEXT("current trackball"),0);
setLight(false); //turn light off.
//resize main window, the screen shot size will be an integer multiple
//of the main window size, I don't know Qt so I call win32 directly,
//would need changing for linux or just do it manually.
hWnd=GetForegroundWindow();
::MoveWindow(hWnd,0,0,1200,1000,1);
//set up the screenshot for twice the current window size
ss.resolution=2;
ss.counter=0;
ss.basename="image";
//to get e.g. image000.png change the following in GLArea::pasteTile()
// .arg(ss.counter++,3,10,QChar('0')); //change 2 to 3
for (thetaDeg=-61.5; thetaDeg>=-151.5; thetaDeg=thetaDeg-0.5)
{
thetaRad=thetaDeg*3.1415926/180.0;
sprintf(s, "trackball(%f,%f,%f,%f,%f,%f,%f,%f,%f)",
0.0,sin(thetaRad),0.0,cos(thetaRad),0.0,0.0,0.0,1.0,0.0);
trackball.SetFromAscii(s);
updateGL();
saveSnapshot();
//need to call updateGL once for each tile in the screenshot!
for(int j=0; j<ss.resolution*ss.resolution; j++) updateGL();
}
updateGL();
}
These images may then be cropped down to the required 1024x768, have their transparency replaced by black, and converted to a 24-bit bmp by the following dos command which calls the convert program from the excellent free ImageMagick suite.
forfiles /M image*.png /C "cmd /c convert @file -fill black -opaque none
-gravity Center -crop 1024x768+0+0 +repage BMP3:@fname.bmp"
I'll update this page when I get the hologram.
Results 2.0
The session went well, Jeanette controlled the cameras, whilst I lifted William up. It became evident after half a dozen shots that the cameras were no longer in synch, they were alternating between the half-press and full-press state, a simple remedy to unplug the usbs from the full-press state and shoot off one shot on the others brought them back into synch. I downloaded the photos with barrel correction as described previously, resized the images from 3072x2304 to 1536x1152 and applied an unsharp mask as some were a bit out of focus, and then submitted to Arc3D without doing any masking this time. (Note for submitting to Arc3D, it appears to crash if you don't have short file names, e.g. C:\arc3d\01.jpg works ok. And any text added to the reference field appears to crash the program.
Arc3D emails back "Your model reconstruction is ready." So it worked. Then comes the experience of opening the data in Meshlab. There are a lot of settings you can alter, a lot. I found the following worked best, keep the subsample to about 6, otherwise meshlab creates multiple coincident meshes, with a rigorous post-processor to smooth the mesh this wouldn't be a problem, but I couldn't find a way to do this in Meshlab - the Poisson function removes the colouring. Set the minumum count to 3 to reduce noise. Set the minimum angle to 80 to get the sides of the face, cameras spaced around the head could reduce this, Feature aware smoothing left at 3, remove pieces, and close holes set to 10. depth filter left at 5x5 and 5x5. fast merge off, all photos selected.
This gave a point cloud which looks really cool, but making a mesh from this proved quite troublesome. (If a 3d pdf could display coloured points I might have left it at that.) To clean up the mesh, I did a variety of "build surface from points" "clustering decimation" "clean unreferenced vertices" and manually deleting parts of mesh that covered the mouth. This I then saved as an IDTF file, and subsequently edited to make the u3d work. Meshlab and/or IDTFConverter and/or Acrobat have some discrepancies in their use of u3d. At somewhere along the chain the vertex colours will only get carried into the pdf if the idtf file is edited to have the following shading definition, i.e. with an additional dummy material without vertex colours!
RESOURCE_LIST "SHADER" {
RESOURCE_COUNT 2
RESOURCE 0 {
RESOURCE_NAME "ModelShader1"
ATTRIBUTE_USE_VERTEX_COLOR "TRUE"
SHADER_MATERIAL_NAME "Material1"
SHADER_ACTIVE_TEXTURE_COUNT 0
}
RESOURCE 1 {
RESOURCE_NAME "ModelShader2"
ATTRIBUTE_USE_VERTEX_COLOR "FALSE"
SHADER_MATERIAL_NAME "Material2"
SHADER_ACTIVE_TEXTURE_COUNT 0
}
}
RESOURCE_LIST "MATERIAL" {
RESOURCE_COUNT 2
RESOURCE 0 {
RESOURCE_NAME "Material1"
MATERIAL_AMBIENT 0.0 0.0 0.0
MATERIAL_DIFFUSE 1.0 1.0 1.0
MATERIAL_SPECULAR 0.0 0.0 0.0
MATERIAL_EMISSIVE 1.0 1.0 1.0
MATERIAL_REFLECTIVITY 0.0
MATERIAL_OPACITY 1.000000
}
RESOURCE 1 {
RESOURCE_NAME "Material2"
MATERIAL_AMBIENT 0.0 0.0 0.0
MATERIAL_DIFFUSE 0.0 0.0 0.0
MATERIAL_SPECULAR 0.0 0.0 0.0
MATERIAL_EMISSIVE 0.0 0.0 0.0
MATERIAL_REFLECTIVITY 0.0
MATERIAL_OPACITY 1.000000
}
}
MODIFIER "SHADING" {
MODIFIER_NAME "VcgMesh01"
PARAMETERS {
SHADER_LIST_COUNT 1
SHADING_GROUP {
SHADER_LIST 0 {SHADER_COUNT 1 SHADER_NAME_LIST {SHADER 0 NAME: "ModelShader1" } }
}
}
}
You also need to correct the MESH_FACE_DIFFUSE_COLOR_LIST order from 0 2 1 to 0 1 2, and also correct the MODEL_DIFFUSE_COLOR_LIST from e.g. 92 82 52 0 to 0.359 0.320 0.203 (as idtf doesn't use an alpha channel, and goes from 0.0-1.0 rather than 0-255). I also rescaled and centred the xyz coordinates from -100 to +100 (and mirrored the x and z), as this works well with the javascript that I use to control the 3D pdf. I did these corrections in Excel. I then decided to go the whole hog and reduce the colour depth from 100,000 diffuse colours to 256, I used paintshoppro to produce a 256 colour palette, then wrote a matlab script (run with the free Octave program) to do a least squares matching for each vertex colour to reduce the file size from 1300K to 500K.
palettelist=[34 28 23 %256 colours from paintshoppro
49 41 24
56 36 25
49 53 45
66 32 20
61 52 36
...
239 231 231
255 255 255];
idtf=[92 82 52 0 %98,313 colours from meshlab's idtf
57 49 28 0
63 51 30 0
...
84 67 37 0
94 75 45 0
88 64 40 0
94 71 39 0];
idtfout=zeros(size(idtf,1),1);
for entry=1:size(idtf,1),
palmatch=1;
leastsqpal=256*256*256;
for pal=1:256,
sqpal=(idtf(entry,1)-palettelist(pal,1))^2+(idtf(entry,2)-palettelist(pal,2))^2
+(idtf(entry,3)-palettelist(pal,3))^2;
if (sqpal<leastsqpal)
leastsqpal=sqpal;
palmatch=pal;
end
end
idtfout(entry)=palmatch-1; %idtf is zero based
end
diary idtfout.txt
idtfout
diary off
The u3d file is embedded in a pdf document using the following MiKTeX script.
\documentclass[a4paper,11pt]{article}
\usepackage[3D]{movie15}
\usepackage[UKenglish]{babel}
\usepackage[pdfpagemode=FullScreen, colorlinks=true]{hyperref}
\usepackage[paperheight=210mm,paperwidth=210mm,margin=0mm]{geometry}
\parindent0pt
\begin{document}
\pagestyle{empty}
\begin{center}
\includemovie[
controls=false,
autoplay,
mouse=false,
poster=william.jpg,
3Dlights=None,
3Djscript=william2.js,
]{210mm}{210mm}{william2-2bh.u3d}
\end{center}
\end{document}
Ideas for version 2.1, try doing a 3d reconstruction using only cameras 1 3 5 7 9 11 and see if that spacing produces acceptable results.
Ideas for version 3.0, space cameras around the head, with some above and some below, may need to buy more cameras. Work out how to use the manual focussing feature of CHDK and use that.
Framework 2.0
Cameras arranged in a quarter-circle, zoomed in to one point. The cameras were aligned by placing a teddy where William's head would be, and using a marker hanging from the ceiling as a guide.
Results 1.0
I must have accidentally trod on one of the usb cables, dislodging the connector slightly, so camera 4 stopped taking photos after the trial run. This is the best set of photos, which I then cropped in paintshoppro to remove superfluous data.
I uploaded the images to Arc3D, and got the following message back "DenseMatching: Failed to compute dense reconstruction. This may be due to a bad reconstruction, bad cameras or because the images were very large. Hint: try to use smaller images."
Ideas for version 2.0. Rather than use a wide angle, zoom in on William's head, in order to keep his head at a precise point, hold him up. Rearrange cameras into an arc with only small steps between each camera. Route cabling off the floor so it doesn't get stepped on. The shutter release system works well, although not all cameras are as focussed as they could be.
Data management.
The photos are downloaded onto the computer by manually re-plugging the usb cables from the remote-trigger into a usb hub. To reduce the possibility of information overload when dealing with hundreds of similar photos, I wrote a script to download all the photos from all the connected cameras and rename each photo to include the camera number, the focal length, and time of shot, this helps in making sure the set of photos you're working with actually belong together.
Most new cameras use WIA (Windows Image Acquisition) to connect to Windows, i.e. no drive letter or volume label is accessible, so rather than using a simple batch file, I used WIA from VBScript to access the serial number of each camera with WIA for renaming the jpegs appropriately, and to apply the appropriate correction for the lens distortion...
<job>
<reference object="WIA.DeviceManager" />
<object id="DeviceManager1" progid="WIA.DeviceManager" />
<object id="CommonDialog1" progid="WIA.CommonDialog" />
<script language="VBScript">
For Each oDeviceInfo In DeviceManager1.DeviceInfos
cameraName="NOTFOUND"
if oDeviceInfo.Properties("PnP ID String").Value = _
"\\?\usb#vid_04a9&pid_314d#996721449c18439fab1e115f71cacea9" & _
"#{6bdd1fc6-810f-11d0-bec7-08002be2092f}" then
cameraName="_Cam01"
elseif oDeviceInfo.Properties("PnP ID String").Value = _
"\\?\usb#vid_04a9&pid_314d#33c90f784d434636b48d78d93e20b26a" & _
"#{6bdd1fc6-810f-11d0-bec7-08002be2092f}" then
cameraName="_Cam02"
elseif oDeviceInfo.Properties("PnP ID String").Value = _
"\\?\usb#vid_04a9&pid_314d#e44d7ef3549c41848873fc02f2990694" & _
"#{6bdd1fc6-810f-11d0-bec7-08002be2092f}" then
cameraName="_Cam03"
' etc. for all cameras
else
MsgBox oDeviceInfo.Properties("PnP ID String").Value
end if
if cameraName <> "NOTFOUND" then
set Dev = oDeviceInfo.Connect
For Each Itm in Dev.Items
dim filesys
Set filesys = CreateObject("Scripting.FileSystemObject")
'either sdm or canon creates a folder which can't be copied
'so check that item is a jpeg first
if itm.Formats(1) = wiaFormatJPEG then
Set Img = Itm.Transfer
'A560 has 8 zoom settings, each setting requires a different barrel
'correction, so add the zoom details to the filename, also helps
'checking that all cameras are the same.
'parse the date to strip out colons which can't appear in file names.
'ie from 2008:09:15 11:12:44 to 20080915111244
fileDate = Img.Properties("DateTime").Value
fileDateOut = "_"&Mid(fileDate,1,4)&Mid(fileDate,6,2)&Mid(fileDate,9,2)& _
Mid(fileDate,12,2)&Mid(fileDate,15,2)&Mid(fileDate,18,2)
fileName = "C:\internet\3d\phototo3d\photos\" & _
Itm.Properties("Item Name").Value & cameraName & _
"_Zoom" & Img.Properties("ExifFocalLength").Value & _
fileDateOut & "." & Img.FileExtension
'savefile won't overwrite so check file doesn't exist first,
'this also avoids any unnecessary copying
if not filesys.FileExists(fileName) Then
Img.SaveFile fileName
Dim WshShell
Set WshShell = WScript.CreateObject("WScript.Shell")
'WshShell.Run command, 1 for normal focus, True for wait until finished
'fulla for barrel correction using PTLens data for A510, using approriate
'factors for each focal length -t 4 for 4 threads for quad-core.
Select Case Img.Properties("ExifFocalLength").Value
Case 5.8
WshShell.Run """C:\Program Files (x86)\Hugin\bin\fulla.exe"" _
-g 0.008099:-0.030970:0.000000:1.000 -t 4 "& fileName, 1, True
Case 7.889
WshShell.Run """C:\Program Files (x86)\Hugin\bin\fulla.exe"" _
-g 0.000000:-0.010876:0.000000:1.000 -t 4 "& fileName, 1, True
Case 9.817
WshShell.Run """C:\Program Files (x86)\Hugin\bin\fulla.exe"" _
-g 0.000000:-0.007821:0.000000:1.000 -t 4 "& fileName, 1, True
Case 11.759
WshShell.Run """C:\Program Files (x86)\Hugin\bin\fulla.exe"" _
-g 0.000000:-0.006082:0.000000:1.000 -t 4 "& fileName, 1, True
Case 13.74
WshShell.Run """C:\Program Files (x86)\Hugin\bin\fulla.exe"" _
-g 0.000000:-0.003660:0.000000:1.000 -t 4 "& fileName, 1, True
Case 15.783
WshShell.Run """C:\Program Files (x86)\Hugin\bin\fulla.exe"" _
-g 0.000000:-0.001945:0.000000:1.000 -t 4 "& fileName, 1, True
Case 18.353
WshShell.Run """C:\Program Files (x86)\Hugin\bin\fulla.exe"" _
-g 0.000000:0.000000:0.000000:1.000 -t 4 "& fileName, 1, True
Case 23.2
WshShell.Run """C:\Program Files (x86)\Hugin\bin\fulla.exe"" _
-g 0.000000:0.001312:0.000000:1.000 -t 4 "& fileName, 1, True
Case Else
MsgBox "No correction applied as zoom value not recognised for file:"_
& vbCrLf & fileName
End Select
end if
end if
Next
end if
Next
</script>
</job>
Lens distortion.
One drawback with using point and shoot cameras is that compromises are made in the design of the lenses, in particular at the wide angle setting you get noticeable barrel distortion as seen below. However this can easily be corrected when downloading the photos to the computer by running the program fulla, which is part of the hugin package.
The lens properties for the A560 are very similar to the A510 which was documented by PTLens before this program stopped being open source. Note each zoom value requires a different correction. There is negligible chromatic aberration (where sharp edges take on coloured fringes around the edges of the photo) in the A560 lens, but this could affect other point and shoot cameras and could similarly be corrected using fulla.
Framework 1.0
For supporting the cameras I used a polymeric hemispherical geodesic dome, yeah ok, it's a climbing frame for kids (it was a clearance item, try googling "Monkey Bar Deluxe Climbing Frame"). The first attempt uses 11 cameras placed around one side of the dome, the bamboo cane trial placement comes from watching too much CSI.
The camera mounts are simply two 1¼" toolclips bolted to a 3" angle, with the camera bolted to the other leg via it's tripod attachment which takes a ¼" UNC bolt (in hindsight a hex head would be easier to tighten than an allen, both for access and because you'll need a 3/16" allen key).
To simplify the background white sheets were zip-tied to the frame.
Camera setup.
Need to capture many images, for a statue or an adult who can keep very still this can be done with one camera, for a dynamic subject i.e. a baby this needs multiple cameras fired simultaneously. Most top of the range digital slrs feature remote firing, wheras no cheap digital cameras have this as a listed feature. However, most Canon digital cameras can be remotely fired using the Canon Hack Development Kit CHDK in particular the StereoData Maker SDM build. The CHDK is software saved on the camera's SD card and loaded into the camera's memory when it boots up. SDM enables the camera to take a photo by cycling the voltage to the camera's usb port. It is fairly straightforward to connect a dozen usb cables to a transformer and switch, From my box of bits I wired up three 4-port usb connectors as seen below.
Once the cameras are fixed to the framework it would be awkward to open them up to remove the SD cards for downloading and batteries for recharging, hence the cameras are powered by universal adaptors (these are much cheaper than Canon's official transformer) My A560's take a female 0.7mm jack 3V centre positive, but beware I've already burnt out the lens motors on one camera by accidentally using 5V, having forgotten to reset the universal adaptor after using it for the USB trigger.
What is the point of a 3D model.
To make a 3D PDF, <as shown above>
To make a hologram: Liti3D will produce a bespoke hologram from 181 images taken at 1 degree intervals around a subject. 9"x12" custom hologram - $149, 4.5"x6" custom hologram - $99. Once you have a 3D model, generating these images is straightforward, and easier than buying 181 cameras.
Laser engraving into glass: because of the setting up overheads would cost £300ish from e.g. Crystal Insights and 3D Crystal Creations
Background.
There are many methods to digitise the real world. Most commercial scanning is based on active methods as this is quicker and more precise. Active methods include, probes, lasers e.g. a hand held laser scanner, and structured light where dots or grids are projected using light or lasers.
For capturing a baby a non-contact passive method is essential. There are several methods to produce 3d models from unstructured photographs. The simplest technique, used by products such as iModeller Pro and 3D Software Object Modeller Pro, is to generate a solid shape which fits within the silhouettes from all the photos, and then project the texture from the photos onto the model. Advantage - simple, disadvantage - poor at modelling curves and cavities.
More advanced methods match salient points between paired photographs to generate 3d coordinates for the salient points. The resulting disparity maps between the image pairs are then combined into depthmaps for every image using a statistical algorithm that checks for consistency of matched pixels over multiple pairs. Quality maps indicating the confidence in the 3D reconstruction of every pixel are generated. The more consecutive views in which a pixel is matched consistently, the more confidence there is in its 3D position, hence the higher the quality measure for this pixel. Such software will also calculate the positions of the cameras, as this is seldom known.
A number of organisations provide software for this (list as of Oct 2008)
- Arc3D provide a free web-based service.
- Libmv will be available as a plugin for Blender, it is currently non-functional.
- Patch-Based Multi-View Stereo Software would be free, but he wouldn't send me a link as I'm not in academia.
- ShapeCapture is a £1500 commercial package.
- Photo-to-3D provide a free web-based service to generate a 3D model from 2 photos, and a service to model a head for £800 using 8 photos, and £1500 using 16 photos.