More on that SmugShot Metadata Mess

A few days after posting a J script that fixes EXIF timestamps in SmugShot images SmugMug changed the filename assigned to downloaded SmugShot images.

Originally the layout was:

SmugMugID_smugshot_SmugShotID.jpg

This layout was inconsistent with the way SmugMug normally handles files. Default SmugMug behavior is to leave filenames unchanged.  This contrasts with other photo sites, (you know who you are), that rename your sacred files to suit their indexing schemes. A few days ago SmugMug corrected this discrepancy and downloaded SmugShot images reverted to their original format:

smugshot_SmugShotID.jpg

This fix broke smugshotexif: see: my previous post. I have modified SmugShotFix.ijs to handle the current format and the older one.  I will keep tweaking this hack until SmugShot’s sloppy EXIF handling is corrected.

SmugShot Metadata Mess

SmugShot iPhone App Logo

While languishing in a series of hospitals and rehab centers recovering from a bad fall I have been amusing myself by taking iPhone pictures and posting them to SmugMug with SmugShot.

SmugShot is a nifty little iPhone app that uploads iPhone pictures and videos directly to your SmugMug galleries. SmugShot is a freebie so I cannot, in good faith, whine about missing features but SmugShot has one serious deficiency: it strips EXIF timestamps. Native iPhone jpegs contain EXIF timestamps but after passing through SmugShot only GPS information survives. I am not sure if this is by design or oversight but I don’t like it!

The fix is straight forward:

1. Download SmugMug metadata.
2. Download all SmugShot pictures.
3. Match pictures with metadata and use upload timestamps as date taken EXIF timestamps.
4. Finally replace the uploaded SmugShot images with the EXIF repaired versions.

Simple but tedious. The following J word manipulates Phil Harvey’s superb command line utility exiftool to cleanup the SmugShot metadata mess. The words used by smugshotexif are in the J script SmugShotFix.ijs. You can also find this script in the J files sidebar of this blog. Look in the J Scripts directory. Let’s hope the next version of SmugShot fixes the date time issue and hacks like this will no longer be necessary.

smugshotexif=:4 : 0

NB.*smugshotexif v--  inserts missing EXIF  metadata  in SmugShot
NB. iPhone uploads.
NB.
NB. The  iPhone  SmugShot  app  removes  EXIF  information during
NB. upload. This verb restores items I care about.
NB.
NB. dyad:  btclSmugMetaData smugshotexif clPathFile
NB.
NB.   NB. update TAB delimited SmugMug metadata tables
NB.   SmugTablesFrXml2 'c:\pd\docs\smugmug\data\smugheavy.xml'
NB.
NB.   NB. load SmugShot specific metadata
NB.   SMUGSHOTMD=: readsmugshots 0
NB.
NB.   SMUGSHOTMD smugshotexif 'c:\pictures\2011\Missouri\wip\1204618219_smugshot_9811805.jpg'
NB.
NB.   NB. fix all jpg files in a directory
NB.   (<SMUGSHOTMD) smugshotexif&> 1 dir 'c:\pictures\2011\Missouri\wip\*.jpg'

NB. extract SmugMug id from SmugShot file name
spid=. <extractsmugid y

NB. all smugshot pids
pids=. x {"1~ (0{x) i. <'PID'

if. spid e. pids do.

  NB. metadata exists for image insert items
  pos=. pids i. spid

  exif=. setartistcopyright y

  NB. use upload date for the missing original datetime
  date=. ;pos { x {"1~ (0{x) i. <'UPLOADDATE'
  date=. '-:' charsub date
  exif=. date setdate y

  caption=. ;pos { x {"1~ (0{x) i. <'CAPTION'
  exif=. caption setdescription y

  lb=. pos { x {"1~ (0{x) i. ;:'LATITUDE LONGITUDE'
  if. *./ 0 < #&> lb do.
    lb=. _999&". &> lb
    exif=. lb setlatlng y
  end.

  NB. rename as iphone file with original smugshot number
  NB. format path chars for windows rename
  newname=. '.jpg'&beforelaststr y
  newname=. 'iphone ',('_'&afterlaststr) @:(' [s'&beforelaststr) newname
  shell '/\' charsub 'rename ' , ; ' ' ,&.> dblquote y;newname,'.jpg'

  newname ; exif
else.
  'No SmugShot metadata for';spid
end.
)

Assigning SmugMug Print Size Keys

I believe that a picture’s aspect ratio,  its width divided by its height, should be tuned to its contents.  Consequently,  I am not a slave to standard aspect ratios and print sizes like the ones shown in the following table.

Aspect Ratio (Short/Long)	Print Sizes (Inches)
0.5	4×8, 5×10, 8×16, 10×20
0.7	3.5×5, 7×10
0.666666667	4×6, 8×12, 10×15, 12×18
0.714285714	5×7
0.772727273	8.5×11

Over the years I have clipped and cropped my pictures producing a large number of nonstandard ratios. This is not a problem until I decide to order prints.

Pictures sites like SmugMug offer many standard sizes but they cannot accommodate every crank with an aspect to grind.  To print a strangely shaped picture on SmugMug you have to open up the website’s print selection tool and fit one side or your picture to a standard print size.  This leaves part of the print blank and is often called the letterbox method.  Letterboxing is not ideal but it gets the job done.

For standard print sizes there is a much better way.  The following J verb takes the pixel dimensions of my images and computes standard, 300 DPI, print size keys like: 4×6, 5×7 and 8×10. After assigning print size keys you can select and print all standard sizes with a few mouse clicks.

smugprintsizes=:3 : 0

NB.*smugprintsizes v-- compute largest print size for given dpi.
NB.
NB. Computes the  largest  print size  (relative  to  DPI x)  for
NB. SmugMug images. Only images that have aspect ratios  close to
NB. the ratios  on  (SMUGPRINTSIZES) are associated  with a print
NB. size.
NB.
NB. monad:  st=. smugprintsizes btclImages
NB.
NB.   'albums images'=. readsmugtables 0
NB.   smugprintsizes images
NB.
NB. dyad:  st=. iaDpi smugprintsizes btclImages
NB.
NB.   200 smugprintsizes images

SMUGPRINTDPI smugprintsizes y
:
nsym=.s:<''

NB. reduce image table on PID
images=. }. y [ imhead=. 0 { y
pidpos=. imhead i. <'PID'
if. 0=#images=. images #~ ~:pidpos {"1 images do. 0 2$nsym return.end.

NB. compute print sizes table
pst=. printsizestable SMUGPRINTSIZES

NB. image dimensions short x long
idims=. _1&".&> (imhead i. ;:'WIDTH HEIGHT') {"1 images
'invalid image dimensions' assert -. _1 e. idims
idims=. (/:"1 idims) {"1 idims

NB. aspect ratio and print area (square inches)
ratios=. SMUGASPECTROUND round %/"1 idims
areas=.  SMUGAREAROUND round (*/"1 idims) % *: x

NB. mask table selecting images with ratio
masks=. (SMUGASPECTROUND round ;0 {"1 pst) =/ ratios
if. -.1 e. ,masks do. 0 2$nsym return.end.

masks=. <"1 masks
pids=.  s:&.> masks #&.> <pidpos {"1 images

NB. largest print area for selected images at current DPI
masks=. (1 {"1 pst) </&.> masks #&.> <areas
pids=.  (<"1&.> masks #"1 &.> pids) -. L: 0 nsym
sizes=.  <"0&.> 2 {"1 pst
; |:&.> ; pids ,: L: 0 (# L: 0 pids) # L: 0 sizes
)

Setting Good SmugMug Key Words

Finding good key words is not easy.  In many ways it is like creating a good book index.  Decent book indexes are carefully constructed by human readers that understand what is being indexed and why certain terms should be included. Superior indexes showcase the gems and bury the garbage. There is a lot of mediocre software out there that purports to automate this task but I remain unimpressed.  Machine indexing is like machine language translation: they both suffer from a lack of real understanding. I was reminded of how difficult indexing is while writing some code to update my SmugMug key words.

My first attempt at computing key words followed this recipe:

1. Run my little C# SmugMug metadata dumper to update image metadata.
2. Sift through the image metadata and extract all current key words.
3. Remove common English words from current key words.
4. Similarly, extract all image caption text and remove common English words.
5. Sort the remaining caption text key words by frequency.
6. Append the frequency sorted caption key words to corresponding current key words.
7. Take at most seven words from the appended list as key words.

My thinking was high frequency uncommon English words would make good keys.  This is generally the case but the removal of common English words from currently assigned keys was a big mistake.

I take care when naming image files.  SmugMug picks up words in image file names and uses them as default key words.  If you have good file names you will get useful keys automatically.  Removing common English words from file names deleted words that were present for good reasons.  For example: two common English words, “before” and “after,” are used in the file names of image restorations like this picture I took in Cyprus way back in 1968.

I think “before” and “after” are perfectly good keys for this image.

To avoid such problems I now leave file name words intact and augment these words with high frequency caption text words and print size keys like: 4×5, 4×6, 5×7 and 8×10.  Print size keys is another story. You can view my entire list of SmugMug keys here.

More on SmugMug Duplicates

In my previous post I described a method for eliminating duplicate SmugMug images based on MD5’s.   This method does not get all the duplicates.

If you uploaded the same image to different galleries,  at different times,  the SmugMug MD5’s may differ.  I don’t know what  SmugMug  rolls into their MD5 ‘s but I suspect it’s more than image data.

To get all the copies you must remove duplicate picture ids ,  file names and MD5’s.  Furthermore, to maintain a pure duplicate free state,  you need to check these items often.   Now,  after uploading or rearranging pictures,  I update my SmugMug metadata and execute this J verb to insure I don’t introduce duplicates.

CheckSmugDups=:3 : 0

NB.*CheckSmugDups v-- checks duplicate SmugMug images.
NB.
NB. monad:  CheckSmugDups uuIgnore

'albums images'=. readsmugtables 0
images=. }. images [ imhead=. 0 { images

NB. images should be unique in three ways:
r=. ,: 'PID unique: '; # ~.(imhead i. <'PID') {"1 images
r=. r, 'MD5 unique: '; # ~.(imhead i. <'MD5') {"1 images
r=. r, 'FILENAME unique: '; # ~.(imhead i. <'FILENAME') {"1 images

if. 1 <# ~. ;{:"1 r do. smoutput 'duplicates present' end.
r
)