Faculty News: Falsified Truths
RITphoto asked Prof Christye Sisson to share the work she and Prof. Ted Kinsman have been working on since June 2016.
Falsified images have long been a part of photographic history. From spirit photography in the late 1800’s, to Stalin retouching out his deputies once they had fallen out of favor, photographs have been manipulated for countless reasons. More recent examples include composites of convenience, such as when National Geographic moved the Pyramids of Giza closer together so they would fit on a vertical cover in 1982. But with the recent advent of “fake news” and the sheer number of imaging devices in the hands of citizen journalists and combined with the ease of image manipulation, images in the digital era have lost credibility as public trust in image truths has eroded. Additionally, government and forensic agencies must often rely on third party experts to analyze images brought in as evidence. Many media forensic algorithms require expert application and “triage”, to determine what algorithm might best be applied for a particular image to confirm its veracity.
To combat many of these issues, the government embarked on a large-scale project with several components. I am the PI or Principal Investigator for RIT’s Project Medisphere. This project’s focus is directed at creating a single, online tool for use by government and law enforcement agencies to detect if an image (or video) has been manipulated or altered. This would include a determination how it has been altered. The project (called MediFor, short for Media Forensics) involves many universities, private organizations, and contractors from all over the world.
RIT and the Photographic Sciences’ role is unique in this project, in that we are providing all of the manipulations for the other teams to test their algorithms of detection. The project currently employs eight undergraduate students for approximately 5-10 hours per week. Five of those students are from the photo sciences program, two are from motion picture sciences, and one is from the advertising photo program. We are also working with the imaging science department at RIT to assist the students to help our cover their tracks” to replicate what real life manipulators with the intent to deceive might attempt.
This past summer (2017), we hired 13 students as full-time coop students. These students came from photo sciences, imaging sciences, motion picture science, and media arts and technology. We are planning on something similar for the summer of 2018. FYI, if you are a student in these majors, or a computer sciences student and interested in this work during the upcoming academic year or summer, I want to talk to you!
The students researchers have two primary responsibilities: creating (capturing) HP data and journaling their work. HP stands for “high provenance”, where the student photographs or records video of a specific subject under stringent capture guidelines. The equipment “cage” in the School of Photographic Arts and Sciences is perfectly suited for this work with its incredible variety of equipment. This inventory allows the project to have a wide selection of image types with numerous capture parameters. The students then ingest the images/video in through a specific process to ensure the metadata is not appended. Having these images as a starting point ensures we can know the exact provenance of all the images that are being used for manipulations going forward.
The students spend the large majority of their time journaling. This is the term we use to describe creating and chronicling image manipulations. They perform the manipulation using HP sources and image processing tools such as Adobe® Photoshop, GIMP®, Adobe® Premiere and AfterEffects or some other editor. They then use a tool created for the project to journal each step they used to create the new image. This tool creates a visual map with masks of each step that acts as a “cheat sheet”. The whole file is then uploaded back to the same browser, where the students can instantly see how the manipulation fared against some standard algorithms of detection. This feedback allows the students to instantly see how well they did (or didn’t do) in escaping detection. Our overarching task is to create manipulations of a certain number, type and category; ranging from challenging to rudimentary; the goal is to mimic what is being observed in real life.
The black and white image displaying a yellow mask is indicative of what we know was manipulated.
Green colorization indicates correct identification of manipulation
Red colorization falsely identified as manipulation
Blue colorization missed manipulation
It’s actually sort of fun pretending to be the bad guys but we use our skills for good. We hope we are making it harder for the bad guys to get one step one of these algorithms as they improve their skills. We also conduct specific experiments to challenge algorithms we know are good at one particular thing. For example, if an algorithm is looking for specific cues based in the physical world (such as shadows or reflections following the laws of physics), we might start our manipulations by subtly breaking those laws and pushing the boundaries to see when it detects manipulation. By challenging our undergraduates with this work, we’re setting a new standard for undergraduate research at RIT in our community. Our school is well suited for this project and I am very proud of the work we’ve done so far.
About the Author:
Christye Sisson joined the Biomedical Photographic Communications department in 1997 after working as an ophthalmic photographer in Boston, Massachusetts and in Rochester. Christye holds a Master’s Degree in Information Technology, a Bachelor of Science in Biomedical Photographic Communications, and is a Certified Retinal Angiographer. She was appointed chair of the newly formed Photographic Sciences department (formerly Biomedical Photographic Communications and Imaging and Photographic Technology programs) in 2010. Christye is an Associate Professor, and teaches various courses including Ophthalmic Photography I and II, Vision and Perception in Imaging, and Careers and Professional Practices courses for the Photographic Sciences department.
Christye holds a Visiting Faculty appointment at the Flaum Eye Institute, part of the Department of Ophthalmology at the University of Rochester Medical Center. She is involved with a number of image quality and color in fundus imaging projects, including Tele-I-Health, a diabetic retinopathy screening program. Christye is also the project director of the Color Eye Model project, a sub-group of the Medical Imaging Working Group with the International Color Consortium (ICC), co-founded by the FDA and ICC.
Christye is active in the Ophthalmic Photographer’s Society and has hosted the OPS’s CRA (Certified Retinal Angiographer) Exam, acting as both site coordinator and examiner. She is a past member of the OPS Board of Certification, which oversees the certification and administration of the Certified Retinal Angiographer’s exam and activities. Christye oversees all aspects of the ophthalmic activities and equipment that the department maintains, and is also a faculty advisor for the Photographic Sciences Club.