Ed Schlotzhauer

An artists journey

Category: Technology

Ideas about the mechanics, techniques, and technology behind image making.

How We Get Color Images
Have you ever considered that that great sensor in your camera only sees in black & white? How, then, do we get color images? It turns out that there is some very interesting and complicated Engineering involved behind the scenes. I will try to give an idea of it without getting too technical.
Sensor
I have discussed digital camera sensors before. They are marvelous, unbelievably complicated and sophisticated chips. But they are, still, a passive collector of photons (light) that falls on them.
An individual imaging site is a small area that collects light and turns it into an electrical signal that can be read and stored. The sensor packs an unimaginable number of these sites into a chip. A “full frame” sensor has an imaging area of 24mm x 36mm, approximately 1 inch by 1.5 inch. My sensor divides that area into 47 million image sites, or pixels. It is called “full frame” because that was the historical size of a 35mm film frame.
But, and this is what most of us miss, the sensor is color blind. It receives and records all frequencies in the visible range. In the film days it would be called panchromatic. That is just a fancy word to say it records in black & white all the tones we typically see across all the colors.
This would be awesome if we only shot black & white. Most of us would reject that.
Need to introduce selective color
So to be able to give us color, the sensor needs to be able to selectively respond to the color ranges we perceive. This is typically Red, Green, and Blue, since these are “primary” colors that can be mixed to create the whole range.
Several techniques have been proposed and tried. A commercially successful implementation is Sigma’s Foveon design. It basically stacks three sensor chips on top of each other. The layers are designed so that shorter wavelengths (blue) are absorbed by the top layer, medium wavelengths (green) are absorbed by the middle layer, and long wavelengths (red) are absorbed by the bottom layer. A very cleaver idea, but it is expensive to manufacture and has problems with noise.
Perfect color separation could be achieved using three sensors with a large color filter over each. Unfortunately this requires a very complex and precise arrangement of mirrors or prisms to split the incoming light to the three sensors. In the process, it reduces the amount of light hitting each sensor, causing problems with image capture range and noise. It is also very difficult and expensive to manufacture and requires 3 full size sensors. Since the sensor is usually the most expensive component of a camera, this prices it out of competition.
Other things have been tried, such as a spinning color wheel over the sensor. If the exposure is captured in sync with the wheel rotation then 3 images could be exposed in rapid sequence giving the 3 colors. Obviously this imposes a lot of limitations on photographers, since the rotation speed has to match the shutter speed. A real problem for very long or very short exposures or moving subjects.
Bayer filter
Thankfully, a practical solution was developed by Bryce Bayer of Kodak. It was patented in 1976, but the patent has expired and the design is freely used by almost all camera manufacturers.
The brilliance of this was to enable color sensing with a single sensor by placing a color filter array (CFA) over the sensor to make each pixel site respond to only one color. You may have seen pictures of it. Here is a representation of the design:
Bayer Filter Array, from Richard Butler, DPReview Mar 29, 2017
The gray grid at the bottom represents the sensor. Each cell is a photo site. Directly over the sensor has been placed an array of colored filters. One filter above each photo site. Each filter is either red or green or blue. Note that there are twice as many green filters as either red or blue. This is important.
But wait, we expect that each pixel in our image contains full RGB color information. With this filter array each pixel only sees one color. How does this work?
It works through some brilliant Engineering with a bit of magic sprinkled in. Full color information for each pixel is constructed by interpolating based on the colors of surrounding pixels.
Restore resolution
Some sophisticated calculations have to be done to calculate the color information for each pixel. This makes each pixel end up with full RGB color values. The process is termed “demosaicking” in tech speak.
I promised to keep it simple. Here is a very simple illustration. In the figure below, if we wanted to derive a value of green for the cell in the center, labeled 5, we could average the green values of the surrounding cells. So an estimate of the green value for cell red5 is (green2+green6+green8+green4)/4
From Demosaicking: Color Filter Array Interpolation, IEEE Signal Processing Magazine, January 2005
This is a very oversimplified description. If you want to get in a little deeper here is an article that talks about some of the considerations without getting too mathematical. Or this one is much deeper but has some good information.
The real world is much more messy. Many special cases have to be accounted for. For instance, sharp edges have to be dealt with specially to avoid color fringing problems. Many other considerations such as balancing the colors complicate the algorithms. It is very sophisticated. The algorithms have been tweaked for over 40 years since Mr. Bayer invented the technique. They are generally very good now.
Thank you, Mr. Bayer. It has proven to be a very useful solution to a difficult problem.
All images interpolated
I want to emphasize a point that basically ALL images are interpolated to reconstruct what we see as the simple RGB data for each pixel. And this interpolation is only one step in the very complicated data transformation pipeline that gets applied to our images “behind the scenes”. This should take away the argument of some of the extreme purists who say they will do nothing in post processing to “damage” the original pixels or to “create” new ones. There really are no original pixels.
I understand your point of view. I used to embrace it, to an extent. But get over it. There is no such thing as “pure” data from your sensor, unless maybe you are using a Foveon-based camera. All images are already interpolated to “create” pixel data before you ever get a chance to even view them in your editor. In addition profiles and lens corrections and other transformations are applied,
Digital imaging is an approximation, an interpretation of the scene the camera was pointed at. The technology has improved to the point that this approximation is quite good. Based on what we have learned, though, we should have a more lenient attitude about post processing the data as much as we feel we need to. It is just data. It is not an image until we say it is, and whatever the data is at that point defines the image.
Black & White
Our digital cameras all have a color filter array over the sensor to enable color image information. But if you create black & white images, you then have to convert the pixels back to monochrome. That seems useless extra work.
But think about it. We have much better control converting a color image to black & white, because more tonal information is available. We can create better separation of tones by using the available color data.
Some die hard black & white photographers have the color filter array removed from their sensor. This converts the camera into a true high resolution black & white device, but the photographer now must use color filters at exposure time to help with the tone separation, just like in the old film days.
It seems better to me to take all the color data into our image processing software and do the conversion there. We are not having to guess what filtration to use in the field. But that’s just my opinion, and what the vast majority of digital photographers do.
The image
I chose the image at the head of this article to illustrate that the Bayer filter demosaicking and other image processing steps gives us very good results. The image is detailed and with smooth, well defined color variation and good saturation. And this is a 10 year old sensor and technology. Things are even better now. I am happy with our technology and see no reason to not use it to its fullest.
Feedback?
I felt a need to balance the more philosophical, artsy topics I have been publishing with something more grounded in technology. Especially as I have advocated that the craft is as important as the creativity. I am very curious to know if this is useful to you and interesting. Is my description too simplified? Please let me know. If it is useful, please refer your friends to it. I would love to feel that I am doing useful things for people. If you have trouble with the comment section you can email me at ed@schlotzcreate.com.
March 2, 2022
Craft Completes Magic
Craft completes magic. I read this in a book on writing poetry by Robert Wallace. This was a new thought to me. It is unusual in my world for a random phrase to seem to crystalize immediately as truth. This did. I have often written about the 2 sides of art as being the creative, the magic, and the technical, the craft. I love the way this brings them together and completes the whole.
The magic
Oftentimes we artists focus almost exclusively on the creative aspects of what we do. After all, we think this is what separated us from other artists. And to a large degree, it is true.
So we look at the work of others we admire. We plan or write or set projects to focus our thoughts. We look for the new and different. The driving challenge is how can we bring a unique perspective to the things we see in the world.
Sometimes the muse visits us and we feel we have truly made magic. It is a great feeling. Creativity breeds creativity. We try to go on to leverage this new stage into even more.
But, have you ever had a guilty feeling, looking at your new creative work, that it could have been executed better? Not necessarily more creatively, but with better craftsmanship? Sometimes we don’t know how to make our great idea into a finished work of art. Concentrating too much on just one aspect can throw us off balance.
The craft
I believe our craftsmanship is as important as our creativity. Not a replacement, but to balance and complete our work. It’s this completion I want to emphasize.
There are 2 tendencies I see in a lot of photographers that disturb me. Some seem to feel that a technically perfect image is a good image. Some others take the attitude that “I’m a creative, I don’t know the ‘techie’ stuff”. I believe that either of these, if they drive your behavior too much, lead to bad ends.
Ansel Adams famously said “There’s nothing worse than a sharp image of a fuzzy concept.” This, to me, is the danger of overemphasizing technical perfection. I see this a lot in online critiques where the objections are things like not enough depth of field or that the color correction may not be completely true to the original scene. The reality in many cases is that no amount of technical improvement is going to give this image life.
If you don’t have an emotional connection with the scene and a definite point of view to share, then it isn’t going to get great by technical skill.
On the other hand, it frustrates me to hear even professional photographers dismissively say they don’t do “tech”. Sorry, but photography is a uniquely technical art form. If you don’t understand and appreciate and know how to control the technical aspects you are at a severe disadvantage. You can end up with images that show a great idea but you were unable to produce a gallery-worthy image.
The whole
There is a symbiotic relationship between the creative and the craft. Mr. Wallace, who I quoted at the start, related it to the two legs of a runner. The creative leg propels you forward. Then the craft leg helps you bring it into being, which also thrusts you forward to another level. These work together, alternating, each with strengths to add. Neither is complete without the other.
A comedian doesn’t just walk out on stage and think up funny things. He spends many hours on each skit, refining and rehearsing and tuning it before you ever hear it. Likewise, a magician spends countless hours working on an illusion to make it smooth and believable, to make the magic happen. A musician practices day in and day out for years to get and stay good. Yes, famous musicians still practice scales. It trains their technique.
Art is hard work. It is hard to do creative things and it requires great skill to make it real. No one can tell you what you can or can’t do, or how you should do your art. But I believe that if we don’t put in as much work on the craft side of our art as on the creative we will never achieve what we could.
A boring image will never be great because it was technically perfect. On the other hand, you don’t get a free pass to ignore the craft because you are a “creative”. As the initial quote says, craft completes the magic.
February 23, 2022
What’s the Noise?
Noise in digital imaging. Is it a problem? Is it part of the art? Should you be concerned? Are there exotic techniques you need to learn to eliminate the noise?
Fear
Many of us have been taught to fear noise. So much so that I know people who would pass up a great shot because the image might be noisy. The fear of noise is an irrational, superstitious fear.
This seems most common in the landscape photography community. Conventional wisdom, and the teaching of most instructors, says we should always shoot at the “native” ISO of the camera for lowest noise. That would be ISO 64 for my camera. If we don’t, we are increasing the noise in our images and that would be a “bad thing”.
Have you ever examined the feared noise yourself? Do you understand what it is and what effect it has on prints? Have you confronted the monster and stared it down?
Noise Technology
The digital sensor is an amazing piece of technology. It has a HUGE grid of photo-receptive sites. My Z7 has nearly 46 million pixels. A strand of the smallest human hair would cover at least 14 of these pixels. No wonder I see dust spots!
Did you know that “digital” imaging is actually an analog process? Each receptor “adds up” the number of photons it receives for a frame. This is a scalar value, an analog signal. Each site is read out to an amplifier and analog-to-digital converter where it is transformed into a digital value. The amplification is determined by the ISO setting – dialing in a greater sensitivity corresponds to more amplification. This amplification is one source of noise. Any time you take a very low level analog signal and amplify it, some noise is inescapable.
But in addition, the sensor chip itself contributes noise. There is a phenomenon called “shot noise”. It is beyond my ability to explain simply, but electrons spontaneously generate noise. It is usually low level and it is temperature dependent. This is why your camera probably has a long exposure compensation mode. It is there to reduce this background noise accumulated over a long exposure. During a long exposure the sensor is powered for long enough to raise its temperature, leading to increased noise. The compensation mode takes another exposure, but with the shutter closed. This reads the background noise level with no light coming in. The camera basically subtracts the noise signal from the original image. It does a pretty good job.
The amplifier and the sensor noise generation are 2 significant sources of noise in digital imaging. Not the only ones, but they are big. Keep in mind that most of the writers you will see do not have a significant technical background. They sometimes give bad advice because they do not really understand what is happeningl
What does the dreaded noise look like?
Take an image with the ISO cranked up pretty high, say 12,800. Look at it at 1-to-1 magnification in your editing software. You will see that it looks kind of like blotchy sandpaper. You are seeing the 2 primary types of digital noise: luminance noise and color noise.
Luminance noise looks kind of like the grain we used to see in fast black and white film. Some people like it and it does add an interesting texture to some images. Color noise is the mottled color patches you may see at high magnification. I don’t know of anyone who likes that. Both of these types of noise can be compensated for significantly by your editing software, like Lightroom Classic.
Let me point out that noise is just a part of the image capture. It is not something that, when they see it, the authorities kick you out of the gallery or revoke your artist’s certificate. You have an artist’s certificate, right? 🙂
Noise Techniques
Noise is part of the technology we deal with because we do digital imaging. We need to be aware of it and know how to deal with it, if it is a problem for us. I mentioned amplification noise and sensor noise.
To minimize noise, keep the ISO low, keep the sensor cool, and minimize long exposures. Simple. But what if that doesn’t work?
I will use me as an example. I often shoot in low light, sometimes hand held with no tripod. And I often use long exposures. Am I doomed?
Here are the decisions I generally make: if I want the image to be free from blurring caused by shake, I up the ISO until the shutter speed is at about 3x the focal length. Yes, conventional wisdom is 2x, but I find that does not work well for very high resolution sensors, even with great image stabilization. If I want a long exposure for the creative effect, I use it. Noise is not a significant consideration most of the time. And, unconventionally, I usually leave the long exposure noise compensation off.
Let me address that last one. Why would I leave the long exposure noise compensation off? The noise here is made worse by sensor heating. I shoot a lot in Colorado. Unless it is mid summer, the sensor stays pretty cool. As a matter of fact, it is often more of a problem keeping the battery warm enough to not shut off. And even in summer, it is not a problem to wait a few seconds between shots to let it cool. There have been a few times where I have gotten in trouble with this, but very few.
So what?
For me, noise is just a part of the creative balance. Sometimes I want to minimize it, sometimes I actually want to introduce it. Even for the vaunted landscapes, noise can introduce a welcome texture at times, maybe to give a grit for effect or to give subtle interest to a featureless sky.
I do not fear noise. ISO 400 is my default setting – that is 3 stops over the camera native ISO. I do not mind going to 3200 or 6400 if I need to as a tradeoff to capture what I want.
With my camera it is hard to discern noise at 400. There is not much to find at 1600. I admit, my old training to favor the lowest ISO sometimes interferes with my artistic judgment. I try to fight it.
The image with this article was shot at ISO 6400, with what is, as of this writing, an 8 year old sensor. I’m not ashamed of the noise. It I didn’t shoot at 6400 I would not have gotten the image. Good tradeoff, to me.
Noise is a traditional part of photography. It is a feature that sets it apart from painting. Black & White photography favored grain for a gritty look. Many artists like the effect. Even the ones who may not use it recognize and accept it as part of the medium. Digital noise is the equivalent.
So don’t fear noise. Accept it. Use it where you can. Understand where it comes from and what control you have. Your editing tools have ways to reduce it, and they do a pretty good job. Luminence noise is not exactly the same thing as grain, but the overall effect is not that different. Perhaps always totally eliminating noise is not the goal. Did you ever see the slider in LIghtroom Classis in the Effects panel to increase grain? Play with it sometime. It may add to your artistic vision.
December 8, 2021
The Paint is Never Dry
I find there are 2 categories of images in my library: ones I am “done” with and ones I want to tweak each time I open the file. Furthermore, it seems the ones I want to do something to each time I see them are the ones I like best. I refer to this as the paint being never dry.
A significant advantage of digital image manipulation is that it is so easy to make changes. This can also be a problem.
Wet paint
Modern technology gives us great freedom to edit and express ourselves. It is so easy to make some changes every time we open the file. Oh, I didn’t see that little flaw. I really don’t like the relation of these tones now that I look at it again. Maybe it would have more punch if I pumped some of these colors some.
But this is a subtle trap. A trap of time, because this is a never ending treadmill of editing, and of lack of confidence. I will write about this confidence problem in the future. Basically, it has been hard to accept that, as an artist, no one can tell me what is “right” or when I am “done”. I am the only one who can decide.
Prints
I wrote once about prints being a frozen moment in time. This is one of the great things about prints. They are not changeable.
A print represents my interpretation of the image at one moment in time. It is very tempting for me to modify it a little every time I print it. But now that I do editions of prints, I have to discipline myself to create exact duplicates for each print in the edition. It would be dishonest and a disservice to the purchasers if each one was different.
Part of the process of growth is deciding that an image is “done” and is ready to be shown and purchased . And I have to be able to stand proudly and represent it as my art, that I am proud of, even if I see opportunities for improvement.
Creative vision
But my creative vision is evolving all the time. It is frustrating to be locked in to printing a series a certain way when I may see it different now. I am resolved, though, that that is the requirement. I will have to exercise my creativity on new images.
The images are my children, in a sense. But any parent finds out that after they grow up, you have to let them go. Send them on their way to be independent. I can no longer control them or manage them. Kind of the same with my images. When one is sold, the whole edition is frozen, out of my control.
Oh, but the new images, the ones that haven’t sold yet. They are free to be interpreted and re-interpreted at will. I love to do this, but I recognize the need to let the paint dry at some point.
As the artist, all my images are resources to me to use any way I wish. Even the editioned ones can be recycled by compositing, over-painting, or radical cropping. Anything that makes it into a whole new work of art. My creative vision can best be applied to new work rather than reworking old things.
Growth
I don’t believe doing a great image “uses me up”. I have to believe I have a boundless well of creativity. It is better to go out and create new work. Learn what I can from the best of what I have done and go on from there. Explore a theme and do variations. Discover new themes.
My curiosity will lead me to new subjects, new visions for old ones, new points of view. I will learn new techniques for shooting and processing.
It would be devastating to feel that my best work is already done. I would have to quit if that were the case. I feel sorry for the old rock bands who still tour. No one wants to hear their new work. They only want to hear the hits of 40 years ago. They are trapped. I couldn’t do it.
So, yes, my tendency is to want to constantly rework and tweak everything. I often see things I would change in my work. But discipline has to be applied. Most old work should be left as a memory and a signpost along the way of my journey. Apply the creativity to the new images. Let the paint dry.
A confession: even after writing all this about letting the paint dry, I went back and did some minor edits on the image with this article. This is an old image, scanned from film. The quality is not up to today’s standards. But I really like the feeling of the image and the memories it brings back of Chartres Cathedral in France. So I indulged myself in one more little tweak. Do what I say, not what I do.
November 10, 2021
Is It Interesting?
I find myself pondering this question a lot these days. More and more I believe the answer to “is it interesting?” overrides many considerations of composition and technique. This is a personal judgment, of course. as is the question of what is interesting.
Learning
Art is almost as much about our training as it is about our natural creativity. We all start somewhere, whether we have formal training or we are self taught. When we are learning a skill or an art we concentrate on the mechanics first.
The tendency is to focus our attention on what we are trying to master. This is natural. What we should recognize, though, is that we may not really be making art in the process. Yes, it is art in the sense that we create it as art, but it is not a mature and well rounded style yet.
Technique
Photography is possibly the most technical of the normal arts. We have to master many layers of technology to get skilled at the craft. There is the camera with its hundreds of settings and controls, each of which may help us make a great image or a terrible one. Then there is the computer system required to store and process the image. And the software we choose to use for managing and editing the image. If you are taking it all the way to the end of the chain, there is the whole printing process to learn.
Each of these areas is a huge field that could require years of study to master.
If this is where you are, plow into it. Work through the learning process. Get to the point where the camera is a comfortable tool that you can use with little thought. Ideally you should be able to adjust all the major setting in the dark, just by feel.
The image processing software is probably an even bigger challenge. Photoshop is one of the deepest tools I have ever used, and that is from the point of view of a long career in very complex software development. There are only a few people in the world I know of who totally “know” Photoshop. Julianne Kost comes to mind, but then she is the chief Photoshop evangelist for Adobe. It is her full time job to be able to train people on any aspect of it. Others at about that level are Ben Willmore and Dave Cross. I study and use Photoshop hours a week but I will never get to their level.
But the good thing is, I don’t have to be a Ben Willmore. As long as I know enough to realize my artistic vision, I’m OK. I know of excellent and successful photographers who I consider to have only a rudimentary knowledge of the tools. They know enough to do what they want to do. I personally can’t be happy unless I feel I have mastered my tools enough to comfortably use them as an extension of my creativity. So I study a lot. But that is just my own burden.
It should be about creating interesting art, not our ability to use the tools.
Composition
The next major pillar of image making is composition. It is another thing that can become a lifelong study in itself. We can burrow into art history, visual theory, Gestalt psychology, and all manner of ideas and opinions.
We start with only an intuitive feel for good composition, based on art we have seen and our inherent notions of what we like. Probably we cannot express in words what good composition is. As we study and practice we get to where we have a more formal view of it. We can critique our own or other images in terms of their design. Eventually, we can compose our images intuitively, without much conscious thought. We can repeatedly produce compositions that please us.
Keep in mind that most of this time, we are producing images that are now technically “correct” and have “good” composition. But maybe nobody wants to look at them yet.
Is it interesting?
This idea was clarified for me in a book about poetry. (Writing Poems, Robert Wallace. The link is for a later edition of the book) Weird, huh? It is a book about writing poetry rather than a regular book of poems. I find hints and ideas to improve and better understand my art from all sorts of diverse sources.
The author made the statement that if the poem is not interesting, what good is it? It can have wonderful form, metaphor, irony, symbolism, etc., but if it is not interesting, no one will read it.
I believe there is something here to apply to our art.
I have seen, and made, too many technically perfect, classically composed images of … nothing memorable. While I value sharp, well executed images, and pleasing compositions with flow and leading lines and great light, I have come to realize that is not enough by itself to really be art. This is, of course, just my personal opinion. But then all art is a personal opinion. 🙂
When you have mastered the basics I suggest you first visualize something that will make a memorable image. Then use your acquired skill to capture it perfectly. Don’t just work on technique. You’re better than that.
October 27, 2021