LumoLabs

Whitepaper
1/16

Testing the AF sensor array

– Whitepaper / Technical Study –

Falk Lumo [LumoLabs (fl)]

Version: Sep 13, 2012, v1.0

Status: Final
Print version: [Link]/lumolabs/articles/D800Focus/[Link]
Publication URL: [Link]/lumolabs/articles/D800Focus/[Link]
Public comments: [Link]/2012/09/[Link]

This technical study describes a method for checking if a camera's entire AF sensor array works as
advertized. It can be used for almost any camera with a phase AF system, i.e., SLRs and SLTs. In
fact, this paper contains examples for two Nikon and one Canon camera.
A main purpose of this paper is to check if a Nikon D800, D800E or D4 are working as advertized.
More about this has previousy been published here:
● [Link]/lumolabs/articles/D800Focus/[Link]

1. Background
Unfortunately, such a check seems to have become suitable after it is now possible that many such
cameras manufactured so far (Aug 2012) suffer from bad functioning left AF sensors. Especially as
Nikon remained silent about the issue and returned many cameras sent to service “as fixed” while
further inspection revealed they're not. Furthermore, without a statement from Nikon there remains
substantial doubt that a camera “fixed” in service (e.g., calibrated for every AF focus point sensor)
performs as well as a camera which was manufactured according to standards in the first place.
There are more recent rumors that Nikon may meanwhile have dispatched a service procedure to
more reliably fix the issue in service.
Therefore, such a check may now be advisable with multiple lenses. Maybe, Nikon could use our
method with multiple lenses to document the performance of a camera after servicing for affected
customers. Last but not least, the protocol from this method may be a good measure to preserve the
value of early D800 on the used market place.

2. Reikan Focal
In order to make the testing procedure published in the article cited above
(/lumolabs/articles/D800Focus/[Link]) as automated, meaningful and quick as possible,
 LumoLabs
Whitepaper
2/16

Rich (the author of Reikan's FoCal software) and myself decided to work together and enable
buyers of SLRs to check their cameras if in doubt.
I fully endorse the FoCal software and Rich deserves any support from us so he can continue to
make his software even more powerful. Note that I have no benefit in making this statement but the
purchase of FoCal Pro was the best purchase I did in a long time.
FoCal is an automated software which via USB determines the optimum AF manual adjustment
(AFMA) aka AF fine tune parameter for a given lens. In order to check the full sensor array you'll
need the Pro version as it includes the Multi Point Focus Test tool
([Link] It currently sells for
70 GBP (cheaper versions are available but they lack support for the array test). If your camera is
affected by the defect, ask Nikon for a refund to make them feel guilty at least ;) After all, they
claim that only a small number of D800s are affected which definitely is a false statement (only
explained by the fact that most customers don't return their camera in lack of a procedure to reliably
fix the defect or exchange for an unaffected one).
If you're already a customer, you need to update to version 1.6.0 (or later). This version and its
features are described here:
● [Link]/focalweb/[Link]/2012/08/focal-1-6-released
In the following, I am going to describe how to use this software for the given task and what results
to expect.

3. Best practice
The following is a best practice method which I also had in mind when working together with Rich
to make version 1.6 happen. Some people have reported issues to get version 1.6.0 to work
properly. I didn't notice any significant issues though. I follow the following steps:
1. Purchase (or update) FoCal Pro 1.6+ ([Link]).
Note: you can register for up to five camera bodies and change registration up to 10 times.
So, you only should need to pay once even if shuffling bodies ;)
2. Print out the target image (FoCal_MultiFocus_Target_1r1ns.pdf (or .png)).
Note: Print out 3 or 12 copies if using 8x11” or Din A4 paper, then cut away the blank
borders, then glue a row of 3 sheets (in portrait orientation), or two rows of 6 sheets (all in
portrait orientation) resp., onto a wall or board. Inkjet is better but laser does ok if
illuminated from the sides.
This gives you a target roughly appropriate to test at a distance of approx. 25x (3 sheets) or
50x (12 sheets) the focal length. 50x is the common vendor recommendation (if you have to
fight your case) but I found that 25x roughly gives the same results for the given problem.
The following is a snapshot from a possible setup using 12 sheets of the target:
 LumoLabs
Whitepaper
3/16

Use FoCal's “Target Setup” feature to adjust distance such that the full target just fills the
area covered by the AF sensor points (the red shaded area in the target setup window). Of
course, some care should be applied to have the target be perpendicular to the camera lens
axis. One method to make sure it is, is to fix a small mirror to the center of the target and
verify that the lens appears in its middle when watching live view. Also, try to illuminate the
target from both sides to avoid reflections (halogen does just fine). You can enter the light's
color temperature into FoCal if you're a perfectionist.
3. Open FoCal, select camera, open “Show Tools”, then “Multi Point Focus Test”.
Select the AF points you want to test for (I usually select five points: the outermost, the
overnext neighbors towards the middle, and the center point). Make sure to include the
center AF point. Use “Target Setup” to verify that all selected focus points match the test
target (red area).
Tick AFMA: From: -20 To: 20. But I recommend you use Count: 6 steps (rather than 5).
Tick Validate Focus Consistency to 2 percent (rather than 6 % and unticked).
Note: the standard “Multi Point Focus Test” will not change the AFMA value and its result
depends on too many variables to be useful.
Start the test.
Note: with some cameras, you'll have to manually enter an AFMA value using the camera's
menu six times throughout the entire test. The rest is automatic.
4. Save the report (PDF).
The relevant output is the single chart entitled “Best AF Fine Tune” on page 3 of the report.
We are going to call the colors in this chart the “AFMA array color chart” (A2C2).
 LumoLabs
Whitepaper
4/16

Note: Due to a bug in version 1.6.0 of FoCal, the plotted AFMA values are truncated to one
digit (negative) or two digits (positive) in this report. Therefore, the numbers are to be
ignored. Use a screenshot of the FoCal window after the test is complete to obtain more
correct numbers. However, the color encoding is correct and I will only refer to colors in
this chart in the remaining part of this article, i.e., I'll refer to the A2C2.
Note: The color is blue or red if the difference of the AFMA value to the center point
exceeds 10. So, by definition, the center point is green always. The color is green, cyan or
yellow if the difference is 10 or smaller.
5. Interprete the result:
No color in the A2C2 should be red or blue (for an f/2.8 lens).
For AF points which came out red or blue, verify that the result is plausible by checking the
interpolation curve for all tested AFMA values further down in the report. It should exhibit a
clear maximum at the left or right edge or even outside the chart.
It seems to be normal to have a few AF points which turn out cyan or yellow.
Note: Read section 4.4 for a direct comparison between results for Nikon and Canon.
Note: The color-coded A2C2 chart is independent of the lens resolution, the camera (sensor)
resolution, the lens' border sharpness, the exact focus distance, focal length, aperture used
during testing, field curvature, other lens aberrations, uneven illumination, slight tilting of
the target, etc. This is why A2C2 like no other chart is so useful to check for the issue.
However, it is (or may be) dependent on a lens' widest aperture and the camera make (cf.
below).
I.e., while the perceived problem may be most prominent with a fast wide lens, it will show
with probably any lens in the A2C2.
Note: At least for Canon, the AFMA values seem to scale proportional with the nominal lens
aperture, i.e., its widest aperture. Therefore, AFMA values for an f/1.4 lens may be twice the
values as for an f/2.8 lens (for the same absolute defocus error). If true (for a given vendor),
then the A2C2 interpretation is only valid for an f/2.8 lens: with f/1.4, larger deviations
would be acceptable and for f/4 or f/5.6 lenses, tighter tolerances would be required to be
acceptable (i.e., an all green A2C2).
Note: Because Canon seems to use a more fine grain AFMA scale (by a factor of 2 or more),
an A2C2 chart for Canon should remain all green even for an f/2.8 lens.
Note: I found that the central AFMA value can change if the test is repeated after the camera
was used meanwhile. But this doesn't seem to affect the A2C2. While the individual AFMA
values may change by more than +/-2, the difference to the center seems to be more stable.
6. Consult Nikon service:
A Nikon camera with A2C2 in red or blue (for an f/2.8 lens) needs to be serviced.
 LumoLabs
Whitepaper
5/16

4. Sample results

4.1. Nikon D800E

The following results are from a D800E which previously was evaluated and found to be affected
by the “leftmost AF sensor issue”. I consider this particular body to be rather typical for the bulk of
D800 cameras manufactured during the first few months. It needs to be serviced but as of now, is in
its state as delivered.

4.1.1. Nikkor 24-70/2.8G

Fig. 1 A2C2 (AFMA array color chart) D800E/24-70/2.8G at 24 mm and 1.20 m distance (50 x).
Above: full chart, below: second run for select points. While the exact AFMA values fluctuate a bit,
the overall colors do not.
Here is the link to the full PDF report for the upper chart:
→ [Link]/lumolabs/articles/D800Focus/images/array/D800E_24-70f_28G
 LumoLabs
Whitepaper
6/16

Fig. 2 A2C2 D800E/24-70/2.8G at 24 mm and 0.60 m distance (25 x).

Fig. 3 A2C2 D800E/24-70/2.8G at 24 mm and 2.40 m distance (100 x).

Note that this was too far away to cover the entire AF array. However, note that the central area does
look worse than at 1.20 m. This may be an indication that the AF accuracy decreases with increasing
distance. However, at 100x the focal distance (which is the case here), the difference between -11 and
-4 is barely perceivable and the measurement would probably have to be repeated to be accurate
enough (the classic depth of field is already almost as large as the distance).
 LumoLabs
Whitepaper
7/16

Fig. 4 A2C2 D800E/24-70/2.8G at 70 mm and 2.40 m distance (34 x).

The secondmost right AF point measures as -20, but the data wouldn't exclude -10 too. Nevertheless,
the result looks a bit worse than at 24 mm and so, it doesn't seem to be a wide angle only issue.

Overall, the results with this lens are not very dependent on the actual focal length or focus
distance. For larger than ~70 x focal distance, the AF system seems to have additional accuracy
problems which however are masked by the fact that the increasing depth of field then doesn't
require utmost precision anymore.
The middle AF bank is ok, the left bank is junk and the right bank is acceptable (just).

4.1.2. Nikkor 14-24/2.8G

Fig. 5 A2C2 D800E/14-24/2.8G at 14 mm and 0.60 m distance (43 x).

 LumoLabs
Whitepaper
8/16

Fig. 6 A2C2 D800E/14-24/2.8G at 24 mm and 0.60 m distance (25 x).

Overall, the results with this lens are similar to the 24-70. However, probably partly due to the high
center calibration (~ -10) necessary for this lens, the results look a bit worse overall and the central
bank shows some weakness too.

4.1.3. Nikkor 70-200/2.8G II

Fig. 7 A2C2 D800E/70-200/2.8GII at 70 mm and 2.40 m distance (34 x).

 LumoLabs
Whitepaper
9/16

Fig. 8 A2C2 D800E/70-200/2.8GII at 200 mm and 2.40 m distance (12 x).

The results with this lens are similar to the 24-70 again, esp. at 70 mm. However, probably partly
due to the close distance (12 x), the results look a bit worse overall. Interestingly, at 70 mm, the
leftmost AF point wasn't the one performing worst. But the left bank is definitely is not accurate
enough with this lens too.
Overall, the results are similar on a qualitative level. The dependence on distance (at least between
25 x and 50 x) seems to be weak, followed by the dependence on the focal length used. The largest
dependence seems to be on the actual lens used.
As a rule of thumb, if the A2C2 chart has red spots for an f/2.8 lens, then it may have them for all
such lenses. Also, the outer leftmost AF point isn't necessarily the worst one. The other AF points
nearby can be equally bad or worse even. So, it is a good idea to test for more than one AF point per
bank.

4.2. Nikon D3

4.2.1. Nikkor 24-200/2.8G

Fig. 9 A2C2 D3/24-70/2.8G at 24 mm and 1.20 m distance (50 x).

 LumoLabs
Whitepaper
10/16

4.2.2. Nikkor 50/1.4G

Fig. 10 A2C2 D3/50/1.4G at 50 mm and 1.20 m distance (24 x). F1.4 chart!
The upper part shows the measurement done at f/2.8, the lower part done at f/1.4. At f/1.4, the depth of
field is smaller and therefore, the dependency of focus changes is larger. But at the same time, lens
softness makes it hard to accurately determine the AFMA value towards the edges. Nevertheless, the
A2C2 is close enough to be deemed consistent.
I currently believe that AFMA values determined for an f/1.4 lens are twice the values as determined
for an f/2.8 lens. This would be in line with what Canon once said how it defines the AFMA scale: as
fractions of shift on the image side to travel a lens' depth of field.
Therefore, above values would have to be divided by two. This yields a max. deviation of 10 which is
the same as for the 24-70 lens. Only difference is the asymmetry which the 24-70 test didn't show.
Also, the direction is opposite. Maybe, this 50 is a slightly decentered sample, who knows (however,
the absolute quality of focus left and right have been about the same).
 LumoLabs
Whitepaper
11/16

4.3. Canon 7D

4.3.1. Sigma 10-20mm F4.0-5.6 EX DC / HSM

Fig. 11 A2C2 7D/10-20/4.0-5.6Sigma at 17 mm and 0.80 m distance (47 x). F4.0-5.6 chart!
This chart shows an “all fields green” result which is great. However, we have to keep in mind that
defocus AFMA values (for Canon) scale with a lens nominal aperture. At 17 mm, this lens has f/5.6 as
its widest aperture but let's assume for a second that the scale factor used is by lens, i.e., 4.0/2.8 or 1.4.
Therefore, the larger deviations become 7 rather than 5. Applying a Canon/Nikon ratio of two
(cf. section 4.4 for a direct comparison between results for Nikon and Canon), 7 becomes 14 (orange).
So, despite the nice green color, the above chart doesn't mean the camera's AF constency is better than
the measured D3.
Also, to be fair, the 7D isn't full frame and it's AF module covers a smaller area than Nikon's full frame
AF modules (if expressed in absolute terms (mm) in the focus plane). But then, it is cheaper too.

4.4. The AFMA scale

For any given camera, it is possible to establish a relationship between the best AFMA value (as
determined by fine calibration) and the defocus error without fine calibration (as expressed by the
shift of the focus plane against the sensor plane, a number in µm).
Or in other words, to relate the dimensionless unit step in a change of AFMA with a shift of the
focus plane in physical dimensions, expressed in µm.
I believe that this relation is universal for a given AF module, probably for a given vendor and
sensor size. And it may or may not be proportional to a lens' nominal (aka widest) aperture (it
probably is).

4.5. Nikon
For the D800E, I measured the (MTF50-related) resolution (expressed as LW/PH) as a function of
adjusted AFMA value (cf. Fig. 12).
 LumoLabs
Whitepaper
12/16

Fig. 12 D800E resolution in LW/PH (determined from slanted edge 10-90% rise width) as a function of AF
manual adjustment value AFMA, as determined for the central AF focus point and with a Nikkor 24-
70/2.8G lens at 24mm f/2.8 at 1.2 m distance.
The measurement was executed twice and the results are plotted in two different colors. Unfortunately,
the D800E's internal default JPG sharpening is too much to reveal much detail around the peak. The
scatter of AF precision can be pragmatically expressed as a variation of about +/-4 on the AFMA scale.
For a well calibrated lens, this has no effect on visible sharpness as it is almost flat over a region of
about 10. This is a first hint that a change of AFMA of 1 is a small change only.
Nevertheless, it is still possible to determine the parameters of the theoretical curve (green line). It
corresponds to:
Peak at AFMA: -6 (+/-1)
Focus shift per AFMA unit change: 3.4 µm (for an f/2.8 lens)
The theory of how to derive the focus shift from resolution changes is at
[Link]/lumolabs/articles/sharpness/[Link] §2.3.4.
However, I should mention that the above result has a significant systematic error (maybe +/-50%)
because we did not calibrate the method for the actual in-camera JPG sharpening parameters.

The above result can be translated to the following change of the circle of confusion:
 LumoLabs
Whitepaper
13/16

An AFMA unit step for Nikon changes the diameter of the resulting circle of confusion
(CoC) by ~1.2 µm which is ~1/25 of the acceptable CoC (30 µm as used in depth of field
(DoF) calculations).
Values of 1 µm (1/30) or 1.5 µm (1/20) are within systematic error bounds.
This means that the AFMA range of -20 … 20 does almost span the full depth of field range.
However, the systematic errors allow for a full coverage of depth of field (1/20) or for 1.0 µm
change per unit change (1/30, a focus shift of N µm where N is the F-stop number).
This means that an AFMA unit change is a smaller focus shift for a fast lens, and a larger one for a
slow lens, as shown below (the conservative, larger value of 1/20 was used for Nikon):

Focus shift df per AFMA Nikon APS-C Nikon FF Canon APS-C Canon FF
unit change
F/1.4 2.1 µm ? 2.1 µm 3.5 µm ? 5.3 µm ?
F/2.8 4.2 µm ? 4.2 µm 7.0 µm ? 10.5 µm ?
F/4.0 5.9 µm ? 5.9 µm 10.0 µm ? 14.8 µm ?
F/5.6 8.4 µm ? 8.4 µm 14.0 µm ? 21.0 µm ?

Because a phase AF module doesn't become more accurate just because the lens has a larger
nominal aperture, this means that a faster lens needs larger correction values and shows larger
fluctuations in determined AFMA values. This dependency on nominal aperture is published for
Canon and assumed behaviour (by me) for Nikon.

4.5.1. Reikan Focal 1.6 and its quality of focus Q

It is interesting to compare Fig. 12 with the corresponding FoCal chart:
 LumoLabs
Whitepaper
14/16

Fig. 13 D800E resolution as FoCal Quality of Focus Q, for same data points as in Fig. 12.
To be fair, it should be noted that the target used was not optimal for the algorithm used by FoCal. But
the smaller drop of Q towards higher deviations of AFMA is typical still.
Peak at AFMA: -2 (+/-1)

Because FoCal determines a slightly different AFMA value (-2 rather than -6) and its Q curve looks
symmetric while the resolution curve does not, we will have a direct look at the results at AFMA =
-20 and 20 (cf. Fig. 14).

Fig. 14 Test shot at AFMA=-20 (left, top) and AFMA=+20 (right, bottom).
At +20, the shot looks significantly less sharp (you may want to look at the samples in full resolution).

The direct inspection reveals that our resolution determination is accurate and the FoCal Q measure
struggles as blur increases. Reikan knows about this issue and is working on that though. For the
time being, its effect on determination of the optimum absolute AFMA values is small (esp. with the
proper FoCal target and distance) and on the A2C2 chart is even smaller.
 LumoLabs
Whitepaper
15/16

4.6. Canon
During the work with Reikan, I also tried to experimentally determine this value for a Canon 7D
with a 100mm f/2.8 lens.
Unlike Nikon (as far as I know), Canon has made a statement about how to interprete the AFMA
values. This and its dependency on a lens' full aperture was communicated in
→ [Link]/resources/articles/2011/af_microadjustment_article.shtml .
Unfortunately, I only had two blur values to work with. With some handwaving arguments, I derive
at the following increment of the circle of confusion CoC per AFMA unit change: 2.1 µm or 1/9.4
of the APS-C 20 µm full circle of confusion at the edge of depth of field.
This is close enough to 1/8 and makes me belief that Canon's statement applies to 1/8 the CoC
rather than DoF (it would have to read 1/16 of DoF then). This is reflected in the statement below
and the table above:
An AFMA unit step for Canon changes the diameter of the resulting circle of confusion
(CoC) by 1/8 of the acceptable CoC (as used in depth of field (DoF) calculations). This is
2.5 µm for APS-C and 3.75 µm for full frame.
So, the entire Canon scale -20 to 20 spans 2.5 x of the full depth of field. In the table above, I
assume that Canon adapts the scale for APS-C while Nikon doesn't. But I don't really know! After
all, the AF module doesn't magically become more accurate only because the sensor is smaller ...
My rule of thumb is that Nikon AFMA steps are more fine grain than Canon's: 2 or 3 Nikon steps
may equal 1 Canon step. In the interpretation guidelines above, I used a factor of two between the
two scales. But it may be any factor between 2 and 3.75.
A consequence is that I wouldn't expect determined AFMA values for Nikon to be as repeatable as
for Canon: If a Canon value is accurate +/-1, the Nikon value may only be accurate +/-3 which
spans a region of 6 AFMA unit steps. But my experience is that this is still an accurate enough
micro adjustment for the Nikon AF. This means I don't advice to enable AF microadjustment for
Nikon with AFMA values smaller than five.
 LumoLabs
Whitepaper
16/16

5. Conclusion
The A2C2 (the AFMA array color chart) as produced by Reikan FoCal 1.6 Pro's Multi Point Focus
Test tool (using the AFMA step option) is ideally suitable to check a camera for proper alignment of
its AF module. As it is robust against most influences normally making an interpretaion of results
difficult or impossible.
The phase AF modules shipping in today's cameras do only have limited accuracy. Based on
experience with various cameras and testing situations (and the above derivation of absolute units),
I say that calibration accuracy of individual AF fields against the central one is limited to about
40 µm, whatever be the camera or vendor. If you ask me, that's at least twice the value I would
desire but it makes no sense to ask for more than the market delivers.
In practical terms, this means that when testing with an F/2.8 lens, the A2C2 chart for Nikon should
be green, yellow and cyan only. And all green for Canon. This means that AFMA deviations from
the central AF field should be smaller than 10 for Nikon and smaller than 5 – 7 for Canon. For an
F/1.4 lens, AFMA deviations must be allowed to become twice as large. For slower lenses, AFMA
deviations must remain smaller.
This is generally the case, Nikon and Canon alike.
However, it appears that up to now, the left bank of unserviced Nikon D800, D800E and D4 mostly
emerge red in the A2C2 chart. Such cameras may require servicing from Nikon and the test may be
used to verify the success of Nikon's work. Or claims that the issue was resolved in factory.