2014-02-10

Who stole my pixels?

Moving away from the overly detailed discussion on alpha layering from my last post, today I want to briefly discuss developing for iOS and taking advantage of the full resolution of retina devices. Although I knew some care was needed in order to use the full high-resolution retina display, for some reason I assumed this was already happening. Recently, however, I realized that that wasn't true. A little Googling found lot's of info on this and it all comes down to scaling between points and pixels.

Scaling the heights

There are many discussions on this topic, so I won't go into too much detail, but I did want to mention how I addressed the problem. Briefly, for those who don't know, when Apple released their first high-resolution retina devices they did it in a careful way that allowed all existing apps to still run and look good. Basically they added a new property of the screen that defines what it's scale is. The official Apple documentation for UIScreen.Scale says:

This value reflects the scale factor needed to convert from the default logical coordinate space into the device coordinate space of this screen. The default logical coordinate space is measured using points. For standard-resolution displays, the scale factor is 1.0 and one point equals one pixel. For Retina displays, the scale factor is 2.0 and one point is represented by four pixels.

To use this scale I needed to do three things:

  1. Set the ContentScaleFactor of my EAGLView, which is a subclass of iPhoneOSGameView to the same scale so that my rendering uses the entire high-resolution screen
  2. Tell my game the full high-resolution size of the screen
  3. Multiply all of my coordinates by the defined scale factor

For the first item I both get and remember the scale factor and set the ContentScaleFactor to it in the initialization of my EAGLView. Because I want my game to run on older devices with older versions of iOS I need to first check that the new scale property exists and then only reference it if it does. The purpose of ContentScaleFactor is defined by Apple as:

Hmm, reading over this now I just noticed that it says I shouldn't have to adjust the value of ContentScaleFactor. I'll need to go back and look at my code again.

The scale factor determines how content in the view is mapped from the logical coordinate space (measured in points) to the device coordinate space (measured in pixels). This value is typically either 1.0 or 2.0. Higher scale factors indicate that each point in the view is represented by more than one pixel in the underlying layer. For example, if the scale factor is 2.0 and the view frame size is 50 x 50 points, the size of the bitmap used to present that content is 100 x 100 pixels.

The default value for this property is the scale factor associated with the screen currently displaying the view. If your custom view implements a custom drawRect: method and is associated with a window, or if you use the GLKView class to draw OpenGL ES content, your view draws at the full resolution of the screen. For system views, the value of this property may be 1.0 even on high resolution screens.

In general, you should not need to modify the value in this property. However, if your application draws using OpenGL ES, you may want to change the scale factor to trade image quality for rendering performance. For more information on how to adjust your OpenGL ES rendering environment, see “Supporting High-Resolution Displays” in OpenGL ES Programming Guide for iOS.

Here's the code that does that:

_screenScale = 1;
if(UIScreen.MainScreen.RespondsToSelector(new Selector("scale"))
   && RespondsToSelector(new Selector("contentScaleFactor")))
{
    _screenScale = ContentScaleFactor = UIScreen.MainScreen.Scale;
}

For the second item I simply multiply the screen size by the scale I determined above via this code snippet: new Size(Frame.Size.Width, Frame.Size.Height) * _screenScale.

Finally, for the third item, I perform another multiplication of the scale when I handle touch events.

/// <summary>Called when touch events end.</summary>
/// <param name="touches">Touches.</param>
/// <param name="e">The touch event.</param>
public override void TouchesEnded(NSSet touches, UIEvent e)
{
    base.TouchesEnded(touches, e);
    UITouch touch = (UITouch)touches.AnyObject;
    _lastTouch = touch.LocationInView(this);
    _screenManager.Screen.HandleTouch(new Point(_firstTouch.X, _firstTouch.Y) * _screenScale, new Point(_lastTouch.X, _lastTouch.Y) * _screenScale, true);
}

/// <summary>Called when a touch event moves.</summary>
/// <param name="touches">Touches.</param>
/// <param name="e">The touch event.</param>
public override void TouchesMoved(NSSet touches, UIEvent e)
{
    UITouch touch = (UITouch)touches.AnyObject;
    _lastTouch = touch.LocationInView(this);
    _screenManager.Screen.HandleTouch(new Point(_firstTouch.X, _firstTouch.Y) * _screenScale, new Point(_lastTouch.X, _lastTouch.Y) * _screenScale, false);
}

All of the above code happens in my iOS specific EAGLView class and the rest of my code needed no adjustments.

Next: coming soon...

2014-02-07

Alpha layering

In a recent post I discussed premultiplied alpha in textures. After that I thought I had my alphas under control, but it turns out I was wrong.

Keeping score

Early in my game development I came up with a slightly interesting way to render text. In the game, text is used only to show the game score and is generally quite large on the screen. The score advances every frame, or 1/60th of a second. Because that advancement would actually change too quickly for the eye to see I actually truncate the last digit and only render the remaining ones. So if the score advances frame-by-frame as 137 → 138 → 139 → 140 → 141, I would actually render it as 13 → 13 → 13 → 14 → 14. Even doing this the rendered score visibly advances every 10 frames, or every 1/6th of a second. This is still pretty rapid, and I wanted to visually smooth that quickly changing score when it's rendered on the screen.

50 shades of grey

The idea I settled on was to keep track of the last N instances of the score each time a frame in rendered. In the above example, if N was 5, that would be the list of numbers shown. One frame later the list would change to 13 → 13 → 14 → 14 → 14, and 7 frames later it would become 14 → 14 → 14 → 14 → 15. After keeping track of these last N scores I then actually rendered them all with the oldest score being very transparent and each successive score being more and more opaque. This gives an effect like the one shown in the image to the right. Actually that image is showing the score rendered as it is also being rotated in order to make the separate layers easier to see, in most cases the score is not rotating, so each layer is stacked directly on top of the previous one.

This layered transparency effect worked out well and I've had it in place since quite early in the game development. The way I actually implemented this was to take my desired alpha and figure out how many fractional shades of that I would need such that when I layered them all together I would get my desired alpha. For example, for the score when it is shown during a game it is dimmed so as not to distract from game play and has an intended alpha of 0.25. After the game when showing the final score it is brightened to full opacity, or alpha 1.0. If I have 5 layers then I decided to break the desired alpha into 15 fractions 1 + 2 + 3 + 4 + 5. This is also \( \frac{N \times (N+1)}{2} \) or \( \frac{N^2 + N}{2} \). So, for the dimmed alpha of 0.25 I calculated the fractional alpha as \( \frac{0.25}{15} = 0.0167 \). Then I rendered the oldest score layer with an alpha of 0.0167, the second oldest as \( 0.0167 \times 2 = 0.0333 \), etc, up to the newest score as \( 0.0167 \times 5 = 0.0833 \). When all of these were layered on top of one another it produced the desired 0.25 alpha...or so I thought.

One problem I periodically ran into was that if I adjusted a parameter related to the transparency, like making the text slightly lighter or darker, or make the number of layers more or fewer, then the resulting aggregated color often ended up being not what I expected. Recently, and after improving my understanding of alpha blending as described in my previous post, I finally spent the time to dig into this and understand this properly. The problem was that my simple math was incorrect and does not represent how layered alphas work.

In the case of my text, I'm layering a pure white opaque texture onto some existing background. For the texture, being pure opaque white, each white pixel is (1, 1, 1, 1). Because I want to dim this, as described above, I adjust the alpha 0.0167 to get (1, 1, 1, 0.0167). To figure out how this blends with the background I return to the alpha formula I described in my previous post, which said regarding GL.BlendFunc(BlendingFactorSrc.One, BlendingFactorDest.OneMinusSrcAlpha);: for the source color use it as is; and for the destination color take the alpha component of the source, subtract that from one, and multiply that by each component of the destination.

Diminishing returns

Let's do the math. The source is (1, 1, 1, 0.0167) and to keep things simple the destination is black, or (0, 0, 1, 1). So, use the source component as is means just keep (1, 1, 1, 0.0167). Next, for the destination color, take the alpha component of the source, subtract that from one, and multiply that by each component of the destination means we take the source alpha, 0.0167, subtract that from 1, \( 1 - 0.0167 = 0.9833 \), and multiply that by the destination, which gives us (0, 0, 0, 0.9833). Now, the actual screen doesn't have an alpha component, it just shows colors, so to get a real color we multiply the alpha by each component and then add the results, or: (1, 1, 1, 0.0167) becomes (0.0167, 0.0167, 0.0167) and (0, 0, 0, 0.98333) becomes (0, 0, 0) and adding those two gives us (0.0167, 0.0167, 0.0167) or   X   . That's very close to black, but that's ok, it's just 1 of 15 shades we're applying. Let's do another.

The source is the same, (1, 1, 1, 0.0167), or multiplied out as (0.0167, 0.0167, 0.0167). The destination is different this time, since we layering on top of what we just calculated, which is (0.0167, 0.0167, 0.0167, 0.9833), or multiplied out as (0.0164, 0.0164, 0.0164). Adding these gives us (0.0331, 0.0331, 0.0331) or   X   . Hmm, that still looks awfully black, but it's just two of 15 shades.

Now, for those of you paying attention, this actually isn't the shade we'd have at this point. I'm supposed to take the fractional alpha for the oldest text plus two of that fraction for the next oldest, which means at this point I should have three shades layered on one another, which would be a tad brighter black. But, actually, even that isn't true, and this is where we start getting to the heart of my problem. You do not get to the same color if you apply the alpha layer three times in succession compared to doing it once and then a second layer at twice the alpha. Let me explain.
If I continued the above calculations for one more layer at the same fractional alpha I would end with a final color of (0.0492, 0.0492, 0.0492). On the other hand, if I took my first layer, (0.0167, 0.0167, 0.0167), and layered on top of it a double fractional shade (1, 1, 1, 0.0334), then I would result with (0.0497, 0.0497, 0.0497).

Now, 0.04918 and 0.04973 are not that different, but if I did this for all 15 layers the difference increases. Here's the complete table:

Shade Applied Cumulative Total Calculated Color
Layer Single Layers Increasing Shades Single Layers Increasing Shades Single Layers Increasing Shades
1 0.0167 0.0167 0.0167 0.0167 0.0167 0.0167
2 0.0167 0.0333 0.0333 0.0500 0.0331 0.0494
3 0.0167 0.0500 0.0500 0.1000 0.0492 0.0970
4 0.0167 0.0667 0.0667 0.1667 0.0650 0.1572
5 0.0167 0.0833 0.0833 0.2500 0.0806 0.2274
6 0.0167 0.1000 0.0959
7 0.0167 0.1167 0.1110
8 0.0167 0.1333 0.1258
9 0.0167 0.1500 0.1404
10 0.0167 0.1667 0.1547
11 0.0167 0.1833 0.1688
12 0.0167 0.2000 0.1826
13 0.0167 0.2167 0.1963
14 0.0167 0.2333 0.2097
15 0.0167 0.2500 0.2228

This shows two methods of adding the layers. Single Layers means layering the fractional alpha multiple times. Increasing Shades means doing that, but increasing the fraction for each layer. I've highlighted the interesting numbers. The Cumulative Total colums just show that in both methods the resulting simple-math total is what we're trying to get to, 0.25. The Calculated Color column shows what we actually end up with, which in both cases is less than our target, or 0.2228 in the case of Single Layers or 0.2274 in the case of Increasing Shades.

The reason both numbers don't add up to what we want is that each successive layer causes less of a change than the previous one. This is a case of diminishing returns. Because of this we need to do some math to calculate the fractional shade such that it takes into account these diminishing returns.

Calculated Color
Layer Single Layers Increasing Shades
1 0.0667 0.0667
2 0.1289 0.1911
3 0.1870 0.3529
4 0.2412 0.5255
5 0.2918 0.6836
6 0.3390
7 0.3830
8 0.4242
9 0.4626
10 0.4984
11 0.5318
12 0.5630
13 0.5922
14 0.6194
15 0.6447

Warning! Math ahead!

I'm going to shift our target to 1 instead of 0.25. This will make the diminishing returns problem more clear. Doing this, and following the same flawed layering strategy described above, the Calculated Color from the above table becomes as shown on the right. You can now see the totals are quite far for our target of 1. In fact, because of diminishing returns, no matter how many layers we add we'll never actually get to 100% opaque. If I extended the above to 100 layers the Single Layers color would be 0.999.

So, what I needed to do was to determine a different fractional alpha to use when layering that would add up to my desired target. To do this I converted my Single Layers iterative algorithm into a formula. First, the iterative version of this is below, where fractional is the fraction of my intended alpha, or \( \frac{1}{15} = 0.067 \).

$$ f(n) = (1 - fractional) \times f(n-1) + fractional $$

This calculates the final color for n where n is the number of layers. To convert this to a formula that does not reference itself I started by listing the successive values of this formula in Google Spreadsheets, just like the above tables, and then I played with it until I came up with the following equivalent formula.

$$ color = 1 - (1 - fractional)^{layers} $$

We can check this by plugging in our fractional alpha of \( \frac{1}{15} = 0.067 \) and our 15 layers as \( 1 - (1 - 0.067)^{15} = 0.645 \), which gives us the same value. So far so good, but what we want is a formula into which we supply the color and the number of layers and it gives us the fractional alpha. That means we want to get the fractional part of the formula on one side by itself. Time to remember how do to formula manipulation. First, get the exponent expression on a side by itself.

$$ 1 - color = (1 - fractional)^{layers} $$

Next, reverse the exponential equation.

$$ (1 - color)^{ \frac{1}{layers} } = 1 - fractional $$

Finally, get fractional by itself and positive.

$$ fractional = 1 - (1 - color)^{ \frac{1}{layers} } $$

Great, now we just plug in our target color, which was fully opaque white or 1, and keeping the layers as 15, we get...oops...hmm, \( 1 - 1 = 0 \) and \( 0^{ \frac{1}{15} } = 0 \), and \( 1 - 0 = 1 \), so the fractional is 1. Ack, we have a problem. This actually makes sense. Remember when I said that with the diminishing returns problem we can add the fractional alpha as many times as we want and never actually get to 100% opaque, well this is telling us the same thing. Actually, if the fractional alpha starts out as 1 then we'll get to 100% opaque, but then we don't get the successive alpha blending effect that we want.

Ok, time to cheat, although I want the final alpha to be 1 I'll settle with it being a single shade off of 1. Since there are 256 possible alpha shades I'll settle with \( \frac{255}{256} = 0.996 \). And plugging that into the formula I get \( 1 - (1 - 0.996)^{ \frac{1}{15} } = 0.308 \). If I layer that 15 times I get this succession of colors: 0.308 → 0.521 → 0.669 → 0.771 → 0.841 → 0.890 → 0.924 → 0.947 → 0.964 → 0.975 → 0.983 → 0.988 → 0.992 → 0.994 → 0.996.

This works and I've implemented this in code now, but it's not actually exactly what I want. This is the Single Layers algorithm, where really I want the Increasing Shades algorithm. The formula for that is more complex, though:

$$ f(n) = \left[ \left(1 - \frac{ n^2 + n }{2} \times fractional \right) \times f(n-1) \right] + \frac{ n^2 + n }{2} \times fractional $$

I've tried to convert this into a single formula that does not reference \( f(n-1) \), but have not been successful yet. For now I'll either stick with the previous formula or use the one above and calculate it recursively.

Next: Who stole my pixels?