A fascination with computer graphics is what led me into programming, so I was naturally drawn to Metal — Apple's framework for harnessing the Graphics Processing Unit (GPU). Metal's ability to perform a large number of calculations simultaneously, in parallel, makes it useful in a variety of applications — from Machine Learning to Cryptocurrency. If you can structure your algorithms to work in parallel, you can unlock all sorts of avenues. The GPU is fertile ground for iOS developers looking to create cutting-edge applications.
As a kind of "Hello, World," I built a basic fluid-solver using Metal Compute Shaders. I wanted it to look visually striking, so I decided to render the scene as a grid of black-and-white points. As the user drags their fingers across the screen, the points react — making waves in a virtual pool of liquid. After user-interaction stops, the points continue to ripple and undulate, driven by the fluid solver running in a Compute Shader.
There are several parts to the application. There's the Metal View — the app's main view; it receives touch interactions and owns the rest of the objects in the app. User touches are passed to a Painter class, which draws airbrush-like strokes to a Metal Texture. The Renderer owns the Metal Compute 'pipeline', and is responsible for presenting graphics to the screen. It receives continuous draw requests from the Metal View, and texture updates from the Painter.
The Painter class uses two UIGraphicsImageRenderer objects to draw a brush stroke from user touches. The first renderer draws a soft airbrush using a Core Graphics gradient. The second renderer applies the brush to a Metal texture. This could be improved by drawing many brush strokes along the 'coalesced' touches — a mix of predicted touch values, and interpolated touch positions between the previous touch location and the current touch location. The limits of the basic approach shown here are that if a touch moves very quickly, 'gaps' in the touches become apparent. The second limitation is that UIGraphicsImageRenderer executes on the CPU. More on that in a bit.
// Render a reusable brush image
lazy var brush: UIImage = {
let brushBounds = CGRect(x: 0, y: 0, width: brushSize, height: brushSize)
let brushRenderer = UIGraphicsImageRenderer(bounds: brushBounds)
return brushRenderer.image { ctx in
let path = UIBezierPath(ovalIn: brushBounds)
let colors = [UIColor.black.cgColor,
UIColor.white.cgColor]
let gradient = CGGradient(colorsSpace: ctx.cgContext.colorSpace,
colors: colors as CFArray,
locations: [0, 1])!
let centerPoint = CGPoint(x: brushBounds.midX, y: brushBounds.midY)
ctx.cgContext.drawRadialGradient(gradient,
startCenter: centerPoint,
startRadius: 0,
endCenter: centerPoint,
endRadius: brushBounds.width * 0.5,
options: [])
}
}()
It's amazing to see the richness that evolves from just a few basic rules.
For the fluid solver I went with a classic 'heightfield' technique used in games for some time; it's fast, and it's simple. In fact, it's remarkably simple. It's amazing to see the richness that evolves from just a few basic rules.
Like I mentioned before, as the user drags their fingers around the screen the Painter class uses those touches to paint to a texture map with airbrush-like strokes. With every display refresh of the screen, the compute shader loads the current texture map and applies the basic fluid solver rules. The way it works is by evaluating each pixel in the texture map, and averaging all the neighboring pixels to find the fluid's velocity. The previous height value is then subtracted from the new velocity, and the new velocity is added to the height texture. Some dampening gets applied to keep the simulation stable. (These magic numbers took quite a bit of experimentation to get right; definitely an 'artistic license' part of the app. Tiny adjustments have outsized effects on the outcome.)
// Fluid Solver Compute Shader
// Create lookup coordinates for neighboring pixels
// max and min clamp values to edges of the texture map
uint eastX = max(gid.x - 1, uint(0));
uint northX = max(gid.y - 1, uint(0));
uint westX = min(gid.x + 1, heightTexture.get_width() - 1);
uint southX = min(gid.y + 1, heightTexture.get_height() - 1);
// read previous frame's height texture values for each neighbor
float east = heightTexture.read(uint2(eastX, gid.y)).r;
float west = heightTexture.read(uint2(westX, gid.y)).r;
float north = heightTexture.read(uint2(gid.x, northX)).r;
float south = heightTexture.read(uint2(gid.x, southX)).r;
float northEast = heightTexture.read(uint2(eastX, northX)).r;
float northWest = heightTexture.read(uint2(westX, northX)).r;
float southEast = heightTexture.read(uint2(eastX, southX)).r;
float southWest = heightTexture.read(uint2(westX, southX)).r;
// get previous velocity and height values for the current pixel
float v = velocityTexture.read(gid).r;
float u = heightTexture.read(gid).r;
// average the neighboring pixels
float average = (north +
east +
west +
south +
northEast +
northWest +
southEast +
southWest) / 8;
// update the velocity and height values, apply dampening,
// and write out to the texture maps
v+= (average - u);
v*= 0.99;
u+= v;
u*= 0.97;
heightTexture.write(u, gid);
velocityTexture.write(v, gid);
If this were done using the CPU, one pixel would be evaluated at a time until every pixel was calculated. Because this works on the GPU, many pixels are evaluated all at once. This is done by creating a Compute Grid that defines how the GPU breaks up the problem into smaller chunks called Thread Groups. The fluid solver can safely run in parallel, because each Thread Group only needs access to the previous frame's texture values; if each Thread Group needed to know what every other Thread Group was doing, things would very quickly get more complicated. (There's an entire branch of computing, known as Parallel Computing, that has algorithms specifically designed to coordinate work across Thread Groups. I hope to illustrate some of these concepts in future write-ups.)
The Vertex Shader, which tells Metal how to transform and 'project' the geometry onto the screen, simply uses the heightfield texture to move the points up and down along the Y axis. The Fragment Shader, which controls the lighting of each point, uses the height values to subtly adjust the brightness of the points as they rise and fall.
There's a cost to transferring data between the CPU and the GPU.
The fluid simulation itself runs reliably at 60 frames per second. The primary bottleneck is the Painter class. There's a cost to transferring data between the CPU and the GPU, and the Painter uses Core Graphics (which runs on the CPU) to generate the 'airbrush' strokes. These strokes are composited to a Core Image texture, which is directly linked to the Metal heightfield texture map. This all works fine, but with more than a few fingers on the screen that CPU/GPU bottleneck starts to slow things down. The solution to this would be to rewrite the Painter class using Core Image's CIImageAccumulator class, which is specifically designed for this very type of application.
Because I love making strange, mysterious, music apps, for the next iteration, I'd like to turn this into a musical instrument. A grid of colors will correspond to notes that can be triggered both by user interaction, and the resulting ripples. For rendering, the heightfield will be used to refract the colorful grid below — like looking at colorful tiles at the bottom of a swimming pool.
Thanks for reading!
Return to Home Page.