A couple of days ago, I finally reached a point where I felt Lifecards was ready to be submitted. It has been a long an tough fight to get it ready for release (check out www.mexircus.com).

Lifecards was not as simple to develop as it may appear on the surface. We originally decided to work on Lifecards back in september. At that time there were practically no card apps on the app-store. Of course the store has been flooded with them since. I have contributed with three of those card apps -- Although all three were seasonal. Neither of those three apps were particular complicated to code. Honestly, I think that one of the hardest parts was to get the snow falling when you shake the phone in xMasizeMe, so it was quite simple stuff. The strategy of those apps were simply to composite a png-file with an alpha mask on top of a photo that you position with your fingers. As a kind of aftermath I added text and later snow. The code was simple to do. The graphics was far more difficult.

Quartz drawing

The technology behind those apps started with a straight-forward use of Quartz. Setup an offscreen graphics context, call quartz to draw photo, call quartz to draw png with alpha-mask, call Quartz to draw title and then out comes the finished card in the offscreen graphics context which is then used to generate a jpg-file that can be saved to the users photo-album. The same system can be used to draw the card on the screen.

Unfortunately Quartz is dead-slow on the iPhone. It isn't really any wonder, because it does a lot of sub-pixel accurate drawing and images are drawn with filtering. The iPhone is not fast. I reckon that the CPU is the equivalent of a 100 MHz Pentium, and on top of having to draw the graphics (it at least appears as if ) the data is transfered to the GPU which composites all the on-screen elements. It is a long and slow process and the iPhone's graphics system is wonderful for static graphics, but less so for dynamic graphics generated by the CPU.

The first couple of projects lived for a very long time in the simulator and so I had no sense of the speed. The simulator isn't really a simulator but merely a copy of the libraries that run on the iPhone compiled to run on Leopard, and the speed of the simulator is typically 20 times faster than the iPhone (depending on what you do). My original intention when I started Lifecards and later Pumpkinzer was to use Quartz even for on-screen drawing while the user was manipulating the card. Already with Pumpknizer I had to give up because the frame rate was too low. Instead, Pumpknizer used a Core Animation layer for the PNG-file and Quartz to draw the photo. The CPU had to redraw only the photo when it was manipulated and then core animation was used to composite the PNG-file on top of the photo as a CALayer. After I disabled filtering I got around 10 frames per second using that strategy. Once the user was finished with the card, it would be drawn just with Quartz.

I forgot why I didn't use a second layer for the photo. The best explanation I can come up with is that it is best to use the same code for as much as possible, since if I had used core animation layers for editing the card, and Quartz for the final rendering, then I would have to build and maintain two types of drawing and risk that a bug in the code would make the card and on screen graphics differ. Therefore when I realized that Quartz was plenty fast to handle drawing of just one photo I settled with that.

Lifecards card description

For Lifecards we had some wild ambitions originally. They all boiled down to giving the card-designer a lot of freedom while ensuring that the user would get the smoothest experience possible. Specifically, you should be able to position multiple photos and zoom in and out freely while you work on the card. All in real-time. To describe the design of the cards we created a file format. Here is a snippet:


----
usercontentimage = {
tag="img3"
path = [(490,430), (630,430), (630,640), (490,640)]
borderstyle = noborder
}

compound = {
pos = (700,-70)
scalerotate=(0.0,0.92)
image = {
pos = (0,0)
size = (800, 290)
borderstyle = noborder
filename = "t_filmstrip3.png"
scalestyle=fit
}
}

}



usercontenttext = {
tag = "text1"
path = [(35,500), (765, 500), (765, 580), (35, 580)]
borderstyle = {
color = (1,1,1)
thickness = 0
}
fillstyle=whitefill
alignment="left"
valignment="center"
size=40
font="Helvetica"
}

---


The format consists of a number of primitives. Primitives can be three things; something that we draw, a placeholder for the user, or a compound. "usercontentimage" is the definition of a photo placeholder for the user. It consists of a clip-path that describes the position of a photo on the card as well as its border. "usercontentimage" also has a tag that is used to uniquely identify each photo objects.

Compounds can consist of a group of sub-primitives. In reality a card is represented by a tree of compounds. The root-node is a compound that describes the entire card. The position and scalerotate vectors describes how members of the compounds are to be drawn on the card. So the coordinates of a compound member is given in a local coordinate system relative to the compound, not the whole card.

The example compound consists solely of an image to be drawn. That particular image is a PNG-file that is a piece of film that will be drawn on top of a photo that the user supplies and which has been positioned, rotated, and scaled:

User provided objects can also consist of text ("usercontenttext"). Again they are described by a tag and clip-path, but they also have default values for horizontal and vertical alignment, font size, font face, and color.

It is an incredibly simple yet very powerful format. If there is one thing Michael can do extremely well it is to make file parsers, and Michael's chief contribution to this project was the parser of the format. I am grateful that he came up with a very clean and simple implementation for this in less than a week (Note that Michael has a full-time job), and the first week we started working on this project even.

Drawing a card

Drawing the file is simple enough. We start with root compound and traverse the compound tree. Every time a primitive is encountered we draw it. If a user content placeholder is encountered we draw the associated user object (if any). If a compound is encountered we transform the coordinate system and draw the children of the compound. Using Quartz this was quite simple to implement and it took me only a day or two.

It worked great in the simulator and I was full of optimism because back in September I thought the simulator was a "simulator" in the traditional *slow* sense. Putting the thing on the phone was a nightmare. It was soooo slow. I think we got 5 frames per second even with minimal quality settings. It wasn't quite satisfactory but for a while we felt like settling for it.

Smoother drawing

Development of Lifecards was interrupted a number of times. We worked on the basic bits in September, but development was interrupted since I focused on getting iPushFit out, because I went to America for a couple of weeks, and when I returned both iPushFit 1.1 and the Turkeynizer got most of the attention. It wasn't really until the middle of November that I started looking at Lifecards again.

At that time I was determined to get smooth graphics. My first thought was to use Core Animation layers which I used for Pumpknizer (in October). Core Animation is one of the things in OS X I really appreciate. Core Animation layers are so simple to work with. You can supply content either as a cgimage, or by assigning a delegate to the layer. If you assign a delegate, then Core Animation will just call drawRect of the delegate whenever redraw is required, and you just have supply the drawing routines in quartz. It is 10 times simpler than generating textures in OpenGL and then compositing them yourself. You can't do everything with Core Animation, but it follows Apple's design philosophy; I.e. what you get is simple but works extremely well, and if you want something more you have to look elsewhere.

Lifecards cards consists a number of primitives that are drawn on top of each other. Some of the primitives are dynamic (i.e. user defined (like photos and text)), while others are static (i.e. defined by us).

Now my main idea was to generate core animation layers for the primitives. Creating a layer for each individual component would result in a lot of layers which doesn't increase the frames per second. Instead, when the tree is traversed I would stop every time I encountered a dynamic primitive. All static primitive encountered since the last dynamic primitive could then be drawn on the same layer. In the end a card with a bit of text and 3-4 photos would consist of just 5-6 layers and each dynamic primitive would have its own layer.

The idea was that it would be faster to update individual layers than redraw the entire card in each frame. So when the user decided to rotate the a photo, only the layer containing the photo would be redrawn. Layers would have large dimensions (about 800x600 pixels) so that when the user would zoom and pan the card, I could just pan and scale the layers without doing any quartz drawing. Unfortunately, although this approach was a bit better, it was still extremely slow and even worse the many layers used a lot of memory.

OpenGL

One of the cool things about CALayer is that you can actually perform any affine transformation to it. The user can translate, rotate, and scale photos and so ideally whenever the user makes a change to a photo, we could just change the transformation matrix of the associated CALayer. Unfortunately, the cards we had in mind required arbitrary clip paths, and there is no way to clip a CALayer on the phone. I think you can do it under Leopard, but on the phone you can only clip layers to a rectangular region.

It seemed inevitable that we had to move drawing into OpenGL. I contemplated pros and cons for a bit and after one or two days work I had OpenGL drawing up and running smoothly. I still used the same layer mechanism, so the card was divided into layers consisting of static and user text primitives. Each layer is drawn on a texture, which is a lot simpler than I first feared. Finally, each individual photo is also copied to a texture.

When I draw the photos I don't actually draw a rectangle and clip it to the clip path. Clipping in two dimensions is relatively simple, but for some reason I was concerned with implementing clipping. The approach I use instead, is to triangulate the clip path, draw each triangle, and assign appropriate texture coordinates to each corner point. I determine the texture coordinates by multiplying each coordinate with the inverse of the transformation matrix applied to the photo -- It is a bit technical but really not too complicated. Inverting the transformation matrix is simple since it is a 3x3 matrix (for handling 2D homogeneous coordinates).

I actually implemented clipping of polygons later for another purpose so in retrospect it would have been simpler to clip a rectangular photo to the clip path.

Reducing memory use

The iPhone has 128 MB of memory with some of it allocated for the GPU. I am not sure anyone outside Apple knows the exact memory layout. However, apps are not allowed to use even remotely 128 MB of memory. I have been able to allocate and use up to around 50 MB at most. In reality, the memory system on the iPhone is a bit peculiar. Often you will receive "memory warnings" from the system when you are using around 10 MB. According to the documentation you should free memory at this point. Apple's default implementation of UIViewController will release the associated UIView to free memory. It is a bit unpleasant.

At around 20-30 MB of used memory you will receive "Urgent memory warnings". Apple's default implementation at this point is to close your app cleanly by giving you a termination event. That will allow you to save everything and exit nicely.

In January I started testing some of the complicated cards on the phone, I discovered many memory warnings. When I started adding things up I realized that Lifecards would quickly reach 20 MB of memory. This occurred especially, since I allocated some large textures and had a couple of large UIImage objects in memory. UIImage is implemented such that the image doesn't have to reside in the memory. I think, they are loaded and unloaded by the system if the image points to a file on the disk, but I am unsure how it works. In fact, I was observing some odd behavior because I initialized a UIImage with a file, and then later deleted that file. Suddenly nothing happened when I wanted draw the image after deleting it, but it didn't happen immediately after deleition. It took me a bit to realize what was going on. It is a good design strategy. It just doesn't really say anything about it in the documentation. The only thing it says is that the system can load and unload the images, but not when it happens.

The first step towards reducing memory was to change the textures. For photos and the first layer of the card I use 16-bit textures since these are completely opaque. For the remaining once I need an alpha channel and so they are 32-bit.

Then next step was to change the texturing system. Originally I allocated 1024x1024 textures (texture dimensions have to be powers of 2) to accommodate layers for a card of size 800x600. In other words I wasted a lot space. There were two fixes. First, I now determine a bounding box of the primitives on each layer to find a smaller size. I.e. a single layer could have primitives occupying 450x230 pixels that would result in a 512x256 texture. Secondly, I break the layers into smaller 128x128 pieces. For an 800x600 layer that gives 7 * 5 128x128 pieces or the equivalent of an 896x640 texture. For a 32-bit texture that is about 2.3 MB bytes rather than the 4 MB that a full 1024x1024 takes up. Furthermore, if an entire 128x128 piece has zero alpha values, I discard and disregard it completely. Overall, I got the memory used for layers (not photos) down from 12 MB to around 2-3 MB for the most complicated cards.

A final thing I did was to change the behavior of the UIImage objects used for photos. Rather than depending on the dodgy system to load and unload them I actually store the UIImages on the disk explicitly. Whenever I need to redraw a photo onto a texture or to the final card, I load the UIImage from disk, draw it, and then immediately release it. That way I can be sure that at most one UIImage resides in memory at any given time. It would be great if Apple gave you the ability to explicitly unload a UIImage, when the functionality to load and unload already is somewhere within the implementation of UIImage.

Overall, I can say that building apps that use a lot of memory on the iPhone is a bit painful, mostly because you can experience things that doesn't happen on conventional operating systems and it takes a bit of time to get used to. It is also a bit frustrating that you can only use about 20 MB of the 128 MB on the system, but I guess it is better than 10 MB.

One of the cool things about using OpenGL is that it allowed me to do the genie effect for the main-menu of the app. It was one of those moments, where I just felt compelled to do something fun. It didn't take much longer than an hour to implement the effect, but it would have if I had not had switched to OpenGL. OpenGL also made it simpler to make the toolbar icons rotate with the phone's orientation. If I had used Core Animation I would have needed a layer for each tool-button. I think the current implementation is slightly simpler.

Building the cards

Designing and building cards was a challenge. Mostly, because it was such a daunting task for someone like me who is not a designer. Initially, we only wanted around a 50-100 cards, but even that number seemed hard to reach. At one point I was
naive and I thought that I could build 50 cards in a few days. I started building cards in the end of November but I had to abandon it after only a few days, simply because I felt I didn't have a plan.

In the middle of January I decided to give it another shot, so I loaded up cgtextures.com and found a bunch of textures to use for backgrounds. Then I spent about a week or so building around 50-70 master card templates. I then used the master card templates to built around 5000 cards, by simply changing the background. Michael and I spend about an hour and a half just scrolling through the first 3000, and by the end Michael was completely dizzy. The reality was that there were way too many. Some backgrounds didn't work and some of the master templates didn't work either. Sometimes the combination of master template and background just didn't work. After about a week or so I had it down to 700 and
last week I got that down to around 360. Looking at the cards now, I think I could probably reduce it by another 50.

The funny thing is that, because the card system was so flexible it was really hard to make the cards. If we had made a simpler card-representation it would have been infinitely faster to create the cards. It is actually a good design philosophy; Don't give yourself too much freedom, because it makes it harder to find out what you want to do with it.

The 10 MB app size limit

Apple has imposed a 10 MB limit for apps that can be downloaded over the phone network from the app store. Any app larger than that must be downloaded over wifi. I don't have any statistical data, but I guess that under-10 MB apps sell more often than over-10 MB apps. Because of this, and because it is always important to minimize the space you use I spend some time getting Lifecards under 10 MB. I compressed the backgrounds a bit harder and reduced the background set significantly. At one point I thought the app was around 11.3 MB and I felt that there was absolutely nothing I could do to reduce the size, but luckily a "clean" build got it down to the 8 MB it occupies today. I guess you have to clean-build once in a while.

The name

Lifecards started life as "Collage" but by the time we were about to release it Collage was already taken and we had to change the name. I am not sure if Lifecards is a good name. The idea is that it should somehow associate to iLife and your daily life. I hope people will get it.

Planning a project

When I first pitched the idea to Michael I targeted a late November release. The release kept sliding for a number of reasons. First of all some of the easier projects got a bit in the way, and secondly several things turned out to be much harder to do than I initially expected. The code for drawing the cards took several revisions and probably a total of about 10 days of fooling around since I had to try several approaches. Creating the card design took about 5-10 times longer than I expected and was the main reason the deadline kept sliding. It has definitely been a fun project so far, even despite the fact the deadline kept sliding, but it has also been a learning experience, and I think I will be very wary to assume a task which involves a similar amount of graphical design in the future.

Just yesterday we discussed possibilities of a spin-off project which will reuse all of the code from Lifecards and I am actually very excited about getting started on it. It doesn't include nearly as much design.

The fact was that 90% of the tasks didn't take longer than I initially anticipated. In fact some of them took shorter because, by the time I got to them I could reuse some of the code I had made for the other apps. It really was what originally seemed like only 10% of the project that just exploded. I honestly thought I was 10% from release on the 1st of December. In reality I had probably just done 25%. And although it is cliche, thinking back on many projects I have worked on it is true, that if you think you are 90% done, you are really only halfway there. In particular I remember being hired in a company almost 10 years ago. The rest of the group I was hired to work with, were just about one month away from finishing a project. 6 months later I left the company, and 7 months later the hardware we worked on was finally released. When I started it was already overdue. It wasn't poor management skills. There were just a lot of unforeseen events and for some tasks a huge underestimation of time.

About 50% into development of an app I set a target release date and start dividing development into tasks that I estimate will take up around 1-4 hours each; but only around 10-20 tasks in total at one time. That actually works well for me and it is also extremely gratifying to see a todo-list getting shorter and shorter as you complete the tasks. Every 3 days or so I build a new todo list. I never finish on the exact day I expect to. Usually I am at least one day off, but having short detailed todo-list helps to push yourself.

With Lifecards I first set a submit date for the 22nd of January. That got pushed another week, because filtering the card templates took longer than I anticipated. I didn't finish on the 29th of January either, but it felt like I was just a few days away, so I set the release date to the 3rd of February. I was working hard on the 3rd to finish and stumbled across some more of the memory problems I have mentioned previously. But finally, the 4th of February I managed to submit it late in the afternoon after spending all day on an icon-redesign. There comes a times where all you want to do is just finish a project and I honestly don't think I had the energy to work on Lifecards one more day at that time.