April 26, 2001: Bang! A trip inside darwin
I have been watching with interest as the cultural icon of apple computer furiously transforms itself, not unlike madonna, from a tired old 80's icon into a fresh and modern brand with real commercial relevance. In february, I wrote an article about apple computer and its history with steve jobs, ross perot and next computer, inc.

Things got interesting in the last few days of 1996, when apple surprised everyone in buying next, inc. and promising to employ next's never realized technological potential in apple's mass market products. Designing a complete, modern operating system is a monumental task. Apple had run aground in managing the effort to retrofit the mac os to support so called modern os features, so they went shopping for one.

They turned down windows nt/2000 because nt is such a poor shamble of compromise. The favorite of many mac zealots, be/os, was turned down because it just wasn't open or ready enough, especially when compared to what next offered.

Next had some incredible pure technology that had been refined for its enterprise customers over the last decade, had been adapted to the web, and was built upon modern, open and complete foundations. What next didn't have is big developers and mass market hardware business, so apple worked for four years to sell to its developers the idea of modernizing their apps to run on the next os, all while working to update and improve the next os in ways next itself had intended, but lacked the time and money to accomplish.

During this time, the next faithful, its former users, watched things develop with a certain anxiety. Would apple screw everything up, and end up trashing the great technology they acquired? Would they dumb things down and make next's os into a closed appliance reminiscent of the old apple?

The answer would be no, but it wasn't always obvious. I watched history unfurl through four years of nail biting as an occasional visitor to the bay area next-users' group meetings, hosted at the apple campus south of the city in cupertino, a san jose suburb.

Last night the meeting was about mac os x's darwin layer, the core os layer just underneath all of the flashy colored gel cap buttons and translucent effects of mac os x. I almost didn't make it, as my car started gasping for breath on the 40 minute drive south, but managed somehow.

The meeting was a presentation called "mac os x under the hood" and given by apple's director of core os engineering, brett halle. Brett has been at apple for a decade, working on everything from a/ux to pink to copland. While the press often jeers apple's failure in delivering a modern os as intended years ago, brett pointed out that through all those projects, apple learned an enormous amount about how to make things work in an operating system. Really, mac os x is the culmination of a lot of the work that had been done at apple.

Brett presented a marketing overview of ten things that makes os x different in the marketplace, and then got into some details on what makes darwin, as the core basis of mac os x, interesting and relevant. In my notes, the ten things turned into thirteen, and there was a lot of overlap between features in mac os x and darwin's details, so rather than closely follow the notes I took, I though I'd just write about things I thought were really interesting.

Brett described the x team, saying "we move fast, work like dogs, eat our own dogfood and are excited by what we do." This excitement about technology and making products that are not just profitable but exciting and usable has resulted in an operating system that is an excellent foundation for further development.

Darwin is both an open source project and another distribution of unix, but more importantly, is also part of a real shipping os product. Brett said apple is serious about keeping code in the darwin project identical with the core code that runs os x. In fact, the darwin distribution can conceptually be swapped out entirely with the code that comes with os x. The only difference is that darwin comes with a limited version of core foundation, the shim that sits between darwin and higher level application layers; if you try to swap in the darwin code, you should preserve the full version of core foundation that comes with os x.

Darwin is based on a mach microkernel, which differs from the macrokernel design of linux. This was much ballyhooed in the press recently, as comments made by the linus torvalds about mach being 'crap' were widely presented as an attack on mac os x by c|net and other sources.

Brett explained that linus' comment about mach was inspired by an ideological difference linus had with andrew tannenbaum, an os guru who beat on linus for not using a mk design in his linux project. Mk architecture was the darling of pure os design, and linus' approach in making linux a more integrated macrokernel os was poopooed by tannenbaum as a 70's design. Linus insisted that performance issues with the pure laboratory technology of mach were more of a problem that the use of a faster, if less 'pure' design he chose for linux.

Mach is a pure research project, and does not exist as a consumer os. Brett explained that the mach mk employed in mac os x is significantly retooled to allow a closer integration between the pure kernel and the personalities that run on it. Essentially, a pure microkernel exibits no policy; that is, it only concerns itself with very low level memory resource management and processor scheduling. There is no direct involvement in the kernel with higher level features such as file systems or input/output systems.

By using a hybrid of pure mk and tight integration with higher bsd personality layers, darwin benefits from the increased speed of linux over pure mach, while retaining a clean separation between low kernel features and higher os policy.

Apple's version of mach 3.0+ also includes scheduling framework features from later mach projects (mach 4.0), and benefits from rt/mach research done by the army to study real-time operating system features. Mach as a modern, practical mk design has also existed in some form for the last decade in nextstep, so it's not an entirely new idea. So, apple's mach has little to do with the mach linus panned as a stuffy, impractical research.

Brett ran through some other improved mach features in darwin, including support for smp (efficient use of multiple processors), real-time (allowing priority to tasks with needs for highly accurate timing, such as video), external paging (allowing for flexible virtual memory paging) and a modular architecture that allows for kernel extensions to load at boot time or dynamically when needed. This allows for adding driver support without recompiling the kernel, a regular task facing linux users.

Speaking of drivers, apple wrote a whiz bang framework called the i/o kit to support any sort of new hardware. It makes it easy for developers to write device drivers, and does much of the work in advance to ensure that all devices provide true plug and play support and sophisticated power management with support for multiple power planes. The i/o kit provides abstractions for most device classes and supports different device families, or ways of i/o interfacing, for everything from scsi to audio devices, all of which require vastly different specifics on how to communicate.

Also new and improved upon is hibernation for laptops. Sleep and wake in os x are immediate, as brett demonstrated by showing a laptop wake up and operate before the screen could fully open. One reason for the dramatic increase in speed over the classic mac os is that os x is fully multithreaded, allowing lots of tasks to occur in parallel. Under mac os 9, the waking laptop had to handle a series of wake up tasks in sequence; os x does them all at once.

Above all of this low level kernel / hardware layer (mach and the i/o kit) darwin layers its personality os: bsd unix. While many distributions of unix are built similar to linux, darwin uses lessons learned with mklinux, copeland's nukernel and next's own mach/bsd work to provide a very real unix userland that is clearly separate, yet highly integrated with mach, the i/o kit, the virtual memory system and a unified buffer cache.

This bsd unit provides the policy or personality of bsd unix, including its process model, security model and its filesystem and networking features. It includes the bsd system framework that exposes posix, bsd and pthreads; a user environment the provides the application environment; and the net clients, file tools and system tools and services associated with bsd unix: things like ping, ls, ps, kill, dns, ftp and apple's own netinfo. Darwin is unix.

Darwin's file system is a component of the bsd subsystem, and employs an improved upon virtual file system with a posix++ interface and support for stackable layers of file system components. In addition to supporting basic file system types such as hsf, nfs and ufs, and in the future ntfs and others, vfs can support modular plug-ins, such as compression or encryption as stackable file system layers.

Brett said darwin is striving to be file system neutral, allowing users their choice of principle file system, although the supplied hfs+ has advantages such as its b-tree design that allows for fast searching.

Darwin's networking is based on the bsd 4.4 ip stack, which has been synched up with freebsd 3.2. It uses sockets technology but provides an abstraction layer to support open transports' streams implementation. Darwin also provides support for ipv4 and appletalk multihoming, or the ability of the os to support multiple network interfaces at once.

In fact, os x provides intelligent and automatic configuration of the most appropriate network interface, matching features recently offered in windows 2000. For example, if you are working with a wireless airport card and then plug into your high speed lan, os x is smart enough to know to make your faster internet connection your principle interface automatically. Under classic mac os, you would have to change your location manager setting to make a different ip setup active.

Even better, os x provides intelligent application update services, so for example, mail.app can change setup appropriately as network settings change. Brett illustrated by pointing out that apple's own airport network sits outside their firewall; moving from wireless to direct connection on apple's private lan requres mail.app to change how it checks for mail.

Darwin also supports full routing over ipv4 and appletalk, although it is not enabled by default for security reasons. Routing allows one network interface to pass traffic correctly to another interface. Darwin also supports firewall and nat mechanisms (creating a private network protected from an outside, public network address space) and service location protocol for discovery services. Other features, such as vpn support, can be added though kernel extensions, providing the modularity of open transport's streams over bsd's sockets code.

Other network features include a shared ip environment for classic, abstractions to support open transport in carbon, ppp for dialup and ppoe for dsl users, a dhcp server and client and support for ad hoc networks (casual crossover or airport peer to peer connections). Apple is working toward support for ipv6 and the related ipsec protocol (for virtual private networking and other security purposes).

Darwin's status as an open source project exposes the functionality of mac os x to developers and changes the rules for os development. Already, developers have worked to port lots of software to support darwin and os x, have fixed lots of bugs and can comment and participate in the direction of the os' development.

Other random ideas brett presented that I thought were interesting:

Carbon developers who are porting classic mac os apps have to deal with differences in semantics and scheduling differences, as classic mac os was not built with the mindset of modern os services. Allowing a gentle migration for users was presented as a top ten differentiation of mac os x.

Another top ten point was multilingual support. Most os products ship each language as different sku, which is expensive and difficult. Mac os x supports multiple languages, a major goal for apple, since half of apple's revenue comes from overseas. Users in mac os x create a preference list for the languages, and individual apps present themselves in the language of choice that they support. Different users on the same machine can have different os language preferences. Unicode support and app package design makes this easier for developers, too.

Apple, as a developer of both the hardware and software, can accelerate new technology adoption and provide better integration than any competitor. This flies in the face of what nearly every analyst and their dog demands, that essentially apple should become a marginalized pc software developer. Thankfully, apple has taken a better path.

Talking about quicktime, brett added that the new sound architecture in os x would not have been possible in os 9. This includes support for multi channel sound, allows multiple applications to produce sound at once, enables live mixing, is capable of 5.1 channel audio and makes efficient use of altivec.

Another top ten difference in mac os x is cocoa, the application frameworks from next providing "lego like reuse." Brett made it clear that objective-c was cocoa's "implementation language", while it also supports java for "a few apps like setup assistant." An audience member added, "thanks for admitting that!"

Support for java 2 is significant and was considered another top ten differentiation. It uses the hotspot java virtual machine, and is 'ideal for cross platform support.' Java apps present themselves with an aqua look and feel. Brett reported that enterprise apps are being ported to os x because of its complete java support.

Apple's team is happy with the results of their work. More than anything, mac os x is a foundation for further progress. In taking the time to build a well thought-out system that leverages not only the massive investment of decades of testing and billions of dollars of research involved with unix, but also the two billion lines of code in applications written for the mac os, apple is building a springing point for developers eager to take on the next generation technology of application frameworks invented at next. And throughout the os, there is pervasive support for open technologies, such as the use of xml in all preferences and configuration files; pdf in display and printing and opengl for 3d.

Now I just need to buy me a copy.

More information about os x, open source software and communicating