Indelible Ink

Enforcing Trusted SSL Certificates on iOS and OS X

2012-10-24T00:00:00+00:00

One of the nice things about iOS and OS X networking is that it generally just works. In particular, connecting to an SSL-protected host is a transparent operation for high-level code based on NSURLConnection. As long as the authenticity of the remote host’s public certificate can be verified, the connection is considered trustworthy.

This broad definition of trust allows connections to arbitrary hosts without requiring prior knowledge of those hosts’ certficates. This convenience is generally desirable, but there are times when we want to restrict our trust to a specific set of known certificates, not just any certificate signed by a well-known authority or registered with the Keychain.

For example, many iOS applications interact with a backend server component. These connections can be “sniffed” by introducing proxy software between the iOS device and the rest of the network. While this can be a wonderful development aid, it highlights the often misunderstood insecurity of these otherwise “private” communication channels. This scenario is known as a man-in-the-middle attack. (It is also how the Siri protocol was originally cracked.)

Fortunately, Apple includes all of the tools needed for an application to evaluate the trusthworthiness of individual network connection attempts:

Trusted Certificates

To begin, the client application needs its own copy of the trusted server’s public certificate. This should be in DER format (which is just a binary version of the more familiar PEM format). To convert a certificate from PEM to DER format:

$ openssl x509 -inform PEM -outform DER -in cert.pem -out cert.der

Then add the certificate to the application’s resource bundle (or otherwise encode it in the application binary). This is safe because it’s only the public portion of the certificate; the private key remains private.

Note that bundling the server’s public certificate with the application does introduce an ongoing maintenance concern. It will need to be updated whenever the server’s certificate changes (such as after a renewal). But this is the simplest approach for the illustrative purposes of this article.

Evaluating Server Trust

The application now needs a way to determine whether or not the remote host’s certificate matches the bundled certificate. This is done by establishing a chain of trust “anchored” on the local certificate. The server’s credentials are accessed via the NSURLProtectionSpace interface.

- (BOOL)shouldTrustProtectionSpace:(NSURLProtectionSpace *)protectionSpace {
    // Load up the bundled certificate.
    NSString *certPath = [[NSBundle mainBundle] pathForResource:@"cert" ofType:@"der"];
    NSData *certData = [[NSData alloc] initWithContentsOfFile:certPath];
    CFDataRef certDataRef = (__bridge_retained CFDataRef)certData;
    SecCertificateRef cert = SecCertificateCreateWithData(NULL, certDataRef);

    // Establish a chain of trust anchored on our bundled certificate.
    CFArrayRef certArrayRef = CFArrayCreate(NULL, (void *)&cert, 1, NULL);
    SecTrustRef serverTrust = protectionSpace.serverTrust;
    SecTrustSetAnchorCertificates(serverTrust, certArrayRef);

    // Verify that trust.
    SecTrustResultType trustResult;
    SecTrustEvaluate(serverTrust, &trustResult);

    // Clean up.
    CFRelease(certArrayRef);
    CFRelease(cert);
    CFRelease(certDataRef);

	// Did our custom trust chain evaluate successfully?
    return trustResult == kSecTrustResultUnspecified;
}

NSURLConnection

Integrating the above code with an NSURLConnection-based system is easy. First, the connection delegate needs to advertise its ability to authenticate “server trust” protection spaces.

- (BOOL)connection:(NSURLConnection *)connection canAuthenticateAgainstProtectionSpace:(NSURLProtectionSpace *)protectionSpace {
    return [protectionSpace.authenticationMethod isEqualToString:NSURLAuthenticationMethodServerTrust];
}

Then, the connection delegate needs to handle the authentication challenge:

- (void)connection:(NSURLConnection *)connection didReceiveAuthenticationChallenge:(NSURLAuthenticationChallenge *)challenge {
    if ([self shouldTrustProtectionSpace:challenge.protectionSpace]) {
        [challenge.sender useCredential:[NSURLCredential credentialForTrust:challenge.protectionSpace.serverTrust] forAuthenticationChallenge:challenge];
    } else {
        [challenge.sender performDefaultHandlingForAuthenticationChallenge:challenge];
    }
}

Trust failures will result in a connection failure:

The certificate for this server is invalid. You might be connecting
to a server that is pretending to be "api.example.com" which could
put your confidential information at risk.

AFNetworking

AFNetworking implements the NSURLConnectionDelegate methods internally. Fortunately, it also exposes a block-based interface for supplying custom implementations of the necessary callbacks. These are applied directly to AFHTTPRequestOperation objects, so a convenient place to set them is in a custom HTTPRequestOperation:success:failure: implementation.

- (AFHTTPRequestOperation *)HTTPRequestOperationWithRequest:(NSURLRequest *)urlRequest
                                                    success:(void (^)(AFHTTPRequestOperation *, id))success
                                                    failure:(void (^)(AFHTTPRequestOperation *, NSError *))failure {
    AFHTTPRequestOperation *operation = [super HTTPRequestOperationWithRequest:urlRequest success:success failure:failure];

    // Indicate that we want to validate "server trust" protection spaces.
    [operation setAuthenticationAgainstProtectionSpaceBlock:^BOOL(NSURLConnection *connection, NSURLProtectionSpace *protectionSpace) {
        return [protectionSpace.authenticationMethod isEqualToString:NSURLAuthenticationMethodServerTrust];
    }];

    // Handle the authentication challenge.
    [operation setAuthenticationChallengeBlock:^(NSURLConnection *connection, NSURLAuthenticationChallenge *challenge) {
        if ([self shouldTrustProtectionSpace:challenge.protectionSpace]) {
            [challenge.sender useCredential:[NSURLCredential credentialForTrust:challenge.protectionSpace.serverTrust]
                 forAuthenticationChallenge:challenge];
        } else {
            [challenge.sender performDefaultHandlingForAuthenticationChallenge:challenge];
        }
    }];

    return operation;
}

Manually Loading nib Files

2011-06-15T00:00:00+00:00

Interface Builder is the standard Mac OS and iOS user interface design tool. Layouts are saved as “nib” (NeXT Interface Builder) files. Recent versions of Interface Builder write these files as XML archives using the .xib file extension.

At runtime, nib files are loaded and assigned an “owner” object that forms the basis of the layout’s object hierarchy. This “owner” is very often a view controller object. In this article, we’ll focus specifically on UIKit and its UIViewController class.

The nib loading process is generally abstracted away as part of the standard UIViewController initialization process. Here’s a simple example:

- (id)init
{
    if ((self = [super initWithNibName:@"SomeNib" bundle:nil])) {
        // additional view controller initialization
    }

    return self;
}

-initWithNibName:bundle: handles all of the nib loading and instantiation work. While this is quite convenient, it has one major limitation: the nib file must have been compiled and bundled (in the NSBundle sense) along with the application itself. Fortunately, it is possible to manually load a nib from arbitrary data and associate its contents with a view controller.

The UINib Class

The UINib class provides all of the functionality needed to manually load nib files. One reason to do this is to pre-cache the contents of a nib file in memory, ready for quick deserialization and instantiation later in the application’s lifetime. Another reason (and our focus here) is to load the raw nib data from an alternate (non-NSBundle) source.

The UINib class provides two methods for loading nibs:

+nibWithNibName:bundle: - load a bundled nib file
+nibWithData:bundle: - load raw nib data

Once loaded, the nib’s objects can be instantiated and assigned an owner using the -instantiateWithOwner:options: instance method.

Manual View Controller Loading

How can the UINib class be used in conjunction with a view controller?

First, the view controller needs to call UIViewController’s designated initializer without a nib name. Note that passing nil as the nib name will trigger an automatic lookup for a nib named the same as the view controller class, so take care not to fall into that case unintentionally.

[super initWithNibName:nil bundle:nil]

Because this view controller hasn’t loaded a view during initialization, we must implement the -loadView method. This is where all of the manual loading is performed.

- (void)loadView
{
    [super loadView];

    UINib *nib = [UINib nibWithNibName:@"SomeNib" bundle:nil];
    [nib instantiateWithOwner:self options:nil];
}

This code effectively reproduces the standard nib loading process that was previously being handled by -initWithNibName:bundle: itself.

Loading Raw nib Data

At this point, our code is now manually loading the nib’s objects, but it’s still reading the nib itself from our application bundle. We can go a step further and load arbitrary nib data using the +nibWithData:bundle: method.

- (void)loadView
{
    [super loadView];

    NSData *data = [NSData dataWithContentsOfFile:@"SomeNib.xib.bin"];

    UINib *nib = [UINib nibWithData:data bundle:nil];
    [nib instantiateWithOwner:self options:nil];
}

The nib data can come from pretty much anywhere, such as from a bundled file or a network-fetched resource. The only requirement is that the nib data be compiled. To compile an existing .xib file, use ibtool:

ibtool SomeNib.xib --compile SomeNib.xib.bin

Additional Considerations

Now that the basic approach has been established, there are a few practical considerations to keep in mind. First, we’ve largely ignored the bundle parameter in the above code examples. This argument instructs the nib loader where to look for dependent resources, such as images. Passing nil implies that the main application bundle ([NSBundle mainBundle]) should be used. There unfortunately doesn’t appear to be a way to redirect these lookups to non-bundle locations, however.

Next, the nib data needs to be available by the time the view controller is initialized. All I/O needs to block, and any stalls could result in a poor user experience. Given that, any non-bundled nib data needs to be fetched before the view is loaded.

Lastly, nibs can contain references to “external” objects. One such example is the “owner” object, which is explicitly resolved by the call to -instantiateWithOwner:options:. If the nib references any additional external objects, the loading code must provide those objects via a dictionary mapping passed through the options: interface’s UINibExternalObjects key.

Protocol Buffer Polymorphism

2010-12-15T00:00:00+00:00

Protocol Buffers provide an efficient way to encode structured data for serialization. The language’s basic organizational type is a message, which can be thought of as a C-style structure. It is named and contains some number of fields. Messages can also be extended, but the method by which this is accomplished differs from familiar C++ or Java-style inheritance. Instead, message extension is implemented by reserving some number of field indices in the base message for use by the extending messages.

message BaseType
{
    // Reserve field numbers 100 to 199 for extensions.
    extensions 100 to 199;

    // All other field numbers are available for use here.
    required string name = 1;
    optional uint32 quantity = 2;
}

extend BaseType
{
    // This extension can only use field numbers 100 to 199.
    optional float price = 100;
}

But can protocol buffers model more flexible inheritance and polymorphism hierarchies?

Optional Message Fields

One approach to implementing polymorphism involves the use of optional message fields defined in either the base message or in an extension. Each “subclass” is defined as an independent message type that can optionally be included in the top-level message.

message Cat
{
    optional bool declawed = 1;
}

message Dog
{
    optional uint32 bones_buried = 1;
}

message Animal
{
    required float weight = 1;
    optional Dog dog = 2;
    optional Cat cat = 3;
}

This composition scheme is simple but has the property of allowing multiple nested message types to be filled out at the same time. While desirable in some contexts, this means that the deserialization code must test for the availability of each and every optional message type (unless an additional type hint field is added). It can could also be considered less than “pure” as a result.

Embedded Serialized Messages

A second approach simply “embeds” the subclass’s serialized contents within a bytes field in the parent message. An explicit type field informs the deserialization code of the embedded message’s type.

message Animal
{
    enum Type
    {
        Cat = 1;
        Dog = 2;
    }

    required Type type = 1;
    required bytes subclass = 2;
}

This brute-force approach to the problem, which effective, is both inelegant and inefficient. It can be useful where it is desirable to defer the deserialization of the embedded message, however, such as in routing systems that are only interested in decoding the outer enveloping message’s fields.

Nested Extensions

The final (and most generally recommended) approach uses a combination of nested messages and extensions to implemented polymorphic message types. This works by having each subclass reference itself from within a nested extension of the base type. An explicit type field is still required to guide the deserialization process.

message Animal
{
    extensions 100 to max;

    enum Type
    {
        Cat = 1;
        Dog = 2;
    }

    required Type type = 1;
}

message Cat
{
    extend Animal
    {
        required Cat animal = 100; // Unique Animal extension number
    }

    // These fields can use the full number range.
    optional bool declawed = 1;
}

message Dog
{
    extend Animal
    {
        required Dog animal = 101; // Unique Animal extension number
    }

    // These fields can use the full number range.
    optional uint32 bones_buried = 1;
}

It may not be immediately obvious how to work with this message structure, so here’s an example using the Python API:

from animals_pb2 import *

# Construct the polymorphic base message type.
animal = Animal()
animal.type = Animal.Cat

# Create the subclass type by referencing the appropriate extension type.
# Note that this uses the self-referential field (Cat.animal) from within
# nested message extension.
cat = animal.Extensions[Cat.animal]
cat.declawed = True

# Serialize the complete message contents to a string.  It will end up
# looking roughly like this: [ type [ declawed ] ]
bytes = animal.SerializeToString()

# ---

# Unpack the serialized bytes.
animal = Animal()
animal.ParseFromString(bytes)

# Determine the appropriate extension type to use.
extension_map = { Animal.Cat: Cat.animal, Animal.Dog: Dog.animal }
extension = animal.Extensions[extension_map[animal.type]]

This approach tends to be the most efficient and robust implementation, albeit a bit non-obvious to protocol buffer newcomers.

Conclusion

As shown above, there are multiple ways to implement polymorphic message behavior using protocol buffers. Like most things in software, the approaches differ in complexity and efficiency.

It’s a little unfortunate that there isn’t a more obvious language syntax for expressing inheritance. For example, many programmers would likely find these definitions familiar:

message Animal { ... }
message Cat : Animal { ... }
message Dog : Animal { ... }

There are of course open questions as to how this syntax would map to the underlying concepts of messages and extensions, but it shouldn’t be too difficult to enforce things like extension number ranges, etc. during the compilation process. Perhaps something along these lines will be undertaken by a motivated developer or an ambitious Google Summer of Code student.

Reloading Python Modules

2010-10-24T00:00:00+00:00

Being able to reload code modules is one of the many nice features of Python. This allows developers to modify parts of a Python application while the interpreter is running. In general, all that needs to be done is pass a module object to the imp.reload() function (or just reload() in Python 2.x), and the module will be reloaded from its source file.

There are a few potential complications, however.

If any other code references symbols exported by the reloaded module, they may still be bound to the original code. For example, imagine if module A contains the constant INTERVAL = 5, and module B imports that constant into its namespace (from A import INTERVAL). If we change the constant to INTERVAL = 10 and just reload module A, any values in module B that were based on INTERVAL won’t be updated to reflect its new value.

The solution to this problem is to also reload module B. But it’s important to only reload module B after module A has been reloaded. Otherwise, it won’t pick up the updated symbols.

PyUnit deals with a variation of this problem by introducing a rollback importer. That approach “rolls back” the set of imported modules to some previous state by overriding Python’s global __import__ hook. PyUnit’s solution is effective at restoring the interpreter’s state to pre-test conditions, but it’s not a general solution for live code reloading because the unloaded modules aren’t automatically reloaded.

The following describes a general module reloading solution which aims to make the process automatic, transparent, and reliable.

Recording Module Dependencies

It is important to understand the dependencies between loaded modules so that they can be reloaded in the correct order. The ideal solution is to build a dependency graph as the modules are loaded. This can be accomplished by installing a custom import hook that is called as part of the regular module import machinery.

import builtins

_baseimport = builtins.__import__
_dependencies = dict()
_parent = None

def _import(name, globals=None, locals=None, fromlist=None, level=-1):
    # Track our current parent module.  This is used to find our current
    # place in the dependency graph.
    global _parent
    parent = _parent
    _parent = name

    # Perform the actual import using the base import function.
    m = _baseimport(name, globals, locals, fromlist, level)

    # If we have a parent (i.e. this is a nested import) and this is a
    # reloadable (source-based) module, we append ourself to our parent's
    # dependency list.
    if parent is not None and hasattr(m, '__file__'):
        l = _dependencies.setdefault(parent, [])
        l.append(m)

    # Lastly, we always restore our global _parent pointer.
    _parent = parent

    return m

builtins.__import__ = _import

This code chains the built-in __import__ hook (stored in _baseimport). It also tracks the current “parent” module, which is the module that is performing the import operation. Top-level modules won’t have a parent.

After a module has been successfully imported, it is added to its parent’s dependency list. Note that this code is only interested in file-based modules; built-in extensions are ignored because they can’t be reloaded.

This results in a complete set of per-module dependencies for all modules that are imported after this custom import hook has been installed. These dependencies can be easily queried at runtime:

def get_dependencies(m):
    """Get the dependency list for the given imported module."""
    return _dependencies.get(m.__name__, None)

Reloading Modules

The next step is to build a dependency-aware reload() routine.

import imp

def _reload(m, visited):
    """Internal module reloading routine."""
    name = m.__name__

    # Start by adding this module to our set of visited modules.  We use
    # this set to avoid running into infinite recursion while walking the
    # module dependency graph.
    visited.add(m)

    # Start by reloading all of our dependencies in reverse order.  Note
    # that we recursively call ourself to perform the nested reloads.
    deps = _dependencies.get(name, None)
    if deps is not None:
        for dep in reversed(deps):
            if dep not in visited:
                _reload(dep, visited)

    # Clear this module's list of dependencies.  Some import statements
    # may have been removed.  We'll rebuild the dependency list as part
    # of the reload operation below.
    try:
        del _dependencies[name]
    except KeyError:
        pass

    # Because we're triggering a reload and not an import, the module
    # itself won't run through our _import hook.  In order for this
    # module's dependencies (which will pass through the _import hook) to
    # be associated with this module, we need to set our parent pointer
    # beforehand.
    global _parent
    _parent = name

    # Perform the reload operation.
    imp.reload(m)

    # Reset our parent pointer.
    _parent = None

def reload(m):
    """Reload an existing module.

    Any known dependencies of the module will also be reloaded."""
    _reload(m, set())

This reload() implementation uses recursion to reload all of the requested module’s dependencies in reverse order before reloading the module itself. It uses the visited set to avoid infinite recursion should individual modules’ dependencies cross-reference one another. It also rebuilds the modules’ dependency lists from scratch to ensure that they accurately reflect the updated state of the modules.

Custom Reloading Behavior

The reloading module may wish to implement some custom reloading logic, as well. For example, it may be useful to reapply some pre-reloaded state to the reloaded module. To support this, the reloader looks for a module-level function named __reload__(). If present, this function is called after a successful reload with a copy of the module’s previous (pre-reload) dictionary.

Instead of simply calling imp.reload(), the code expands to:

# If the module has a __reload__(d) function, we'll call it with a
# copy of the original module's dictionary after it's been reloaded.
callback = getattr(m, '__reload__', None)
if callback is not None:
    d = _deepcopy_module_dict(m)
    imp.reload(m)
    callback(d)
else:
    imp.reload(m)

The _deepcopy_module_dict() helper routine exists to avoid deepcopy()-ing unsupported or unnecessary data.

def _deepcopy_module_dict(m):
    """Make a deep copy of a module's dictionary."""
    import copy

    # We can't deepcopy() everything in the module's dictionary because
    # some items, such as '__builtins__', aren't deepcopy()-able.
    # To work around that, we start by making a shallow copy of the
    # dictionary, giving us a way to remove keys before performing the
    # deep copy.
    d = vars(m).copy()
    del d['__builtins__']
    return copy.deepcopy(d)

Monitoring Module Changes

A nice feature of a reloading system is automatic detection of module changes. There are many ways to monitor the file system for source file changes. The approach implemented here uses a background thread and the stat() system call to watch each file’s last modification time. When an updated source file is detected, its filename is added to a thread-safe queue.

import os, sys, time
import queue, threading

_win = (sys.platform == 'win32')

class ModuleMonitor(threading.Thread):
    """Monitor module source file changes"""

    def __init__(self, interval=1):
        threading.Thread.__init__(self)
        self.daemon = True
        self.mtimes = {}
        self.queue = queue.Queue()
        self.interval = interval

    def run(self):
        while True:
            self._scan()
            time.sleep(self.interval)

    def _scan(self):
        # We're only interested in file-based modules (not C extensions).
        modules = [m.__file__ for m in sys.modules.values()
                if '__file__' in m.__dict__]

        for filename in modules:
            # We're only interested in the source .py files.
            if filename.endswith('.pyc') or filename.endswith('.pyo'):
                filename = filename[:-1]

            # stat() the file.  This might fail if the module is part
            # of a bundle (.egg).  We simply skip those modules because
            # they're not really reloadable anyway.
            try:
                stat = os.stat(filename)
            except OSError:
                continue

            # Check the modification time.  We need to adjust on Windows.
            mtime = stat.st_mtime
            if _win32:
                mtime -= stat.st_ctime

            # Check if we've seen this file before.  We don't need to do
            # anything for new files.
            if filename in self.mtimes:
                # If this file's mtime has changed, queue it for reload.
                if mtime != self.mtimes[filename]:
                    self.queue.put(filename)

            # Record this filename's current mtime.
            self.mtimes[filename] = mtime

An alternative approach could use a native operation system file monitoring facility, such as the Win32 Directory Change Notification system.

The Reloader object polls for source file changes and reloads modules as necessary.

import imp
import reloader

class Reloader(object):

    def __init__(self):
        self.monitor = ModuleMonitor()
        self.monitor.start()

    def poll(self):
        filenames = set()
        while not self.monitor.queue.empty():
            try:
                filenames.add(self.monitor.queue.get_nowait())
            except queue.Empty:
                break
        if filenames:
            self._reload(filenames)

    def _reload(self, filenames):
        modules = [m for m in sys.modules.values()
                if getattr(m, '__file__', None) in filenames]

        for mod in modules:
            reloader.reload(mod)

In this model, the reloader needs to be polled periodically for it to react to changes. The simplest example would look like this:

r = Reloader()
while True:
    r.poll()
    time.sleep(1)

The complete source code is on GitHub. The package distribution is available as reloader on the Python Package Index.

Android Text Links Using Linkify

2010-04-09T00:00:00+00:00

The Android framework provides an easy way to automatically convert text patterns into clickable links. By default, Android knows how to recognize web URLs, email addresses, map addresses, and phone numbers, but it also includes a flexible mechanism for recognizing and converting additional text patterns, as well.

The Android Developers Blog has an article entitled Linkify your Text! that provides a nice overview of the system. It discusses how the Linkify class can be used to enable the default link patterns and then continues with a more advanced WikiWords example that demonstrates custom links. That article is a fine introduction to the system, so the rest of this article will primarily focus on details not covered therein.

All of the examples in this article are based on the TextView widget. The Linkify class can also be used to add links to Spannable text, but those use cases won’t be covered here because their usage is nearly identical to the TextView cases.

TextView AutoLinking

The TextView widget features an android:autoLink attribute that controls the types of text patterns that are automatically recognized and converted to clickable links. This attribute is a convenient way to enable one or more of the default link patterns because it can be configured directly from a layout without involving any additional code.

However, for those cases where programmatically setting this value is useful, the setAutoLinkMask() function exists.

There is one important caveat to using this “auto-linking” functionality, however. It appears that when “auto-linking” is enabled, all additional Linkify operations are ignored. It’s unclear whether this behavior is intentional or inadvertent, so it’s possible things could change in future released of the Android SDK. Consider disabling “auto-linking” before using any of the Linkify operations discussed below.

// Disable the text view's auto-linking behavior
textView.setAutoLinkMask(0);

Default Link Patterns

Enabling support for one of Android’s default link patterns is very easy. Simply use the addLinks(TextView text, int mask) function and specify a mask that describes the desired link types.

import android.text.util.Linkify;

// Recognize phone numbers and web URLs
Linkify.addLinks(text, Linkify.PHONE_NUMBERS | Linkify.WEB_URLS);

// Recognize all of the default link text patterns 
Linkify.addLinks(text, Linkify.ALL);

// Disable all default link detection
Linkify.addLinks(text, 0);

Custom Link Patterns

Detecting additional types of link patterns is easy, too. The addLinks(TextView text, Pattern pattern, String scheme) function detects links based on a regular expression pattern.

import java.util.regex.Pattern;
import android.text.util.Linkify;

// Detect US postal ZIP codes and link to a lookup service
Pattern pattern = Pattern.compile("\\d{5}([\\-]\\d{4})?");
String scheme = "http://zipinfo.com/cgi-local/zipsrch.exe?zip=";
Linkify.addLinks(text, pattern, scheme);

The text is scanned for pattern matches. Matches are converted to links that are generated by appending the matched text to the provided URL scheme base.

Note that the scheme doesn’t have to be an external web-like URL. It could also be an Android Content URI that can be used in conjunction with a content provider to reference application resources, for example.

Match Filters

Regular expressions are a very powerful way to match text patterns, but sometimes a bit more flexibility is needed. The MatchFilter class provides this capability by giving user code a chance to evaluate the link worthiness of some matched text.

import java.util.regex.Pattern;
import android.text.util.Linkify;
import android.text.util.Linkify.MatchFilter;

// A match filter that only accepts odd numbers.
MatchFilter oddFilter = new MatchFilter() {
    public final boolean acceptMatch(CharSequence s, int start, int end) {
        int n = Character.digit(s.charAt(end-1), 10);
        return (n & 1) == 1;
    }
};

// Match all digits in the pattern but restrict links to only odd
// numbers using the filter.
Pattern pattern = Pattern.compile("[0-9]+");
Linkify.addLinks(text, pattern, "http://...", oddFilter, null);

A more complex (but useful!) example would involve matching valid dates. The regular expression could be generous enough to match strings like “2010-02-30” (February 30, 2010), but a match filter could provide the logic to reject bogus calendar dates.

Transform Filters

Up until this point, the final link was always being generated based on the exact matched text. There are many cases where that is not desirable, however. For example, it’s common to mention a username using the @username syntax, but the resulting link should only include the username portion of the text. The TransformFilter class provides a solution.

import java.util.regex.Pattern;
import android.text.util.Linkify;
import android.text.util.Linkify.TransformFilter;

// A transform filter that simply returns just the text captured by the
// first regular expression group.
TransformFilter mentionFilter = new TransformFilter() {
    public final String transformUrl(final Matcher match, String url) {
        return match.group(1);
    }
};

// Match @mentions and capture just the username portion of the text.
Pattern pattern = Pattern.compile("@([A-Za-z0-9_-]+)");
String scheme = "http://twitter.com/";
Linkify.addLinks(text, pattern, scheme, null, mentionFilter);

This approach uses the regular expression’s capture syntax to extract just the username portion of the pattern as a uniquely addressable match group. Alternatively, the transform filter could just return all of the matched text after the first character (@), but the above approach is nice because it keeps all of the pattern’s details within the regular expression.

Of course, transform filters can be combined with match filters for ultimate flexibility. The Android SDK uses this approach to detect wide ranges of phone number formats (many of which include various parentheses and dashes) while always generating a simplified link containing only digits.

Password Composer for iPhone

2009-10-25T00:00:00+00:00

I often use Password Composer (written by Johannes la Poutré) to generate unique, per-site passwords. It does an excellent job because it’s simple, unobtrusive, and reliable. The one downside is that you need to have it available in order to (re)generate the password for a given web site, and that isn’t always convenient, despite the large number of existing Password Composer implementations.

The main place I find myself missing Password Composer is on my iPhone. The only current solution that works from Mobile Safari is the public web-based interface, and I’m security-conscious enough to avoid using that version whenever possible. I decided the only reasonable solution would be to find a way to run Password Composer locally on my iPhone.

I first considered writing a full-blown native iPhone application. I know a bit of Objective-C and have experimented with the iPhone SDK, but, additional learning opportunities aside, this approach felt like overkill.

I then discovered the option of writing an iPhone-enhanced offline web application. That would let me reuse Password Composer’s existing static web form without having to work through the iPhone SDK. I found the early release of Jonathan Stark’s Building iPhone Applications with HTML, CSS, and JavaScript and set out to adapt Password Composer to the iPhone.

Basic Application

The basic application started as a single HTML file containing a form and some JavaScript.

<html>
<head>
    <title>Password Composer</title>
</head>
<body>

<script type="text/javascript">
// Password Composer JavaScript
</script>

<h1>Password Composer</h1>

<form>
<table>
<tr>
    <td>Master Key:</td>
    <td><input type="password" id="masterpwd1" onkeyup="mpwd_generate()"
        onchange="mpwd_generate()"/></td>
</tr>
<tr>
    <td>Domain:</td>
    <td><input type="text" id="domain1" onkeyup="mpwd_generate()"
        onchange="mpwd_generate()"/></td>
</tr>
<tr>
    <td>Password:</td>
    <td><input type="text" id="genpwd1" onkeyup="mpwd_generate()"
        onchange="mpwd_generate()"/></td>
</tr>
</table>
</form>

</body>
</html>

The JavaScript and form were copied directly from Password Composer’s public web-based interface and worked just fine in Mobile Safari.

Styling the Application

Next, I set about styling the look and feel of the page to more closely resemble an iPhone application. I started by adding a stylesheet file. I could have used inline CSS, and maybe I’ll merge the styles back into the main page at some point in the future, but using a separate file keeps things more manageable for the time being.

body {
    background-color: #ddd;
    color: #222;
    font-family: Helvetica; 
    font-size: 14px;
    margin: 0;
    padding: 0;
}

h1 {
    background-image: -webkit-gradient(linear, left top, left bottom,
                                        from(#ccc), to(#aaa));
    background-color: #ccc;
    border-bottom: 1px solid #666;
    color: #222;
    font-size: 20px;
    font-weight: bold;
    padding: 10px 0;
    text-align: center;
    text-decoration: none;
    text-shadow: 0px 1px 0px #ddd;
}

form {
    background-color: #FFFFFF;
    border: 1px solid #999999;
    color: #222222;
    font-size: 15px;
    padding: 12px 10px;
    margin: 10px;
    -webkit-border-radius: 8px;
}

td, input {
    font-size: 14px;
}

Most of these styles are influenced by examples from Jonathan Stark’s book.

Viewport Definition

The next change is the addition of a viewport meta tag:

<meta name="viewport" content="user-scalable=no, width=device-width" />

This viewport definition convinces Mobile Safari not to rescale the HTML content as though it were a “normal” 980px-wide web page.

Input Field Types

I was also interested in configuring the form’s <input> fields to pop up the most appropriate type of on-screen keyboard on the iPhone. Master Key was already being recognized as a password field (type=password), but I wanted the Domain field to use the iPhone’s special URL keyboard.

A little research indicated that Mobile Safari recognizes a small subset of the HTML5 input types, including the url type.

<input type="url" id="domain1">

Custom Icon

Because I want Password Composer to feel like a natural, first-class iPhone application, it deserves to get a real icon. This icon is used when a link to the application is saved to the user’s home screen. Home screen icons are 57×57 pixel PNG images.

There are two ways for a web application to specify its home screen icon:

Provide a file named apple-touch-icon.png in the site’s root.
Add a <link rel="apple-touch-icon" href="icon.png" /> tag to the page’s <head> block.

I chose the second approach because I wasn’t planning on using an entire domain to serve the Password Composer application. It also feels like better encapsulation for a single-page web application.

Offline Application Cache

The last bit of work adds support for the iPhone’s offline application cache. This cache allows the files used by a web-based applications to be stored locally on the iPhone so that they can still be accessed when the phone doesn’t have an active network connection. This “offline mode” is described quite well in Stark’s book: Chapter 6: Going Offline.

Briefly, all that’s required is the creation of a “manifest” file, as specified by HTML5. My manifest file is named pwdcomposer.manifest:

CACHE MANIFEST
index.html
iphone.css
icon.png

The manifest lists all of the files used by the web application. Manifest files can be much more involved than this example. They can specify fallback images for use when network-based files are inaccessible, for example. But for my current purposes, the simple manifest above works great.

Manifest files must be served using the text/cache-manifest MIME type. For Apache-like web servers, this can be configured using a directive like:

AddType text/cache-manifest .manifest

Lastly, the web page needs to specify its associated manifest file using the manifest attribute of the <html> element:

<html manifest="pwdcomposer.manifest">

Deployment

All that is left to do is deploy the application. Simply serve up the source files using a web server, and iPhone users can choose to bookmark the application’s link just like they would for any other web page. The offline application caching system will work automatically. When the iPhone is connected to a network, Mobile Safari will attempt to fetch the original manifest file to check for updates, but the application will otherwise be running entirely from the cache, essentially making it a local iPhone application.

My version is deployed here: https://www.indelible.org/pwdcomposer/

The complete source code is available on GitHub.

Vim Color Schemes

2009-04-21T00:00:00+00:00

The Vim text editor supports highly-configurable color schemes which build upon the editor’s rich syntax highlighting system. The stock Vim distribution includes a number of color schemes, and many more are available from the Vim Scripts repository.

Color scheme definitions are simply normal Vim scripts that live in the colors/ directory of the Vim runtime hierarchy (see :help runtimepath).

Color schemes are loaded using the :colorscheme command. The scheme’s name is determined by the filename of its script file (minus the .vim extension). For example, to load the stock blue color scheme (which is defined by the colors/blue.vim script):

:colorscheme blue

Creating Color Schemes

Creating a custom color scheme is quite easy. Start by creating a new Vim script file in the colors/ directory based on the name of the new scheme. Start the script with the following commands:

set background=dark
"or background=light
highlight clear
if exists("syntax_on")
    syntax reset
endif
let g:colors_name = "example"

These commands will reset the syntax highlighting system to its default state. Note that some color scheme scripts might prefer a light background, so that first line should be changed accordingly. (highlight clear uses the background value, so background must be set first.)

The final line sets the global colors_name variable to the scheme’s name (example, in this example).

The rest of the script defines the color scheme itself. This is accomplished primarily through the highlight (or hi) command. Each highlight command sets the colors for a single syntax group. Setting the colors for the Comments group might look like this:

hi Comment ctermbg=black ctermfg=darkgrey guibg=#000000 guifg=#777777

To see the full list of Vim’s syntax groups (along with their current highlight settings), run the following command from within the editor:

:source $VIMRUNTIME/syntax/hitest.vim

Testing Runtime Features

Because the color scheme is simply a Vim script, you can conditionalize the definitions based on various runtime values. The presence of the gui_running feature indicates that the Vim GUI is running, for example:

if has('gui_running')
    " GUI colors
else
    " Non-GUI (terminal) colors
endif

And the terminal’s color range – the number of available colors – can be queried via the &t_Co variable:

if &t_Co > 255
    " More than 256 colors are available
endif

Additional Configuration

Color scheme scripts can support basic configuration using global variables.

if exists("g:example_force_dark")
    set background=dark
endif

The user should set this variable in his .vimrc file before loading the color scheme script.

let g:example_force_dark = 1
colorscheme example

(Global variables can be “unset” using the unlet command.)

Virtual Ethernet Tunneling

2009-03-03T00:00:00+00:00

This paper discusses the implementation of virtual Ethernet tunnels using OpenBSD. The current release of OpenBSD at the time of writing (2001) was version 2.9, so some of the material may be fairly dated. I haven’t revisited the details since then.

Overview

Without going too deep into the technical details, a virtual Ethernet tunnel uses packet encapsulation, Ethernet bridging, and IPSec encryption to tunnel a subnet from one host to another host over a public network (generally, the Internet).

The best way to explain how this works is by describing each of the pieces involved.

We’ll start with packet encapsulation. Ethernet frames can be encapsulation within an IP packet and transported to a remote host, who can then strip off the surrounding IP headers and retrieve the original, unmodified Ethernet frame. This is the essence of tunneling packets from one network to another over an intermediate network. OpenBSD supports this type of encapsulated tunnel using the gif interface.

Bridging is definitely not a new networking technology, but once the OpenBSD code was expanded to allow the inclusion of virtual interfaces in bridge groups along with traditional physical interfaces, it opened up a number of new possibilities. In our case, it allows us to group the generic gif interfaces (which represent our Ethernet tunnel, as previously described) with a physical interface. This allows the physical interfaces to share traffic with the tunnel interface as though they existing on the same subnet. And because the tunnel interface is really a stream of encapsulated packets from some other host, we can effectively link two segments of the same subnet using two bridges (one on each host) and an encapsulated tunnel to span the hosts.

     (bridged interfaces)                  (bridged interfaces)
        |            |                        |            |
        v +--------+ v                        v +--------+ v
     /----| Host 1 |-----[ public network ]-----| Host 2 |----\
     |    +--------+ ^    <-(gif tunnel)->    ^ +--------+    |
  private            |                        |            private
  network           gif                      gif           network

The final piece of the system is IPSec. While not absolutely required to tunnel between our bridged interfaces, it adds a welcome layer of security and helps maintain the privacy of our “private” network. The IPSec part of the equation is perhaps the most complex and difficult to set up, but it scarcely causes problems once it’s running, and it’s not really as hard to get going as it may sound.

Required sysctl Settings

Before your system can forward packets between interfaces or handle the IPSec protocols, the following sysctls need to be enabled (i.e. set to 1):

net.inet.ip.forwarding
net.inet.esp.enable
net.inet.ah.enable
net.inet.etherip.allow

OpenBSD can enable these settings on boot based on the contents of /etc/sysctl.conf. Make sure you have at least the following lines uncommented:

net.inet.ip.forwarding=1        # 1=Permit forwarding (routing) of packets
net.inet.esp.enable=1           # 1=Enable the ESP IPSec protocol
net.inet.ah.enable=1            # 1=Enable the AH IPSec protocol

These parameters can also be modified at runtime using the sysctl tool.

ipsecadm and Security Associations

OpenBSD includes a tool named ipsecadm that is used for managing the system’s security associations (SA’s) and flows. The tool accepts a number of arguments, but we’ll mainly be working with the SA creation syntax:

ipsecadm new esp -spi 2000 -dst 66.24.105.57 -src 129.21.111.216 -enc 3des \
-auth sha1 -key d09fffc3ebaee12362d65b38068dd381df89e4961ed282b3 -authkey \
5ee0fc2cc2197fe24417934cac6db483b53eace3

This command will create a new security association using the esp IPSec protocol. We’ve assigned this entry an SPI (unique index) of 2000. We also set the source and destination addresses for this SA. Note that these are specific to the SA and don’t assume anything about the host, meaning that the source address isn’t necessarily the local machine’s address.

We continue by defining the desired encryption algorithm (3des), the authentication algorithm, and the two encryption keys. We’ll cover key generation in the Generating Keys section below.

The following is an example of a second, complementary SA:

ipsecadm new esp -spi 2001 -dst 129.21.111.216 -src 66.24.105.57 -enc 3des \
-auth sha1 -key d09fffc3ebaee12362d65b38068dd381df89e4961ed282b3 -authkey \
5ee0fc2cc2197fe24417934cac6db483b53eace3

This entry is nearly the same as the previous SA. Note, however, that we’ve assigned it a different SPI (2001). We’ve also swapped the source and destination addresses.

We now have two SA’s defined, one for each direction of packet flow. These two SA’s must be defined identically on both hosts – don’t change the ordering of the addresses or alter the encryption protocols or keys on one of the hosts! Because the SPI is included with each encrypted packet - and each host uses the SPI to determine how the packet should be routed, encrypted, or decrypted - the ordering must remain consistent. In other words, security associations are unique to the connection, not the hosts.

The last thing we need to do is define a flow. A flow determines what security parameters a packet should have for either input or output. Here’s an example:

ipsecadm flow -dst 66.24.105.57 -out -transport etherip -require -addr \
129.21.111.216 255.255.255.255 66.24.105.57 255.255.255.255

This command defines a flow for packets destined for 66.24.105.57. We specify that we’ll be transporting these packets using “Ethernet in IP” encapsulation, and the packets will be traveling from us at 129.21.111.216 to our destination at 66.24.105.57.

Unlike security associations, flows are unique to the individual host’s configuration. The command for the opposite host would be:

ipsecadm flow -dst 129.21.111.216 -out -transport etherip -require -addr \
66.24.105.57 255.255.255.255 129.21.111.216 255.255.255.255

Note that all we’ve done here is change the flow’s destination address and swap the source and destination address of the connection.

Generating Keys

The key’s size depends on the encryption protocol. DES and 3DES use 8 and 24 bits, respectively. The sizes for the other protocols (Rijndael, Blowfish, CAST) may vary.

Keys can be generated using the following command:

openssl rand 24 | hexdump -e '24/1 "%02x"' && echo ""

This will generate a hexadecimal representation of a 24-bit key. If a different key size is needed, replace the occurrences of 24 with the desired bit size.

Configuring the Ethernet Interfaces

Two Ethernet interfaces on each host are required for this sort of tunneling. Each interface must sit on a different subnet. One of those subnets should obviously be the one whose addresses you want to tunnel. In our example, that network is 129.21.60.0/24. The other network can be pretty much anything (a cable modem network, for example).

The easiest way to configure the network interfaces is by using the /etc/hostname.interface file. OpenBSD will execute the interface configuration commands listed inside this file upon boot, which is generally desirable for this kind of setup.

We’ll begin by configuring the “public” interface (/etc/hostname.sis0, in my case):

inet 129.21.111.216 255.255.255.128 NONE

Be sure to set up this interface correctly (including the netmask!). In this case, the interface sits on a nine-bit subnet, so we set the netmask accordingly. Test that this interface works correctly before proceeding.

Now we’ll configure the second network interface, the one that sits on the subnet that we want to tunnel (/etc/hostname.dc0, for me):

inet 129.21.60.107 255.255.255.0 NONE

A Note on Gateway Addresses

It is important to note that each host’s gateway address must be set to the “public” network’s gateway (not the subnet that you are tunneling!). Things will not work correctly otherwise.

The system’s gateway address can be statically configured using the /etc/mygate file. It might contain a line like this:

129.21.111.254

If your host uses DHCP to configure its gateway address (in the case of a cable modem provider), this will all be handled for you and there is no need to configure your gateway address by hand.

The Generic Interface

The generic interface (gif) is used for the actual tunneling between the hosts. This interface is purely virtual, meaning it is not necessarily bound to any physical interface on the system. Instead, it is given a source and a destination address between which to tunnel its encapsulated by packets.

More detailed information is available in the manpage.

The gif interface can be configured at boot via the /etc/hostname.gif0 file. Here is the /etc/hostname.gif0 file that I use:

giftunnel 129.21.111.216 66.24.105.57
up

The first line establishes the tunnel between the local (source) address and the remote (destination) address. The second line activates the interface.

Both of these command strings are passed directly to ifconfig.

The Bridge Interface

OpenBSD includes excellent Ethernet bridging support. Each bridge is represented by a bridge interface (e.g. /dev/bridge0). Bridge configuration is performed using the brconfig tool.

Each bridge can have an arbitrary number of interfaces added to it. These interfaces can either be physical network interfaces or virtual encapsulation interfaces (such as the gif interface).

More detailed information is available in the manpage.

Bridge configuration can also be performed upon boot. This is accomplished through the /etc/bridgename.bridge0 file. Here’s mine:

add gif0
add dc0
#
!ipsecadm flush
!ipsecadm new esp -spi 2000 -dst 66.24.105.57 -src 129.21.111.216 -enc 3des   \
    -auth sha1 -key d09fffc3ebaee12362d65b38068dd381df89e4961ed282b3 -authkey \
    5ee0fc2cc2197fe24417934cac6db483b53eace3
!ipsecadm new esp -spi 2001 -dst 129.21.111.216 -src 66.24.105.57 -enc 3des   \
    -auth sha1 -key d09fffc3ebaee12362d65b38068dd381df89e4961ed282b3 -authkey \
    5ee0fc2cc2197fe24417934cac6db483b53eace3
!ipsecadm flow -dst 66.24.105.57 -out -transport etherip -require -addr       \
    129.21.111.216 255.255.255.255 66.24.105.57 255.255.255.255
#
up

The contents of this file will require a little bit of explanation.

The first two lines add the gif0 and dc0 interfaces to this bridge interface. Once these two interfaces are placed in a bridge group, packets will be able to move freely between them, as if they existing on the same physical Ethernet segment.

The second set of commands are all prefixed with a exclamation point (!). This indicates that these commands are not ifconfig parameters. Instead, they should be executed on their own during the configuration sequence.

These commands set up the various security associations using ipsecadm, as was discussed earlier. Note that the first thing we do is flush any existing security associations and flows. This starts us with a clean slate every time.

The last line simply activates the bridge interface.

Additional Information

Using kernfs

OpenBSD’s kernfs virtual file system exports some useful information on the system’s current IPSec settings. This information can be very handy when debugging setups and gathering statistics.

To begin, you must first mount the kernfs file system (which isn’t done as part of a default installation). You’ll need to create a new mount point (/kern is typical):

# mkdir /kern

Next, you’ll need to add the following line to you /etc/fstab file:

/kern /kern kernfs ro 0 0

You can now mount the kernfs file system using the command:

# mount /kern

The information we want is stored in /kern/ipsec. It can be viewed simply by cat‘ing the file:

% cat /kern/ipsec

After you’ve set up a couple of security associations, your output will look something like this:

Hashmask: 31, policy entries: 1
SPI = 00002000, Destination = 66.24.105.57, Sproto = 50
        Established 1283589 seconds ago
        Source = 129.21.111.216
        Flags (00000000) = 
        Crypto ID: 1
        xform = 
                Encryption = <3DES>
                Authentication = 
        174365335 bytes processed by this SA
        Expirations:
                (none)

SPI = 00002001, Destination = 129.21.111.216, Sproto = 50
        Established 1283589 seconds ago
        Source = 66.24.105.57
        Flags (00000000) = 
        Crypto ID: 2
        xform = 
                Encryption = <3DES>
                Authentication = 
        405224 bytes processed by this SA
        Expirations:
                (none)

Increasing the Number of Interfaces

Because each virtual Ethernet tunnel requires one gif interface and one bridge interface, a stock OpenBSD installation can only support two tunnels (only two bridge interfaces are available by default in OpenBSD 2.9).

Increasing the number of available is fairly trivial, however. The downside is that it currently requires a kernel recompilation, which inherently requires a reboot in order to see the effects of the new kernel.

Note that the details of building a kernel under OpenBSD are outside the scope of this document, but the topic is conferred pretty well as part of the OpenBSD FAQ.

Begin by editing your kernel configuration file. Note that the definitions for the gif and bridge devices are in the GENERIC kernel configuration file, which your configuration file probably includes. We’ll simply be overriding these default entries.

Add the following lines to your kernel configuration file:

pseudo-device   gif             8   # IPv[46] over IPv[46] tunnel (RFC1933)
pseudo-device   bridge          8   # network bridging support

Feel free to replace the number of devices (8, in this case) with a number of your own choosing. It’s up to you and your setup.

Continue on building your custom kernel. Note that config might complain about the redefinition of the above interfaces, but the warning is only informational. Your new values will override the defaults.

A reboot using your new kernel is required for the new devices to become available.

References

Classless in-addr.arpa. Delegation

2009-01-18T00:00:00+00:00

Classless in-addr.arpa. delegation allows network administrators to provide authoritative reverse DNS on subnets that don’t fall on octet boundaries. This is especially useful for subnets comprised of less than eight bits in the host portion of the address (i.e. smaller than a class C).

There are two important things to remember: first, we’re dealing with classless subnets, meaning they don’t align themselves neatly with IPv4’s octet boundaries (like a class A, B, C, D, or E network); and second, only one name server can be the primary authoritative controller of a given class of IP addresses.

In order for a non-octet subnet network administrator to properly provide reverse DNS for his hosts, he will require the cooperation of his upstream provider. This goes back to the second rule above: the upstream provider technically controls the complete class of addresses out of which the downstream subnet receives its address space.

I found myself in this position years ago while working for a small web-based startup. We were allocated a block of 32 addresses as a /27 subnet (netmask: 255.255.255.224). What follows is a brief summary (with examples) of how we went about establishing our name servers as the authorities for reverse DNS in our zone.

All given configurations were tested using BIND 8 under both OpenBSD and FreeBSD circa 2000, but I expect they are still relevant today.

Our assigned subnet was 206.126.7.64/27 (in CIDR notation). Thus, our network address was 206.126.7.64 and our broadcast address was 206.126.7.95, giving us 30 usable host addresses in between.

We begin by creating a standard zone file (example-hosts) for the example.com domain (for “forward” DNS queries):

@               IN      A       206.126.7.73
ns              IN      A       206.126.7.65
[...]
gateway         IN      A       206.126.7.94

There really isn’t anything special going on here. Simply add this entry to your named.conf file:

zone "example.com" {
    type master;
    file "example-hosts";
};

Next, we define the reverse lookup mappings for the 206.126.7.64/27 subnet in a file named 64-27.7.126.206.rev:

65      IN      PTR     ns.example.com.
[...]
94      IN      PTR     gateway.example.com.

Now, for this file, we add a slightly different entry to named.conf:

zone "64/27.7.126.206.in-addr.arpa" {
    type slave;     
    file "64-27.7.126.206.rev";
    masters {
            206.126.7.65;
    };
};

Be sure to list your provider’s name server(s) in the masters {} block. As far as the provider is concerned, they are still the authoritative “master” of this IP address zone even though they delegate the mappings within our subnet to us. We must act as the slave and participate in zone transfers.

That’s really all the subnet administrator has to worry about. The rest of the work needs to be done on the provider’s side. In order for them to delegate the reverse lookups to the downstream name servers, we employ (or perhaps abuse) the CNAME record. These entries go in a file named 7.126.206.rev. Note that this file contains entries for the entire 206.126.7.0/24 class (C) of addresses, not just our 206.126.7.64/27 subnet.

;; example.com
;
; Name servers for 206.126.7.64/27
;
64/27           NS      ns.example.com.     ; 206.126.7.65
64/27           NS      raq.example.com.    ; 206.126.7.66
;
; Classless in-addr.arpa. delegation for 206.126.7.64/27
;
65              CNAME   65.64/27.7.126.206.in-addr.arpa.
[...]
94              CNAME   94.64/27.7.126.206.in-addr.arpa.

The provider’s named.conf should contain an entry like:

zone "7.126.206.in-addr.arpa" {
    type master;
    file "7.126.206.rev";
};

That’s really all there is to it. It took me a couple of hours to muddle my way through all of the details (and making a few silly mistakes along the way didn’t help much either), but it’s really not that complicated. When a reverse DNS query is issued, it’s sent to the authoritative name server (the provider’s), who in turn delegates the request to the subnet’s name servers, who then get the final say. This is transparent to the querying client, and everyone is happy in the end.

For more (reputable) information on this topic, see RFC 2317.

You may also want to check out Avoid RFC 2317 style delegation of “in-addr.arpa.”, which describes an alternative to RFC 2317. I offer no opinion as to which method is the best solution for a given situation, however.

Twisted Python and Bonjour

2009-01-10T00:00:00+00:00

Bonjour (formerly Rendezvous) is Apple’s service discovery protocol. It operates over local networks via multicast DNS. Server processes announce their availability by broadcasting service records and their associated ports. Clients browse the network in search of specific service types, potentially connecting to the service on the advertised port using the appropriate network protocol for that service.

A common example of Bonjour in action is iTunes’ music library sharing feature. iTunes sharing uses DAAP (Digital Audio Access Protocol). iTunes uses Bonjour to announce its local shared libraries as well as to browse the network for remote DAAP servers.

Twisted Python, while supporting a wide range of network protocols by default, currently lacks an official Bonjour or multicast DNS implementation. The start of a multicast DNS implementation exists in Itamar’s sandbox, but it hasn’t been updated since 2004.

Given that, applications that want to use Bonjour-based service discovery must provide their own implementation. Unfortunately, there can only be one Bonjour “responder” running on a system at one time. If multiple applications attempted to advertise services by standing up competing responders, a port conflict would arise. Therefore, all of the Bonjour-aware applications running on a system must coordinate.

On Mac OS 10.2 and later, applications can simply communicate with the operating system’s built-in Bonjour service. Most other operation systems don’t provide native Bonjour functionality, but support is generally available via third-party packages. Apple provides Bonjour for Windows, and the LGPL-licensed Avahi runs on most other platforms.

Supporting multiple potential Bonjour interfaces can be a burden for application developers. Fortunately, for Python-based projects, pybonjour exists to provide a very nice ctypes-based abstraction layer to all of the Bonjour-compatible libraries mentioned above.

pybonjour’s public API is based on “service descriptors”. Each operation returns a service descriptor reference and signals the caller via a callback when the operation completes. Service descriptors can generally be treated like read-only file descriptors, but all read operations must be done using pybonjour’s DNSServiceProcessResult() function.

We can easily wrap a pybonjour service descriptor in an object that implements Twisted’s IReadDescriptor interface:

import pybonjour
from twisted.internet.interfaces import IReadDescriptor
from zope import interface

class ServiceDescriptor(object):

    interface.implements(IReadDescriptor)

    def __init__(self, sdref):
        self.sdref = sdref

    def doRead(self):
        pybonjour.DNSServiceProcessResult(self.sdref)

    def fileno(self):
        return self.sdref.fileno()

    def logPrefix(self):
        return "bonjour"

    def connectionLost(self, reason):
        self.sdref.close()

Then, it’s simply a matter of writing some operation-specific functions that join the pybonjour interface with Twisted’s event-driven concepts. These functions initiate the pybonjour request, handle the pybonjour callback, and dispatch the result using a Twisted Deferred.

The following example broadcasts a new service record. Note that the local callback function handles both the success and error results.

from twisted.internet.defer import Deferred

def broadcast(reactor, regtype, port, name=None):
    def _callback(sdref, flags, errorCode, name, regtype, domain):
        if errorCode == pybonjour.kDNSServiceErr_NoError:
            d.callback((sdref, name, regtype, domain))
        else:
            d.errback(errorCode)

    d = Deferred()
    sdref = pybonjour.DNSServiceRegister(name = name,
                                        regtype = regtype,
                                        port = port,
                                        callBack = _callback)

    reactor.addReader(ServiceDescriptor(sdref))
    return d

The caller can then provide callback and errback functions that will be invoked using Twisted’s event-based reactor machinery.

from twisted.internet import reactor
from twisted.python import log

sdref = None

def broadcasting(args):
    global sdref
    sdref  = args[0]
    log.msg('Broadcasting %s.%s%s' % args[1:])

def failed(errorCode):
    log.err(errorCode)

d = broadcast(reactor, "_daap._tcp", 3689, "DAAP Server")
d.addCallback(broadcasting)
d.addErrback(failed)

To stop broadcasting, simply close the service descriptor (sdref.close()).