This article is based on Scala in Action , to be published on Fall 2011. It is being reproduced here by permission from Manning Publications. Manning publishes MEAP (Manning Early Access Program,) eBooks and pBooks. MEAPs are sold exclusively through Manning.com. All pBook purchases include free PDF, mobi and epub. When mobile formats become available all customers will be contacted and upgraded. Visit Manning.com for more information. [ Use promotional code 'java40beat' and get 40% discount on eBooks and pBooks ]

Examining Functional Concepts in Google Collections

Introduction

The Google Collections API adds a lot of power to the standard Java collections. It brings a nice set of efficient immutable data structures and some functional ways of interacting with your collections, primarily the Function interface and the Predicate interface. These interfaces are used from the Iterables and Iterators classes. Let’s take a look at the Predicate interface (listing 1) and its uses.

Listing 1 Google Collections Predicate interface

1
2
3
4
5

interface Predicate {
		public boolean apply(T input);	 				#A
		public boolean equals(Object other);
	}
	#A Matches Function1.apply

The predicate interface is rather simple. Besides equality, it contains an apply method, which returns true or false against its argument. This is used in Iterators/Iterables filter method. The filter method takes a collection and a predicate. It returns a new collection containing only elements that pass the predicate apply method. Predicates are also used in the find method. The find method looks in a collection for the first element passing a Predicate and returns it.

The filter and find method signatures are shown in listing 2.

Listing 2 Iterables filter and find methods

1
2
3
4
5
6
7
8
9

class Iterables {
		public static <T> Iterable<T> filter(Iterable<T> unfiltered,
		Predicate<? super T> predicate) {...} 				#A
		public static <T> T find(Iterable<T> iterable,
		Predicate<? super T> predicate) {...} 				#B
		...
	}
	#A Filters using predicate
	#B Finds using predicate

There is also a Predicates class that contains static methods for combining predicates (ands/ors) as well as standard predicates for use, such as not null. This simple interface creates some powerful functionality through the potential combinations that can be achieved with very terse code. Also, since the predicate itself is passed into the filter function, the function can determine the best way or time to execute the filter. The data structure may be amenable to lazily evaluating the predicate, thus making the iterable return a “view” of the original collection.

It might also determine that it could best optimize the creation of the new iterable through some form of parallelism. The fact is that this has been abstracted away, so the library could improve over time with no code changes on our part.

The Predicate interface itself is rather interesting because it looks like a very simple function. This function takes some type T and returns a Boolean. In Scala, this would be represented as T => Boolean. Let’s rewrite the filter/find methods in Scala and see what their signatures would look like (listing 3).

Listing 3 Iterables filter and find methods in Scala

1
2
3
4
5
6
7

object Iterables {
		def filter[T](unfiltered : Iterable[T],
		predicate : T => Boolean) : Iterable[T] = {...} 		#A
		def find[T, U :> T](iterable : Iterable[T], predicate : U => Boolean) : T = {...}
		...
	}
	#A No need for ?

You’ll immediately notice that in Scala we aren’t using any explicit ? super T type annotations. This is because the Function interface in Scala is appropriately annotated with covariance and contravariance. Covariance (+T or ? extends T) is when a type can be coerced down the inheritance hierarchy. Contravariance (-T or ? super T) is when a type can be coerced up the inheritance hierarchy. Invariance is when a type cannot be coerced at all. In this case, a Predicate’s argument can be coerced up the inheritance hierarchy as needed. This means, for example, that a predicate against mammals could apply to a collections of cats, assuming a cat is a subclass of mammal. In Scala, you specify co-/contra-/in-variance at class definition time.

What about combining predicates in Scala? We can accomplish a few of these rather quickly using some functional composition. Let’s make a new Predicates module in Scala that takes in function predicates, and provides commonly used function predicates. The input type of these combination functions should be T => Boolean and the output should also be T => Boolean. The predefined predicates should also have a type T => Boolean.

Listing 4 Predicates in Scala

1
2
3
4
5
6
7
8
9

object Predicates {
		def or[T](f1 : T => Boolean, f2 : T => Boolean) =
		(t : T) => f1(t) || f2(t) #A
		def and[T](f1 : T => Boolean, f2 : T => Boolean) =
		(t : T) => f1(t) && f2(t) #B
		val notNull[T] : T => Boolean = _ != null
	}
	#A explicit anonymous function
	#B placeholder function syntax

We’ve now started to delve into the realm of functional programming. We are defining first-class functions and combining them to perform new behaviors. You’ll notice the or method takes two predicates, f1 and f2. It then creates a new anonymous function that takes an argument t and ors the results of f1 and f2.

This is the essence of functional programming. Playing with functions also makes more extensive use of generics and the type system. Scala has put forth a lot of effort to reduce the overhead for generics in daily usage.

Summary

As Scala blends together various concepts, users of Scala will find themselves striking a delicate balance between functional programming techniques, object orientation, integration with existing Java applications, Expressive library APIs, and enforcing requirements through the type system.

This article is based on Scala in Action , to be published on Fall 2011. It is being reproduced here by permission from Manning Publications. Manning publishes MEAP (Manning Early Access Program,) eBooks and pBooks. MEAPs are sold exclusively through Manning.com. All pBook purchases include free PDF, mobi and epub. When mobile formats become available all customers will be contacted and upgraded. Visit Manning.com for more information. [ Use promotional code 'java40beat' and get 40% discount on eBooks and pBooks ]

Methods and Pattern Matching

Introduction

Expression-oriented programming challenges some good practices from other languages. A common practice in Java programming is having a single point of return for any method. This means that, if there is some kind of conditional logic, the developer creates a variable that contains the eventual return value. As the method flows, this variable is updated with what the method returns. The very last line in every method is a return statement. See an example in listing 1.

Listing 1 Java Idiom: One return statement

1
2
3
4
5
6
7
8
9
10
11
12
13
14

def createErrorMessage(errorCode : Int) : String = {
		var result : String = _ 			#1
		errorCode match {
			case 1 =>
			result = “Network Failure” 		#2
			case 2 =>
			result = “I/O Failure”
			case _ =>
			result = “Unknown Error”
		}
		return result;
	}
	#1 Initializes to default
	#2 Directly assigns result

As you can see, the result variable is used to store the final result. The code falls through a pattern match, assigning error strings as appropriate and then returns the result variable. We can improve this code slightly using the expression-oriented syntax that pattern matching allows. A pattern match actually returns a value. The type of the value is determined as a common super type from all case statement returns. Pattern matching also throws an exception if no pattern is matched, so we’re guaranteed a return or error here. Let’s translate the code for an expression-oriented pattern match (listing 2).

Listing 2 Updated createErrorMessage with expression-oriented pattern match

1
2
3
4
5
6
7
8
9
10

def createErrorMessage(errorCode : Int) : String = {
		val result = errorCode match { 			#1
			case 1 => “Network Failure” #2
			case 2 => “I/O Failure” #2
			case 3 => “Unkonwn Error” #2
		}
		return result
	}
	#1 Assigning pattern match
	#2 Returns expression

You’ll notice two things. First, we changed the result variable to a val and let the type inferencer determine the type. This is because we no longer have to change the val after assignment, the pattern match should determine the unique value. So, not only have we reduced the size and complexity of the code, we’ve also increased immutability in the program. Immutability refers to the unchanging state of an object or variable. Immutability is the opposite of mutability. Mutability is the ability of an object or variable to change or mutate during its lifetime. You’ll frequently find that expression-oriented programming and immutable objects work very well together.

The second thing we’ve done is remove any kind of assignment from the case statements. The last expression in a case statement is the “result” of that case statement. We could have embedded further logic in each case statement if necessary, as long as we eventually had some kind of expression at the bottom. The compiler will also warn us if we accidentally return the wrong type.

The code is looking a lot more concise; however, we can still improve it somewhat. In Scala, most developers avoid return statements in their code, preferring to have the last expression be the return value (similar to all the other expression-oriented styles). In fact, for the createErrorMessage method, we can remove the intermediate result variable altogether. Let’s take a look at listing 3 for the final transformation.

Listing 3 Final expresison-oriented createErrorMessage method

1
2
3
4
5

def createErrorMessage(errorCode : Int) : String = errorCode match {
		case 1 => “Network Failure”
		case 2 => “I/O Failure”
		case _ => “Unknown Error”
	}

Notice how we haven’t even opened up a code block for the method? The pattern match is the only statement in the method, and it returns an expression of type String.

Summary

We’ve completely transformed the method into an expression-oriented syntax. Notice how much more concise and expressive the code is. Also note that the compiler will warn us of any type infractions or unreachable case statements.

This article is based on MongoDB in Action ( Buy from Amazon), to be published on July, 2011. It is being reproduced here by permission from Manning Publications. Manning publishes MEAP (Manning Early Access Program,) eBooks and pBooks. MEAPs are sold exclusively through Manning.com. All pBook purchases include free PDF, mobi and epub. When mobile formats become available all customers will be contacted and upgraded. Visit Manning.com for more information. [ Use promotional code 'java40beat' and get 40% discount on eBooks and pBooks ]

Introduction

In this article, we present parts of MongoDB’s query language. Specifically, we discuss general description queries, their semantics, and types.

Selector matching

The simplest way to specify a query is with a selector whose key-value pairs literally match the document you’re looking for. A couple of examples follow.

1
2

db.users.find({last_name: "Banker"})
	db.users.find({first_name: "Smith", age: 40})

The second query reads, “Find me a user with a first name is Smith and aged 40.” Notice that, whenever you pass more than one key-value pair, both have to match; the query conditions are additive.

Ranges

It’s frequently necessary to query for documents whose values span a certain range. In SQL, we’d use, and >=; with MongoDB, we get the analogous set of operators $gt, $gte, $lt, and $lte. Their behavior is mostly predictable. However, beginners sometimes struggle with combining these operators. A common mistake is to repeat the search key:

1

db.users.find({age: {$gte: 0}, age: {$lte: 30})

But, because keys cannot be repeated, this query selector is invalid. This query is properly expressed as follows:

1

db.users.find({age: {$gte: 0, $lte: 30})

The only other surprise regarding the range operators involves types. In short, range queries will only match values having the same type as the value passed in. For example, suppose you have a collection with the following documents:

1
2
3
4
5
6

{ "_id" : ObjectId("4caf82011b0978483ea29ada"), "value" : 97 }
	{ "_id" : ObjectId("4caf82031b0978483ea29adb"), "value" : 98 }
	{ "_id" : ObjectId("4caf82051b0978483ea29adc"), "value" : 99 }
	{ "_id" : ObjectId("4caf820d1b0978483ea29ade"), "value" : "a" }
	{ "_id" : ObjectId("4caf820f1b0978483ea29adf"), "value" : "b" }
	{ "_id" : ObjectId("4caf82101b0978483ea29ae0"), "value" : "c" }

You then issue the following query:

1

db.items.find({value: {$gte: 97})

Now, you may think to yourself that this query should return all six documents since the strings are numerically equivalent to the integers 97, 98, and 99. This is not the case. If you want the integer results, you need to issue the above query. If you want the string result, here’s the query you’ll want:

1

db.items.find({value: {$gte: "a"})

Of course, you won’t have to worry about this type restriction as long as you never store multiple types for the same key within the same collection. Definitely a good base rule of thumb.

Set operators

Three query operators –$in , $all, and $nin– take a list of one or more values as their predicate. $in, the most used of the three, returns a document if any of the given values matches the search key. We might use this operator to return all products belonging to some discrete set of categories. If the following list of category ids correspond to the lawn mowers, hand tools, and work clothing categories:

1
2
3
4

[ObjectId("6a5b1476238d3b4dd5000048"),
	ObjectId("6a5b1476238d3b4dd5000051"),
	ObjectId("6a5b1476238d3b4dd5000057")
	]

Then finding all products belonging to these categories looks like this:

1
2
3
4

db.products.find(main_cat_id: { $in:
		[ObjectId("6a5b1476238d3b4dd5000048"),
		ObjectId("6a5b1476238d3b4dd5000051"),
		ObjectId("6a5b1476238d3b4dd5000057") })]

$in is frequently used with lists of ids. $nin matches only when none of the given elements matches. We might use $nin to find all products that are neither black nor blue:

1

db.products.find(‘details.color’: { $nin: ["black", "blue"] }

Finally, $all matches if all the given elements match the search key. If we wanted to find all products tagged as gift and garden, $all would be a good choice:

1

db.products.find(‘tags’: { $all: ["gift", "garden"] }

Naturally, this query only makes sense if tags is an array.

Logical operators

MongoDB’s logical operators include $ne, $not, $or, and $exists. There is no $and, since adding conditions to a query selector always implies AND.

$ne, the not equal operator, works as expected. In practice, it’s best used in combination with at least one other operator; otherwise, it’s likely to be pretty inefficient. We might use $ne to find all products manufactured by ACME that aren’t tagged “gardening.”

1

db.products.find(‘details.manufacturer’: ‘ACME’, tags: {$ne: "gardening"} }

$ne works on keys pointing to single values and to arrays, as shown in the foregoing example.

While $ne matches the negation of the specified values, $not will negate the result of another MongoDB operator or regular expression query. Keep in mind that most query operators already have a negated form ($in and $nin, $gt and $lte, and so on); $not shouldn’t be used with any of these. Reserve $not for times when the operator or regex you’re using lacks a negated form. If you wanted to query for all users with last names not beginning with B, you could use $not like so:

1

db.users.find(last_name: {$not: /^B/} }

$or expresses the logical disjunction of two values for two different keys. This is an important point: if the possible values are scoped to the same key, use $in instead. Trivially, finding all products that are either blue or green looks like this:

1

db.products.find(‘details.color’: {$in: [‘blue’, ‘green’]} }

But finding all products that are either blue or made by ACME requires $or:

1

<code> db.products.find({ $or: [{‘details.color’: ‘blue’}, ‘details.manufacturer’: ‘ACME’}] }) </code>

Notice that $or takes an array of query selectors, where each selector can be arbitrarily complex and contain other query operators.

Because collections don’t enforce a fixed schema, we sometimes need a way to query for documents containing a particular key. For this, we can use the $exists operator. Suppose we had planned to use a product’s details attribute to store custom fields. Assuming that not all products specify a set of colors, we can query for them:

1

db.products.find({ $or: [{‘details.color’: ‘blue’}, ‘details.manufacturer’: ‘ACME’}] })

The opposite query is also possible:

1

db.products.find({‘details.color’: {$exists: true}})

Now, before moving on, it’s worth noting that there’s another way to check for existence that’s practically equivalent: by querying for null.2 This would alter the above two queries slightly. The first could be expressed like so:

1

 db.products.find({‘details.color’: null})

And the second:

1

db.products.find({‘details.color’: {$ne: null}})

The main advantage of querying for null is that such a query can use an index. While $exists always performs a full table scan, these queries for a null or non-null value will use an index if it exists. That’s a good thing. So prefer the query for null if your application allows it.

Objects

Some of the entities in our e-commerce data model have keys that point to a single embedded object. The product details attribute is one good example. Here’s part of the relevant document, expressed as JSON:

1
2
3
4
5
6

{ _id: {sym: ‘gs’, date: 20101005}
		open: 40.23,
		high: 45.50,
		low: 38.81,
		close: 41.22
	}

We can query on such objects by separating the relevant keys with a dot. For instance, if we want to find all products manufactured by Acme, here would be our query:

1

<code> db.products.find({‘details.manufacturer_id’: 432}); </code>

Such queries can be specified arbitrarily deep. Thus, supposing we had the following slightly modified representation:

1

<code> { _id: ObjectId("4c4b1476238d3b4dd5003981"), slug: "wheel-barrow-9092", sku: "9092", details: { model_num: 4039283402, manufacturer: { name: "Acme", id: 432 }, color: "Green" } } </code>

The query would be as expected:

1

 db.products.find({‘details.manufacturer.id’: 432});

But, in addition to querying on a single object attribute, we can query on an object as a whole. This sometimes comes in quite handy. For instance, imagine you’re using MongoDB to store stock data. To save space, you may want to forgo the standard object id and replace it with a compound key consisting of the stock symbol and a timestamp. Here’s how a representative document might look:

1

 { _id: {sym: ‘gs’, date: 20101005} open: 40.23, high: 45.50, low: 38.81, close: 41.22 } </code>

We could then find the summary of GS for October 5, 2010 with the following _id query:

1

 db.ticks.find({_id: {sym: ‘gs’, date: 20101005} });

Keep in mind that these queries do a strict, byte-by-byte comparison, which means that the order of the keys matters. The following query is not equivalent and won’t match our sample document:

1

 db.ticks.find({_id: {date: 20101005, sym: ‘gs’} });

And while the order of keys will be preserved in JSON documents entered via the shell, this is not necessarily true for document representations in the various language drivers. In particular, hashes in Ruby 1.8 are not order-preserving. To preserve order in Ruby 1.8, you’ll need to explicitly construct an OrderedHash:

1
2
3
4

doc = BSON::OrderedHash.new
	doc[‘sym’] = ‘gs’
	doc[‘date’] = 20101005
	@ticks.find(doc)

Arrays

Arrays give the document model its power. They’re used to store lists of strings, object ids, and even other documents. Arrays afford us rich yet comprehensible documents; it stands to reason that MongoDB would let us query and index the array type with ease. Indeed, the simplest array queries look just like queries on any other document type. Take product tags, for example. These tags are represented as a simple list of strings:,/p>

1
2
3
4
5

{ _id: ObjectId("4c4b1476238d3b4dd5003981"),
		slug: "wheel-barrow-9092",
		sku: "9092",
		tags: ["tools", "equipment", "soil"]
	}

Querying for products with the tag soil is trivial:

1

 db.products.find({tags: "soil"})

This query is also efficient so long as we have an index on the tags field:

1

 db.products.ensureIndex({tags: 1})

When we need more control over our queries, we can use dot-notation to query for a value at a particular position within the array. Here’s how we’d restrict the previous query to the first of a product’s tags:

1

 db.products.find({‘tags.0’: "soil"})

It might not make much sense to query tags in this way, but imagine we’re dealing with user addresses. In this case, we have an array of sub-documents.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

{ _id: BSON::ObjectID("4c4b1476238d3b4dd5000001")
		username: "kbanker",
		addresses: [
			{name: "home",
				street: "588 5th Street",
				city: "Brooklyn",
				state: "NY",
				zip: 11215},
			{name: "work",
				street: "1 E. 23rd Street",
				city: "New York",
				state "NY",
				zip 10010},
		]
	}

We might stipulate that the first of these addresses always be the user’s primary shipping address. Thus, to find all users whose primary shipping address is in New York, we could again use the specify the zero position and combine that with a dot to target the state field:

1

 db.users.find({‘addresses.0.state’: "NY"})

We can just as easily omit the position and use a specify a field alone. Following query will return a user document if any of the addresses in the list are in New York.

1

 db.users.find({‘addresses.state’: "NY"})

As we did before, we can index this dotted field:

1

 db.users.ensureIndex({‘addresses.state’: 1})

Notice that the same dot-notation is used regardless of whether a field points to a subdocument or to an array of subdocuments. This is powerful, and the consistency is reassuring. However, ambiguity can arise when querying for more than attribute within an array of subobjects. For example, suppose we want to fetch a list of all users whose home address is in New York. Can you think of a way to express this query?

1

 db.users.find({‘addresses.name’: ‘home’, ‘addresses.state’: ‘NY’})

The problem with this query is that the field references aren’t restricted to a single address. In other words, this query will match as long as one of the addresses is designated as “home” and one is in New York, but these need to be the same address. To restrict multiple conditions to the same subdocument, you have to use the $elemMatch operator. Here’s how we’d use $elemMatch to ensure that home and NY appear in the same subdocument:

1

db.users.find({addresses: {$elemMatch: {name: ‘home’, state: ‘NY’}}})

Logically, $elemMatch is used only when you need to match two or more attributes in a subdocument.

JavaScript

If you can’t express your query with the tools described thus far, then you may have to write some JavaScript. You can use the special $where operator to pass a JavaScript expression to any query. Within the JavaScript context, this refers to the current document. To take an example:

1

db.reviews.find({$where: "function() { return this.helpful_votes > 3; }"}})

There’s also an abbreviated form for simple expressions like this one:

1

 db.reviews.find({$where: "this.helpful_votes > 3"}})

This query works but can be easily expressed with the standard query language. JavaScript expressions can’t use an index, and they incur a substantial cost because they must be evaluated within a JavaScript interpreter context. For these reasons, you should issue JavaScript queries only when the standard query language can’t express your query. If you do find yourself needing JavaScript, try to combine the JavaScript expression with another query operator. The standard query operator will prompt the server to pare down the result set, thus limiting the number of documents that must be loaded into a JS context. Let’s take a quick example to see how this would work.

Imagine that, for each user, we’ve calculated a rating reliability factor. This is essentially an integer that, when multiplied by the user’s rating, results in a more normalized rating. Suppose further that we want to query a particular user’s reviews and only return a normalized rating greater than 3. Here’s how that query would look:

1

 db.reviews.find({user_id: ObjectId("4c4b1476238d3b4dd5000001"), $where: "(this.rating * .92) > 3"})

This query meets both recommendations: it uses a standard query, on a presumably-indexed field, and it employs a JavaScript user_id expression that’s absolutely beyond the capabilities of the standard query language.

In addition to the attendant performance penalties, it’s good to be aware of the possibility of JavaScript injection attacks. An injection attack becomes possible whenever a user is allowed to enter code directly into a JavaScript query. While there’s never any danger of the user writing or deleting data, a user might be able to read sensitive data. A naive JavaScript query in Ruby might look something like this:

1

@users.find({$where => "this.#{attribute} == #{value}"})

Assuming that the user controls the values of attribute and value, he can use manipulate query to search the users collection on any attribute pair. While this might not be the worst imaginable intrusion, we’d be wise to prevent this.

Regular expressions

You can use a regular expression within a query. You would use, for example, a prefix expression, /^B/, to find last names beginning with a B, and this query would use an index. In fact, much more is possible. MongoDB is compiled with PCRE,5 which supports a huge gamut of regular expressions.

Now, with the exception of the prefix-style query just described, regular expressions queries can’t use an index. Thus, I recommend using them as you would a JavaScript expression, that is, in combination with at least one other query term. Here’s how you might query a given user’s reviews for review text containing one of the words “best” or “worst.” Notice that we use the i regex flag to indicate case insensitivity:

1

 db.reviews.find({user_id: ObjectId("4c4b1476238d3b4dd5000001"), text: /best|worst/i })

If the language you’re using has a native regex type, you can use a native regex object to perform the query. The above query, in Ruby, is practically identical:

1

 @reviews.find({:user_id => BSON::ObjectId("4c4b1476238d3b4dd5000001"), :text => /best|woest/i })

If you’re querying from an environment that doesn’t support a native regex type, you can use the special $regex and $options operators. Using these operators from the shell, the previous query would be expressed like so:

1

 db.reviews.find({user_id: ObjectId("4c4b1476238d3b4dd5000001"), text: {$regex: "best|worst", $options: "i" })

Special query operators

There are two more query operators that aren’t easily categorized and, thus, deserve their own section. The first is $mod, which allows you to query for the modulo of a given value.

Finally, there’s the $type operator, which matches a value if it’s represented as a particular BSON type. I don’t recommend storing multiple types for the same field within a collection but, if the situation ever arises, you have a query operator that essentially lets you test against type. This came in handy recently when we had discovered a user whose _id queries weren’t always matching when they should have. This problem, it turned out, was that the user had been storing ids as both strings and as proper object ids. These are represented as BSON types 2 and 7, respectively; to a new user, this distinction is easy to miss.

Correcting the issue first required finding all documents with ids stored as strings. The $type operator can help us to just that:

1

db.users.find({_id: {$type: 2}})

Summary

Queries allow us to get the data as it’s stored. You’ve learned about general description queries, their semantics, and types. If you’re ever unsure of how a particular combination of query operators will serve you, the shell is always a ready test bed.

This article is based on MongoDB in Action ( Buy from Amazon), to be published on July, 2011. It is being reproduced here by permission from Manning Publications. Manning publishes MEAP (Manning Early Access Program,) eBooks and pBooks. MEAPs are sold exclusively through Manning.com. All pBook purchases include free PDF, mobi and epub. When mobile formats become available all customers will be contacted and upgraded. Visit Manning.com for more information. [ Use promotional code 'java40beat' and get 40% discount on eBooks and pBooks ]

Introduction

Indexes are enormously important. With the right indexes in place, MongoDB can use its hardware efficiently and serve your application’s queries quickly. With the wrong indexes, you’ll see the exact opposite effect: slow queries and poorly utilized hardware. It stands to reason, then, that anyone wanting to use MongoDB effectively and make the best use of hardware resources must understand indexing.

We’re going to look at some refinements on the kinds of indexes that can be created in MongoDB. We’ll then proceed to some of the niceties of administering those indexes.

Index types

All indexes in MongoDB use the same underlying data structure, but several kinds of indexes are nevertheless permitted. In particular, unique, sparse, and multikey indexes are frequently used, and here we describe them in some detail.

Unique indexes

To create a unique index, specify the unique option:

1

 db.users.ensureIndex({username: 1}, {unique: true})

Unique indexes ensure that all index entries are unique. Thus, if we try to insert a document into the collection with an already-indexed username, users that insert will fail with the following exception:

1

E11000 duplicate key error index: gardening.users.$username_1 dup key: { : "kbanker" }

If using a driver, this exception will be caught only if you perform the insert using your driver’s safe mode. If you need a unique index on a collection, it’s usually best to create the index before inserting any data. If you create the index in advance, you guarantee the uniqueness constraint from the start. Creating a unique index on a collection that already contains data, you run the risk of failure since it’s entirely possible that duplicate keys will exist in the collection. When duplicate keys exist, the index creation fails.

If you do find yourself needing to create a unique index on an established collection, you have a couple of options. The first is to repeatedly attempt to create the unique index and use the failure messages to intelligently remove the documents with duplicate keys. But if the data isn’t so important, you can also instruct the database to drop documents with duplicate keys automatically using the dropDups option. So, to take an example, if our users collection already contains data and if we don’t care that documents with duplicate keys are removed, then we can issue the index creation command like this:

1

 db.users.ensureIndex({username: 1}, {unique: true, dropDups: true})

Sparse indexes

Indexes are dense by default. This means that, for every document, in an indexed collection, there will be a corresponding entry in the index even if the document lacks the indexed key. Suppose you have a products collection in an e-commerce data model, and imagine that you’ve built an index on the product attribute category_ids. Now, suppose that there are a few products that haven’t been assigned to any categories. For each of these category-less products, there will still exist a null entry in the category_ids index. We can query for those null values:

1

db.products.find({category_ids: null})

When searching for all products lacking a category, the query optimizer will still be able to use the category_ids index to locate those products. When the number of null values in the index is small, this is still a good thing because this query can use the index.

But, there are two cases where a dense index is undesirable. The first is when you want a unique index on a field that doesn’t appear in every document in the collection. For instance, we definitely want a unique index on every product’s sku field. But, suppose that, for some reason, products are entered into the system before a sku is assigned. If we have a unique index on sku and we attempt to insert more than one product without a sku, then those inserts will fail because there will already be an entry in the index where sku is null. This is, therefore, a case where a dense index doesn’t serve our purpose. What we want instead is a sparse index.

In a sparse index, only those documents having a non-null value for the indexed attribute will appear. If we want to create a sparse index in MongoDB, all we have to do is specify {sparse: true}. So, for example, we can create a unique, sparse index on sku like so:

1

db.products.ensureIndex({sku: 1}, {unique: true, sparse: true})

There’s another case where a sparse index is desirable: when a large number of documents in a collection don’t contain the indexed key. For example, suppose we allowed anonymous reviews on our e-commerce site. In this case, half the reviews might lack a user_id field and, if that field were indexed, then half the entries in that index would be null. This would be pretty inefficient for two reasons. First, it would increase the size of the index. Second, it would require updates to the index when adding and removing documents with null user_id fields.

If we rarely (or never) expected queries on anonymous reviews, we might elect to build a sparse index on user_id. Again, setting the sparse option is simple:

1

 db.reviews.ensureIndex({user_id: 1}, {sparse: true})

Now, only those reviews linked to a user via the user_id field would be indexed.

Multikey indexes

Indexing fields whose values are arrays is made possible by what’s known as a multikey index, which allows multiple entries in the index to reference the same document. This makes sense if we take a simple example. Suppose we have a product document with a few tags like this:

1
2
3

{ name: "Wheelbarrow",
		tags: ["tools", "gardening", "soil"]
	}

If we create an index on tags, then each value in this document’s tags array will appear in the index. This means that a query on any one of these array values can use the index to locate the document. This is the idea behind a multikey index: multiple index entries, or keys, end up referencing the same document.

Multikey indexes are always enabled in MongoDB. Anytime an indexed field contains an array, each array value will be given its own entry in the index. The intelligent use of multikey indexes is essential to proper MongoDB schema design.

Index administration

When it comes to administering indexes in MongoDB, there may be some gaps in your understanding. Here we’ll examine index creation and deletion in detail and address questions surrounding compaction and backups.

Creating and deleting indexes

If you’ve created a few indexes, there should be no mysteries surrounding the index creation syntax. Simply call one of the index creation helper methods, either in the shell or with your language of choice and a document defining the new index will be placed into the special system.indexes collection.

While it’s usually easier to use a helper method to create an index, you can insert an index specification manually (this is, after all, what the helper methods do). You just need to be sure you’ve specified the minimum set of keys: ns, key, and name. ns is the namespace, key is the field or combination of fields to index, and name is a name used to refer to the index. Any additional options, like sparse, can also be specified here. So to have an example, let’s create a sparse index on our users collection:

1
2

spec = {ns: "green.users", key: {‘addresses.zip': 1}, name: ‘zip'}
	db.system.indexes.insert(spec, true)

If no errors are returned on insert, then the index now exists and we can query the system.indexes collection to prove it.

1
2
3

db.system.indexes.find()
	{ "_id" : ObjectId("4d2205c4051f853d46447e95"), "ns" : "green.users",
	"key" : { "addresses.zip" : 1 }, "name" : "zip", "v" : 0 }

If you’re running MongoDB v1.8 or later, you’ll see that an extra key, v, has been added. This version field allows for future changes in the internal index format but should be of little concern to application developers.

To delete an index, you might think that all you need do is remove the index document from system.indexes, but this operation is prohibited. Instead, you can delete indexes using the database command deleteIndexes. As with index creation, there are helpers for deleting indexes but, if you want to run the command itself, you can do that too. It takes as its argument a document containing the collection name and either the name of the index to drop or “*” to drop all indexes. To manually drop the indexes we just created, issue the command like so:

1
2

use green
	db.runCommand({deleteIndexes: "users", index: "zip"})

That’s the basics of creating and deleting indexes. For what to expect when an index is created, read on.

Building indexes

Most of the time, you’ll want to declare your indexes before putting your application in production. This allows indexes to be built incrementally, as the data is inserted. But there are two cases where you might choose to build an index after the fact. The first case is when you need to import a lot of data before switching to production. For instance, you might be migrating an application to MongoDB and need to seed the database with user information from a data warehouse. You could certainly create the indexes on your user data in advance, but doing so after you’ve imported the data will ensure an ideally balanced index from the start. Furthermore, the net time updating will be minimized. The second and more obvious case of creating indexes on existing data sets is when you have to optimize for new queries.

Regardless of why you’re creating new indexes, the process isn’t always pleasing. For large data sets, building an index can take hours, even days. You can see its progress in the MongoDB logs. Let’s take an example from a data set that we’ll be using in the next section. First, we declare an index to be built:

1

db.values.ensureIndex({open: 1, close: 1})

The index builds in two steps. In the first step, the values to be indexed are sorted. A sorted data set makes for a much more efficient insertion into the B-tree. Notice that the progress of the sort is indicated by the ratio of the number of documents sorted to the total number of documents:

1
2
3
4
5
6

Tue Jan 4 09:58:17 [conn1] building new index on { open: 1.0, close: 1.0 } for stocks.values
	1000000/4308303 23%
	2000000/4308303 46%
	3000000/4308303 69%
	4000000/4308303 92%
	Tue Jan 4 09:59:13 [conn1] external sort used : 5 files in 55 secs

For step two, the sorted values are inserted into the index. Progress is indicated in the same way, and, when complete, the time it took to complete the index build is indicated as the insert time into system.indexes:

1
2
3
4
5
6
7
8
9

1200300/4308303 27%
	2227900/4308303 51%
	2837100/4308303 65%
	3278100/4308303 76%
	3783300/4308303 87%
	4075500/4308303 94%
	Tue Jan 4 10:00:16 [conn1] done building bottom layer, going to commit
	Tue Jan 4 10:00:16 [conn1] done for 4308303 records 118.942secs
	Tue Jan 4 10:00:16 [conn1] insert stocks.system.indexes 118942ms

In addition to examining the MongoDB log, you can check the index build progress by running the currentOp() command:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

> db.currentOp()
	{
		"inprog" : [
		{
			"opid" : 58,
			"active" : true,
			"lockType" : "write",
			"waitingForLock" : false,
			"secs_running" : 55,
			"op" : "insert",
			"ns" : "stocks.system.indexes",
			"query" : {
			},
			"client" : "127.0.0.1:53421",
			"desc" : "conn",
			"msg" : "index: (1/3) external sort 3999999/4308303 92%"
		}
		]
	}

WARNING: Be careful declaring indexes

Because it’s so easy to declare indexes, it’s also quite easy to inadvertently trigger an index build. If the data set is large enough, then the build will take a long time. And, in a production situation, this can be a nightmare since there’s no easy way to kill an index build. If this ever happens to you, you’ll have to fail over to a secondary node—if you have one. But the most prudent advice is to treat an index build as a kind of database migration and ensure that your application code never declares indexes automatically.

BACKGROUND INDEXING

If you’re running in production and simply can’t afford to halt access to the database, you can specify that an index be built in the background. Although the index build will still take a write lock, the job will pause to allow other readers and writers to access the database. If your application typically exerts a heavy load on MongoDB, then a background index build will certainly degrade performance, but this may be acceptable under certain circumstances. For example, if you know that the index can be built within a time window where application traffic is at a minimum, background indexing in this case might be a good choice. To build an index in the background, simply specify {background: true when you declare the index. The previous index can be built in the background like so:

1

 db.values.ensureIndex({open: 1, close: 1}, {background: true})

OFFLINE INDEXING

If your production data set is too large to be indexed within a few hours, then a more sophisticated plan will be required. This will usually involve taking a replica node offline, building the index on that node by itself, and then allowing the node to catch up with the master replica. This does presume that your replication oplog is large enough to prevent the offline node from becoming stale during the index build.

Backups

Because indexes are hard to build, you definitely want to back them up. Unfortunately, not all backup methods include indexes. For instance, you might be tempted to use mongodump and mongorestore, but these utilities handle documents only. This means that, when you run mongorestore, all the indexes defined for any collections you’ve backed up will be recreated. As always, if your data set is large, the time it takes to build all indexes may be unacceptable.

Consequently, if you want your backups to include indexes, then you’ll want to opt for backing up the MongoDB data files themselves.

Compaction

If your application heavily updates existing data or performs a lot of large deletions, then you may end up with a highly fragmented index. The primary symptom of this is an index size much larger than you’d expect for the given data size, but you may also see certain performance problems. In these cases, you may want to consider rebuilding one or more indexes. You can do this by recreating individual indexes or by running the reIndex command, which will rebuild all indexes in a given collection.

1

db.values.reIndex();

Be careful about reindexing: the command will take out a write lock for the duration of the rebuild, rendering your MongoDB instance unusable for that time. This is best done offline.

Summary

For many developers, indexes are a topic shrouded in mystery. This need not be the case. The purpose of this article was to help you develop a mental model for thinking clearly about indexes. We discussed unique, sparse, and multikey indexes, and provided a number of pointers on index administration.

This article is based on MongoDB in Action ( Buy from Amazon), to be published on July, 2011. It is being reproduced here by permission from Manning Publications. Manning publishes MEAP (Manning Early Access Program,) eBooks and pBooks. MEAPs are sold exclusively through Manning.com. All pBook purchases include free PDF, mobi and epub. When mobile formats become available all customers will be contacted and upgraded. Visit Manning.com for more information. [ Use promotional code 'java40beat' and get 40% discount on eBooks and pBooks ]

Introduction

This article takes a closer look at the document-oriented data model and at how data is organized at the database, collection, and document level in MongoDB. We start with a brief, general discussion of schema design. This is helpful because a large number of MongoDB users have never designed schemas outside the realm of a traditional relational database management system (RDBMS). This exploration of principles helps set the stage for the second part of the article, where we examine the design of an e-commerce schema in MongoDB. Along the way, I’ll explain how this schema differs from an equivalent RDBMS schema, and we’ll learn how common relationships between entities, such as one-to-many and many-to-many, are replicated in MongoDB.

Principles of schema design

Database schema design is the process of choosing the best representation for a data set given the features of the database system, the nature of the data, and the requirements of the application. The principles of schema design for relational database systems are well established. With RDBMSs, we’re encouraged to shoot for a normalized data model, which helps to ensure generic queryability and avoid updates to the data that might result in inconsistencies. Developers aren’t wondering how to create one-to-many and many-to-many relationships in an RDBMS. But, having said that, schema design is not a hard science even with relational databases. Performance-intensive applications or applications that have to consume data flexibly, may require a more denormalized data model. Some applications are so demanding in their storage and scaling requirements that they’re forced to break all the old rules of schema design. FriendFeed is a great example of this, and reading about their data model is well worth your time.

If you’re moving to MongoDB from the RDBMS world, you may be troubled by the lack of hard schema design rules. Good practices are beginning to emerge but, frankly, there’s often more than one good way to model a given data set. The premise of this section is that principles can drive schema design. But those principles are, in practice, variable. Here, then, are few questions that can be applied when modeling data with any database system.

• What’s the basic unit of data?

  In an RDBMS, we have tables with columns and rows. In a key-value store, we have keys and amorphous values. And, in MongoDB, the basic unit of data is the BSON document.

• How can we query and update that data?

  Once we have a basic unit of data, we need to know how we can manipulate it. RDBMSs feature indexed dynamic queries and joins. MongoDB also has indexed dynamic queries, but joins aren’t supported. And a simple key-value store can only get and put values based on a single key. Databases also diverge in the kinds of updates they permit. With an RDBMS, we can update documents in sophisticated ways using SQL and we can wrap such updates in a transaction to get atomicity and rollback. MongoDB doesn’t support transactions, but it does support a variety of atomic update operations that can work on the internal structures of a complex document. With simple key-value stores, we might be able to update a value but every update involves replacing the value completely. The essential point of this second question is that building the best data model means understanding the features of your database. If you want to model data well in MongoDB, you need to know exactly what sorts of queries and updates it performs the best.

• What are our application access patterns?

  In addition to understanding our basic unit of data and database features, we also need to pin down the needs of our application. If you read the FriendFeed article suggested earlier, you see that the idiosyncrasies of an application can easily demand a schema that goes against firmly-held data modeling principles. Thus, numerous questions can be asked about the application to help determine the data model. What’s the read/write ratio? What’s sorts of queries do we need? How is the data updated? What concurrency issues are involved?

The best schema designs are always the product of deep knowledge of the database you’re using, good judgment about the requirements of the application at hand, and plain old experience.

Designing an e-commerce data model

Demonstrations of next-generation data stores typically revolve around social media: Twitter-like demo apps are the norm. Unfortunately, such apps tend to have rather simple data models. E-commerce has the advantage of including a large number of familiar data modeling patterns. It’s not difficult to imagine how products, categories, product reviews, and orders would typically be modeled using an RDBMS. This should make the upcoming examples more instructive, as we’ll be able to contrast them with our preconceived notions of schema design.

E-commerce has always been a domain exclusive to RDBMSs for a couple of reasons. The first is that e-commerce sites generally require transactions, and transactions are a strong point of RDBMSs. The second is that, up until very recently, domains that require rich data models and dynamic queries have been assumed to fit best within the realm of RDBMS. The examples below call into question this second assumption.

Building an entire e-commerce back-end is beyond this article’s scope. What we’re going to do instead is pick out a handful of e-commerce entities and show how they might be modeled in MongoDB. In particular, we’re going to look at products and categories, users and orders, and product reviews. For each entity, we’ll show an example document. Then, we’ll hint at some of the database features that make that particular structure viable.

Before moving on, we must say a few words about object mappers. When developing applications on an RDBMS, you’ve probably used an object-relational mapping library, such as Java’s Spring framework or Ruby’s ActiveRecord. Such libraries are absolutely necessary for building applications efficiently on an RDBMS. But they’re less necessary with MongoDB. This is due in part to documents already being an object-like representation. It’s also partly due to the good usability of drivers, which provide a high-level interface to MongoDB.

Indeed, applications can frequently be built on drivers alone. Having said that, object mappers are nice for larger projects, since they usually provide validation, type checking, and associations for free. And there are, undoubtedly, a number of mature object mappers for MongoDB that work on top of the basic language drivers. Ultimately, regardless of the object mapper, what gets stored in MongoDB are documents. Knowing the shape of documents in a good MongoDB schema will prepare you to work with the database efficiently with or without an object mapper.

Products and categories

Products and categories are mainstays of any e-commerce site. Products in a normalized RDBMS model tend to require a large number of tables. There’s a table for basic product info, such as the name and SKU, but there will also be other tables to relate shipping information and pricing histories. If the system allows creating products with arbitrary attributes, a complicated series of tables will be added to define and store those attributes. All of this will be facilitated by the RDBMS’s ability to join tables.

Modeling a product in MongoDB should be less complicated. Because collections don’t enforce a schema, any product document will have room for whichever dynamic attributes the product dictates. And, because we can use arrays to contain inner document structures, we can typically condense a multitable RDBMS representation into a single MongoDB collection. Listing 1 shows a product from a sample gardening store.

Listing 1 A sample product document

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32

@doc =
	{ :_id => BSON::ObjectID("4c4b1476238d3b4dd5003981"),
		:slug => "wheel-barrow-9092",
		:sku => "9092",
		:name => "Extra Large Wheel Barrow",
		:description => "Heavy duty wheel barrow...",
		:details => {
			:weight => 47,
			:weight_units => "lbs",
			:model_num => 4039283402
		},
		:tags => ["tools", "equiptment", "soil"],
		:pricing => {
			:retail => 589700,
			:sale => 489700,
		},
		:price_history => [
			{:retail => 529700,
				:sale => 429700,
				:start => Time.utc(2010, 5, 1),
				:end => Time.utc(2010, 5, 8)
			},
			{:retail => 529700,
				:sale => 529700,
				:start => Time.utc(2010, 5, 9),
				:end => Time.utc(2010, 5, 16)
			},
		],
		:category_ids => [BSON::ObjectID("6a5b1476238d3b4dd5000048"),
		BSON::ObjectID("6a5b1476238d3b4dd5000048")],
		:average_review => 4.5
	}

The document contains the basic name, sku, and description fields. There’s also the standard MongoDB object id stored in the _id field. In addition, we’ve defined a slug, “wheel-barrow-9092,” to provide a meaningful URL. MongoDB users sometimes justifiably complain about the ugliness of object ids in URLs. Naturally, we don’t want URLs looking like this:

1

http://mygardensite.org/products/4c4b1476238d3b4dd5003981

Meaningful ids are so much better:

1

http://mygardensite.org/products/wheel-barrow-9092

As a result, we generally recommend building out a slug field if a URL will be generated for the document. Such a field should have a unique index on it so that it can be used as a primary key. Here’s how we’d create the unique index in Ruby:

1

@products.create_index('slug')

If we have a unique index on slug, we’ll need to insert our product document using safe mode so that we’ll know if the insert fails. That way, we can retry with a different slug if necessary. To take an example, imagine our garden store has multiple wheel barrows for sale. We decide to start selling a new wheel barrow, so our code will need to generate a unique slug for the new product. Here’s how we perform the insert:

1
2
3
4

@products.insert({:name => "Extra Large Wheel Barrow",
		:sku => "9092",
		:slug => "wheel-barrow-9092"},
		:safe => true)

If the insert succeeds without raising an exception, we know we’ve chosen a unique slug. But if an exception is raised, our code will have to retry. In any case, the management of slugs is easy with unique indexes.

Continuing on, we have a key, details, that points to a subdocument containing various product details. We’ve specified the weight, weight units, and the manufacturer’s model number. But, we might store other ad-hoc attributes here as well. For instance, if we were selling seeds, we might include attributes here for the expected yield and time to harvest, and, if we were selling lawn mowers, we could include horse power, fuel type, and mulching options. The details attribute provides a nice container for these types of dynamic attributes.

Notice that we’re also able to store the product’s current and past prices in the same document: pricing simply points to an object containing retail and sale prices; price history, by contrast, references a whole array of pricing options. This is a common technique for versioning a document.

Next, we see an array of tag names for the product. Since we can index array keys, this is the simplest and best way of storing relevant tags on an item while at the same time assuring efficient queryability.

But what about relationships? We’re able to use rich document structures, like embedded documents and arrays, to store product details, prices, and tags¡Xall in a single document. And yet, we do eventually have to relate to documents in other collections. To start, we’re going to relate products to a category structure. This relationship between products and categories is usually conceived as many to many, where each product can belong to more than one category, and each category can contain multiple products. In an RDMBS, we’d use a join table to represent a many-to-many relationship like this. Join tables are great because they store all the relationship references between two tables in a single table. Using a SQL join, it then becomes possible to issue a single query to pull back a product and all its categories, and vice versa.

Now, MongoDB doesn’t support joins, so we need a different many-to-many strategy. Looking at our wheelbarrow document, you’ll see a field called category_ids containing an array of object ids. Each of these object ids is a reference to the _id field of some category document. For reference, you can see an example category document in listing 2.

Listing 2 A category document

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

doc =
	{ :_id => BSON::ObjectID("6a5b1476238d3b4dd5000048"),
		:slug => "gardening-tools",
		:parents => [{:name => "Home",
			:_id => BSON::ObjectID("8b87fb1476238d3b4dd5000003"),
			:slug => "home"
		},
		{:name => "Outdoors",
			:_id => BSON::ObjectID("9a9fb1476238d3b4dd5000001"),
			:slug => "outdoors"
		}
		],
		:name => "Gardening Tools",
		:description => "Gardening gadgets galore!",
	}

If you look carefully at the object ids in the category_ids fields, you’ll see that our product is related to the “Gardening Tools” category just shown. Having the category_ids array key in the product document enables all of the kinds of queries we’d want to do on a many-to-many relationship. For instance, to query for all products in the “Gardening Tools” category, the Ruby code is quite simple:

1
2

gardening_id = @category['_id']
	@products.find({:category_ids => gardening_id})

To query for all categories for our products, we pass the array of category ids using the special $in operator:

1

@categories.find({:_id => {"$in" => @product['category_ids']}})

Queries like this, with the $in operator, are not unlike SQL’s IN directive. With the many-to-many relationship out of the way, we must say a few words about the category document itself. You’ll notice the standard _id, slug, name, and description fields. These are straightforward, but the array of parent documents probably won’t be. Why are we storing such a large percentage of each parent category document? The fact is that categories are always conceived as a hierarchy and the ways of representing such a hierarchy in a database are many.

The strategy we choose is always dependent on the needs of the application. In this case, because MongoDB doesn’t support joins, we’ve elected to cache the parent category names in each child document. This way, when querying for the “Gardening Products” category, there’s no need to perform additional queries to get the names and URLs of the parent categories, “Outdoors” and “Home.”

Some developers would, no doubt, consider this level of denormalization unacceptable. In those cases, there are other options for representing a tree. For the moment, try to be open to the possibility that what best determine the schema are the demands of the application, and not necessarily the dictates of theory.

Users and orders

Looking at how we model users and orders illustrates another common relationship: one to many. In this case, a user is said to have many orders. In an RDBMS, we’d use a foreign key in our orders table; here, the convention is similar. Examine listing 3.

Listing 3 An e-commerce order as a document with line items, pricing, and a shipping address

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30

doc =
	{ :_id => BSON::ObjectID("6a5b1476238d3b4dd5000048")
		:user_id => BSON::ObjectID("4c4b1476238d3b4dd5000001")
		:state => "CART",
		:line_items => [
			{ :_id => BSON::ObjectID("4c4b1476238d3b4dd5003981"),
				:sku => "9092",
				:name => "Extra Large Wheel Barrow",
				:pricing => {
					:retail => 5897,
					:sale => 4897,
				}
			},
			{ :_id => BSON::ObjectID("4c4b1476238d3b4dd5003981"),
				:sku => "10027",
				:name => "Rubberized Work Glove, Black",
				:pricing => {
					:retail => 1499,
					:sale => 1299,
				}
			}
		],
		:shipping_address => {
			:street => "588 5th Street",
			:city => "Brooklyn",
			:state => "NY",
			:zip => 11215
		},
		:subtotal => 6196
	}

The second attribute, user_id, stores a given user’s _id. That points to our sample user, shown in listing 4 (we discuss this listing presently).

Listing 4 A user document with addresses and payment methods

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

doc =
	{ :_id => BSON::ObjectID("4c4b1476238d3b4dd5000001")
		:username => "kbanker",
		:first_name => "Kyle",
		:last_name => "Banker",
		:hashed_password => "bd1cfa194c3a603e7186780824b04419",
		:addresses => [
			{:name => "home",
				:street => "588 5th Street",
				:city => "Brooklyn",
				:state => "NY",
				:zip => 11215},
			{:name => "work",
				:street => "1 E. 23rd Street",
				:city => "New York",
				:state => "NY",
				:zip => 10010},
		],
		:payment_methods => [
			{:name => "VISA",
				:last_four => 2127,
				:crypted_number => "43f6ba1dfda6b8106dc7",
				:expiration_date => Time.utc(2014, 5)
			}
		]
	}

This modeling arrangement makes it easy to query either side of the relationship. To find all orders for a given user is simple:

1

@orders.find({:user_id => some_user_id})

The query for getting the user for a particular order is equally simple:

1
2

user_id = @order['user_id']
	@users.({:_id => user_id})

Thus, a one-to-many relationship between orders and users is easily accomplished using a foreign key. Let’s look at some other salient aspects of our order document. In general, we’re using the rich representation afforded to us by the document data model to represent the object holistically. You’ll see that the document includes both the line items and the shipping address, which, in a normalized relational model, would be located in separate tables. Our line items consist of an array of subdocuments, each describing a product in our shopping cart. The shipping address attribute points to a single object containing address fields.

Let’s take just a moment to discuss the merits of this representation. First, there’s a win for the human mind. Our entire concept of an order, including line items, shipping address, and, eventually, payment information, can be encapsulated in a single entity. When querying the database, we can return the entire order object without a single join. What’s more, the products, as they appeared when purchased, are effectively captured within our order document. The order document can be easily queried and modified.

The user document presents similar patterns in storing a list of address document along with a list of payment method documents. In addition, at the top level of the document, we store the basic attributes common to any user model.

And, like the slug field on our product, the username field here would have a unique index.

Reviews

We’re going to close out our sample data model with product reviews. Relationally speaking, each product has many reviews. Here, that relationship is encoded using a foreign key, review_id, which you can see in our sample review document in listing 5.

Listing 5 A document representing a product review

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

doc =
	{ :_id => BSON::ObjectID("4c4b1476238d3b4dd5000041"),
		:product_id => BSON::ObjectID("4c4b1476238d3b4dd5003981"),
		:date => Time.at(2010, 6, 7),
		:title => "Amazing",
		:text => "Has a squeaky wheel, but still the best wheel barrow I've ever owned.",
		:rating => 4,
		:user_id => BSON::ObjectID("4c4b1476238d3b4dd5000041"),
		:username => "dgreenthumb",
		:helpful_votes => 3,
		:voter_ids => [ BSON::ObjectID("4c4b1476238d3b4dd5000041"),
			BSON::ObjectID("7a4f0376238d3b4dd5000003"),
			BSON::ObjectID("92c21476238d3b4dd5000032")
		]
	}

Most of the remaining attributes are self-explanatory: we store the review’s date, title, and text; the rating provided by the user; and the user’s id. But it may come as a surprise that we store the username as well. After all, if this were an RDBMS, we’d be able to pull in the username with a join on the users’ table. Since we don’t have the join option with MongoDB, we can proceed in one of two ways: either query against the users collection for each review or accept some denormalization. Issuing a query for every review seems unnecessarily costly when the attribute being queried, the username, is extremely unlikely to change. So it makes sense to take the denormalized route. This does mean that a username update is more expensive, since a username will have to change in every place that it appears but, again, that happens infrequently enough to justify this design choice.

Also noteworthy is our decision to store votes in the review document itself. Of course, it’s common for users to be able to vote on reviews. Here, we store the object id of each voting user in an array of voter ids. This allows us to prevent users from voting on a review more than once, and it also gives us the ability to query for all the reviews a user has voted on. Notice that we also cache the total number of helpful votes, which allows, among other things, sorting of reviews based on helpfulness. With that, we’ve covered a basic data model for e-commerce.

Summary

We’ve covered a lot of ground in this article! We began with a theoretical discussion of schema design and then proceeded to outline the data model for an e-commerce application. This gave you a chance to see what documents might look like in a production system and it should have gotten you thinking in a more concrete way about the differences between schemas in RDMBSs and MongoDB. Granted, some of the rationale for the design of these documents won’t become clear until you get more experience with MongoDB’s query and update powers. But the general pattern of using a rich document data structure should be coming to the fore.

This article is based on Liferay in Action, to be published on August, 2011. It is being reproduced here by permission from Manning Publications. Manning publishes MEAP (Manning Early Access Program,) eBooks and pBooks. MEAPs are sold exclusively through Manning.com. All pBook purchases include free PDF, mobi and epub. When mobile formats become available all customers will be contacted and upgraded. Visit Manning.com for more information. [ Use promotional code 'java40beat' and get 40% discount on eBooks and pBooks ]

Introduction

Liferay’s code generator for database persistence, which is called Service Builder, jump-starts portlet development. This utility (which ships as part of Liferay) creates code and SQL for accessing a database from within portlets. Since it uses Spring and Hibernate to implement this, it’s not much different from what you would already do manually, with the important exception that it does much of this “grunt work” automatically, freeing time for you to implement your business logic.

In this article, we’ll continue working with the data layer of a product registration portlet for a fictional company named Inkwell, which we started in part 1 of this series. In part 1, you used Service Builder to generate your persistence objects. Now we’ll see how to use that persistence layer to store and retrieve objects.

How Service Builder works

As shown in figure 1, Service Builder automatically generates several files for you. Among them are both the Data Access Object and the Data Transfer Object layers.

Let’s see how this actually works.

Implementing the DTO layer

Though not created yet, Inkwell’s design calls for a form in Edit mode of the portlet. This form allows users to enter products that appear in a dropdown selection box in the actual registration form that makes up the portlet’s View mode. Only two fields will be coming from this form because the third is the primary key, which will be generated:

• Product name
• Product’s serial number mask, which is set by the manufacturing department and can be used for field validation

So, how do we get the data the user entered into the database? Remember, we’re after the idea of loose coupling. This means that we don’t want to have any database insertion code in our business logic. So our business logic (when we get to it) will need to call a generic method that has nothing to do with the database. This is where the DTO layer comes in. It acts as a buffer between your business logic and your underlying database code. Service Builder generates both layers, leaving the DTO layer as a stub for you to implement as you wish.

Create a generic method

Presumably, we’ll want a method that has two parameters in the method signature for our two values—the product name and the product’s serial number mask. So let’s create it.

Open the file ProductLocalServiceImpl.java. You’ll find this in a newly generated package called com.inkwell.internet.productregistration.service.impl. This class extends a generated class called ProductLocalServiceBaseImpl (see figure 2).

If you’re using an IDE (such as Eclipse) that does not automatically recognize when another tool adds files, you may need to click on the project name and press F5 to refresh the project in order to see this package.

Adding a product

Another great thing about the design of Service Builder is that you never have to touch or modify any of the classes it generates. All of the Spring dependency injection, the Hibernate session management, and the query logic stay in classes you never have to touch. You add code in a class that extends the generated class and, if you add something that changes the interface/implementation contract, those changes are propagated up the chain so the contract is never broken. We’ll see an example of this a bit later. For now, you’ll see that this file is just an empty stub. This is because you have yet to implement anything. So we’ll implement our first method, which will take the values passed to it and call the underlying database code to persist the data to the database (listing 1).

Listing 1 Adding a product to the database

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

public Product addProduct(PRProduct newProduct, long userId)
throws SystemException, PortalException {
	PRProduct product = prProductPersistence.create(counterLocalService.increment(PRProduct.class.
	GetName())); 1
	resourceLocalService.addResources(newProduct.getCompanyId(),
	newProduct.getGroupId(), userId,
	Product.class.getName(), product.getPrimaryKey(), false, true, true); 2
	product.setProductName(newProduct.getProductName());
	product.setSerialNumber(newProduct.getSerialNumber());
	product.setCompanyId(newProduct.getCompanyId());
	product.setGroupId(newProduct.getGroupId());
	return prProductPersistence.update(product, false);
}
 
 
#1 Create empty object
#2 Create permissions resources

The first thing we do (#1) in this method is create a new Product object. Product is the Interface (inherited from PRProductModel), and PRProductImpl (which extends PRProductModelImpl) is the implementation. Note that all we have to do is specify the Interface here since a Factory design pattern is used to create the object for us using the prProductPersistence object that has been injected into this class by Spring. The interface and implementation of this object were both generated automatically by Service Builder, and the fields within the object map directly to the fields in the PRProduct table that we defined. To obtain a new instance of this object, we call a create method, which was also generated by Service Builder.

Because Liferay is database agnostic, it does not use any database-specific means of generating primary keys for its database. Instead, it provides its own utility for generating primary keys. Since the create method that was generated requires a primary key, we need to call Liferay’s Counter utility to generate this. Here’s the cool thing about how this is done: the Counter is automatically injected into the class we’re working on. Why? Because, if you’re doing work with databases and Service Builder, you’re 100 percent likely to need the Counter to create new objects that will be persisted.

Notice that in #2, we next make a call to resourceLocalService to persist resources. Resources are used to define permissions on the objects that are persisted. We added CompanyId and GroupId fields to our tables in order to make our portlet a non-instanceable portlet and to enable us to later implement Liferay’s permissions system.

These two variables track the portal instance and the community/organization, respectively. For example, say a user adds our portlet to a page in the Guest community of the default portal instance. Users then begin adding Products to the database. Because we’ve added the Company ID and the Group ID to the product entities, users can add our portlet to another community, organization, or portal instance and the set of records returned will be different based on the Company ID and Group ID. Our Finder methods filter the records being returned by Company ID and Group ID.

This is how Liferay allows you to place, for example, a Message Boards portlet in two different communities, and have completely different content in each. We’ll do the same thing as we write our portlet.

Once we have our object, it’s a simple matter to set the proper values in the object and then persist it to the database. We then return the object that we’ve created back to the layer that called this method in the first place.

But we’re not quite done yet.

Propagating your changes up the chain

Recall that changes to the interface/implementation contract get propagated up the chain automatically by Service Builder. We were just working in an implementation class, which extends another class, which we call a -BaseImpl class. All of the methods generated by Service Builder have entries in their Interfaces and actual implementations in their BaseImpl class. If developers wish to add more, they’re added in the -LocalServiceImpl class.

Because we’ve just added a new method to a class that implements an Interface, there is no method stub in the Interface for the method we just created. To continue without errors, we need to run Service Builder again. This can be done by running the build-service Ant task. When you run the task, Service Builder will regenerate the interface, which is called PRProductLocalService, to include a stub for the method you created in the implementation class.

This is generally the point where people scratch their heads and say, “Isn’t this backwards? Aren’t you supposed to define the Interface first and then write the implementation?”

In a sense, yes, that’s correct. But, remember that we’re working with a code generator, and it’s that code generator’s job to make things easier for us as developers. By using Service Builder, we are defining the Interface first, as much as we can do that up front. We did this in one step, when we defined our database table in service.xml. Service Builder generated our Interface as well as our default implementation—as best as it could guess at what we needed. That’s what went in PRProductLocalServiceBaseImpl.java. If you need further customization—and you generally do—you can add your own methods. These need to be added to a class that is free of the code generator, so that your changes don’t get overwritten. So PRProductLocalServiceImpl.java is provided as a DTO layer, which allows you to add any methods you may need.

You’ll generally only need to make customizations in one class: the -LocalServiceImpl class. In some cases, you may need to make customizations in -Impl classes in the model.impl package. The only thing we need to worry about right now is the -LocalServiceImpl class.

Now that we’re done with adding products, let’s try deleting products.

Deleting Products

If you think about how you might want to delete products, there are two ways:

• You already have a Product object, and you just want to delete it.
• You have a Product’s primary key, and you want to delete it that way without having to retrieve the whole Product.

To make things easier for us in the Controller layer, we’ll overload the delete method by creating versions of it that can handle both cases:

1
2
3
4
5
6
7
8
9
10
11
12

public void deleteProduct(long productId, long companyId)
throws NoSuchProductException, SystemException, PortalException {
	PRProduct product = prProductPersistence.findByPrimaryKey(productId);
	deleteProduct(product, companyId);
}
public void deleteProduct(Product product, long companyId)
throws PortalException, SystemException {
	resourceLocalService.deleteResource(
	companyId, Product.class.getName(),
	ResourceConstants.SCOPE_INDIVIDUAL, product.getPrimaryKey());
	productPersistence.remove(product);
}

What we’ve done here is enable us to delete a PRProduct using its primary key or the object itself. If we’re using the primary key, we’ll first retrieve the PRProduct object and then call deleteProduct on it. Note that we delete the resource as well as deleting the product. This keeps Liferay’s permissions tables free of “dead” data that might otherwise be left there simply taking up space. When you’re finished modifying your -Impl class, run Service Builder again using ant build-service. As before, this adds your new methods to the interface.

The only thing remaining to do is to query the database for the products we’ve entered because, presumably, users will want to view them and/or edit them.

Querying the database

Just like adding and deleting, getting products out of the database is really easy. You’ll probably remember that we had Service Builder generate a finder for this, which queries the database for products by the GroupId. We always want to query by GroupId because our portlet is non-instanceable. So we’ll implement this in our DTO layer like this:

1
2
3
4
5

public List<PRProduct> getAllProducts(long groupId)
throws SystemException {
	List<Product> products = prProductPersistence.findByGroupId(groupId);
	return products;
}

This will return a List of PRProducts, which can be used in the UI layer to display products for viewing or editing purposes. And, as you can see, whatever calls this method (it will be our portlet class) does not need to know anything about JDBC or databases. It’s simply requesting a List. This frees you to do something like swap out Service Builder later, if you find that it doesn’t meet your needs.

Summary

Service Builder makes it easy to generate a whole persistence layer of an application. Using well-known design patterns such as Data Access Objects and Data Transfer Objects, it not only implements a consistent design but also helps the developer to do so. Finders are automatically generated, which allow developers to access their data as Java objects.

This article is based on Liferay in Action, to be published on August, 2011. It is being reproduced here by permission from Manning Publications. Manning publishes MEAP (Manning Early Access Program,) eBooks and pBooks. MEAPs are sold exclusively through Manning.com. All pBook purchases include free PDF, mobi and epub. When mobile formats become available all customers will be contacted and upgraded. Visit Manning.com for more information. [ Use promotional code 'java40beat' and get 40% discount on eBooks and pBooks ]

Introduction

If you’re an experienced developer, it’s likely that you’ve dealt with a data-driven application before. Liferay ships with a tool called Service Builder, which makes the creation of data-driven applications very easy. I highly recommend that you use Service Builder when writing applications on Liferay’s platform because it will get you going very quickly. How? By generating a lot of the database plumbing code for you so you can concentrate on your application’s functionality.

A database-persistence code generator

With Service Builder, you can simply define a database table in an XML file. Based on that definition, Service Builder then generates the entire Hibernate configuration, the entire Spring configuration, finder methods, your model layer, the SQL to create the table on all leading databases, and your entire Data Access Object layer at one fell swoop. Figure 1 illustrates this.

Did you know?

Service Builder is a proven tool that produces code suitable for enterprise deployments. In fact, all of Liferay’s internal database persistence is generated using Service Builder.

If you’re like me, I know what popped into your head as soon as I said “code generator.” It was, “Oh, no. Code generators are bad.” And then you began justifying that statement with many sound, accurate, and excellent arguments. Believe me, I agree with you. But Service Builder is different. From one code generator “hater” to another: Service Builder is designed to enable you to write custom code, not prevent it. It just takes care of the mundane stuff you hate writing, anyway. That is a code generator I’m on board with. I think you’ll like it, too.

We’re going to create a product registration application for a fictional company named Inkwell. This application is designed to replace those warranty registration cards that ship with electronic equipment. You’ll use Service Builder to generate the database tables as well as database persistence. To start, you’ll create that one XML file that is the key to generating the rest of the code.

Creating the service.xml file

Create a file called service.xml in the WEB-INF folder of your project. This file will contain your table definitions. You’ll populate this file with all the information Service Builder needs to generate the SQL for the table—for all the databases Liferay supports as well as database persistence objects you can use in your Java code.

We’ll use the simplest of the tables as our example, which is the Product table. The following listing defines the table for Service Builder.

Listing 1 Defining the Product table using Service Builder

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36

<?xml version="1.0" encoding="UTF-8" ?>
<!DOCTYPE service-builder PUBLIC "-//Liferay//DTD Service Builder 6.0.0//EN" "http://www.liferay.com/dtd/liferay-service-builder_6_0_0.dtd">
<service-builder package-path="com.inkwell.internet.productregistration"> 1
<author>Rich Sezov</author> 2
<namespace>PR</namespace> 3
<entity name="PRProduct" local-service="true" remote-service="false"> 4
<column name="productId" type="long" primary="true" /> 5
<column name="productName" type="String" /> 6
<column name="serialNumber" type="String" />
<column name="companyId" type="long" /> 7
<column name="groupId" type="long" />
<order by="asc"> 8
<order-column name="productName" />
</order>
<finder name="G_PN" return-type="Collection"> 9
<finder-column name="groupId" />
<finder-column name="productName" />
</finder>
<finder name="GroupId" return-type="Collection">
<finder-column name="groupId" />
</finder>
<finder name="CompanyId" return-type="Collection">
<finder-column name="companyId" />
</finder>
</entity>
</service-builder>
 
#1 Java package for code
#2 Author for JavaDoc
#3 Namespace for tables
#4 Name of entity
#5 Primary key
#6 Additional fields
#7 Foreign Keys
#8 Order data is returned
#9 Finder methods

This file is pretty easy to read, so we’ll attack it section by section instead of line by line.

DEFINE JAVA PACKAGE

You can define the Java package (#1) into which Service Builder generates the code that powers this database table. Service Builder also generates JavaDoc, and the name you place in the Author tags (#2) will wind up in the JavaDoc as the author of the code. By default, tables you define with Service Builder will go in the Liferay database. To set them apart from the rest of Liferay’s tables, you can prefix them with a namespace (#3). This table, when created, will actually be called PR_PRProducts in the database.

DEFINE ENTITY

Now we get to the cool stuff. The database entity—which, for us, is a Product—is defined using the Entity tag (#4). The two parameters in this tag define how you want Service Builder to generate the service that retrieves and saves these entities. At minimum, you need to have a local service, but you can also have a remote service. This is not an EJB. It is instead a web service, complete with a WSDL document describing it so that your service may participate as part of a Services Oriented Architecture (SOA).

DEFINE COLUMNS

Next, we define the columns and finder methods. Our first column (#5) is defined as a primary key in the database. Other fields we want to store come next (#6). We are also defining two foreign keys (#7): a companyId and a groupId. The DBA team did not specify these two foreign key fields in the tables, but we added them anyway. These fields are internal to Liferay and are used for context purposes in non-instanceable portlets. The companyId corresponds to the portal to which the user has navigated, and the groupId corresponds to the community or organization to which the user has navigated. Since we will be using these field values as parameters in all of our queries, our portlet will have different data in different portals, communities, and organizations.

Quick test

Without reading the next sentence, answer this question: Is our portlet instanceable or non-instanceable? The portlet is non-instanceable because we’ll be using these fields to make sure that the portlet ties all data to the portal and to the community or organization upon which the portlet is placed.

#8 defines a default ordering of the entities when they are retrieved from the database. You can choose to have them returned in ascending or descending order. You aren’t limited to one column here; you can specify multiple columns, and the ordering will happen by priority in order of the columns.

DEFINE FINDER METHODS

The finder methods (#9) actually go and retrieve the objects. Specifying these finders means that Service Builder will automatically generate methods that retrieve objects from the database using the parameters you specify in the finder. For example, the first finder returns Products by groupId and productName.
Now that you’ve defined your table, it’s time to run Service Builder.

Running Service Builder

Save the service.xml file and then run the Ant task called build-service. You’ll see several files get generated and the task will complete with a BUILD SUCCESSFUL message. If you’re using an IDE that compiles source files automatically, you’ll notice errors suddenly appear in your project. Don’t worry about this; it’s easy to fix.

What has happened is that Service Builder generated several Java source files. Some of them are in your existing src folder. Others are in a new source folder called service that it also generated. This folder will resideinside of WEB-INF like the src folder you already have. To fix the errors in your project, just use your IDE’s facility to add another source folder to your project. You may also need to refresh your project. Once you do that, the errors will go away.

Service Builder divides the source it generates into two layers:

• An interface layer gets generated in the aforementioned service folder. You’ll never change anything in the interface layer manually; Service Builder always generates the code that’s found there.
• An implementation layer gets generated in your src folder and is initially just skeleton code that allows you to implement the functionality you need.

You’ll notice that there’s also a new file in the root of your src folder called service.properties. This file was also generated by Service Builder. It contains properties that Service Builder needs at runtime to perform its functions. The most important of these properties is a list of Spring configuration files that were also generated.

Another new construct that was generated was a META-INF folder in your src folder. This folder contains all of the XML configuration files that Service Builder needs, including Spring configuration files and Hibernate configuration.

What all of this means is that Service Builder has taken care of your database persistence configuration for you. If you’ve ever used Hibernate alone or the combination of Hibernate and Spring, you know that there are multiple configuration files to juggle between the two. Service Builder automatically configures all of that for you and provides you with static classes that you can use to perform your database persistence functions. It provides both a Data Access Object (DAO) layer and a Data Transfer Object (DTO) layer for you automatically.

Summary

Liferay’s code generator for database persistence, which is called Service Builder, jump-starts portlet development. This utility (which ships as part of Liferay) creates code and SQL for accessing a database from within portlets. Since it uses Spring and Hibernate to implement this, it is not much different from what developers would already do manually, with the important exception that it does much of this “grunt work” automatically, freeing time for developers to implement their business logic.

In part 2 of this 2-part series, I’ll show you how to provide the functionality in your DTO to keep your portlet code from being dependent on anything related to SQL databases.

This article is based on Liferay in Action, to be published on August, 2011. It is being reproduced here by permission from Manning Publications. Manning publishes MEAP (Manning Early Access Program,) eBooks and pBooks. MEAPs are sold exclusively through Manning.com. All pBook purchases include free PDF, mobi and epub. When mobile formats become available all customers will be contacted and upgraded. Visit Manning.com for more information. [ Use promotional code 'java40beat' and get 40% discount on eBooks and pBooks ]

Introduction

Liferay’s Service Builder encourages the proper layering of functionality within a portlet application. Generally, when building an application, it’s good practice to separate the various layers of the application: UI, model, persistence, and so on. This is sometimes called separation of concerns. By keeping the layers as separate as possible, you gain the ability to change the implementation of any one layer more easily—if for some reason you find a better way to do it later.

To some, this must seem like some unnecessary work, but let me give you an example as to why this is important.

The spaghetti code story

Suppose you have a poorly designed application that consists of only two layers—your JSPs and your portlet class. You develop the application by creating action URLs in your JSPs, which correspond to various methods within the portlet. These methods use JDBC to connect to a database and select, insert, update, and delete data. Your database, validation, and page navigation code is all in the portlet or the JSPs, resulting in very large files with lots of logic in them. You test the portlet as well as you can, and it then gets deployed to production.

Within the first day of its use, a bug is found. Your application has a last name field, and a user tried to insert the last name O’Bannon. Because you coded all of your database interaction manually, you forgot that sometimes you need to escape out certain characters. In this case, the apostrophe in the name is causing your database insert to fail.

With the poor design you used for your portlet, you’ll now have to modify the Insert and Update methods of your portlet class and add some code that checks the values your users are inserting for apostrophes. This is an example of a tightly-coupled design: your database code is inside your portlet, so your implementation of code that communicates with the database is intertwined with your business and display logic. Because you were recently working on this portlet, you know where the problem is and so you fix it and redeploy the application. The users work with it for several months before another bug is found. It seems that, somewhere in the portlet, a database connection isn’t being closed and is causing the database server to run out of connections.

Now some significant time has passed, and you’re onto another project. You don’t remember exactly what you did when you wrote this one. You now have to slog through mounds of spaghetti code in your one portlet class looking for the problem—database code mixed in with all kinds of other functionality, such as JavaScript functions, field validation code, database access code, and so on. It’s become a hard-to-maintain mess. After hours of debugging, you finally find the condition where the database connection wasn’t being closed and it was part of a long block of if-then-else conditions, which included field validation logic, business logic, and persistence logic. It took you almost an entire day to find this one problem, because it was buried in a lot of code that should have been separated out into different layers of code logic.

If you had properly layered your application so that the UI code was in one place, the business logic was another, and the persistence logic was in another, you’d have a much more maintainable project. And you might decide at that point that you could use something like Hibernate for your database persistence, and you’d only have to replace a few classes in the one layer. Doing that would solve all of your persistence problems, because Hibernate escapes characters and closes connections automatically. You might then go another step further and replace your homegrown application logic with an MVC framework like JSF or Struts. On each refactor, you could modify one layer of your application at a time. Let’s see how to do that for the persistence layer.

Using two layers for persistence

While the above is a simplistic example, it serves as an example we can use to show how Service Builder encourages proper separation of concerns. For database persistence, two of these layers are the Data Access Object and the Data Transfer Object (see figure 1). These two layers can be written by developers manually, and often a lot of time is spent writing and debugging code in this layer. Service Builder automatically generates both layers for you.

This article is based on iPhone and iPad in Action, published on August, 2010. It is being reproduced here by permission from Manning Publications. Manning publishes MEAP (Manning Early Access Program,) eBooks and pBooks. MEAPs are sold exclusively through Manning.com. All pBook purchases include free PDF, mobi and epub. When mobile formats become available all customers will be contacted and upgraded. Visit Manning.com for more information. [ Use promotional code 'java40beat' and get 40% discount on eBooks and pBooks ]

Introduction

When you’re using the iPhone’s orientation notification, the frameworks are doing the work for you: they’re taking low-level acceleration reports and turning them into more meaningful events. It’s similar to the concept of iPhone actions, which turn low-level touch events into high-level control events.

Notifications aren’t going to be sufficient if you would prefer to program entire interfaces that effectively use the iPhone’s movement in three-dimensional space as a new user-input device. For that, you’ll need to access two iPhone classes: UIAccelerometer and UIAcceleration. We’ll talk about UIAccelerometer.

The UIAccelerometer is a class that you can use to receive acceleration-related data. It is a shared object, like UIApplication and UIDevice. The process of using it is shown in listing 1.

Listing 1 Accessing the UIAccelerometer takes just a few steps

1
2
3
4
5
6
7
8
9
10
11

- (void)viewDidLoad {
UIAccelerometer *myAccel =
#1
[UIAccelerometer sharedAccelerometer]; #1
myAccel.updateInterval = .1; #2
myAccel.delegate = self; #3
[super viewDidLoad];
}
#1 Creates shared accelerometer
#2 Updates 10 times a second
#3 Delegates accelerometer protocol

The first step is to access the accelerometer, which is done with another call to a shared-object method (#1). Having this step on its own line is probably unnecessary, because you could perform the other two steps as nested calls, but we find this a lot more readable.

Next, you select your update interval (#2), which specifies how often you’ll receive information on acceleration. This is hardware limited, with a current default of 100 updates per second. That’s most likely just right if you’re creating a game using the accelerometer, but excessive for other purposes. We’ve opted for 10 updates per second, which is an updateInterval of 0.1. You should always set the lowest acceptable input to preserve power on the iPhone.

Finally, you must set a delegate for the accelerometer (#3), which is how you receive data on accelerometer changes. The delegate will need to respond to only one method, accelerometer:didAccelerate:, which sends a message containing a UIAcceleration object whenever acceleration occurs (to the limit of the updateInterval).

Summary

The iPhone’s accelerometers can give you access to a variety of information about where an iPhone exists in space. By measuring gravity, you can easily discover an iPhone’s precise orientation. By measuring movement, you can see how an iPhone is being guided through space. Finally, you can build more complex movements into three-dimensional gestures, such as the shake.

Accelerometers allow users to make simple adjustments to a program. We can imagine game controls and painting programs both built entirely around the accelerometers.

This article is based on iPhone and iPad in Action, published on August, 2010. It is being reproduced here by permission from Manning Publications. Manning publishes MEAP (Manning Early Access Program,) eBooks and pBooks. MEAPs are sold exclusively through Manning.com. All pBook purchases include free PDF, mobi and epub. When mobile formats become available all customers will be contacted and upgraded. Visit Manning.com for more information. [ Use promotional code 'java40beat' and get 40% discount on eBooks and pBooks ]

Introduction

Adding a map to any application is very similar to adding any other view. You can either do it through Interface Builder or programmatically. The choice is up to you depending on how you like to work with UI elements. We will show you both methods.

The view that displays a map is called MKMapView. This class contains quite a bit of functionality including how to display the map, annotations, and user location. We will first be discussing how to add an MKMapView through Interface Builder. Once we add the map to the view, we will connect it to an IBOutlet and set its delegate class.

Adding the map using Interface Builder

Before you start coding the MKMapView, you must first import MapKit.framework. This will provide you will all of the libraries needed to work with maps.

To add a map to your view, your must first create an MKMapView IBOutlet in the class that will be using the map. Listing 1 shows how to do this.

Listing 1 Declaring an IBOutlet for an MKMapView

1
2
3
4
5

#import <UIKit/UIKit.h>
#import <MapKit/MapKit.h>
@interface SimpleMapViewController : UIViewController<MKMapViewDelegate> {
IBOutlet MKMapView * theMap;
}

@property (nonatomic, retain) IBOutlet MKMapView * theMap;
The first thing we want to point out is you must import the MapKit.h header file. This is included in MapKit.framework and is needed for all MapKit interaction. The second is, our class implements the MKMapViewDelegate interface. It will allow this class to become the delegate of the MKMapView and receive messages from it. In addition to making a property, you should also synthesize the map property in the .m file.

Now that the IBOutlet has been declared, you need to add the MKMapView to the view in the nib file. To do this open up the nib file associated with the class that contains your IBOutlet. Now, select MKMapView from the Library and drag it on to your view. Figure 1 shows what this object should look like.

The last thing that needs to be done is set the delegate of the MKMapView. To do this, right click on the map and drag to the File’s Owner object. It should pop up a bubble that reads, “delegate”. Click the word delegate to make the connection. Figure 2 shows what the Connection Inspector should look like after you have completed all of these steps.

Adding the map programmatically

As noted before, whether you add the map using Interface Builder or with code is up to you. It is completely dependent on your preferences and organization technique. Listing 2 demonstrates how to add an MKMapView to your view controller’s view using code.

Listing 2 Adding an MKMapView programmatically

1
2
3
4
5
6
7

- (void)viewDidLoad {
[super viewDidLoad];
MKMapView * map = [[MKMapView alloc] initWithFrame:
CGRectMake(0, 0, 320, 480)];
Map.delegate = self;
[self.view addSubview:map];
}

This code is actually quite short. We create the map using a frame, set its delegate to our class, and add it to our class’ view. As with any other view, modifying the frame passed to the map will alter its size and position.

Controlling the map

By default, the map has gives the user a bit of control. Without any additional code, they have the ability to scroll all over the world by flicking the map. Also, the map comes with the ability to zoom in and out by using the pinch and pull gestures.

To navigate the map programmatically, you must specify a region. Doing this will allow you to move the map to a specified location. You will also have the ability to set the zoom level.

Let’s examine the region properties and methods for navigating the map. Table 1 discusses these properties and methods and their uses.

The code below shows you how to create an MKCoordinateRegion and move the map to display it on the screen. The method we created here fires when a user presses a button titled Apple. It will move the map from its current location to center on Apple’s headquarters in Cupertino.

Listing 3 moving the map to a given location

1
2
3
4
5
6
7
8

- (IBAction) apple:(id) sender {
CLLocationCoordinate2D coords;
coords.latitude = 37.33188;
coords.longitude = -122.029497;
MKCoordinateSpan span = MKCoordinateSpanMake(0.002389, 0.005681);
MKCoordinateRegion region = MKCoordinateRegionMake(coords, span);
[theMap setRegion:region animated:YES];
}

The first thing we see here is a CLLocationCoordinate2D being created. This is simply a struct that holds two doubles for latitude and longitude. Next, we assign them to the coordinates of Apple using the WGS 84 reference frame.

Following the coordinates, we create a span using the MKCoordinateSpanMake method. An MKCoordinateSpan is a struct made of two doubles that represent a delta for latitude and longitude. The span represents the amount of area to view and is used when setting the zoom level. A larger number here tells the map to show a larger view area resulting in the map zooming out. Similarly, a smaller number tells the map to show less area and causes it to zoom in.

Once the coordinates and span have been initialized, we can create a region using the MKCoordinateRegionMake method. Finally, we set the region of the map to our newly created one and animate the transition. In addition to controlling the map’s location, you can also control how the user can interact with it. Table 2 details these properties.

As mentioned above, the mapType property allows you to display the map in three different ways. Most likely, you will want to make this configurable by the user, as preferences tend to vary about how the map should display. Figure 3 shows the difference between the map types.

The other two are quite similar. The only difference between them is the hybrid view contains the road names in addition to displaying the satellite photos.