MongoDB Client Framework for .Net

10gen has an official Mongo C# driver which provides API for basic Mongo interaction and  excellent LINQ support. However, there is a lot more that goes into writing Data Access Layer (DAL) code which typically includes constructs like Data Access Objects, Repository Pattern, ORM configuration, Transient Fault Handling et al. This is where the Mongo.Framework tries to bridge the gap and provide the patterns that have become industry standard which everybody needs.

Introduction

Mongo.Framework builds on top the 10gen’s official Mongo C# driver and provides standard DAL patterns and constructs that enable rapid application development.

The framework provides following capabilities:

• Constructs for Data Access Layer (DAL) abstraction
• Repository pattern to interface with Mongo driver
• Standardized pattern to define Object Model, its ORM and versioning scheme
• Transient fault handling and configurable retry logic using the Enterprise Library
• Command pattern to provide stored procedure like semantics
• Constructs to enable ACID semantics using Compare And Swap(CAS) approach
• Constructs to generate various hashes (MD5, Murmur2)
• Constructs to generate identifiers similar to Identity column in SQL
• Fault injection framework to test ACID semantics of commands under faults

The source code is available at GitHub.

Theory

This section explains the design of the framework and elaborates upon some of the key constructs and design philosophies.

Data Access

The center piece of Mongo Framework is the MongoDataAccess class. This class is responsible for executing Mongo Commands and providing transient fault handling by using configurable retry logic. It is injected with IMongoDataAccessAbstractFactory which provides access to various factories (each of which is covered in sections below). The purpose of this class is to serve as the base class for concrete implementations of Data Access Object (DAO)  which enables the derived class to leverage all the boilerplate code MongoDataAccess provides.

MongoCommand in turn uses another abstraction layer called MongoRepository to interface with the underlying Mongo cluster. The rationale for this abstraction is explained in “Mongo Repository” section below. But, for now remember that it contains Mongo specific CRUD operations on entities.

Mongo Command

MongoCommand plays the same role that Stored Procedure plays in SQL world. The purpose of this class it to provide the smallest unit of transaction. Generally, for each of the methods in the DAO interface a corresponding command implementation is created. It is important to note that in case of transient faults the whole command is replayed which implies interesting dynamics which will be covered in future posts.

BaseMongoCommand implements most of the common command logic and provides utility methods for concrete command implementations. It internally has 2 abstract template methods (ExecuteInternal and ValidateInternal) which is all that a concrete command needs to implement.

Mongo Repository

IMongoRepository is based on Repository Pattern which abstracts the CRUD operations specific to Mongo. This additional layer provides the flexibility to use different Mongo driver/connector implementations. The “Default” implementation is based on 10gen’s official C# driver. Another important reason for putting this abstraction in place is to allow mocking of IMongoRepository interface and do fault simulation to test the transient fault handling/ retry logic.

The repository interface is templatized and the template parameter is data model object (aka entity) which represents a document in a Mongo collection. This implies that for each collection within Mongo there will be a corresponding Model object in DAL and the DAL will access that document by creating a MongoRepository instance with that Model object as the template parameter.

var post = new Post()
{
    Title = "My First Post",
    Body = "This isn't a very long post.",
    Tags = new List<string>(),
    Comments = new List<Comment>  {
        { new Comment() { TimePosted = new DateTime(2013,1,3),
                          Email = "jackie@gmail.com",
                          Body = "Why are you wasting your time!" }},
    }
};

GetRepository<Post>().Add(post);

Transient Fault Handling

Transient faults, as the name implies, are temporary exception scenarios (like network failures, primary mongodb going down etc.) from which the system can recover within a short period of time. Any seasoned developer knows that one of the most important considerations while writing fault tolerant code is retry logic to handle transient faults. It is unfortunate that 10gen driver doesn’t provide this out-of-the-box. But, don’t sweat the framework is your savior.

 The framework provides transient fault handling capability by leveraging the Enterprise Library Azure Integration Pack.

• The Enterprise Library provides a RetryPolicy interface which comprises of ITransientErrorDetectionStrategy (to detect whether a given exception is transient or not) and RetryStrategy (which specifies the retry logic – fixed interval, incremental or exponential back off).
• The framework provides its own implementation for ITransientErrorDetectionStrategy to detect various Mongo driver and DB specific transient fault conditions.
• The framework currently uses FixedInterval based RetryStrategy but it can easily be changed to incremental or exponential backoff.

ID Generation

Mongo has no concept of Identity column as a result we have to provide our own implementation in order to achieve the same effect. The approach used for id generation is to create an “Identity” collection which contains the last id of various types as shown below. And FindAndModify method  is used to atomically update the Id of a given type.

{ "_id" : "Server", "seq" : 736 }
{ "_id" : "Share", "seq" : 1185 }

 The implementation is wrapped into IIdGenerator and IMongoIdGeneratorFactory interface to have necessary abstractions in place for future enhancements and mocking ability for testing purposes.

Hash Generation

Hash generation provides the ability to generate hash code (MD5, SHA1 etc.) of a given string. This is required for generating hash fields which are then used as shard key for random distribution of documents. The rationale for wrapping this into various interfaces is same as that for Id Generation.

The framework currently supports MD5 and Murmur2 hash codes.

Data Model

Last but not least aspect of the framework is the construction of data model used to serialize and de-serialize data from MongoDB. A data model is basically a class which is the application code counterpart of a document in Mongo.

For every Mongo collection you want to access in DAL you need to create a corresponding data model class. This model will be the template parameter that you will pass into the IMongoRepository interface to access/manipulate the collection. It is important to note that 10gen C# driver takes care of all the ORM for us and the only thing we need to do is correctly setup the class mapping (i.e. which field in data model maps to which field in the mongo collection).

Following are the design principles you need to adhere to for creating the data model:

1. The collection name is specified by setting the MongoCollectionName attribute on model class.
2. Derive the class from IVersionedEntity to enable schema versioning (if desired). I will explain in later posts about how this can be used for Schema Evolution.
3. Declare all the data fields as properties in the “Data Model” region.
4. Declare all the object mapping in the static constructor of the high level data model class.
   • Mapping for any dependent/contained classes needs to be defined in the high level data model class only.
   • Use the SetIgnoreExtraElements to ensure backward compatibility.
   • Declare all the field names as const variables in “Field Names” region so that same can be used during query construction to ensure consistency.
   • It is important to note that MongoRepository invokes the static constructor of data model at appropriate time to register the mapping at runtime.
5. Declare all the adapter logic for to/from conversion of data model to VO or DTO in the “Adapter interface to convert to/from model” region.

Following is an example data model for illustration:

[MongoCollectionName(COLLECTION_NAME)]
public class Post
{
    #region Field names

    public const string COLLECTION_NAME = "post";
    public const string FN_TITLE = "title";
    public const string FN_COMMENTS = "comments";
    public const string FN_COMMENT_TIME = "time";
    ...

    #endregion

    #region Data model

    public ObjectId Id { get; set; }
    public string Title { get; set; }
    public IList<Comment> Comments { get; set; }

    #endregion

    #region Object mapping

    static Post()
    {
        // Setup the mapping from collection columns to model data members
        BsonClassMap.RegisterClassMap<Post>(cm =>
        {
            cm.AutoMap();
            cm.SetIgnoreExtraElements(true); // For backward compatibility
            cm.SetIdMember(cm.GetMemberMap(c => c.Id));
            cm.GetMemberMap(c => c.Title)
              .SetElementName(FN_TITLE);
            cm.GetMemberMap(c => c.Comments)
              .SetElementName(FN_COMMENTS);
        });

        BsonClassMap.RegisterClassMap<Comment>(cm =>
        {
            cm.AutoMap();
            cm.SetIgnoreExtraElements(true); // For backward compatibility
            cm.GetMemberMap(c => c.TimePosted)
              .SetElementName(FN_COMMENT_TIME)
              .SetSerializationOptions(new DateTimeSerializationOptions(DateTimeKind.Utc));
            ...
        });
    }

    #endregion
}

public class Comment
{
    public DateTime TimePosted { get; set; }
    ...
}

Practice

This section will elaborate upon how to actually use the framework with the help of an example. The complete reference implementation of this example is available on GitHub in Mongo.Framework.Example project.

Example Use-Case

Your company is planning to create yet another blogging platform. But, this one is going to be different :-). It will change the whole blogging space once for all.

The product manager has come up with following amazing ideas:

• Each blog post will consist of title and body
• Users will be able to comment on posts
• Each comment will consist of the time it was posted, the email of the commenter and actual text.

The CTO after lot of thinking has decided to use Mongo as the persistence store and has tasked you to create the DAL layer with following API requirements:

• Ability to create a new post
• Ability to get all the posts
• Ability to get the posts which were commented by given person
• Ability to delete posts

You are super excited and want to show to the world how awesome you are by implementing this in less than a day. You are actually really awesome because you decide to use the Mongo.Framework.

Procedure

1. Once you are done setting by the DAL project (and including the Mongo.Framework), the first thing you need to do is define the contract which maps the above requirements into an interface. That is, you need to create the DAO interface.

   public interface IPostDataAccess
   {
      void CreatePost(Post post);
   
      Post[] GetPosts();
   
      string[] FindPostWithCommentsBy(string commenter);
   
      void DeleteAllPost();
   }
   

2. Implement the concrete data access implementation deriving from MongoDataAccess to get all the goodies. 

   class PostDataAccess : MongoDataAccess, IPostDataAccess
   {
     public PostDataAccess(ConnectionInfo connectionInfo, IMongoDataAccessAbstractFactory factories)
       : base(connectionInfo, factories)
     {
     }
   
     public PostDataAccess(ConnectionInfo connectionInfo)
       : this(connectionInfo, new DefaultMongoAbstractFactory(connectionInfo))
     {
     }
   
     public void CreatePost(Post post)
     {
        throw new NotImplementedException();
     }
     ...
   }
   

3.  Implement the data model following the guidelines described in the Theory section. Surprisingly, the example data model in illustration of that section applies to this use-case.
4. For each of the methods in the DAO implement a corresponding command class. Derive the command class from BaseMongoCommand to get all the boilerplate code.

   class CreatePostCommand : BaseMongoCommand
   {
     public Post PostToCreate { get; set; }
   
     protected override object ExecuteInternal()
     {
        GetRepository<Post>().Add(PostToCreate);
   
       return RETURN_CODE_SUCCESS;
     }
   }
   

5. Glue the command to the DAO by invoking the command in the corresponding method.

   class PostDataAccess : MongoDataAccess, IPostDataAccess
   {
      public void CreatePost(Post post)
      {
         var command = new CreatePostCommand() { PostToCreate = post };
         ExecuteCommand<int>(command);
      }
   }
   

6. Add the connection string for the mongo instance in you app.config file

   <configuration>
     <connectionStrings>
        <add name="PostsDB"
             providerName="Mongo"
             connectionString="server=10.0.20.200:47017;database=posts;journal=true;w=majority;" />
     </connectionStrings>
   </configuration>
   

7. That is it my friend. Now, you are ready to use your newly minted DAL code.

   // Instantiate the data access for posts
   var connectionInfo = ConnectionInfo.FromAppConfig("PostsDB");
   var postDataAccess = new PostDataAccess(connectionInfo);
   
   var post = new Post()
   {
      Title = "My Second Post",
      Body = "This isn't a very long post either.",
      Tags = new List<string>(),
      Comments = new List<Comment>()
   };
   
   // Insert the posts into the DB
    postAccess.CreatePost(post);
   

In future posts, I will explain some of the more advance scenarios where the framework really shines like CAS approach for atomic transaction guarantees and fault injection tests to ensure ACID semantics under transient faults.