Configuration builder in NodeJS with TypeScript

In recent project that I am working on we had a requirement of creating a configuration builder that satisfies following requirements:

  • Ability to read default configuration from a file
  • Ability to read environment specific configuration from a file
  • Ability to override any defaults by command line options
  • Ability to override and/or access any defaults by the set as environment variables
  • Ability to update configuration by any module in the application without modifying default configuration module
  • Ability to read configuration from the database

Some background

The application in which this configuration builder needs to be developed follows a modular approach where each logically grouped functionalities are defined in a module and registers themselves to the main application. A similar approach to that of Angular 1.x modules. Each module receives application object while core app is going through module registration code, so the module can access any application properties that are public. The application is developed in NodeJS and the language of code that we are using TypeScript, so we had the ability to write classes, interfaces, and decorators to simplify code and make it more maintainable.

To put in simple words, we wanted a configuration module which can read arguments, environment variables, configuration files, and database while at the same time provide the ability to any other module in application to easily access and update configuration if required. Other modules will mostly add their configuration options into default config object, at application startup, so other places it can be accessed.

Making configuration builder

configuration_builder

To start with, I created default configuration class, which will do the default configuration read and hold the result in itself. This will then be attached to main Application object as the configuration of the application, every update to configuration will alter this class only.

export default class AppConfig implements IConfig {
    additionalConfigs: Types.Map;

    constructor() {
        this.additionalConfigs = {};
    }

    putExtra(key: string, value: any, override?: boolean): boolean {
        if (this.additionalConfigs[key] && !override) {
            return false;
        }
        this.additionalConfigs[key] = value;
        return true;
    }

    getExtra(key: string): any {
        return this.additionalConfigs[key];
    }
}

IConfig is an interface where I define how default configuration is going to look like. The additionalConfigs and related extra methods were added just in case someone decides not to populate default configuration space and want to add it in additional space then they can use it while updating configuration object.

Once default configuration structure is ready, I defined an interface called IConfigurationBuilder with a method called buildConfiguration that accepts an AppConfig object as an argument. The idea behind this is that if any module wants to contribute in configuration building activity then they can implement this interface, register itself to ConfiurationBuilder and rest will be handled by the builder to make sure that it’s buildConfiguration is executed while configuration was being built.

export interface IConfigBuilder {
    configure(config: AppConfig): Promise<any>;
}

After that, I defined a ConfigurationBuilder, which is responsible for building configuration on application startup by following all above requirements. It will first read default configuration and initialise AppConfig instance, then reads environment files and update configuration object, then read command line options and environment variables and add them to configuration object and finally go through different ConfigurationBuilders registered by different modules and call their buildConfiguration method which updates configuration object based on their requirement. Some default configuration builders we implemented are EnvAndOptConfigReader and DatabaseConfigReader.

export default class ConfigBuilder {
    private builders: Array<IConfigBuilder>;
    private config: AppConfig;

    constructor(configFilePath?: string) {
        this.builders = [];
        if (!configFilePath && fs.existsSync("config.json")) {
            configFilePath = path.join(process.cwd(), "config.json");
        }
        this.config = new AppConfig();
        const customConfig = require(configFilePath);
        _.merge(this.config, customConfig);
    }

    addConfigBuilder(builder: IConfigBuilder): ConfigBuilder {
        this.builders.push(builder);
        return this;
    }

    build(): Promise<any> {
        let defers: Array<Promise<any>> = [];
        this.builders.forEach((builder: IConfigBuilder) => {
            defers.push(builder.configure(this.config));
        });
        return Promise.all(defers).then(() => {
            return this.config;
        });
    }
}

As you can see in the constructor it reads default config.JSON file and initialise AppConfig with it. The `addConfigBuilder` method is supposed to be used by other modules to register their ConfigurationBuilders and finally `build` method loops over all registered builders and waits for them to finish updating configuration object by using deferred promise.

Advantages

Following are some advantages of this:

  • Clean and maintainable code
  • Easy to understand
  • Easy to extend – to add new configuration for a specific module is easy and does not require to update core code
  • Isolation

What else can be done?

In addition to this to make it easier, decorators can be added to the system which will auto-register your class to the builder. Same way decorators for other facilities can also be added like environment specific class loading. To do this you can either use some available auto-discovery plugin for typescript or implement a custom solution using metadata features of a decorator.

Do not reinvent wheel

Search for an available solution before implementing your own, there are some good solutions available online which can do similar stuff. The available solution at this point of time was not sufficient to cover all our application related structure and requirements which led to defining custom solution but for you, you might be able to find readily available solution in NPM repository.

Final note

Above code is just for illustration purpose and not production ready, you need to tweak it and add more validations/checks to make it more suitable for production.

Advertisements

Mongoose gotchas

I am using Mongoose since last few months and while querying MongoDB in Node.js and getting results. While doing that I came up with some cases that made me scratch my head for more than couple hours just to figure our it is little ‘gotcha’ (may be because I did not read through all the docs of Mongoose).

Listing my cases below so that if anyone ever comes to same mistake and tries to find for solution online, this can be helpful.

I have tried all following scenarios in 4.0.6 version of Mongoose with MongoDB 3.0.4.

1. When you query using a model and in result you see bunch of fields but when you try to access them you get undefined.

Ex.: (assuming you have user collections with name and email field in it.)

var UserSchema = new Schema({
 name: String
});

var User = mongoose.model('User', UserSchema);
User.find({}, function(err, users) {
 users.forEach(function(user) {
 console.log(user); // this will print something like {'name':'harsh', 'email': 'abc@abc.com'}
 console.log(user.email); // this will print 'undefined'
 });
});

Solution:
You need to add missing field entries in Schema definition, mongoose only binds fields that are in schema definition while creating objects.
Change above schema to following:


var UserSchema = new Schema({
 name: String,
 email: String
});

And now when you run same example above it should print email id of that particular record.

2. Auto conversion of field values in ‘find’ and ‘aggregate’ query.

Ex.:


var UserSchema = new Schema({
 name: String,
 cTime: Number
});

var User = mongoose.model('User', UserSchema);
User.find({
 cTime: '123456' // here we are passing value as string but it works
 // some how mongoose converts it before query may be
}, function(err, users) {
 console.log(users); // print users that match the cTime value
});

Now if you try same thing with aggregate function then it will not work:


User.aggregate({
 $match: { cTime : '1427807167' } // mongoose does not convert it to number this time
}, function(err, users) {
 console.log(users); // this will be empty, no match
});

In case of aggregate you have to take care of conversion by your own. Most likely case for this is getting time or other numeric values in req.query and then directly using it in $match query of aggregate. Instead you should first convert it to specific type of field and then use that in query.


User.aggregate({
 $match: { cTime : parseInt(req.query.time) } // assuming req.query.time has the value for cTime
}, function(err, users) {
 console.log(users); // now this will be not empty and prints the matching records.
});

3. Updating array items and calling save

When you just update an item in array of objects and call save on it, it will update the array in database. To make it work you need to mark the array field as modified.
user.markModified(‘array’); // this will mark array field of user object
if you call user.save after this then it will update array in database.

More info on this: http://stackoverflow.com/questions/17865081/updating-array-within-mongodb-record-with-mongoose

4. Listen to db connection drop event

This is probably not a gotcha, but it really made me work about an hour before knowing the solution. When connection to database is dropped for some reason then you will get it as ‘error’ event and not in ‘disconnected’ event. I tried to listed to ‘disconnected’ event and do reconnect but found that it does not even call it when db connection is crashed for some reason. It listens on ‘error’ event in that case so if you are planning to catch event for db connection failure and do operation based on it.

That’s all for now.

If you have seen any more while working with it, do let all know by commenting on the post. I will also update it with more as I come to know.

%d bloggers like this: