Implement MongoDB replication in 3 simple steps

November 2, 2012 Leave a comment

After we find out how replication works with MySQL lets look at mongoDB

Use the following steps to implement mongoDB Replication:

1) Create the data directories
2) Create the replication set and instances
3) Configure, primary, secundaries and an arbiter

Donwload MongoDB? Goto the Download site

Step 1) Create the data directories

Start by creating a data directory for each replica set member, one for the primary and one for the secundary. We add also an arbiter. The arbiter does not relpicate data, but choose a new primary in case there is an outage of the existing primary.

mkdir /data/node1
mkdir /data/node2
mkdir /data/arbiter

Step 2) Create the replication set and instances

Next, start each member as a separate mongod. Since you’ll be running each process on the same machine, it’s probably easiest to start each mongod in a separate terminal window:

mongod --replSet person --dbpath /data/node1 --port 40001
mongod --replSet person --dbpath /data/node2 --port 40002
mongod --replSet person --dbpath /data/arbiter --port 40003

Step 3) Configure, primary, secundaries and an arbiter

Logon on the primary node to proceed, you need to configure the replica set, because if you examine the mongod log output, the first thing you’ll notice are error messages saying that the configuration can’t be found.

mongo localhost:40001
MongoDB shell version: 2.2.0
connecting to: localhost:40001/test
> rs.initiate()
"info2" : "no configuration explicitly specified -- making one",
"me" : "Computername.local:40001",
"info" : "Config now saved locally. Should come online in about a minute.",
"ok" : 1

Now connect again to the primary node, and add the secondary node including the arbiter node:

person:PRIMARY> rs.add(Computername:40002)
{ "ok" : 1 }
person:PRIMARY> rs.add(Computername:40003, {arbiterOnly:true})
{ "ok" : 1 }

Check if the configuration is ok, with rs.status():

person:PRIMARY> rs.status()
 "set" : "person",
 "date" : ISODate("2012-10-28T19:50:52Z"),
 "myState" : 1,
 "members" : [
 "_id" : 0,
 "name" : "Computername.local:40001",
 "health" : 1,
 "state" : 1,
 "stateStr" : "PRIMARY",
 "uptime" : 1266,
 "optime" : Timestamp(1351453811000, 1),
 "optimeDate" : ISODate("2012-10-28T19:50:11Z"),
 "self" : true
 "_id" : 1,
 "name" : "Computername.local:40002",
 "health" : 1,
 "state" : 2,
 "stateStr" : "SECONDARY",
 "uptime" : 41,
 "optime" : Timestamp(1351453811000, 1),
 "optimeDate" : ISODate("2012-10-28T19:50:11Z"),
 "lastHeartbeat" : ISODate("2012-10-28T19:50:51Z"),
 "pingMs" : 0
 "ok" : 1
 "_id" : 1,
 "name" : "Computername.local:40003",
 "health" : 1,
 "state" : 3,
 "stateStr" : "ARBITER",
 "uptime" : 14,
 "optime" : Timestamp(1351453811000, 1),
 "optimeDate" : ISODate("2012-10-28T19:50:11Z"),
 "lastHeartbeat" : ISODate("2012-10-28T19:50:51Z"),
 "pingMs" : 0
 "ok" : 1

And now its time to check if it works. We put a person in our primary database:

person:PRIMARY> use portraitGallery
switched to db portraitGallery
"name" : "Maikel",
"group" : [ "Oracle", "ExaData", "Big Data"],
} )

Logon on the secondary and check if the data is there, and don’t forget to enable reading with rs.slaveOk() or db.getMongo().setSlaveOk()

mongo localhost:40002
MongoDB shell version: 2.2.0
connecting to: localhost:40002/test
person:SECONDARY> rs.slaveOk()
person:SECONDARY> use portraitGallery
switched to db portraitGallery
person:SECONDARY> db.person.find()
{ "_id" : ObjectId("508d971dda0730903bcbb612"), "name" : "Maikel", "group" : [ "Oracle", "ExaData", "Big Data" ] }

Now we can test it with a filler script. Type in the primary something like:

person:PRIMARY> for(i=0; i<1000000; i++) {{person: i}); }

And in the secondary check if the collection is filled:

person:SECONDARY> db.person.find()
 { "_id" : ObjectId("508f95e9e38917f43ae20db3"), "person" : 0 }
 { "_id" : ObjectId("508f95e9e38917f43ae20db4"), "person" : 1 }
 { "_id" : ObjectId("508f95e9e38917f43ae20db5"), "person" : 2 }
 { "_id" : ObjectId("508f95e9e38917f43ae20db6"), "person" : 3 }
 { "_id" : ObjectId("508f95e9e38917f43ae20db7"), "person" : 4 }
 { "_id" : ObjectId("508f95e9e38917f43ae20db8"), "person" : 5 }
 { "_id" : ObjectId("508f95e9e38917f43ae20db9"), "person" : 6 }
 { "_id" : ObjectId("508f95e9e38917f43ae20dba"), "person" : 7 }
 { "_id" : ObjectId("508f95e9e38917f43ae20dbb"), "person" : 8 }
 { "_id" : ObjectId("508f95e9e38917f43ae20dbc"), "person" : 9 }
 { "_id" : ObjectId("508f95e9e38917f43ae20dbd"), "person" : 10 }
 { "_id" : ObjectId("508f95e9e38917f43ae20dbe"), "person" : 11 }
 { "_id" : ObjectId("508f95e9e38917f43ae20dbf"), "person" : 12 }
 { "_id" : ObjectId("508f95e9e38917f43ae20dc0"), "person" : 13 }
 { "_id" : ObjectId("508f95e9e38917f43ae20dc1"), "person" : 14 }
 { "_id" : ObjectId("508f95e9e38917f43ae20dc2"), "person" : 15 }
 { "_id" : ObjectId("508f95e9e38917f43ae20dc3"), "person" : 16 }
 { "_id" : ObjectId("508f95e9e38917f43ae20dc4"), "person" : 17 }
 { "_id" : ObjectId("508f95e9e38917f43ae20dc5"), "person" : 18 }
 { "_id" : ObjectId("508f95e9e38917f43ae20dc6"), "person" : 19 }
 Type "it" for more
 person:SECONDARY> db.person.count()
 person:SECONDARY> db.person.count()
 person:SECONDARY> db.person.count()
 person:SECONDARY> db.person.count()

Works, succes with mongoDB!!!

If you wan to do the mongoDB intro lab goto


Implement MySQL Replication in 3 simple steps

July 14, 2012 Leave a comment

For a project we need a second MySQL database for reporting, so whe can split the day to day business (forms) and the analytic reporting. Another thing is that the data must be very actual, so a nightly backup is not really an option.

Use the following steps to implement MySQL Replication:

1) Configure master and slave instances
2) Create Replication user on the master
3) Connect from the slave to the master

Step 1) Configure master- and slave instances

Every server need a unique server ID. We use for the master server-id=1 and for the slave server-id=2.

Binary logging must be enabled (log-bin=mysql-bin) on the master because the binary log is the basis for sending data changes from the master to its slaves. If binary logging is not enabled, replication will not be possible.

For the greatest possible durability and consistency in a replication setup using InnoDB with transactions, you should use innodb_flush_log_at_trx_commit=1 and sync_binlog=1.

Find the MySQL configuration files (on Ubuntu it is located in /etc/mysql/my.cnf). Add the following lines in both the master- and slave configuration files.




Restart the master- and slave instances.

/etc/init.d/mysql restart

Step 2) Create Replication user on the master

Now we need to create a Replication user on the master that is used for the connection from the slave.

Logon on you’re master with the root user:

mysql -u root -p

and type in the MySQL prompt:


Now we use the ‘show master status’ statement to determine the current binary log file name and position. Remember this carefully! In our example we get the following output.

mysql> show master status;
| File | Position | Binlog_Do_DB | Binlog_Ignore_DB |
| mysql-bin.000153 | 106 | | |
1 row in set (0.00 sec)

Step 3) Connect from the slave to the master

To set up the slave to communicate with the master for replication, you must tell the slave the necessary connection information.

First logon on you’re slave with the root user:

mysql -u root -p

and type in the MySQL prompt:

-> MASTER_HOST='ip-number of master',
-> MASTER_USER='repl',
-> MASTER_LOG_FILE='mysql-bin.000153',

And … the last command simple type:


… and you are running MySQL in with Replication! Try to make some changes in the master. For example make a new database and a table and fill it with some rows. Look in the slave and see that the changes are processed.

If there is existing data in the master then create a data dump with the ‘mysqldump’ command on the master. Once the data dump has been completed, you then import this data into the slave before starting the above Replication process.

More information MySQL Replication How To

Automate deployment from CloudBees / Jenkins to Amazon Beanstalk

October 10, 2011 4 comments

In my latest post about Amazon Beanstalk I explained how to deploy to Amazon Beanstalk in 3 simple steps. Of  course this a manual activity. We often deploy new versions of apps to Amazon Beanstalk, so it is a ‘must have’ to automate this proces.

Amazon delivers the AWS SDK for Java API to access Amazon Beanstalk. Lets investigate what we need.

We use the AWSCredentials interface from the com.amazonaws.auth package, to login with the AWS credentials.

To access the Amazon S3 we use the AmazonS3 interface and the AmazonS3Client class from the package.

We use the AWSElasticBeanstalk interface and the AWSElasticBeanstalkClient class from the package, to use the following methods:

  • deleteApplicationVersion – Deletes the specified version from the specified application.
  • createStorageLocation – Creates the Amazon S3 storage location for the account, to upload the WAR.
  • createApplicationVersion – Creates an application version for the specified application.
  • updateEnvironment – Updates the environment description, deploys a new application version, updates the configuration settings to an entirely new configuration template, or updates select configuration option values in the running environment.
From the package we use the following methods in combination with AWSElasticBeanstalkClient
  • DeleteApplicationVersionRequest – Deletes the specified version from the specified application.
  • S3Location – A specification of a location in Amazon S3.
  • CreateApplicationVersionRequest – Creates an application version for the specified application.
  • UpdateEnvironmentRequest – Updates the environment description, deploys a new application version, updates the configuration settings to an entirely new configuration template, or updates select configuration option values in the running environment.
  • DescribeEnvironmentsRequest – Returns descriptions for existing environments.
  • DescribeApplicationVersionsRequest – Returns descriptions for existing application versions.

Lets make a script using the Groovy language:

import com.amazonaws.auth.*

target(deployBeanstalk: 'Deploy to AWS Beanstalk') {
 // Check existing version and check if environment is production
 depends(checkExistingAppVersion, prodEnviroment, war)

 // Log on to AWS with your credentials
 def credentials = credentials
 AmazonS3 s3 = new AmazonS3Client(credentials)
 AWSElasticBeanstalk elasticBeanstalk = new AWSElasticBeanstalkClient(credentials)

 // Delete existing application
 if (applicationVersionAlreadyExists(elasticBeanstalk)) {
 println "Delete existing application version"
 def deleteRequest = new DeleteApplicationVersionRequest(applicationName:     applicationName,
 versionLabel: versionLabel, deleteSourceBundle: true)

 // Upload a WAR file to Amazon S3
 println "Uploading application to Amazon S3"
 def warFile = projectWarFilename
 String bucketName = elasticBeanstalk.createStorageLocation().getS3Bucket()
 String key = URLEncoder.encode(, 'UTF-8')
 def s3Result = s3.putObject(bucketName, key, warFile)
 println "Uploaded application $s3Result.versionId"

 // Register a new application version
 println "Create application version with uploaded application"
 def createApplicationRequest = new CreateApplicationVersionRequest(
 applicationName: applicationName, versionLabel: versionLabel,
 description: description,
 autoCreateApplication: true, sourceBundle: new S3Location(bucketName, key)
 def createApplicationVersionResult = elasticBeanstalk.createApplicationVersion(createApplicationRequest)
 println "Registered application version $createApplicationVersionResult"

 // Deploy the new version to an existing environment
 // If you don't have any AWS Elastic Beanstalk environments yet, you
 // can easily create one with with:
 // The AWS Management Console -
 // The AWS Toolkit for Eclipse -
 // The Elastic Beanstalk CLI -
 println "Update environment with uploaded application version"
 def updateEnviromentRequest = new UpdateEnvironmentRequest(environmentName:  environmentName, versionLabel: versionLabel)
 def updateEnviromentResult = elasticBeanstalk.updateEnvironment(updateEnviromentRequest)
 println "Updated environment $updateEnviromentResult"

From CloudBees / Jenkins we make a separate build job ‘Deployment_Amazon’ where we can easily put the Grails command line to execute the above script.

So we have seen in this post that we can easy setup a Build environment using CloudBees / Jenkins and Deploy automatically via the ‘AWS SDK for Java API’ to Amazon Beanstalk. Lets enjoy Develop, Build and Deploy in the Cloud!

*** Update ***

Hereby the complete script with the private functions. Just add this script right after the above script. Fill in own credentials(access- and secretkey, application- and environmentname. Succes.



private boolean applicationVersionIsDeployed(elasticBeanstalk) {
    def search = new DescribeEnvironmentsRequest(applicationName: applicationName, versionLabel: versionLabel)
    def result = elasticBeanstalk.describeEnvironments(search)

private boolean applicationVersionAlreadyExists(elasticBeanstalk) {
    def search = new DescribeApplicationVersionsRequest(applicationName: applicationName, versionLabels: [versionLabel])
    def result = elasticBeanstalk.describeApplicationVersions(search)

private AWSCredentials getCredentials() {
    def accessKey = '@@@@@@@@@@@@@@@'
    def secretKey = '@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@'
    def credentials = new BasicAWSCredentials(accessKey, secretKey)

private String getApplicationName() {

private String getEnvironmentName() {

private String getDescription() {
    applicationName + " via 'grails deploy-beanstalk' build on ${new Date().format('yyyy-MM-dd')}"

private String getVersionLabel() {
    def applicationVersion = metadata.getApplicationVersion()
    applicationVersion + '-production'

private File getProjectWarFilename() {
    new File(basedir, 'target/scs-' + versionLabel + '.war')

Setting up a Yahoo email account on Android Honeycomb 3.x

August 18, 2011 4 comments

Since I have an Android Honeycomb Asus Transformer the default email adres is from Google’s Gmail. The standard Yahoo mail App is not really nice and you want to integrate all the mail accounts into the mail system of Android Honeycomb. Adding a yahoo mail account is easy:

  1. Open email client and add new account
  2. E-mailadres: Enter yahoo email
  3. Password:Enter Yahoo password
  4. Select manual setup en enter IMAP
  5. Change IMAP-server in:
  6. Security type: SSL
  7. Port: 993
  8. Select next
  9. SMPT-server:
  10. Security type: None
  11. Port: 587
  12. Then select done or finished. That’s all!

Tested on my Asus Transformer with Android Honeycomb 3.2

Continuous deployment Grails Apps with Cloudbees Jenkins GitHub

May 16, 2011 Leave a comment

After deploying Grails Apps to Amazon Beanstalk, it’s now time to look at CloudBees. At the moment CloudBees offers the only Platform as a Service (PaaS) that spans the complete develop-to-deploy lifecycle of Java web applications in the cloud; without any servers, any virtual machines or any IT staff. See the Cloudbees website for more info. For a random Grails App named ‘KnowledgeMatch’ we follow some simple steps:

  • Sign up to CloudBees (RUN@Cloud is free and there is a free plan for DEV@Cloud) (this step is not described in this post)
  • Install and configure Jenkins plugins (DEV@Cloud)
  • Configure a new Jenkins job (DEV@Cloud)
  • We make a change to our source code repo in GitHub and see if it all works
From the Jenkins Dashboard go to Manage Jenkins > Manage Plugins and install the ‘Cloudbees Deployer Plugin’ and the ‘Hudson Grails plugin’ and restart Jenkins. Check if the plugins are installed.

Go to Manage Jenkins > Configure System. At the CloudBees area fill in your API- and Secret Key. Just click the question mark and copy and paste those fields.

Now its time to configure a new Jenkins job. Choose a JDK version. In the source code management area choose for the ‘Git’option.  Copy the Publickey and goto to you Github  > Account settings and make a new SSH Public Key and call it for example ‘CloudBees’.

At the build triggers area. Choose for ‘Poll SCM’ and fill in ‘* * * * * ‘ which mean every minute. You can change this to every poll schedule you need. See question mark for examples.

In the build area press the button ‘Add build step’ and choose ‘Build With Grails‘. Now fill in the grails version (in our case 1.3.6) and type in the target field your grails command: clean “war target/knowledgematch.war“. Also I checked the field ‘force upgrade’

And finnaly in the post-build action area. We fill in:

Archive the artifacts : target/knowledgematch.war

Email notification:

Cloudbees Deployment

  • Cloudbees Site : maikel
  • Application ID: maikel/knowledgematch
  • Filename Pattern: target/knowledgematch.war

In GitHub we change the domain class kandidaat.groovy. This domain class has a default controller wit scaffolding implemented.

class Kandidaat
 String Name
 String Adres
 Date BirthDate

 static constraints = {}

Now lets look at the DEV@Cloud / Jenkins Dashboard if the job is starting.

Yes, the job is start the building proces.

Now we look at RUN@Cloud if the WAR file is deployed.

Yes the WAR file is deployed. Lets look at if the Grails App is available.



CloudBees use the underlying Amazon platform. Tomcat is used as Java container and MySQL is used as database. There is direct access with local MySQL tools to restore data.

CloudBees RUN@Cloud is a serious alternative to Amazon Elastic Beanstalk and RDS. Cloudbees is cheaper and is easier to use. The combination DEV@Cloud and RUN@Cloud is a really good platform to automate the (Agile) software development proces.

Implement Business Rules using Grails (Part 1)

February 24, 2011 1 comment


Implementing business rules can be a tedious and error-prone task in application development. Business rules are often ambiguously defined and recorded. Since implementing business rules is usually a major task in each project, this can introduce a significant risk to the success of your Grails application.

We assume the following three main types of business rules:

  • constraint rules
  • change event rules
  • authorization rules

Constraint rules define a restriction to the state of the system or the change of the system state. Change event rules define automatic actions triggered by a change to the system state. The automatic action can either be a change in data (insert, update, delete) or an action outside the database such as sending e-mail or printing a report. Authorization rules define a restriction on the authorized use of the system. A very good blogpost to implement the Spring Security Plugin is described here:

Contraint rules

Constraint rules define a restriction to the state of the system or to allowed changes in the system state. Restrictions to the state of system are known as Invariants. Restrictions to the allowed changes to the system state are known as PreConditions. PreConditions can only be checked when actually performing the change to the data since they are checking whether the change itself is allowed. Invariants can be checked as you perform an action, but can also be checked against existing system objects.

The following subtypes of constraint rules can be identified:

  • Attribute
  • Instance
  • Entity
  • Inter Entity

Classifying the rules into the above subcategories is quite simple, since it is very easy to see if the rule involves:

  • Only one attribute in one entity object instance (Attribute)
  • Two or more attributes in the same instance (Instance)
  • More than one instance of the same entity object (Entity)
  • More than one instance across multiple entity objects (Inter Entity)

Essentially this subdivision is in increasing order of complexity. When estimating the time it will take to implement a given rule, take into account that Attribute rules are relatively easy to implement, instance rules are more difficult, and so forth.

Lets implement two examples in Grails using custom validators.

Attribute Rules
An Attribute Rule defines which values are allowed for an attribute.

Code example: Date must be in the future

date(validator: {it <= new Date()}) 

Instance Rules
Instance Rules depend on the value of two or more attributes within the same entity
object instance.

Code example: End Date must be after the start date

 static constraints = {
        startDate(nullable: true)
        endDate(nullable: true, validator: { endDate, instance ->
            if (endDate && endDate < instance.startDate) {
                return 'validation.period.endDate.greater.than.startDate'

This example appears often in several domain objects an of course is much better to install the constraints plugin, to put this constraint in it’s own class and re-use it in the domain objects. So we don’t have to repeat ourselves and prevent code duplication.

Next time we look at the more difficult constraints Entity- and Inter Entity constraints.

Deploy Grails Apps in 3 simple steps to Amazon Beanstalk

February 18, 2011 12 comments

We can easy deploy the Grails WAR file to Amazon Beanstalk in 3 simple steps:

  • We have to create an Amazon Web Services Account (this part is not described is this post)
  • We have to make a MySQL database. We use Amazon RDS
  • We have to make the Amazon Elastic Beanstalk and deploy the WAR

Important note before you start!

We need security groups, to connect local client tools and Amazon Elastic Beanstalk to the MySQL Database Instance.  To succeed to make this connection it is important to install Amazon RDS and Amazon Elastic Beanstalk in the same region. At this moment you can choose only select one region for setting up Amazon Elastic Beanstalk (US  East) so your region for Amazon RDS must be ‘US East’.

Make MySQL Database Instance with Amazon RDS

1) Sign in to the AWS Management Console and choose for the RDS tab

2) Press the Launch DB Instance Button

You see the first screen. In our case we fill in the following:

Allocated Storage: 5
DB Instance Identifier: db
Master User Name: root
Master User Password: mysql

The other fields will have the default value

3) In the second screen you can enter some additional configuration

Database name: database

The other fields will have the default value

4) In the other screen you can enter some management options like backup scedule etc.

5) After you review all the info and pressing the Launch button you get an available MySQL Database Instance!

Fill in the DB security group to authorize access from local client tools and elastic beanstalk

Enter in the default security group:

1) a CIDR/IP connection type with IP/32 of the machine with client tools. Fill in your IP, to haven direct access with your own client tools
2) a EC2 Security Group type with Security Group:elasticbeanstalk-default and enter your AWS Account ID

Make the elastic beanstalk and deploy the Grails WAR

1) We use the production evironment option in the datasource.groovy to put in the Amazon information. Use the endpoint of the Amazon RDS / MySQL Database Instance as the machinename and the Amazon Beanstalk environment in the jdbc connection.

production {
        dataSource {
            pooled = true
            driverClassName = "com.mysql.jdbc.Driver"
            dbCreate = "update" // one of 'create', 'create-drop','update'
            username = 'root'
            password = 'mysql'
            url = 'jdbc:mysql://'endpoint':3306/'environment''
            dialect = org.hibernate.dialect.MySQL5InnoDBDialect
            properties {
                validationQuery = "SELECT 1"
                testOnBorrow = true
                testOnReturn = true
                testWhileIdle = true
                timeBetweenEvictionRunsMillis = 1000 * 60 * 30
                numTestsPerEvictionRun = 3
                minEvictableIdleTimeMillis = 1000 * 60 * 30

2) Create the Grails WAR with the correct datasource.groovy file.

Enter an environment is our case : environment

Upload a Grails WAR file: appname.war

You can edit your environment. See the manual Elastic Beanstalk guide

After successfully upload en deploy of the Grails WAR file you see a little green square!

At this point you have successfully upload and deploy your Grails App to the Amazon Cloud!!!

Look at and enjoy!!!

%d bloggers like this: