How to Fetch RSS feeds into MongoDB with Groovy

Suppose we will fetch some Amazon AWS news into a MongoDB database. These few lines made it possible with the use of Groovy and the Gmongo module:

// To download GMongo on the fly and put it at classpath
@Grab(group='com.gmongo', module='gmongo', version='1.0')
import com.gmongo.GMongo

// Instantiate a com.gmongo.GMongo object instead of com.mongodb.Mongo
// The same constructors and methods are available here
def mongo = new GMongo("127.0.0.1", 27017)

// Get a db reference
def db = mongo.getDB("amazonnews")

// Give the url of the RSS feed
def url = "http://aws.amazon.com/rss/whats-new.rss"

// Parse the url with famous XML Slurper
def rss = new XmlSlurper().parse(url) 

// Write the title and link into the news collection
rss.channel.item.each {
    db.news.insert([title: "${it.title}", link: "${it.link}"])
}

Connect to the MongoDB database if there some documents:

> use amazonnews
switched to db amazonnews
> db.news.find({})
{ "_id" : ObjectId("519a2e980364e3901f41827d"), "title" : "Amazon Elastic Transcoder Announces Seven New Enhancements, Including HLS Support", "link" : "http://aws.amazon.com/about-aws/whats-new/2013/05/16/amazon-elastic-transcoder-announces-seven-new-features/" }
{ "_id" : ObjectId("519a2e980364e3901f41827e"), "title" : "Amazon DynamoDB Announces Parallel Scan and Lower-Cost Reads", "link" : "http://aws.amazon.com/about-aws/whats-new/2013/05/15/dynamodb-announces-parallel-scan-and-lower-cost-reads/" }
{ "_id" : ObjectId("519a2e990364e3901f41827f"), "title" : "AWS Management Console in AWS GovCloud (US) adds support for Amazon SWF", "link" : "http://aws.amazon.com/about-aws/whats-new/2013/05/14/aws-management-console-in-aws-govcloud-us-adds-support-for-amazon-swf/" }
{ "_id" : ObjectId("519a2e990364e3901f418280"), "title" : "AWS OpsWorks launches Amazon CloudWatch metrics view", "link" : "http://aws.amazon.com/about-aws/whats-new/2013/05/14/aws-opsworks-cloudwatch-view/" }
{ "_id" : ObjectId("519a2e990364e3901f418281"), "title" : "AWS OpsWorks supports Elastic Load Balancing", "link" : "http://aws.amazon.com/about-aws/whats-new/2013/05/14/aws-opsworks-supports-elb/" }
{ "_id" : ObjectId("519a2e990364e3901f418282"), "title" : "AWS Direct Connect location in Seattle and access to AWS GovCloud (US) now available", "link" : "http://aws.amazon.com/about-aws/whats-new/2013/05/08/aws-direct-connect-location-seattle-and-access-govcloud/" }
{ "_id" : ObjectId("519a2e990364e3901f418283"), "title" : "Announcing AWS Management Pack for Microsoft System Center ", "link" : "http://aws.amazon.com/about-aws/whats-new/2013/05/08/aws-management-pack-for-microsoft-system-center-2012/" }
{ "_id" : ObjectId("519a2e990364e3901f418284"), "title" : "Raising the bar: Amazon announces 4,000 IOPS per EBS Volume and Provisioned IOPS products on AWS Marketplace", "link" : "http://aws.amazon.com/about-aws/whats-new/2013/05/07/announcing-4000-iops-per-piops-volume-and-marketplace-support/" }
{ "_id" : ObjectId("519a2e990364e3901f418285"), "title" : "Announcing General Availability of the AWS SDK for Node.js", "link" : "http://aws.amazon.com/about-aws/whats-new/2013/05/06/announcing-general-availability-of-the-aws-sdk-for-node-js/" }
{ "_id" : ObjectId("519a2e990364e3901f418286"), "title" : "Amazon Elastic MapReduce (EMR) now supports S3 Server Side Encryption", "link" : "http://aws.amazon.com/about-aws/whats-new/2013/05/01/amazon-elastic-mapreduce-now-supports-S3-server-side-encryption/" }

Works!

More info:
Groovy http://groovy.codehaus.org/
Gmongo https://github.com/poiati/gmongo

Automate deployment from CloudBees / Jenkins to Amazon Beanstalk

In my latest post about Amazon Beanstalk I explained how to deploy to Amazon Beanstalk in 3 simple steps. Of  course this a manual activity. We often deploy new versions of apps to Amazon Beanstalk, so it is a ‘must have’ to automate this proces.

Amazon delivers the AWS SDK for Java API to access Amazon Beanstalk. Lets investigate what we need.

We use the AWSCredentials interface from the com.amazonaws.auth package, to login with the AWS credentials.

To access the Amazon S3 we use the AmazonS3 interface and the AmazonS3Client class from the com.amazonaws.services.s3 package.

We use the AWSElasticBeanstalk interface and the AWSElasticBeanstalkClient class from the com.amazonaws.services.elasticbeanstalk package, to use the following methods:

• deleteApplicationVersion – Deletes the specified version from the specified application.
• createStorageLocation – Creates the Amazon S3 storage location for the account, to upload the WAR.
• createApplicationVersion – Creates an application version for the specified application.
• updateEnvironment – Updates the environment description, deploys a new application version, updates the configuration settings to an entirely new configuration template, or updates select configuration option values in the running environment.

From the com.amazonaws.services.elasticbeanstalk.model package we use the following methods in combination with AWSElasticBeanstalkClient

• DeleteApplicationVersionRequest – Deletes the specified version from the specified application.
• S3Location – A specification of a location in Amazon S3.
• CreateApplicationVersionRequest – Creates an application version for the specified application.
• UpdateEnvironmentRequest – Updates the environment description, deploys a new application version, updates the configuration settings to an entirely new configuration template, or updates select configuration option values in the running environment.
• DescribeEnvironmentsRequest – Returns descriptions for existing environments.
• DescribeApplicationVersionsRequest – Returns descriptions for existing application versions.

Lets make a script using the Groovy language:

import com.amazonaws.auth.*
import com.amazonaws.services.s3.*
import com.amazonaws.services.elasticbeanstalk.*
import com.amazonaws.services.elasticbeanstalk.model.*

target(deployBeanstalk: 'Deploy to AWS Beanstalk') {
 // Check existing version and check if environment is production
 depends(checkExistingAppVersion, prodEnviroment, war)

 // Log on to AWS with your credentials
 def credentials = credentials
 AmazonS3 s3 = new AmazonS3Client(credentials)
 AWSElasticBeanstalk elasticBeanstalk = new AWSElasticBeanstalkClient(credentials)

 // Delete existing application
 if (applicationVersionAlreadyExists(elasticBeanstalk)) {
 println "Delete existing application version"
 def deleteRequest = new DeleteApplicationVersionRequest(applicationName:     applicationName,
 versionLabel: versionLabel, deleteSourceBundle: true)
 elasticBeanstalk.deleteApplicationVersion(deleteRequest)
 }

 // Upload a WAR file to Amazon S3
 println "Uploading application to Amazon S3"
 def warFile = projectWarFilename
 String bucketName = elasticBeanstalk.createStorageLocation().getS3Bucket()
 String key = URLEncoder.encode(warFile.name, 'UTF-8')
 def s3Result = s3.putObject(bucketName, key, warFile)
 println "Uploaded application $s3Result.versionId"

 // Register a new application version
 println "Create application version with uploaded application"
 def createApplicationRequest = new CreateApplicationVersionRequest(
 applicationName: applicationName, versionLabel: versionLabel,
 description: description,
 autoCreateApplication: true, sourceBundle: new S3Location(bucketName, key)
 )
 def createApplicationVersionResult = elasticBeanstalk.createApplicationVersion(createApplicationRequest)
 println "Registered application version $createApplicationVersionResult"

 // Deploy the new version to an existing environment
 // If you don't have any AWS Elastic Beanstalk environments yet, you
 // can easily create one with with:
 // The AWS Management Console - http://console.aws.amazon.com/elasticbeanstalk
 // The AWS Toolkit for Eclipse - http://aws.amazon.com/eclipse
 // The Elastic Beanstalk CLI - http://aws.amazon.com/elasticbeanstalk
 println "Update environment with uploaded application version"
 def updateEnviromentRequest = new UpdateEnvironmentRequest(environmentName:  environmentName, versionLabel: versionLabel)
 def updateEnviromentResult = elasticBeanstalk.updateEnvironment(updateEnviromentRequest)
 println "Updated environment $updateEnviromentResult"
}

From CloudBees / Jenkins we make a separate build job ‘Deployment_Amazon’ where we can easily put the Grails command line to execute the above script.

So we have seen in this post that we can easy setup a Build environment using CloudBees / Jenkins and Deploy automatically via the ‘AWS SDK for Java API’ to Amazon Beanstalk. Lets enjoy Develop, Build and Deploy in the Cloud!

*** Update ***

Hereby the complete script with the private functions. Just add this script right after the above script. Fill in own credentials(access- and secretkey, application- and environmentname. Succes.

 

setDefaultTarget(deployBeanstalk)

private boolean applicationVersionIsDeployed(elasticBeanstalk) {
    def search = new DescribeEnvironmentsRequest(applicationName: applicationName, versionLabel: versionLabel)
    def result = elasticBeanstalk.describeEnvironments(search)
    !result.environments.empty
}

private boolean applicationVersionAlreadyExists(elasticBeanstalk) {
    def search = new DescribeApplicationVersionsRequest(applicationName: applicationName, versionLabels: [versionLabel])
    def result = elasticBeanstalk.describeApplicationVersions(search)
    !result.applicationVersions.empty
}

private AWSCredentials getCredentials() {
    def accessKey = '@@@@@@@@@@@@@@@'
    def secretKey = '@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@'
    def credentials = new BasicAWSCredentials(accessKey, secretKey)
    credentials
}

private String getApplicationName() {
    '@@@@@@@@'
}

private String getEnvironmentName() {
    '@@@@@@@@'
}

private String getDescription() {
    applicationName + " via 'grails deploy-beanstalk' build on ${new Date().format('yyyy-MM-dd')}"
}

private String getVersionLabel() {
    def applicationVersion = metadata.getApplicationVersion()
    applicationVersion + '-production'
}

private File getProjectWarFilename() {
    new File(basedir, 'target/scs-' + versionLabel + '.war')
}

Implement Business Rules using Grails (Part 1)

General

Implementing business rules can be a tedious and error-prone task in application development. Business rules are often ambiguously defined and recorded. Since implementing business rules is usually a major task in each project, this can introduce a significant risk to the success of your Grails application.

We assume the following three main types of business rules:

• constraint rules
• change event rules
• authorization rules

Constraint rules define a restriction to the state of the system or the change of the system state. Change event rules define automatic actions triggered by a change to the system state. The automatic action can either be a change in data (insert, update, delete) or an action outside the database such as sending e-mail or printing a report. Authorization rules define a restriction on the authorized use of the system. A very good blogpost to implement the Spring Security Plugin is described here: http://blog.springsource.com/2010/08/11/simplified-spring-security-with-grails/

Contraint rules

Constraint rules define a restriction to the state of the system or to allowed changes in the system state. Restrictions to the state of system are known as Invariants. Restrictions to the allowed changes to the system state are known as PreConditions. PreConditions can only be checked when actually performing the change to the data since they are checking whether the change itself is allowed. Invariants can be checked as you perform an action, but can also be checked against existing system objects.

The following subtypes of constraint rules can be identified:

• Attribute
• Instance
• Entity
• Inter Entity

Classifying the rules into the above subcategories is quite simple, since it is very easy to see if the rule involves:

• Only one attribute in one entity object instance (Attribute)
• Two or more attributes in the same instance (Instance)
• More than one instance of the same entity object (Entity)
• More than one instance across multiple entity objects (Inter Entity)

Essentially this subdivision is in increasing order of complexity. When estimating the time it will take to implement a given rule, take into account that Attribute rules are relatively easy to implement, instance rules are more difficult, and so forth.

Lets implement two examples in Grails using custom validators.

Attribute Rules
An Attribute Rule defines which values are allowed for an attribute.

Code example: Date must be in the future

date(validator: {it <= new Date()}) 

Instance Rules
Instance Rules depend on the value of two or more attributes within the same entity
object instance.

Code example: End Date must be after the start date

 static constraints = {
        startDate(nullable: true)
        endDate(nullable: true, validator: { endDate, instance ->
            if (endDate && endDate < instance.startDate) {
                return 'validation.period.endDate.greater.than.startDate'
            }
        })
}

This example appears often in several domain objects an of course is much better to install the constraints plugin, to put this constraint in it’s own class and re-use it in the domain objects. So we don’t have to repeat ourselves and prevent code duplication.

Next time we look at the more difficult constraints Entity- and Inter Entity constraints.