Real Time Analytics with Spring XD

This article is about how to configure and run simple example using Spring XD (eXtreme Data).

Have a look at the latest presentation about it on the spring youtube channel.

Spring XD installation

  1. Download the latest version of spring XD from: http://projects.spring.io/spring-xd. I've been using ver. 1.0.0.M5.

  2. Unzip it to any folder you like e.g. /Users/kubam/development/spring-xd-1.0.0.M5

  3. Create symbolic link spring-xd pointing to your version
    ln -s /Users/kubam/development/spring-xd-1.0.0.M5/ spring-xd

  4. Add XD_HOME environment variable pointing to the installation.
    add export XD_HOME=/Users/marcinj1/development/spring-xd/xd in your .bash_profile file

  5. Start Spring XD server
    In case of OutOfMemoryError modify DEFAULT_JVM_OPTS to DEFAULT_JVM_OPTS="-Xmx512M" in the startup script xd-singlenode

  6. Check if the server is running at: http://localhost:9393/. It should respond with JSON with available streams, counters, modules etc.

Streams and metrics configuration

  1. Open Spring XD shell (prefarably in another bash session)

  2. Create stream used to acquire and process incoming data
    stream create --name cars --definition "http --port=8091 | file"
    This stream opens TCP port 8091 and listens for POST request. All acquired data are stored by default in the /tmp/xd/output/ folder. In this case /tmp/xd/output/cars.out. There are plenty of options available. Have a look at them in the documentation: http://docs.spring.io/spring-xd/.

  3. Create simple counter
    stream create --name carsCounter --definition "tap:stream:cars > counter"
    Try sending some data and check if counter is working:
    curl -d '{"brand": "ALFA", "model": "ALFA164"}' localhost:8091
    You can test it: http://localhost:9393/metrics/counters/carsCounter

  4. Let's now create a counters that will count brands and models separately
    stream create --name carsBrandCounter --definition "tap:stream:cars > field-value-counter --fieldName=brand"
    stream create --name carsModelsCounter --definition "tap:stream:cars > field-value-counter --fieldName=model"
    Send some more requests and check how they are working:
    curl -d '{"brand": "ALFA", "model": "ALFA164"}' localhost:8091
    curl -d '{"brand": "ALFA", "model": "ALFA8C"}' localhost:8091
    curl -d '{"brand": "AUDI", "model": "AUDI100"}' localhost:8091
    Counters should be available at: http://localhost:9393/metrics/field-value-counters/carsBrandCounter and http://localhost:9393/metrics/field-value-counters/carsModelsCounter

  5. Then create aggregare counter (a counter that stores timestamp with incoming data, so you may measure by last year, month, etc.)
    stream create --name carsAggregateCounter --definition "tap:stream:cars > aggregatecounter"
    You may test it here: http://localhost:9393/metrics/aggregate-counters/carsAggregateCounter

Real Time Analytics

To display the real time analytics I've used the sample dashboard from https://github.com/spring-projects/spring-xd-samples.

  1. Clone spring xd samples repository from github
    git clone https://github.com/spring-projects/spring-xd-samples.git

  2. Open analytics-dashboard project
    cd spring-xd-samples/analytics-dashboard

  3. Start nginx server
    If you don't have ngnix installed you have to download it first. I had some problems with that on a mac, but following commands solved it for me.
    sudo brew install nginx
    brew reinstall openssl
    brew link --force openssl
    sudo brew reinstall nginx

  4. Open dashboard: http://localhost:9889/dashboard.html
    You should see the data you have seen before.

  5. Post some more request and watch it change. Looks great :)
    I've created very simple project for testing purposes. Give it a try: https://bitbucket.org/jakubmarciniak/spring-xd-cars-stream

  6. Enjoy your real time analytics :) It should look like this:

Thanks to Zhou Yangbo for the feedback and pointing out the mistakes :)