Dockerizing a NodeJS and MongoDB Application, monitored by DataDog

This post explains the setup of Node.js application connecting to MongoDB, monitored by DataDog with each running in its own container.

TL;DR: Refer Github.

Node.js

Our Node.js application is a simple app (using express framework), sample code below.

const express = require('express');
const useragent = require('express-useragent');

const app = express();
const port = process.env.PORT || 3000;

app.get('/', (req, res) => {
    res.send('Hello World!')
});

app.get('/dashboard', async (req, res) => {
    res.json({});
});

app.listen(port, () => {
  console.log(`Example app listening at http://localhost:${port}`)
})

This app starts a server and listens on port 3000 for connections. The app responds with “Hello World!” for requests to the root URL (/) and responds with JSON data (which we will fetch from MongoDB, explained later in this post) for requests to dashboard URL (/dashboard).

MongoDB

We will be using mongoose - Elegant MongoDB object modeling for Node.js - library for interacting with MongoDB from our Node.js app.

connect

The first thing we need to do is include mongoose in our index.js and open a connection to the demodb database running on docker (explained later in the post).

// snippet of index.js
const useragent = require('express-useragent');

const mongoose = require("mongoose");
mongoose
  .connect(
    'mongodb://mongo:27017/demodb', {
    useUnifiedTopology: true,
    useNewUrlParser: true
  })
  .then(() => console.log('MongoDB Connected'))
  .catch(err => console.log(err));

const app = express();

schema

We will record the user details like user IP address, date, and user-agent details (we are using express-agent module).

MongoDB schema:

{
  ipaddress: String,
  date: {
    type: Date,
    default: Date.now,
    expires: 7 * 24 * 60 * 60
  },
  useragent: {
    browser: String,
    version: String,
    os: String,
    platform: String,
    source: String,
  }
}

Including schema in our index.js

const Schema = mongoose.Schema;
const requestSchema = new Schema(_getSchema(), {
  collection: 'request-data'
});

requestSchema.index({
  // index on date
  date: -1 // descending order
});

const RequestModel = mongoose.model('Request', requestSchema);

function _getSchema() {
  return {
    ipaddress: String,
    date: {
      type: Date,
      default: Date.now,
      expires: 7 * 24 * 60 * 60
    },
    useragent: {
      browser: String,
      version: String,
      os: String,
      platform: String,
      source: String,
    }
  };
}

function _getRequestSchema(req) {
  return {
    ipaddress: req.ip || req.connection.remoteAddress,
    useragent: {
      browser: req.useragent.browser,
      version: req.useragent.version,
      os: req.useragent.os || 'unknown',
      platform: req.useragent.platform,
      source: req.useragent.source,
    },
  };
}

We will save user details, IP and user-agent, in MongoDB when users visit our root / route.

// save to mongodb
let requestModelObj = new RequestModel(_getRequestSchema(req));
requestModelObj.save(function (err, savedObj) {
  if (err) {
    console.error(err);
    return;
  }
});

fetch

Add the following code to route /dashboard to fetch the last 10 recorded details in MongoDB

app.get('/dashboard', async (req, res) => {
  // limited to 10 records
  const result = await RequestModel.find({}, null, { limit: 10 });
  res.json(result);
});

Visit route / from different browsers and when you visit /dashboard, you will get a similar response structure as shown below. _id is a unique ID generated by MongoDB and _v is the version key - used to track the revisions of a document.

[
  {
    "useragent": {
      "browser": "Firefox",
      "version": "80.0",
      "os": "Linux 64",
      "platform": "Linux",
      "source": "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:80.0) Gecko/20100101 Firefox/80.0"
    },
    "_id": "5f7885bbab824d002b27f1a4",
    "ipaddress": "::ffff:172.25.0.1",
    "date": "2020-10-03T14:07:55.900Z",
    "__v": 0
  },
  {
    "useragent": {
      "browser": "IE",
      "version": "8.0",
      "os": "Windows XP",
      "platform": "Microsoft Windows",
      "source": "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0)"
    },
    "_id": "5f7abae7ab824d002b27f1a6",
    "ipaddress": "::ffff:192.168.18.12",
    "date": "2020-10-05T06:19:19.032Z",
    "__v": 0
  }
]

Datadog

We will be using hot-shots module to send metrics to datadog.

The first thing we need to do is to include and setup the configuration in our index.js, the code snippet below.

const dogstatsd = new StatsD({
  host: process.env.DD_AGENT_HOST,
  globalTags: {
    env: process.env.NODE_ENV,
  },
  errorHandler: function (error) {
    console.error('Cannot connect to Datadog agent: ', error);
  }
});

To send metrics refer hot-shots module for methods corresponding to the type of metrics. We will send metrics, code below, to count the number of clients connected to the route.

// send metrics
dogstatsd.increment('client_connected');

Refer index.js for the complete code.

Docker

Dockerfile

We will use a very basic Dockerfile to set up a docker image for nodejs. We will include wait-for-it.sh - a pure bash script that will wait on the availability of a host and TCP port.

FROM node:13

COPY wait-for-it.sh /usr/wait-for-it.sh
RUN chmod +x /usr/wait-for-it.sh

WORKDIR /usr/src/app

To build image, demo_node:latest, run docker build -t demo_node:latest .

docker-compose.yml

We will setup three different services for each of datadog, nodejs, and mongodb and connect them over the same network - demo_network.

datadog

For an explanation of various environment options for datadog, refer https://docs.datadoghq.com/agent/docker/?tab=standards.

datadog:
  image: datadog/agent:latest
  environment:
    - DD_API_KEY=${DD_API_KEY}
    - DD_DOGSTATSD_NON_LOCAL_TRAFFIC=true
    - DD_AGENT_HOST=datadog
    - DD_HEALTH_PORT=5555
  networks:
    - demo_network
  volumes:
    - /var/run/docker.sock:/var/run/docker.sock
    - /proc/:/host/proc/:ro
    - /sys/fs/cgroup:/host/sys/fs/cgroup:ro
  deploy:
    restart_policy:
      condition: on-failure
        max_attempts: 3

mongodb

mongo:
  image: mongo
  ports:
    - "27017:27017"
  environment:
    - MONGO_DATA_DIR=/data/db
    - MONGO_INITDB_DATABASE=demodb
  networks:
    - demo_network
  volumes:
    - mongodb_data:/data/db
  deploy:
    restart_policy:
      condition: on-failure
      max_attempts: 3
  command: --quiet

nodejs

We will expose port 5555 for datadog service as health check port, DD_HEALTH_PORT=5555, and the port 27017for mongodb service. We will use wait-for-it.sh to wait for datadog - port 5555 - and mongodb - port 27017 - services to be up and running before we start nodejs application.

app:
  image: demo_node:latest
  volumes:
    - ./:/usr/src/app
  networks:
    - demo_network
  ports:
    - 3000:3000
  environment:
    - NODE_ENV=${NODE_ENV:-development}
    - PORT=3000
    - DD_AGENT_HOST=datadog
  depends_on:
    - mongo
    - datadog
  command:
    sh -c '/usr/wait-for-it.sh --timeout=0 datadog:5555 && /usr/wait-for-it.sh --timeout=0 mongo:27017 && npm i && node index.js'
  deploy:
    restart_policy:
      condition: on-failure

For code and installation visit the project on Github - https://github.com/raunakkathuria/learning/tree/master/docker/nodejs-monogodb-datadog.

References

• https://docs.datadoghq.com/agent/docker/?tab=standards
• https://mongoosejs.com
• https://expressjs.com/
• https://www.npmjs.com/package/express-useragent
• https://www.npmjs.com/package/hot-shots