In this article, I'll illustrate 5 steps that you can use to start shifting security left, into the build phase of the application lifecycle, in a DevSecOps approach. Within, I focus on using GitHub actions as my choice of CI/CD platform, however these concepts can be lifted and applied to almost any other platform of your choice.

📝 Note: The aim of this article is to be a overview of 5 security practices you can incorporate into your CI/CD process, specifically when building your application. Every application, as well as runtime environment, is different, so this is not a guaranteed guide to security. This article serves as a starting point, where you can then carry out further research. I am not a security professional, however these are all steps I have used in either a professional or personal capacity.

🔐 Scanning for Secrets

When developing applications, there is often an abundance of credentials and secrets that are required within the project. These can range anywhere from AWS credentials for accessing their services (think DynamoDB or Kinesis streams), to API keys for third party API usage and even encryption secrets for entities such as JWT, Cookie and password encryption mechanisms. Due to the sensitive and potentially destructive nature (in the case of a malicious party gaining access to hosting environments/databases) of these secrets, they should not be committed into Git history. Files containing these secrets should be added to a repository .gitignore file to prevent this from happening.

Sometimes, however, these secrets manage to slip into code repositories. As this is a matter of when, not if, it is best to plan ahead and detect these as they are added to the repository, alerting for security visibility and tracking.

The following example makes use of gitLeaks, however other options include Whispers and Yelp's Detect-Secrets:

name: gitleaks

on: [ push , pull_request ]

jobs:
  gitleaks:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v2
      with:
        fetch-depth: '2'

    - name: gitleaks
      uses: zricethezav/gitleaks-action@master

Should any secrets make it into the source code (or pre-existing secrets are discovered), the first step that should be carried out is to immediately disable the leaked key. This will prevent any malicious actor from compromising the associated account/service, giving you time to rotate the key and to remove the credential from the repository.

📝 Note: It is also worth noting that secret scanning can also be performed before commiting to a repository, using frameworks such as pre-commit or husky, which helps shift security even further left and would be a preferred solution. However everyone's development environment is different and as such this is not always an option, but it is worth digging into.

📄 Linting Code

Linting code is the process of statically scanning code for errors such as styling or syntactical issues, to better organize and secure your application. By ensuring code follows secure standards and is formatted in a sensible fashion, repositories become easier to maintain and to detect errors (such as memory leaks or other vulnerabilities) which could compromise your application. Having a standardized styling pattern can bring other benefits too, such as a better understanding/collaboration process on the codebase and an easier debugging experience. This is a good, quick first scan that can be run to help increase security and efficiency within your pipeline, however it is not always mandatory.

In this snippet, Hadolint, a Dockerfile linter, is used to ensure standards are maintained. One such standard that Hadolint searches for, is the versioning of applications installed within containers from package managers. This is an important, lesser known tip that helps to keep maintainability, preventing Docker builds from breaking between machines because of an unknown package version upgrade. For alternatives, check out the documentation for GitHub's Super-Linter, which houses linters for various programming languages.

name: Hadolint

on: [ push, pull_request ]

jobs:
  hadolint:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2 

      - name: hadolint 
        uses: hadolint/hadolint-action@v1.5.0
        with:
          dockerfile: Dockerfile

😒 Scanning for Known Vulnerable Components

Scanning for known vulnerable components, aka insecure dependencies, is a big step in reducing the potential attack vector within your application. With some applications requiring hundreds, if not thousands of dependencies (once factoring the entire dependency tree), a compromise within one could leave your application wide open to attack. The good thing, however, is that there are several tools that can be used to identify any insecure dependencies within your application, with some even generating the fixes required for you.

Dependabot is a tool directly integrated into GitHub code repositories, allowing for a configurable dependency scanning experience, with scans being able to run for multiple different package managers across multiple languages, all in the same repository. Upon detecting of a vulnerable dependency, Dependabot will then automatically open a pull request against the codebase, bumping the dependency to a fixed/secure version, without the need for a developer to carry out the upgrade.

# Basic set up for two package managers

version: 2
updates:

  - package-ecosystem: "github-actions"
    directory: "/"
    schedule:
      interval: "weekly"
    labels:
      - "actions"
      - "dependencies"

  - package-ecosystem: "npm"
    directory: "/" 
    schedule:
      interval: "daily"
    labels:
      - "npm"
      - "dependencies"
    open-pull-requests-limit: 10

If, instead, you would prefer a tool that you can use locally, as well as between different CI/CD providers (e.g. if in a larger, segmented organization), then I would recommend checking out Dependency Check, as it is backed by the Open Web Application Security Project (OWASP).

🔎 Run SAST Scans

Static Application Security Testing, or SAST, is similar running a code linter. It scans your codebase, looking for known or custom patterns and reporting on the findings. However, where SAST differs is that instead of looking for potential styling or syntax errors, scans are run to detect insecure code. These scans commonly use pre-defined rules from industry professionals to run, with some options going the extra mile to provide links to resources explaining the risk of the issues detected. By nature, SAST scans are not context aware, so if you have addressed the security concerns to do with detected snippets elsewhere, the scan will still report a false-positive result. However, these false-positives can be an eye opener into the architectural decisions as to why these vulnerable code-snippets are in the code base, regardless of the security measures put in place. After all, even after adding layers of security, should you really be calling dangerouslySetInnerHTML? 🤔

One such SAST tool is Semgrep. Semgrep allows you to hand-pick the rules to run on the code, or alternatively to use a pre-defined collection of rules (rulesets) tailored to specific languages and/or security topics. With over 1400 rules, at the time of writing, these can be explored here.

name: Semgrep

on:
  # Scan changed files in PRs
  [ pull_request ] 
  # Scan all files on branch
  push:
    branches: 
      - "main"

jobs:
  semgrep:
    runs-on: ubuntu-latest
    # Skip any PR created by dependabot to avoid permission issues
    if: (github.actor != 'dependabot[bot]')
    steps:
      - uses: actions/checkout@v2

      - name: semgrep
        uses: returntocorp/semgrep-action@v1
        with:
          auditOn: push # Never fail the build due to findings on push
          config: >- 
            p/security-audit
            p/nodejsscan
            p/expressjs            
        env:
          SEMGREP_TIMEOUT: 300

🤔 Collect Information and Act on it

Finally, after performing any of these steps the most important thing to do is to act on the findings. This is crucial to ensuring the security of applications, as without acting on disclosures the whole process becomes a zero sum equation. If you find a leaked secret, or an insecure dependency, fixing it could be the difference between the application carrying on as normal and data loss/breach of GDPR - which could incur eye-watering fines if this occurs with Personally Identifiable Information (PII).

📎 TL;DR

In summary, some of the key ways of introducing security (along with stability and visibility) into CI/CD can be achieved by introducing the following to your procedures and pipelines:

• Scan for hard-coded secrets
• Lint for inconsistent syntax and code styling
• Search out insecure dependencies to reduce your surface area of attack
• Test for insecure code within your repository with SAST tools
• Act on your findings

There is always more that can be done to improve security (such as environment scanning, DAST scanning, etc), however by introducing some basics, you are setting off in a good direction.

🛠 Setting up the Test Environment

📝 Note: The aim of this article is to be a quick demo of mocking the @aws-sdk, not a Jest integration overview. As such, I will not be going into the internals of Jest. You can do that here.

Before starting, make sure that you have Jest and any AWS modules that you plan to test with, installed. In this article, I will be using the DynamoDB client, however this works with any clients that make use of the send() function for their operations.

npm install jest @aws-sdk/client-dynamodb

As standard, add the following to your package.json file

scripts: {
    # Rest of your scripts go here...
    "test": "jest",
}

As a demo, I will setup a file which I wish to test: dynamoDBController.js. For simplicity, this file will only contain one method: PutItem(). The base code for this is as follows:

// dynamoDBController.js

const { 
  DynamoDBClient, 
  PutItemCommand 
} = require("@aws-sdk/client-dynamodb");

const dynamoDBClient = new DynamoDBClient({ region: "local" });

const PutItem = async (params) => {
  try {
    return await dynamoDBClient.send(new PutItemCommand(params));
  } catch (error) {
    return { statusCode: 500 };
  }
};

module.exports = { PutItem };

🛠 Enabling Mocking of AWS Modules

Once setup, one minor refactor to the existing code unlocks the ease of mocking. By refactoring the client initialization into a separate file, it becomes easy to mock the AWS module. In the current example, I will add a file aws.js to setup the DynamoDB client:

// aws.js

const { DynamoDBClient } = require("@aws-sdk/client-dynamodb");

const dynamoDBClient = new DynamoDBClient({
  region: "local",
});

module.exports = { dynamoDBClient };

And the minor adjustment to dynamoDBController.js:

// updated dynamoDBController.js

const { PutItemCommand } = require("@aws-sdk/client-dynamodb");
const { dynamoDBClient } = require("./aws");

const PutItem = async (params) => {
  try {
    return await dynamoDBClient.send(new PutItemCommand(params));
  } catch (error) {
    return { statusCode: 500 };
  }
};

module.exports = { PutItem };

This small change separates the external dependency initialization from the actual usage of the module, breaking up this operation into separate units, making it easier to mock and test.

🛠 Mocking and Testing AWS Modules

Now that the DynamoDB module has been setup for mocking, it's time to create the test file. In a new dynamoDBController.test.js file, import both the local method to test as well as the initialized dynamoDBClient:

// dynamoDBController.test.js

const { PutItem } = require("./dynamoDBController");
const { dynamoDBClient } = require("./aws");

Using the Jest mock function, we can add the following line to mock DynamoDB module:

jest.mock("./aws.js");

Now we can move onto writing the actual tests. As we are mocking the initialized client, we have access to the exposed client send() function, allowing us to intercept the fundamental DynamoDB call used within the PutItem() function. Within our test, we can intercept this call and return our own test value, by using the following command:

dynamoDBClient.send.mockResolvedValue({ isMock: true });

We can now write the rest of the test and call the PutItem() command as normal. The rest of the test is as follows:

describe("@aws-sdk/client-dynamodb mock", () => {
  it("should successfully mock dynamoDBClient", async () => {
    dynamoDBClient.send.mockResolvedValue({ isMock: true });

    const params = {
      TableName: "CUSTOMER",
      Item: {
        CUSTOMER_ID: { N: "001" },
        CUSTOMER_NAME: { S: "Richard Roe" },
      },
    };

    const response = await PutItem(params);

    expect(response.isMock).toEqual(true);
  });
});

Running npm test now will result in a successful test run.

> jest

 PASS  ./dynamoDBController.test.js
  @aws-sdk/client-dynamodb mock
    √ should successfully mock dynamoDBClient (2 ms)

Test Suites: 1 passed, 1 total
Tests:       1 passed, 1 total
Snapshots:   0 total
Time:        2.307 s, estimated 3 s
Ran all test suites.

As the PutItem() function also accounts for and thrown errors, it is also possible to fully test this by using the mockRejectedValue() function. A test demonstrating this is documented below:

  it("should throw error and return 500 status code", async () => {
    dynamoDBClient.send.mockRejectedValue(new Error("Async error"));

    const params = {
      TableName: "CUSTOMER",
      Item: {
        CUSTOMER_ID: { N: "001" },
        CUSTOMER_NAME: { S: "Richard Roe" },
      },
    };

    const response = await PutItem(params);

    expect(response.statusCode).toEqual(500);
  });

Running npm test now will yield two successful tests:

> jest

 PASS  ./dynamoDBController.test.js
  @aws-sdk/client-dynamodb mock
    √ should successfully mock dynamoDBClient (2 ms)
    √ should throw error and return 500 status code (2 ms)

Test Suites: 1 passed, 1 total
Tests:       2 passed, 2 total
Snapshots:   0 total
Time:        3.529 s
Ran all test suites.

🛠 Preventing Race Conditions and Mock Data Leakage in your Tests

If your tests are sending multiple call to the mocked AWS modules, it is good practice to add a Jest call to the beforeEach() so that any mocked return values can be reset before each test. This lets you dynamically code different responses for each individual test, as well as preventing test data leakage and race conditions. To do this, simply append the following snippet before the start of your test blocks:

beforeEach(() => {
  dynamoDBClient.send.mockReset();
});

🤔 Conclusion

The V3 update to the AWS JavaScript SDK is great. It reduces bundle sizes and increases modularity. Upgrading to it is great for developers and packages. Refactoring existing tests for methods moving this new major version does not have to be hard, especially when you embrace the modular first approach of the new SDK.

Docker. It's everywhere. From hosting simple applications to running the micro-services required by your favorite streaming platforms. This magical technology unfortunately did not make the cut for the University course I attended. As such, it has been my hobby to containerize and manage small applications in the effort to help me learn more about this technology. I have personally run into each of following Docker security conundrums and have learnt by developing and experimenting, as most junior developers should.

Problem #1 - Vulnerabilities

After developing my first full-stack app with Node.js, I naturally decided that I would bundle and containerize the application to allow for better portability and distribution. So the first thing I wrote, much like many other developers, is the conventional FROM node.

Did you see the problem? No? Neither did I at first, but the people at Snyk did. By using the node base image, I had inadvertently inherited a large number of pre-existing vulnerabilities found within the base image.

Solution ✔

By changing to a more secure base image, such as ubuntu, instantly the number or vulnerabilities drop dramatically. Taking the node image for example, by simply building Node from source within a Ubuntu 20.04 base, the number of vulnerabilities shoot down from nearly 600, to a small amount of only 23 (as report by Snyk).

This can be achieved simply, using the following as a baseline for your Dockerfile:

FROM ubuntu:latest

# Install Node.js and dependencies
RUN apt-get update -yq \
    && apt-get install curl gnupg -yq \
    && curl -sL https://deb.nodesource.com/setup_14.x | bash \
    && apt-get install nodejs -yq 

# Copy and Setup App
WORKDIR /app
COPY . /app
RUN npm install

# Expose Port and Run 
EXPOSE 5000
CMD ["npm", "start"]

Problem #2 - Execution Within A Container

The next problem that arises is the use of the Root user within a Docker container. By default, if a user is not specified, commands executed within a container (as well as running applications within the container) are run as root. This opens up vulnerabilities that could potentially give backdoor root access into the host system 🙅‍♂️. You don't need to be a security expert to see how bad this can be.

Solution ✔

By utilizing the USER directive, you can easily create and swap to a less-privileged user to execute the application. It is best to do this near the end of the Dockerfile, allowing for the application to be set up correctly (without any file permission errors). Using the Dockerfile described previously, this would look like the following:

...

# Copy and Setup App
WORKDIR /app
COPY . /app
RUN npm install

# Expose Port 
EXPOSE 5000

# Create and Change to User 'app' and Run
USER app
CMD ["npm", "start"]

Problem #3 - Saving Files to Host

So you've used a more secure base image and swapped to a less-privileged user, but you want to write data to the host to persist across the container life-cycle, such as configuration data? No problem, you just specify a Bind Mount or Volume by using the docker run -v command. Simple.

But wait. We've swapped to a less-privileged user that doesn't have write permission to the host machine. How do we perform this simple operation now, without risking the security of the application? This was a question that I was stuck on for a good deal of time when containerizing my application. Until I found a helpful answer on Stack Overflow.

Solution ✔

This answer, written by Dimitris, kindly explains that the docker container should fix any file permissions for mounted directories, before running the application, and points towards the implementation of the Reddis Docker container. From instecting this repository (along with some further reading), it became clear the power of the ENTRYPOINT command. By using the entrypoint of the Dockerfile to run a script, instead of running the application, the container can be started with directories mounted (make note, this is key), with operations being performed immediately before application start.

Say you have a configuration folder within the container at /app/config, the Dockerfile to achieve this, based on the previous ubuntu image generated, is as follows:

FROM ubuntu:latest

# Add User/Group 'app'
RUN groupadd -r app \
    && useradd -r -s /bin/false -g app app 

# Install Node.js and dependencies
RUN apt-get update -yq \
    && apt-get install curl gnupg gosu -yq \
    && curl -sL https://deb.nodesource.com/setup_14.x | bash \
    && apt-get install nodejs -yq \
    && chown -R app:app /app

# Copy and Setup App
WORKDIR /app
VOLUME /app/config
COPY . /app
RUN npm install \
    && chmod +x docker-entrypoint.sh

ENTRYPOINT ["./docker-entrypoint.sh"]

EXPOSE 5000
CMD ["npm", "start"]

To note, the new internal user must be created at the beginning of the file, using the base image OS's way of creating a user, to ensure permissions are handled successfully. Also, gosu is installed to handle the step-down to the less-privileged user. The contents of the docker-entrypoint.sh are found below:

#!/bin/bash

# Exit if a process exits with an error code
set -e

# if 'CMD' Directive in Dockerfile begins with 'npm'
if [ "$1" = 'npm' ]; then

    # Make Owner of Configuration Folder/Files Newly Created User
    chown -R app:app /app/config

    # Optional
    echo "Finished Fixing Permissions"

    # Change User to 'app' and Run 'CMD' in Dockerfile
    exec gosu app "$@"

fi

# Change Process to PID 1 for Monitoring
exec "$@"

Breaking down the file above, the following functionality is achieved:

• Checks if the CMD directive within the Dockerfile begins with 'npm'. This allows for the potential of different flows for different environments, e.g. Development or Production
• Changes the now mounted configuration folders' permissions to the 'app' user we created earlier, enabling write permissions to the mounted host folder (remember, the folder was mounted within the Dockerfile. This script executes when the container has just started, not on build or extraction)
• Uses gosu to change user to that of 'app' and execute the Dockerfiles' CMD directive as this user (hence, the application is run as 'app')

By using the docker-entrypoint.sh script, the mounted configuration folder has it's ownership changed to the created user, with the application run as said user. This allows the application to write files to the configuration folder to the host machine, without any permissions errors. Allowing for the storage of vital data, without compromising the security of the host/Docker container.

The Takeaway

There are may ways in which one can improve the security of Docker containers, for the benefit of both the host and the container itself. These are only the initial steps anyone should take to help lock-down their container, with a main point to ensure security being enforcing correct security guidelines when developing the application itself. The container could be the Fort Knox of containers, yet with application code not being developed securely, security risks are still a very big possibility.