3 Musketeers

# Environment variables

Development following the twelve-factor app use the environment variables to configure their application.

Often there are many environment variables and having them in a .env file becomes handy. Docker and Compose do use environment variables file to pass the variables to the containers.

# Files env.template and env.example

env.template and env.example files provide some help when managing environment variables in a project.

WARNING

As env.template and env.example files are meant to be part of the source code, never include sensitive values like passwords.

# env.template

env.template contains names of all environment variables the application and pipeline use. No values are set here. # description can be used to describe an environment variable. env.template is mainly used as a template to .env in a CI/CD pipeline.

# env.template
ENV_VAR_A
ENV_VAR_B

# env.example

env.example defines values so that it can be used straight away with Make like $ make envfile test ENVFILE=env.example. It also gives an example of values that are being used in the project which is very useful for the developers.

# env.example
ENV_VAR_A=a
ENV_VAR_B=b

# Pros

  • Simple
    • Understanding the concept is pretty straight forward
    • Does not require any script
  • Application agnostic
    • This pattern can be used for any environment variable of any kind of application
  • Descriptive and explicit
    • env.template tells what environment variables are used by the project
    • env.example shows what value those environment variables can have
    • Environment variables needs to explicitly be added
  • Flexible
    • The way the environment variables are set is up to you. They can be included in the .env file when developing locally or exported in a CD/CI host

# Cons

  • Environment variable management is not centralized
    • Adding, modifying, or deleting environment variables may impact multiple files such as
      • env.template
      • env.example
      • Makefile
      • docker-compose.yml
      • application source code
      • pipeline-as-code file
  • Error prone
    • It is easy to forget to add a new environment variable to the env.template/env.example files

# CI/CD pipeline

Given all environment variables are set in your CI/CD pipeline, creating a .env file based on env.template allows values of those environment variables to be passed to the Docker container environments. See this tutorial to better understand the use of .env file with Docker and Compose.

# Day-to-day development

In a day-to-day development process, you could create a file named .env.dev with the config of your dev environment and copy the contents of it into .env so that you can manually deploy/delete/etc your app for testing. This allows you to not accidentally lose the values if the .env file is replaced. There are few ways to copy the contents of your file to .env:

# Understanding .env file with Docker and Compose

This tutorial shows how .env file works with Docker and Docker Compose.

# Files

Create the following files:

# file: env.template
ECHO_MESSAGE

# file: env.example
ECHO_MESSAGE="Hello, 3 Musketeers!"

# file: docker-compose.yml
version: '3'
services:
  musketeers:
    image: flemay/musketeers
    env_file: .env
    volumes:
      - .:/opt/app
    working_dir: /opt/app

# file: Makefile
COMPOSE_RUN_MUSKETEERS = docker-compose run --rm musketeers

# ENVFILE is env.template by default but can be overwritten
ENVFILE ?= env.template

# envfile creates or overwrites .env with $(ENVFILE)
envfile:
	cp -f $(ENVFILE) .env

# shell creates a shell environment inside a container.
shell:
	$(COMPOSE_RUN_MUSKETEERS) sh -l

# cleanDocker remove all containers/network created by Compose
cleanDocker:
	docker-compose down --remove-orphans

# clean cleans the current directory
clean: cleanDocker
	rm -f .env

# Steps

# Let's clean first
$ make clean

# Let's go inside a container.
$ make shell
# Opps! Error. Compose needs a .env file. Let's create one...
$ make envfile
$ make shell
# See the new .env file?
$ ls -la
$ cat .env
# Let's look at the environment variable ECHO_MESSAGE
$ env | grep ECHO_MESSAGE
# What is the value of ECHO_MESSAGE?
$ exit
$ export ECHO_MESSAGE="Hello, World!"
$ make shell
# Any change to .env?
$ cat .env
$ env | grep ECHO_MESSAGE
# What is the value of ECHO_MESSAGE?
$ exit
# Create a .env file based on the env.example
$ make envfile ENVFILE=env.example
$ make shell
# What happened to our .env file?
$ cat .env
$ env | grep ECHO_MESSAGE
# What's the value of ECHO_MESSAGE? Why isn't it "Hello, World!"?
$ exit
# Clean the work directory
$ make clean

# The two previous steps can be combined into one
$ make envfile shell ENVFILE=env.example
$ exit

# Clean your current repository
$ make clean

# Create .env file with Make and Compose

This section shows some ways to create .env file with Make and Compose with the given docker-compose.yml file:

# docker-compose.yml
version: "3.7"
services:
  alpine:
    image: alpine
    env_file: ${ENVFILE:-.env}
    volumes:
      - type: bind
        source: "."
        target: /opt/app
    working_dir: /opt/app

TIP

Even though docker-compose.yml has the variable substitution env_file: ${ENVFILE:-.env}, which suggests the ENVFILE environment variable can be used instead of .env, the ways below always use .env. However, the line is required when creating the .env file, otherwise docker-compose fails if .env file is specified but does not exist.

# Explicit

Targets requiring .env file will fail if the file does not exist. The .env file can be created with envfile target.

# Makefile
COMPOSE_RUN_ALPINE = docker-compose run alpine
ENVFILE ?= env.template

envfile:
	ENVFILE=$(ENVFILE) $(COMPOSE_RUN_ALPINE) cp $(ENVFILE) .env

target: .env
	$(COMPOSE_RUN_ALPINE) cat .env

# clean removes the .env
clean: .env
	$(COMPOSE_RUN_ALPINE) rm .env

# fail if .env does not exist
$ make target
# overwrite .env based on env.template
$ make envfile
# overwrite .env with a specific file
$ make envfile ENVFILE=env.example
# execute a target with a specific .env file
$ make envfile target ENVFILE=env.example

# Semi-Implicit

Targets requiring .env file will get it created if it does not exist. The .env file can be overwritten by calling make envfile ENVFILE=.env.example.

# Makefile
COMPOSE_RUN_ALPINE = docker-compose run alpine
ENVFILE ?= env.template

# .env creates .env based on $(ENVFILE) if .env does not exist
.env:
	$(MAKE) envfile

# envfile overwrites .env with $(ENVFILE)
envfile:
	ENVFILE=$(ENVFILE) $(COMPOSE_RUN_ALPINE) cp $(ENVFILE) .env

# target requires .env
target: .env
	$(COMPOSE_RUN_ALPINE) cat .env

# clean removes the .env
clean: .env
	$(COMPOSE_RUN_ALPINE) rm .env

# create, if it does not exist, .env based on env.template
$ make target
# create .env, if it does not exist, based on env.template
$ make .env
# create .env, if it does not exist, based on $(ENVFILE)
$ make .env ENVFILE=env.example
# overwrite .env based on env.template
$ make envfile
# overwrite .env with a specific file
$ make envfile ENVFILE=env.example
# execute a target with a specific .env file
$ make envfile target ENVFILE=env.example

# Implicit

Targets requiring .env file will get it created if it does not exist. The .env file can be overwritten by setting ENVFILE environment variable.

# Makefile
COMPOSE_RUN_ALPINE = docker-compose run alpine
ifdef ENVFILE
	ENVFILE_TARGET=envfile
else
	ENVFILE_TARGET=.env
endif

# Create .env based on env.template if .env does not exist
.env:
	$(MAKE) envfile ENVFILE=env.template

# Create/Overwrite .env with $(ENVFILE)
envfile:
	ENVFILE=$(ENVFILE) $(COMPOSE_RUN_ALPINE) cp $(ENVFILE) .env

# target requiring $(ENVFILE_TARGET)
target: $(ENVFILE_TARGET)
	$(COMPOSE_RUN_ALPINE) cat .env

# clean removes the .env
clean: $(ENVFILE_TARGET)
  $(COMPOSE_RUN_ALPINE) rm .env

# create .env, if it does not exist, based on env.template
$ make target
# create .env, if it does not exist, based on env.template
$ make .env
# create .env, if it does not exist, a specific file
$ make .env ENVFILE=env.example
# overwrite .env with a specific file
$ make envfile ENVFILE=env.example
# execute a target with a specific .env file
$ make envfile target ENVFILE=env.example
# or (no need to specify envfile)
$ make target ENVFILE=env.example

# Create .env file with Make and Docker

This section shows some ways to create .env file with Make and Docker.

TIP

Examples below use Alpine container (Docker pattern) to create .env file. However, in most cases, using the host cp (and rm) is fine.

# Explicit

Targets requiring .env file will fail if the file does not exist. The .env file can be created with envfile target.

# Makefile
MAKEFILE_DIR = $(dir $(abspath $(firstword $(MAKEFILE_LIST))))
DOCKER_RUN_ALPINE = docker run --rm -v $(MAKEFILE_DIR):/opt/app -w /opt/app alpine
ENVFILE ?= env.template

# envfile overwrites .env with $(ENVFILE)
envfile:
	$(DOCKER_RUN_ALPINE) cp $(ENVFILE) .env

# target requires .env
target: .env
	$(DOCKER_RUN_ALPINE) cat .env

# clean removes the .env
clean:
  $(DOCKER_RUN_ALPINE) rm .env

# fail if .env does not exist
$ make target
# overwrite .env based on env.template
$ make envfile
# overwrite .env with a specific file
$ make envfile ENVFILE=env.example
# execute a target with a specific .env file
$ make envfile target ENVFILE=env.example

# Semi-Implicit

Targets requiring .env file will get it created if it does not exist. The .env file can be overwritten by calling make envfile ENVFILE=.env.example.

# Makefile
MAKEFILE_DIR = $(dir $(abspath $(firstword $(MAKEFILE_LIST))))
DOCKER_RUN_ALPINE = docker run --rm -v $(MAKEFILE_DIR):/opt/app -w /opt/app alpine
ENVFILE ?= env.template

# .env creates .env based on $(ENVFILE) if .env does not exist
.env:
	$(MAKE) envfile

# envfile overwrites .env with $(ENVFILE)
envfile:
	$(DOCKER_RUN_ALPINE) cp $(ENVFILE) .env

# target requires .env
target: .env
	$(DOCKER_RUN_ALPINE) cat .env

# clean removes the .env
clean:
  $(DOCKER_RUN_ALPINE) rm .env

# create, if it does not exist, .env based on env.template
$ make target
# create .env, if it does not exist, based on env.template
$ make .env
# create .env, if it does not exist, based on $(ENVFILE)
$ make .env ENVFILE=env.example
# overwrite .env based on env.template
$ make envfile
# overwrite .env with a specific file
$ make envfile ENVFILE=env.example
# execute a target with a specific .env file
$ make envfile target ENVFILE=env.example

# Implicit

Targets requiring .env file will get it created if it does not exist. The .env file can be overwritten by setting ENVFILE environment variable.

# Makefile
MAKEFILE_DIR = $(dir $(abspath $(firstword $(MAKEFILE_LIST))))
DOCKER_RUN_ALPINE = docker run --rm -v $(MAKEFILE_DIR):/opt/app -w /opt/app alpine
ifdef ENVFILE
	ENVFILE_TARGET=envfile
else
	ENVFILE_TARGET=.env
endif

# Create .env based on env.template if .env does not exist
.env:
	$(MAKE) envfile ENVFILE=env.template

# Create/Overwrite .env with $(ENVFILE)
envfile:
	$(DOCKER_RUN_ALPINE) cp $(ENVFILE) .env

# target requiring $(ENVFILE_TARGET)
target: $(ENVFILE_TARGET)
	$(DOCKER_RUN_ALPINE) cat .env

# clean removes the .env
clean:
  $(DOCKER_RUN_ALPINE) rm .env

# create .env, if it does not exist, based on env.template
$ make target
# create .env, if it does not exist, based on env.template
$ make .env
# create .env, if it does not exist, a specific file
$ make .env ENVFILE=env.example
# overwrite .env with a specific file
$ make envfile ENVFILE=env.example
# execute a target with a specific .env file
$ make envfile target ENVFILE=env.example
# or (no need to specify envfile)
$ make target ENVFILE=env.example

# Environment variable check for Make target

Here is a way of checking the presence of environment variables before executing a Make target.

echo: env-MESSAGE
	@docker run --rm alpine echo "$(MESSAGE)"

env-%:
	@docker run --rm -e ENV_VAR=$($*) alpine echo "Check if $* is not empty"
	@docker run --rm -e ENV_VAR=$($*) alpine sh -c '[ -z "$$ENV_VAR" ] && echo "Error: $* is empty" && exit 1 || exit 0'

# Call echo without a MESSAGE
$ make echo
Check if MESSAGE is not empty
Error: MESSAGE is empty
make: *** [env-MESSAGE] Error 1

# Call echo with a MESSAGE
$ make echo MESSAGE=helloworld
Check if MESSAGE is not empty
helloworld

# AWS environment variables and ~/.aws

When using AWS, you can use environment variables. This is useful when you assume role as usually a tool like assume-role would set your environment variables.

# env.template
AWS_REGION
AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY
AWS_SESSION_TOKEN
AWS_PROFILE

or share your AWS folder like the following:

# docker-compose.yml
services:
  serverless:
    image: flemay/aws
    env_file: .env
    volumes:
      - ~/.aws:/root/.aws:Z

Or both can be used. In this case, environment variables will take precedence over ~/.aws when using AWS cli.

← Other tips →