Environment variables
Development following the twelve-factor appopen in new window use the environment variables to configureopen in new window their application.
Often there are many environment variables and having them in a .env
file becomes handy. Docker and Compose do use environment variables fileopen in new window to pass the variables to the containers.
.env file and expectations
With the following .env
file:
# make sure these env vars are not set in the system
ENV_A
ENV_B=
ENV_C=env_c
2
3
4
And the docker-compose.yml
file:
services:
alpine:
image: alpine
env_file: .env
2
3
4
The expected results are:
$ docker run --rm --env-file=.env alpine env
ENV_B=
ENV_C=env_c
# ENV_A is not set and ENV_B is set to empty
$ docker-compose run --rm alpine env
ENV_B=
ENV_C=env_c
# Same as Docker
2
3
4
5
6
7
8
9
WARNING
The command compose
of Docker has a very different result than Docker and Docker-Compose.
$ docker compose run --rm alpine env
ENV_C=env_c
ENV_A=ENV_B=
2
3
Files env.template and env.example
env.template
and env.example
files provide some help when managing environment variables in a project.
WARNING
As env.template
and env.example
files are meant to be part of the source code, never include sensitive values like passwords.
env.template
env.template
contains names (key-only) of all environment variables the application and pipeline use. No values are set here. # description
can be used to describe an environment variable. env.template
is mainly used as a template to .env
in a CI/CD pipeline.
# env.template
ENV_VAR_A
ENV_VAR_B
2
3
env.example
env.example
defines values so that it can be used straight away with Make like $ make envfile test ENVFILE=env.example
. It also gives an example of values that are being used in the project which is very useful for the developers.
# env.example
ENV_VAR_A=a
ENV_VAR_B=b
2
3
Pros
- Simple
- Understanding the concept is pretty straight forward
- Does not require any script
- Application agnostic
- This pattern can be used for any environment variable of any kind of application
- Descriptive and explicit
env.template
tells what environment variables are used by the projectenv.example
shows what value those environment variables can have- Environment variables needs to explicitly be added
- Flexible
- The way the environment variables are set is up to you. They can be included in the
.env
file when developing locally or exported in a CD/CI host
- The way the environment variables are set is up to you. They can be included in the
Cons
- Environment variable management is not centralized
- Adding, modifying, or deleting environment variables may impact multiple files such as
- env.template
- env.example
- Makefile
- docker-compose.yml
- application source code
- pipeline-as-code file
- Adding, modifying, or deleting environment variables may impact multiple files such as
- Error prone
- It is easy to forget to add a new environment variable to the
env.template/env.example
files
- It is easy to forget to add a new environment variable to the
CI/CD pipeline
Given all environment variables are set in your CI/CD pipeline, creating a .env
file based on env.template
allows values of those environment variables to be passed to the Docker container environments. See this tutorial to better understand the use of .env
file with Docker and Compose.
Day-to-day development
In a day-to-day development process, you could create a file named .env.dev
with the config of your dev environment and copy the contents of it into .env
so that you can manually deploy/delete/etc your app for testing. This allows you to not accidentally lose the values if the .env
file is replaced. There are few ways to copy the contents of your file to .env
:
- manually
- make envfile ENVFILE=yourfile
Understanding .env file with Docker and Compose
This tutorial shows how .env
file works with Docker and Docker Compose.
Files
Create the following files:
# file: env.template
ECHO_MESSAGE
2
# file: env.example
ECHO_MESSAGE="Hello, 3 Musketeers!"
2
# file: docker-compose.yml
version: '3'
services:
musketeers:
image: flemay/musketeers
env_file: .env
volumes:
- .:/opt/app
working_dir: /opt/app
2
3
4
5
6
7
8
9
# file: Makefile
COMPOSE_RUN_MUSKETEERS = docker-compose run --rm musketeers
# ENVFILE is env.template by default but can be overwritten
ENVFILE ?= env.template
# envfile creates or overwrites .env with $(ENVFILE)
envfile:
cp -f $(ENVFILE) .env
# shell creates a shell environment inside a container.
shell:
$(COMPOSE_RUN_MUSKETEERS) sh -l
# cleanDocker remove all containers/network created by Compose
cleanDocker:
docker-compose down --remove-orphans
# clean cleans the current directory
clean: cleanDocker
rm -f .env
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
Steps
# Let's clean first
$ make clean
# Let's go inside a container.
$ make shell
# Opps! Error. Compose needs a .env file. Let's create one...
$ make envfile
$ make shell
# See the new .env file?
$ ls -la
$ cat .env
# Let's look at the environment variable ECHO_MESSAGE
$ env | grep ECHO_MESSAGE
# What is the value of ECHO_MESSAGE?
$ exit
$ export ECHO_MESSAGE="Hello, World!"
$ make shell
# Any change to .env?
$ cat .env
$ env | grep ECHO_MESSAGE
# What is the value of ECHO_MESSAGE?
$ exit
# Create a .env file based on the env.example
$ make envfile ENVFILE=env.example
$ make shell
# What happened to our .env file?
$ cat .env
$ env | grep ECHO_MESSAGE
# What's the value of ECHO_MESSAGE? Why isn't it "Hello, World!"?
$ exit
# Clean the work directory
$ make clean
# The two previous steps can be combined into one
$ make envfile shell ENVFILE=env.example
$ exit
# Clean your current repository
$ make clean
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
Overwriting .env or not
Examples in this page use .env
to pass environment variables to a container. The file .env
can be overwritten when setting the environment variable ENVFILE
. This has few advantages:
- You know the file
.env
will always be used - Compose uses
.env
when doing variable substitutionopen in new window
Create .env file with Make and Compose
This section shows some ways to create .env
file with Make and Compose with the given docker-compose.yml
file:
# docker-compose.yml
version: "3.7"
services:
alpine:
image: alpine
env_file: ${ENVFILE:-.env}
volumes:
- type: bind
source: "."
target: /opt/app
working_dir: /opt/app
2
3
4
5
6
7
8
9
10
11
The docker-compose.yml
above has the variable substitutionopen in new window env_file: ${ENVFILE:-.env}
, which allows the use of a different file that .env
by defining the environment variable ENVFILE
. This was required for using Compose otherwise Compose would simply fail. Examples in this section will use .env
except when generating the file.
Explicit
Targets requiring .env
file will fail if the file does not exist. The .env
file can be created with envfile
target.
TIP
Explicit is the method I personally prefer.
# Makefile
COMPOSE_RUN_ALPINE = docker-compose run alpine
ENVFILE ?= env.template
envfile:
ENVFILE=$(ENVFILE) $(COMPOSE_RUN_ALPINE) cp $(ENVFILE) .env
targetA: .env
$(COMPOSE_RUN_ALPINE) cat .env
targetB:
$(COMPOSE_RUN_ALPINE) echo "Hello, World!"
# clean removes the .env
clean: .env
$(COMPOSE_RUN_ALPINE) rm .env
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
# fail if .env does not exist
$ make targetA
# fail if .env does not exist even if targetB does not require .env because Compose does in our case
$ make targetB
# overwrite .env based on env.template
$ make envfile
# overwrite .env with a specific file
$ make envfile ENVFILE=env.example
# execute a target with a specific .env file
$ make envfile target ENVFILE=env.example
2
3
4
5
6
7
8
9
10
Semi-Implicit
Targets requiring .env
file will get it created if it does not exist. The .env
file can be overwritten by calling make envfile ENVFILE=.env.example
.
# Makefile
COMPOSE_RUN_ALPINE = docker-compose run alpine
ENVFILE ?= env.template
# .env creates .env based on $(ENVFILE) if .env does not exist
.env:
$(MAKE) envfile
# envfile overwrites .env with $(ENVFILE)
envfile:
ENVFILE=$(ENVFILE) $(COMPOSE_RUN_ALPINE) cp $(ENVFILE) .env
# target requires .env
target: .env
$(COMPOSE_RUN_ALPINE) cat .env
# clean removes the .env
clean: .env
$(COMPOSE_RUN_ALPINE) rm .env
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
# create, if it does not exist, .env based on env.template
$ make target
# create .env, if it does not exist, based on env.template
$ make .env
# create .env, if it does not exist, based on $(ENVFILE)
$ make .env ENVFILE=env.example
# overwrite .env based on env.template
$ make envfile
# overwrite .env with a specific file
$ make envfile ENVFILE=env.example
# execute a target with a specific .env file
$ make envfile target ENVFILE=env.example
2
3
4
5
6
7
8
9
10
11
12
Implicit
Targets requiring .env
file will get it created if it does not exist. The .env
file can be overwritten by setting ENVFILE
environment variable.
# Makefile
COMPOSE_RUN_ALPINE = docker-compose run alpine
ifdef ENVFILE
ENVFILE_TARGET=envfile
else
ENVFILE_TARGET=.env
endif
# Create .env based on env.template if .env does not exist
.env:
$(MAKE) envfile ENVFILE=env.template
# Create/Overwrite .env with $(ENVFILE)
envfile:
ENVFILE=$(ENVFILE) $(COMPOSE_RUN_ALPINE) cp $(ENVFILE) .env
# target requiring $(ENVFILE_TARGET)
target: $(ENVFILE_TARGET)
$(COMPOSE_RUN_ALPINE) cat .env
# clean removes the .env
clean: $(ENVFILE_TARGET)
$(COMPOSE_RUN_ALPINE) rm .env
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
# create .env, if it does not exist, based on env.template
$ make target
# create .env, if it does not exist, based on env.template
$ make .env
# create .env, if it does not exist, a specific file
$ make .env ENVFILE=env.example
# overwrite .env with a specific file
$ make envfile ENVFILE=env.example
# execute a target with a specific .env file
$ make envfile target ENVFILE=env.example
# or (no need to specify envfile)
$ make target ENVFILE=env.example
2
3
4
5
6
7
8
9
10
11
12
Create .env file with Make and Docker
This section shows some ways to create .env
file with Make and Docker.
TIP
Examples below use Alpine container (Docker pattern) to create .env
file. However, in most cases, using the host cp
(and rm
) is fine.
Explicit
Targets requiring .env
file will fail if the file does not exist. The .env
file can be created with envfile
target.
# Makefile
MAKEFILE_DIR = $(dir $(abspath $(firstword $(MAKEFILE_LIST))))
DOCKER_RUN_ALPINE = docker run --rm -v $(MAKEFILE_DIR):/opt/app -w /opt/app alpine
ENVFILE ?= env.template
# envfile overwrites .env with $(ENVFILE)
envfile:
$(DOCKER_RUN_ALPINE) cp $(ENVFILE) .env
# target requires .env
target: .env
$(DOCKER_RUN_ALPINE) cat .env
# clean removes the .env
clean:
$(DOCKER_RUN_ALPINE) rm .env
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
# fail if .env does not exist
$ make target
# overwrite .env based on env.template
$ make envfile
# overwrite .env with a specific file
$ make envfile ENVFILE=env.example
# execute a target with a specific .env file
$ make envfile target ENVFILE=env.example
2
3
4
5
6
7
8
Semi-Implicit
Targets requiring .env
file will get it created if it does not exist. The .env
file can be overwritten by calling make envfile ENVFILE=.env.example
.
# Makefile
MAKEFILE_DIR = $(dir $(abspath $(firstword $(MAKEFILE_LIST))))
DOCKER_RUN_ALPINE = docker run --rm -v $(MAKEFILE_DIR):/opt/app -w /opt/app alpine
ENVFILE ?= env.template
# .env creates .env based on $(ENVFILE) if .env does not exist
.env:
$(MAKE) envfile
# envfile overwrites .env with $(ENVFILE)
envfile:
$(DOCKER_RUN_ALPINE) cp $(ENVFILE) .env
# target requires .env
target: .env
$(DOCKER_RUN_ALPINE) cat .env
# clean removes the .env
clean:
$(DOCKER_RUN_ALPINE) rm .env
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
# create, if it does not exist, .env based on env.template
$ make target
# create .env, if it does not exist, based on env.template
$ make .env
# create .env, if it does not exist, based on $(ENVFILE)
$ make .env ENVFILE=env.example
# overwrite .env based on env.template
$ make envfile
# overwrite .env with a specific file
$ make envfile ENVFILE=env.example
# execute a target with a specific .env file
$ make envfile target ENVFILE=env.example
2
3
4
5
6
7
8
9
10
11
12
Implicit
Targets requiring .env
file will get it created if it does not exist. The .env
file can be overwritten by setting ENVFILE
environment variable.
# Makefile
MAKEFILE_DIR = $(dir $(abspath $(firstword $(MAKEFILE_LIST))))
DOCKER_RUN_ALPINE = docker run --rm -v $(MAKEFILE_DIR):/opt/app -w /opt/app alpine
ifdef ENVFILE
ENVFILE_TARGET=envfile
else
ENVFILE_TARGET=.env
endif
# Create .env based on env.template if .env does not exist
.env:
$(MAKE) envfile ENVFILE=env.template
# Create/Overwrite .env with $(ENVFILE)
envfile:
$(DOCKER_RUN_ALPINE) cp $(ENVFILE) .env
# target requiring $(ENVFILE_TARGET)
target: $(ENVFILE_TARGET)
$(DOCKER_RUN_ALPINE) cat .env
# clean removes the .env
clean:
$(DOCKER_RUN_ALPINE) rm .env
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
# create .env, if it does not exist, based on env.template
$ make target
# create .env, if it does not exist, based on env.template
$ make .env
# create .env, if it does not exist, a specific file
$ make .env ENVFILE=env.example
# overwrite .env with a specific file
$ make envfile ENVFILE=env.example
# execute a target with a specific .env file
$ make envfile target ENVFILE=env.example
# or (no need to specify envfile)
$ make target ENVFILE=env.example
2
3
4
5
6
7
8
9
10
11
12
Environment variable check for Make target
Here is a way of checking the presence of environment variables before executing a Make target.
echo: env-MESSAGE
@docker run --rm alpine echo "$(MESSAGE)"
env-%:
@docker run --rm -e ENV_VAR=$($*) alpine echo "Check if $* is not empty"
@docker run --rm -e ENV_VAR=$($*) alpine sh -c '[ -z "$ENV_VAR" ] && echo "Error: $* is empty" && exit 1 || exit 0'
2
3
4
5
6
# Call echo without a MESSAGE
$ make echo
Check if MESSAGE is not empty
Error: MESSAGE is empty
make: *** [env-MESSAGE] Error 1
# Call echo with a MESSAGE
$ make echo MESSAGE=helloworld
Check if MESSAGE is not empty
helloworld
2
3
4
5
6
7
8
9
10
AWS environment variables and ~/.aws
When using AWS, you can use environment variables. This is useful when you assume role as usually a tool like assume-roleopen in new window would set your environment variables.
# env.template
AWS_REGION
AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY
AWS_SESSION_TOKEN
AWS_PROFILE
2
3
4
5
6
or share your AWS folder like the following:
# docker-compose.yml
services:
serverless:
image: flemay/aws
env_file: .env
volumes:
- ~/.aws:/root/.aws:Z
2
3
4
5
6
7
Or both can be used. In this case, environment variables will take precedence over ~/.aws
when using AWS cli.