How to configure a local dbt development environment for free
In this article, we will learn how to configure a local dbt development environment with the free dbt-core. In this context, we will use the free and open-source relational database management system PostgreSQL and Visual Studio Code as our IDE.
Why choose dbt?
dbt is a popular data transformation framework because anyone familiar with SQL can safely use it, and it supports multiple adapters including Databricks, PostgreSQL and Snowflake among others.
Unlike many alternative data transformation frameworks such as Matillion or Coalesce, which are low-code and GUI oriented, dbt is a code-first framework SQL focused. This means that you can write code using your favorite editor and run it against any database, or even easily migrate away from dbt to another tool.
dbt is available in 3 products: core, fusion and cloud. dbt core is a free and open-source framework, fusion provides additional features and cloud provides a hosted service. In this article, we will focus on dbt core.
Why use Docker?
Docker is a platform that allows to run applications in virtualized containers, providing safely isolated environments, infrastructure-agnostic deployments and portability.
Whether you are developing an application for personal use or business-critical solution for enterprise, technologies evolve and change, and Docker provides a way to test your code against different versions of software, libraries and frameworks.
What you will need?
- Docker Desktop
- Visual Studio Code
- Local folder for your project files, including dbt models and PostgreSQL database
How to install and configure dbt-core and dbt-postgres with Docker?
Download and install Docker Desktop https://www.docker.com/products/docker-desktop/
Launch Docker Desktop.
Launch Visual Studio Code and open your project folder.
For PostgreSQL, we will download from existing Docker image.
In Visual Studio Code, open a terminal window.
Pull a PostgreSQL Docker image by running this command in the terminal:
docker pull postgres:13Here you can specify another PostgreSQL version or "latest" if you prefer, but it is recommended to always target a version.
For dbt, we will build our image with dbt-core and dbt-postgres from existing Python Docker image.
Create a new file named
Dockerfilein your project folder.Add the following content into the file:
FROM python:3.13-slim # Install system dependencies including git RUN apt-get update && apt-get install -y \ gcc \ g++ \ postgresql-client \ libpq-dev \ python3-dev \ build-essential \ git \ && rm -rf /var/lib/apt/lists/* # Install dbt packages RUN pip install --no-cache-dir \ dbt-core \ dbt-postgres WORKDIR /usr/app CMD ["bash"]Again, you can choose a different version of Python. To see the available list of Python Docker images, visit this link https://hub.docker.com/_/python/
Build the dbt image by running this command in the terminal:
docker build -t dbt-image .If you are running on an ARM64 architecture, you need to add the
--platform linux/amd64flag to the build command, because dbt-postgres does not support that architecture.docker build --platform linux/amd64 -t dbt-image .
Open Docker Desktop and look under "Images", you will see both images "postgres" and "dbt-image".
Create your dbt profile configuration which will be used by dbt-core.
Create a new file named
.dbt/profiles.ymlin your user home directory. On Windows, this is typicallyC:\Users\your_username\.dbt\profiles.yml.Add the following content into the file:
your_dbt_project_name: outputs: dev: dbname: your_postgres_db_name host: postgres # This is the service name in docker-compose password: your-postgres-password port: 5432 schema: your_schema_name threads: 1 type: postgres user: your-postgres-user target: devReplace all the placeholders with your actual names and credentials.
Create an environment file to declare credentials for your project.
Create a file named
.envin your project directory.Add the following content into the file:
POSTGRES_DB=your_postgres_db_name POSTGRES_USER=your-postgres-user POSTGRES_PASSWORD=your-postgres-passwordReplace all the placeholders with your actual credentials.
Create the containers to run the images for your project.
Create a new file named
docker-compose.ymlin your project folder.Add the following content into the file:
services: dbt: image: dbt-image # Use your built dbt image volumes: - .:/usr/app - ${USERPROFILE}/.dbt:/root/.dbt # Windows-friendly path working_dir: /usr/app depends_on: - postgres networks: - dbt-network stdin_open: true tty: true postgres: image: postgres:13 # Use your postgres image environment: - POSTGRES_DB=${POSTGRES_DB} - POSTGRES_USER=${POSTGRES_USER} - POSTGRES_PASSWORD=${POSTGRES_PASSWORD} ports: - "5432:5432" volumes: - postgres_data:/var/lib/postgresql/data networks: - dbt-network volumes: postgres_data: networks: dbt-network: driver: bridgeStart the container by running this command in the terminal:
docker compose up -dConfirm the container is running by running this command in the terminal, or simply looking at the list of containers on Docker Desktop:
docker psTo stop the containers from running, run this command in the terminal:
docker compose downSince the container has now been added to Docker Desktop, you can also start and stop the containers from there.
Install the Visual Studio Code extension to manage your containers.
Search for "Microsoft Dev Container" in the Visual Studio Code extensions marketplace, then install it.
Open the "Containers" extension to see the list of containers through this extension, make sure Docker Engine is running.
Right-click on your container to start it.
We will now use dbt to initialize your project.
Right-click on your "dbt-image" container and select "Attach Shell".
From the terminal, run
dbt initto initialize your project. dbt-core will add folders and files to your project. A sample of models will be created in your project.- Make sure your
dbt_project.ymlprofile setting is mapped to the profile defined in yourprofiles.ymlfile.# This setting configures which "profile" dbt uses for this project. profile: 'your_dbt_project_name'
- Make sure your
Run
dbt debugto check your connection.Run
dbt buildto materialize your models into tables on your PostgreSQL instance.
That's it, you have now created your dbt project and build your models into your database instance, all running in a Docker container.
