How to run dbt on Docker

 

How to configure a local dbt development environment for free

In this article, we will learn how to configure a local dbt development environment with the free dbt-core. In this context, we will use the free and open-source relational database management system PostgreSQL and Visual Studio Code as our IDE.

 

Why choose dbt?

dbt is a popular data transformation framework because anyone familiar with SQL can safely use it, and it supports multiple adapters including Databricks, PostgreSQL and Snowflake among others.

Unlike many alternative data transformation frameworks such as Matillion or Coalesce, which are low-code and GUI oriented, dbt is a code-first framework SQL focused. This means that you can write code using your favorite editor and run it against any database, or even easily migrate away from dbt to another tool.

dbt is available in 3 products: core, fusion and cloud. dbt core is a free and open-source framework, fusion provides additional features and cloud provides a hosted service. In this article, we will focus on dbt core.

Why use Docker?

Docker is a platform that allows to run applications in virtualized containers, providing safely isolated environments, infrastructure-agnostic deployments and portability.

Whether you are developing an application for personal use or business-critical solution for enterprise, technologies evolve and change, and Docker provides a way to test your code against different versions of software, libraries and frameworks.

What you will need?

• Docker Desktop
• Visual Studio Code
• Local folder for your project files, including dbt models and PostgreSQL database

How to install and configure dbt-core and dbt-postgres with Docker?

1. Download and install Docker Desktop https://www.docker.com/products/docker-desktop/

2. Launch Docker Desktop.

3. Launch Visual Studio Code and open your project folder.

4. For PostgreSQL, we will download from existing Docker image.

   1. In Visual Studio Code, open a terminal window.

   2. Pull a PostgreSQL Docker image by running this command in the terminal:

      docker pull postgres:13
      

      Here you can specify another PostgreSQL version or "latest" if you prefer, but it is recommended to always target a version.

5. For dbt, we will build our image with dbt-core and dbt-postgres from existing Python Docker image.

   1. Create a new file named Dockerfile in your project folder.

   2. Add the following content into the file:

      FROM python:3.13-slim
      
      # Install system dependencies including git
      RUN apt-get update && apt-get install -y \
          gcc \
          g++ \
          postgresql-client \
          libpq-dev \
          python3-dev \
          build-essential \
          git \
          && rm -rf /var/lib/apt/lists/*
      
      # Install dbt packages
      RUN pip install --no-cache-dir \
          dbt-core \
          dbt-postgres
      
      WORKDIR /usr/app
      
      CMD ["bash"]
      

      Again, you can choose a different version of Python. To see the available list of Python Docker images, visit this link https://hub.docker.com/_/python/

   3. Build the dbt image by running this command in the terminal:

      docker build -t dbt-image .
      

      If you are running on an ARM64 architecture, you need to add the --platform linux/amd64 flag to the build command, because dbt-postgres does not support that architecture.

      docker build --platform linux/amd64 -t dbt-image .
      

6. Open Docker Desktop and look under "Images", you will see both images "postgres" and "dbt-image".

7. Create your dbt profile configuration which will be used by dbt-core.

   1. Create a new file named .dbt/profiles.yml in your user home directory. On Windows, this is typically C:\Users\your_username\.dbt\profiles.yml.

   2. Add the following content into the file:

      your_dbt_project_name:
        outputs:
          dev:
            dbname: your_postgres_db_name
            host: postgres  # This is the service name in docker-compose
            password: your-postgres-password
            port: 5432
            schema: your_schema_name
            threads: 1
            type: postgres
            user: your-postgres-user
        target: dev
      

      Replace all the placeholders with your actual names and credentials.

8. Create an environment file to declare credentials for your project.

   1. Create a file named .env in your project directory.

   2. Add the following content into the file:

      POSTGRES_DB=your_postgres_db_name
      POSTGRES_USER=your-postgres-user
      POSTGRES_PASSWORD=your-postgres-password
      

      Replace all the placeholders with your actual credentials.

9. Create the containers to run the images for your project.

   1. Create a new file named docker-compose.yml in your project folder.

   2. Add the following content into the file:

      services:
        dbt:
          image: dbt-image  # Use your built dbt image
          volumes:
            - .:/usr/app
            - ${USERPROFILE}/.dbt:/root/.dbt  # Windows-friendly path
          working_dir: /usr/app
          depends_on:
            - postgres
          networks:
            - dbt-network
          stdin_open: true
          tty: true
      
        postgres:
          image: postgres:13  # Use your postgres image
          environment:
            - POSTGRES_DB=${POSTGRES_DB}
            - POSTGRES_USER=${POSTGRES_USER}
            - POSTGRES_PASSWORD=${POSTGRES_PASSWORD}
          ports:
            - "5432:5432"
          volumes:
            - postgres_data:/var/lib/postgresql/data
          networks:
            - dbt-network
      
      volumes:
        postgres_data:
      
      networks:
        dbt-network:
          driver: bridge
      

   3. Start the container by running this command in the terminal:

      docker compose up -d
      

      Confirm the container is running by running this command in the terminal, or simply looking at the list of containers on Docker Desktop:

      docker ps
      

      To stop the containers from running, run this command in the terminal:

      docker compose down
      

      Since the container has now been added to Docker Desktop, you can also start and stop the containers from there.

10. Install the Visual Studio Code extension to manage your containers.

    1. Search for "Microsoft Dev Container" in the Visual Studio Code extensions marketplace, then install it.

    2. Open the "Containers" extension to see the list of containers through this extension, make sure Docker Engine is running.

    3. Right-click on your container to start it.

11. We will now use dbt to initialize your project.

    1. Right-click on your "dbt-image" container and select "Attach Shell".

    2. From the terminal, run dbt init to initialize your project. dbt-core will add folders and files to your project. A sample of models will be created in your project.

       • Make sure your dbt_project.yml profile setting is mapped to the profile defined in your profiles.yml file.

         # This setting configures which "profile" dbt uses for this project.
         profile: 'your_dbt_project_name'
         

    3. Run dbt debug to check your connection.

    4. Run dbt build to materialize your models into tables on your PostgreSQL instance.

That's it, you have now created your dbt project and build your models into your database instance, all running in a Docker container.