Package Organization Approaches in Go

Inevitably, every Go developer asks the following question:

How do I organize my code?

There are many articles and approaches, and while some work well for some, they may not work well for others. Go has no official conventions and preferences on how you should structure your packages and where your non-code resources should reside.

Unlike other languages though, Go does not allow circular package imports. Projects thus require additional planning when grouping code into packages to ensure that dependencies do not import each other.

The goal of this article is not to specify a strict convention, but rather advocate building robust mental models for reasoning about your problem domains and how to represent them in your project layouts.

Project Layout Types

No approach is perfect, but there are a few that have gained widespread adoption. Please check the following resources:

Ben Johnson's article on Standard Package Layout.
Golang Standards Project Layout.
Kat Zien's excellent talk, presentation and code samples can be found here.

Now we are going to explore what options are available for structuring your applications and distinguish between some good and some bad practices.

Flat Structure

Rather than spending time trying to figure out how to break your code into packages, an app with a flat structure would just place all of the .go files in a single package.
At first, this sounds awful, because we do not use packages that separate concerns while making it easier to navigate to the correct source files quickly.

Recommendations

When using a flat structure you should still try to adhere to coding best practices. Here are some helpful tips:

You want to separate different parts of your application using different .go files:

restaurant-app/  
   customer.go  
   data.go  
   handlers.go  
   main.go  
   reservation.go  
   server.go  
   storage.go  
   storage_json.go   
   storage_mem.go

Global variables can still become problematic, so you should consider using types with methods to keep them out of your code:

package main

import(
    "net/http"
    "some"
    "someapi"
)

type Server struct {
    apiClient *someapi.Client
    router *some.Router
}

func (s *Server) ServeHTTP(w http.ResponseWriter, r *http.Request) {
    s.router.ServeHTTP(w, r)
}

Your main() function should still be stripped of most logic outside of setting up the application.
A possible improvement to the flat structure is to again put all of your code into a single package, but separate the main package where you define the entry point of the application. This would allow you to use the common cmd sub-directory pattern:

restaurant-app/
cmd/
  web/
    # package main
    main.go
  cli/
    # package main
    main.go
# package restaurantapp
server.go
customer_handler.go
reservation_handler.go
customer_store.go

This project layout is usually suitable for small libraries or CLI tools and very simple projects.

Anti Pattern: Group by Function (Layered Architecture)

Layered architecture patterns are n-tiered patterns where the components are organized in layers.

This is the traditional method for designing most software and is meant to be self-independent, i.e. all the components are interconnected, but do NOT depend on each other.

We have all heard about the famous 3-tier MVC (Model-View-Controller) architecture where we split our application into the following 3 distinct layers:
— Presentation / User Interface (View)
— Business Logic (Controller)
— Storage / External Dependencies (Model)

This architecture translated into our example project layout would look like this:

restaurant-app/
  # package main
  data.go
  handlers/
    # package handlers
    customers.go
    reservations.go
  # package main
  main.go
  models/
    # package models
    customer.go
    reservation.go
    storage.go
  storage/
    # package storage
    json.go
    memory.go
  ...

This type of layout should NOT compile at all due to circular dependencies.
The storage package uses the models package to get the definitions for a Customer and a Reservation and the models package uses the storage package to make calls to the database.
Another disadvantage of this structure is that it does not guide us about what the application does (at least not more than the flat structure).
This type of layout is strongly NOT recommended when writing applications in Go so try to avoid it.

Anti Pattern: Group by Module

Grouping by Module offers us a slight improvement over the layered approach:

reservationapp/
  customers/
    # package customers
    customer.go
    handler.go
  # package main
  main.go
  reservations/
    # package reservations
    reservation.go
    handler.go
  storage/
    # package storage
    data.go
    json.go
    memory.go
    storage.go
  ...

Now our application is structured logically, but that is probably the only advantage of this approach.
It is still hard to decide, e.g., if reservation should go to the customers package because they are customer reservations or are they suited for having their own package.
Naming is worse because we now have reservations.Reservation and customers.Customer which introduces stutter.
Worst of all is the possibility of circular dependencies again if the reservations package needs to reference the customers package and vice versa.

Group by Context (Domain Driven Design)

This way of thinking about your applications is called Domain Driven Design (DDD).

In its essence it guides you to think about the domain you are dealing with and all the business logic without even writing a single line of code.

Three main components need to be defined:
— Bounded contexts
— Models within each context
— Ubiquitous language

Bounded Contexts

A bounded context is a fancy term defining limits upon your models. An example would be, e.g., a User entity that might have different properties attached to it based on the context:

A User in a sales department context might have properties like leadTime, costOfAquisition, etc.
A User in a customer support context might have properties like responseTime, numberOfTicketsHandled, etc.
This showcases that a User means different things to different people, and the meaning depends heavily on the context.

The bounded context also helps in deciding what has to stay consistent within a particular boundary and what can change independently.

If we decide later on to add a new property to the User from the sales department context, that would not affect the User model in the customer support context.

Ubiquitous Language

Ubiquitous Language is the term used in Domain Driven Design for the practice of building up a common, rigorous language between developers and users.

This language is based on the Domain Model used in the software, and it evolves up to the point of being able to express complex ideas by combining simple elements of the Domain Model.

Categorizing the building blocks

Based on the DDD methodology we will now start to reason about our domain by constructing its building blocks.

If we take our Restaurant Reservation System example, we would have the following elements:

Context: Booking Reservations.
Language: reservation, customer, storage, …
Entities: Reservation, Customer, …
Value Objects: Restaurant, Host, …
Aggregates: BookedReservation
Service: Reservation lister/listing, Reservation adder/adding, Customer adder/adding, Customer lister/listing, …
Events: ReservationAdded, CustomerAdded, ReservationAlreadyExists, ReservationNotFound, …
Repository: ReservationRepository, CustomerRepository, …

Now after defining those blocks we can translate them into our project layout:

restaurant-app/
  adding/
    endpoint.go
    service.go
  customers/
    customer.go
    sample_customers.go
  listing/
    endpoint.go
    service.go
  main.go
  reservations/
    reservation.go
    sample_reservations.go
  storage/
    json.go
    memory.go
    type.go

The main advantage here is that our packages now communicate what they PROVIDE and not what they CONTAIN.

This makes it easier to avoid circular dependencies, because:

adding and listing talk to storage.
storage pulls from customers and reservations.
Model packages like reservations and customers do not care about storage directly.

Group by Context (Domain Driven Design + Hexagonal Architecture)

So far we managed to structure our application according to DDD, eliminated circular dependencies and made it intuitive what each package does only by looking at the directory and file names.

We still have some problems though:

How can we start a version of our application that contains sample data*?*
In our current version sample data is bundled with the application’s entry point in main.go and we have only one main.go.
We have NO option to run a sample data version separately from the main version of the application.
Maybe we want a pure CLI version of the application where instead of adding reservations through HTTP requests, we want the command line to prompt us for each reservation property?

Hexagonal Architecture

This type of architecture distinguishes the parts of the system which form your core domain and all the external dependencies are just implementation details.

Fig. 1: Hexagonal Architecture Layers Diagram

External dependencies could be databases, external APIs, mail clients, cloud services etc., anything that your application interacts with.

The problem this solves is giving you the ability to change one part of the application without affecting the rest, e.g., swapping databases or transport protocols (HTTP to gRPC).

This is not in any way similar to the MVC (layered) model, because:

MVC tends to look at inputs and outputs in a top to bottom way (input -> main logic -> output).

Fig. 2: Layered Architecture Dependency Direction

Hex treats inputs and outputs on the same level. It does not care if something is an input or an output, it is just an external interface.

Fig. 3: Dependency Inversion Direction

The key rule in the hex model is that dependencies only point INWARDS (only outer layers depend upon inner layers and not the other way around). This is called the Dependency Inversion Principle. Check this excellent article by Martin Fowler to learn more.

Recommendations

Based on this approach our project structure could look like this:

restaurant-app/
  cmd/
    # HTTP server
    restaurant-server/
      main.go
    # CLI app
    restaurant-cli/
      main.go
    # HTTP server with seeded data
    restaurant-sample-data/
      main.go
      sample_reservation.go
      sample_customers.go
  pkg/
    adding/
      reservation.go
      endpoint.go
      service.go
    listing/
      customer.go
      reservation.go
      endpoint.go
      service.go
    transport/
      http/
        server.go
    main.go
    storage/
      json/
        customer.go
        repository.go
        reservation.go
      memory/
        customer.go
        repository.go
        reservation.go

To solve the multiple app version binaries problem we utilize the cmd sub-directory pattern which we mentioned as an improvement to the flat structure layout.

We are now able to produce 3 different binaries used to serve different purposes:

restaurant-server - main version of the app deploying an HTTP server.
restaurant-cli - a CLI version with a removed transport layer offering a CLI interface for interaction.
restaurant-sample-data - a sample data seeded version used mainly for testing.

We introduce the pkg package which separates our Go code from the cmd binaries and non-code resources, e.g. DB scripts, configs, documentation, etc. which should be found on the same level under the project's root directory.

NOTE

Using the cmd and pkg directories has become somewhat of a trend in the Go community. It is not a standard by any means, but a good recommendation that should be considered.

According to DDD we keep the adding and listing packages that represent our core domain.
We remove reservations and customers packages and instead introduce models in each of the core domain packages, e.g., adding.Reservation, adding.Customer, listing.Reservation, etc.
The advantage here is that we have separate representations per model according to the bounded context (adding or listing). This allows decoupled model modification and avoids circular dependencies.
We introduce a transport package which contains all transport protocol implementations, e.g. HTTP or maybe gRPC in their respective sub-packages.
The storage package is another bounded context that features its model representations on a storage level and sub-packages for storage implementations, e.g. json, memory, etc.
Again, main.go ties everything together and should not contain any logic that would require testing.

Conclusion

Unfortunately, there is no single right answer, but at least we outlined some examples of problem domains, how to reason about them and how to translate that reasoning into Go package organization. Great freedom comes with great responsibility! Use the following guidelines wisely:

“Make everything as simple as possible, but not simpler.” — Albert Einstein.
Maintain consistency.
Flat and simple is OK for small projects.
Avoid global scope and init().
Separate depended upon code into its own package.
Two top-level directories:
— cmd (for your binaries)
— pkg (for your packages)
All other project files (DB scripts, fixtures, resources, docs, Docker configs, etc.) — should reside under your project’s root directory.
Group by context, not generic functionality. Try DDD/Hex.
main package initializes and ties everything together.

Command Palette

Comments