Building a Terraform Provider — Part I — Initial Setup

Image from unsplash.com

Introduction

This document is Part I of a series I wish to write and to add my experience in building a terraform provider. To start with, its not an easy task, since I am just a beginner in Golang and I happen to do a bit of programming on the Python side, I thought to give it a try.

Background

A requirement popped up where I need to create DNS A records for Private Endpoints created in Azure. The DNS records creation was a bit tricky, because it was created using a custom DNS RestAPI and the read, update and delete methods on this resource was done using Azure CLI commands. The reason for custom API is to authenticate the creation of DNS records, and add Owners for the DNS records

There was couple of ideas that came up, like adding null resource which can run scripts. This is kind of ugly, because we cannot maintain the state of the resource created.

Then the first valid idea was to use Shell provider from Terraform and create a shell script resource, which reads a single script with multiple positional parameters and different methods. Inside the script, add code to create record via custom API and read, update, delete resource via Azure CLI commands. this was actually a simple idea, and works.

Then I had a different thought on building a provider, just for fun and to see how far I can go.

Provider Design

I did a bit research online and came up with a solution to build the provider in such a way that I can extend the provider with additional stuff, like different resources, in future if its required.

I wanted to start with this Terraform Provider Scaffold, but it seemed too much for me as a noob. So I started from scratch.

Structure

First create a directory where the provider will be, I named the directory as terraform-provider-rest, so that I can reuse this provider for doing other RestAPI stuff. So inside this folder, created a couple of directories and empty files. So the structure looks like this

.
├── Makefile
├── README.md
├── client
│   ├── auth.go
│   ├── client.go
│   ├── models.go
│   └── dns_records.go
├── examples
│   ├── main.tf
│   └── outputs.tf
├── main.go
└── records
    └── provider.go

So basically we need couple of important files

• main.go : The main entrypoint for the provider
• client: This directory contains files which are required for the HTTP client to work properly
• records: This is the provider which we are going to create, this is a package in Golang term
• examples: Files to test the terraform provider. We don’t need this until we finish ;)
• Makefile: To speed up the development process. This was needed for me for developing each function
• README: The most important stuff. Documentation about the provider. IMO, if there is no documentation, then its less likely read and less likely to be used.

Coding

Provider Config

Time to add some code. I am working on Linux Terminal, I hope the commands is same for Windows as well for running Go. Make sure, you are inside the terraform-provider-rest directory and run

go mod init terraform-provider-rest
go mod tidy

So we are initializing this project as a module with the proper Go version and Go will add or download all necessary packages required and create two files go.mod and go.sum . go.mod file contains information of all necessary or dependent packages required for the module and go.sum contains information of all the checksums for the packages that is added in go.mod file.

Open main.go file and add the code

package main
import (
  "github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema"
  "github.com/hashicorp/terraform-plugin-sdk/v2/plugin"
  "terraform-provider-rest/records"
)
// Main function, calling the provider
func main() {
  plugin.Serve(&plugin.ServeOpts{
    ProviderFunc: func() *schema.Provider {
      return records.Provider()
    },
  })
}

So we are importing Terraform sdk packages and importing the records package. And then the main function is actually calling the Provider function from the schema in terraform sdk and initializing the provider.

In the records/provider.go file, we add the code

package records
import (
   "github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema"
)
  
// Provider -
func Provider() *schema.Provider {
   return &schema.Provider{
     ResourcesMap: map[string]*schema.Resource{},
     DataSourcesMap: map[string]*schema.Resource{},
   }
}

Here we are creating a dummy provider which has a ResourcesMap that has resources map and a DataSourcesMap for datasources. For the project, we are not doing anything with Datasources.

For our purpose, we need two RestAPI clients, one is the custom RestAPI to create the DNS Recorc and an Azure client to do Read, Update and Delete functions. So we need information on both of the clients. The custom Rest uses a username and password and the Azure one, uses the Azure cli requirements like client_id, client_secret, subscription_id, tenant_id.

So we first configure the provider.go to get these information. So replace the Provider() function with the following code

// Provider - Initialize all vars for Provider config
func Provider() *schema.Provider {
  return &schema.Provider{
    Schema: map[string]*schema.Schema{
      "custom_client_url": &schema.Schema{
         Type:        schema.TypeString,
         Optional:    false,
         DefaultFunc: schema.EnvDefaultFunc("CUSTOM_HOST", nil),
      },
"username": &schema.Schema{
         Type:        schema.TypeString,
         Optional:    false,
         DefaultFunc: schema.EnvDefaultFunc("CUSTOM_USERNAME", nil),
      },
"password": &schema.Schema{
         Type:        schema.TypeString,
         Optional:    false,
         Sensitive:   true,
         DefaultFunc: schema.EnvDefaultFunc("CUSTOM_PASSWORD", nil),
      },
      "client_id": &schema.Schema{
         Type:        schema.TypeString,
         Optional:    false,
         DefaultFunc: schema.EnvDefaultFunc("ARM_CLIENT_ID", nil),
      },
      "client_secret": &schema.Schema{
        Type:        schema.TypeString,
        Optional:    false,
        Sensitive:   true,
        DefaultFunc: schema.EnvDefaultFunc("ARM_CLIENT_SECRET", nil),
      },
      "tenant_id": &schema.Schema{
        Type:        schema.TypeString,
        Optional:    false,
        Sensitive:   true,
        DefaultFunc: schema.EnvDefaultFunc("ARM_TENANT_SECRET", nil),
      },
"subscription_id": &schema.Schema{
         Type:        schema.TypeString,
         Optional:    false,
         Sensitive:   true,
         DefaultFunc: schema.EnvDefaultFunc("CUSTOM_SUBSCRIPTION_ID", nil),
      },
    },
    // Define the function to call the resource.
    ResourcesMap: map[string]*schema.Resource{},
    DataSourcesMap: map[string]*schema.Resource{},
    ConfigureContextFunc: providerConfigure,
  }
}

Notice that I am adding the variables for Provider config as a variable also as an option to read it as Environment Variables. This is actually a good practice, so that we can inject the secrets like passwords and secrets only when we are running the code. A bit secure, by not committing passwords in Version Control.

The last line ConfigureContextFunc: providerConfigure is something to take notice. Because this is the function that will configure the provider. Just the credentials does not make sense. So we need to somehow work out with the credentials to create a new Client.

Client config

So lets design the client config. Modify the file client/client.go and add the client config

package client
// Client Configuration-
type Client struct {
	CustomHostURL      string
	CustomHTTPClient   *http.Client
	CustomToken        string
	CustomAuth         CustomAuthStruct
	AzHostURL          string
	AzHTTPClient       *http.Client
	TenantID           string
	SubscriptionID     string
	AzToken            string
	AzAuth             AzAuthStruct
}

So we have

• 2 HTTP clients, one for the custom RestAPI and one for Azure RestAPI.
• 2 URLs used for connecting to the RestAPIs
• 2 Tokens to store the bearer tokens.
• 2 Authentication structs to pass the credentials for the respective HTTP clients
• TenantID and Subscription ID for the Azure Client

Lets add more code for defining the datatypes. Add the following in the same file.

// CustomHostURL AzHostURL Default URL, empty one
const CustomHostURL string = ""
const AzHostURL string     = ""

// AzAuthStruct Azure Azuth Credentials
type AzAuthStruct struct {
	ClientID     string
	ClientSecret string
}

// CustomAuthStruct Auth Credentials -
type CustomAuthStruct struct {
	Username string `json:"username"`
	Password string `json:"password"`
}

// CustomAuthResponse Auth Response -
type CustomAuthResponse struct {
	TokenType string `json:"token-type`
	Token     string `json:"token"`
}

// AzAuthResponse Azure Auth Response -
type AzAuthResponse struct {
	TokenType    string `json:"token_type`
	ExpiresIn    string `json:"expires_in"`
	ExtExpiresIn string `json:"ext_expires_in"`
	ExpiresOn    string `json:"expires_on"`
	NotBefore    string `json:"not_before"`
	Resource     string `json:"resource"`
	AccessToken  string `json:"access_token"`
}

So we are defining constants for the URLS. Its not actually required. But if you are working locally with a local RestAPI server, then this would be useful.

Note that there is 2 additional structs, CustomAuthResponse and AzAuthResponse. This is to parse the incoming json response into Go Objects. So its easy to work with. For this to be created, you will need to know what will be the response of the RestAPI call for getting the token, then create the struct. I will also show how to work without structs when there is a JSON response, you can find that later in the article, when I am working with responses.

Now lets add the function to create a new client. Add the following code to the same file

// NewClient Initialize a new Client
func NewClient(custom_client_url, username, password, clientId, clientSecret, tenantId, subID *string) (*Client, error) {

	// Initialize the client
	client := Client{
		CustomHTTPClient: &http.Client{Timeout: 120 * time.Second},
		AzHTTPClient: &http.Client{Timeout: 120 * time.Second},
		// Set Default URLs
		CustomHostURL: CustomHostURL,
		AzHostURL: AzHostURL,
	}

	// Add Host URL for Custom client
	if host != nil {
		client.CustomHostURL = *host
	}

	// Prepare Host URL for Azure Client
	if tenantId != nil {
		client.AzHostURL = fmt.Sprintf("https://login.microsoftonline.com/%s/oauth2/token", *tenantId)
	}

	if subID != nil {
		client.SubscriptionID = *subID
	}
	// If necessary variables are empty then return and error
	if username == nil || password == nil || clientId == nil || clientSecret == nil || tenantId == nil || subID == nil {
		return &client, nil
	}

	// Prepare struct for Cloud Auth Credentials
	client.CustomAuth = CustomAuthStruct{
		Username: *username,
		Password: *password,
	}

	// Get The token for Custom Client
	ctt, err := client.GetCustomClientToken()
	if err != nil {
		return nil, err
	}

	// Set the received token to the CustomClient
	client.CustomToken = ctt.Token

	// Prepare struct for Azure Auth Credentials
	client.AzAuth = AzAuthStruct{
		ClientID:     *clientId,
		ClientSecret: *clientSecret,
	}

	// Get The token for Azure Client
	azt, err := client.GetAzureToken()
	if err != nil {
		return nil, err
	}

	// Set the received token to the Azure Client
	client.AzToken = azt.AccessToken

	// Return the client
	return &client, nil
}

In here, we are:

• Creating a Client Object,
• Setting default HostUrls for the clients
• Checking if vars are defined, if not, then return error
• Preparing the Credentials struct for both clients
• Call functions to get the Bearer Tokens for both clients
• Setting the tokens in the Client Object
• Returning the client

Don’t forget to add the imports on top of the file under package. If you are using some IDE with Go Plugin, then this should be automatically populated for you. If not, then add it manually.

import ( 
    "fmt" 
    "net/http" 
    "time"
)

Okay, I know this is too much to wrap around for now. I will stop here for now, and continue in next part. The next part will be creating functions for Getting the tokens for both clients. Which is basically authentication setup.

Hope you liked it! Please continue here, if you liked this doc