I started, for a professional project, to look at how Terraform plugins work so that I could create custom resources using the same Infrastructure as Code base that we already use to provision resources for open-source providers like Kafka.

This time, it's about creating our own provider that will interact with an API.

Be careful though, this need appears especially when you want to create and update quasi-static resources that will only require manual action of updating via code and re-provisioning.

How a Terraform plugin works

As described in the diagram above, the provider plugins (or provisioners) communicate with the core of Terraform via gRPC, but this is abstracted by a library that is simple to understand and use.

Each plugin then communicates with its client library, e.g. the Amazon Web Services provider plugin communicates with the AWS API, the GitHub provider communicates with the GitHub API, etc... It will therefore be necessary to create a plugin to communicate in any way with your service.

• A provider allows you to create, manage or update resources such as virtual machines on AWS, for example,
• A provisioner can be used to define specific actions on the local machine or on a remote machine to prepare a resource: for example, running a command on a virtual machine you are creating.

The idea is that each provider can be easily installed on an environment. Terraform plugins are therefore written (like Terraform itself) in Go language, so that you have a simple binary to install in your ~/.terraformd/plugins directory in order to start using a plugin.

As stated on the Providers documentation page, the naming of your binary must be in the format terraform-provider-<NAME>_vX.Y.Z, so they are clearly identified and versioned.

So before writing our Go code, we can already write the Makefile that will build our binary for our tests and place it in this directory with the correct name:

build-dev:
    @[ "${version}" ] || ( echo ">> please provide version=vX.Y.Z"; exit 1 )
    go build -o ~/.terraform.d/plugins/terraform-provider-myprovider_${version} .

.PHONY: build-dev

So we will use the following syntax to build our binary and make it ready to use:

$ make build-dev version=v0.0.1

Writing the provider plugin

Let's now start to write our provider's code by creating a main.go file that will instantiate the provider plugin that we will name myprovider and by respecting the interfaces provided by the Terraform plugin SDK:

package main

import (
	"github.com/hashicorp/terraform-plugin-sdk/plugin"
	"github.com/eko/terraform-provider-myprovider/internal/myprovider"
)

func main() {
	plugin.Serve(&plugin.ServeOpts{
		ProviderFunc: myprovider.Provider,
	})
}

Provider Definition

Now let's define the myprovider package by creating the internal/myprovider/provider.go file:

package myprovider

import (
	"github.com/hashicorp/terraform-plugin-sdk/helper/schema"
	"github.com/hashicorp/terraform-plugin-sdk/terraform"
)

// Provider returns a schema.Provider for my provider
func Provider() terraform.ResourceProvider {
	p := &schema.Provider{
		Schema: map[string]*schema.Schema{
			"url": {
				Type:        schema.TypeString,
				Required:    true,
				DefaultFunc: schema.EnvDefaultFunc("MYPROVIDER_URL", nil),
				Description: "URL of my provider service API.",
			},
		},

		DataSourcesMap: map[string]*schema.Resource{},

		ResourcesMap: map[string]*schema.Resource{
			"myprovider_query": resourceQuery(),
		},
	}

	p.ConfigureFunc = providerConfigure(p)

	return p
}

func providerConfigure(p *schema.Provider) schema.ConfigureFunc {
	return func(d *schema.ResourceData) (interface{}, error) {
		url := d.Get("url").(string),
		client := NewClient(url)

		return client, nil
	}
}

The Provider structure provided by the SDK asks us to declare the following resources:

• Schema: corresponding to the schema of the provider block, here we provide the URL of the API which will allow to access our service,
• ResourcesMap: these are the resources that it will be possible to manage via the provider, so here we will be able to create, modify or delete query resources,
• DataSourcesMap: this is the data of the resources you will be able to retrieve, in this example I didn't provide any but the logic is the same as for resources,
• ConfigureFunc: a function to initialize the provider, as we see in the example, we return here an interface{} which will simply be our HTTP client that will allow our resources to communicate with our API. This client will be provided as an argument to the resources.

For more information, I invite you to have a look at the different types available on this structure: https://pkg.go.dev/github.com/hashicorp/terraform-plugin-sdk/helper/schema?tab=doc#Provider

Defining a resource

Finally, let's create the function to define the different actions available on our query' resource:

package persistedqueries

import (
	"context"
	"fmt"
	"log"
	"time"

	"github.com/davecgh/go-spew/spew"
	"github.com/hashicorp/terraform-plugin-sdk/helper/schema"
	"github.com/hashicorp/terraform-plugin-sdk/helper/validation"
)

func resourceQuery() *schema.Resource {
	return &schema.Resource{
		Create: resourceQueryCreate,
		Read:   resourceQueryRead,
		Update: resourceQueryUpdate,
		Delete: resourceQueryDelete,

		Timeouts: &schema.ResourceTimeout{
			Create: schema.DefaultTimeout(10 * time.Minute),
			Update: schema.DefaultTimeout(10 * time.Minute),
			Delete: schema.DefaultTimeout(10 * time.Minute),
		},

		Schema: map[string]*schema.Schema{
			"query_id": {
				Type:         schema.TypeString,
				Required:     true,
				ValidateFunc: validation.NoZeroValues,
			},

			"payload": {
				Type:         schema.TypeString,
				Required:     true,
				ValidateFunc: validation.NoZeroValues,
			},
		},
	}
}

Again, we follow the model provided by the SDK on [Resource] structure (https://pkg.go.dev/github.com/hashicorp/terraform-plugin-sdk/helper/schema?tab=doc#Resource).

Here we define several methods that we will use, depending on the state of the state, to create, update, delete or simply read (to refresh the state) our query resource.

The schema of our resource, which will be named myprovider_query is composed of the following attributes in this example:

• query_id: a query id,
• payload: the payload we wish to assign to this query.

Note that both are here of type string but you can of course implement the type of your choice. You are also free to implement a custom validation on each attribute of your resource.

In order to make our resource work, all you have to do is declare the function code. To lighten this article, I will simply show you the resourceQueryCreate function:

func resourceQueryCreate(d *schema.ResourceData, meta interface{}) error {
        client := meta.(*Client)

        ctx, cancel := context.WithCancel(context.Background())
        defer cancel()

        queryId := d.Get("query_id").(string)
        payload := d.Get("payload").(string)

        log.Printf("[INFO] create a myprovider query for identifier '%s'", queryId)
        log.Printf("[TRACE] query id: '%s', payload: %v", queryId, spew.Sdump(payload))

        returnedId, err := client.Create(ctx, queryId, payload)
        if err != nil {
                return fmt.Errorf("error creating a myprovider query: %v", err)
        }

        d.SetId(returnedId)

        log.Printf("[INFO] myprovider query created")

        return resourceQueryRead(d, meta)
}

As we have seen before, we first get the result of our ConfigureFunc, our HTTP client.

Then, we get the attributes of our resource and simply call a Create(ctx, queryId, payload) method on our HTTP client which will take care of interacting with the API of the service our provider supports.

If you get an error, just return the error and if all goes well, the important thing is to define via d.SetId() a unique identifier for your resource.

This ID will then be stored in the state, and on the creation and update methods, you can simply return nil or like me, chain to the resourceQueryRead() method which will confirm that your resource has been created.

Use our provider!

Your provider is finished, all you have to do is use it, to do so, install it as seen before by running the Make make build-dev version=v0.0.1 command, then, define a main.tf file in a new directory.

You should then be able to use your:

provider "myprovider" {
  url = "http://my-provider-api.svc.local"
}

resource "myprovider_query" "a_sample_query" {
  query_id = "my-sample-query"
  payload = <<-EOT
  {
    "hello": "world"
  }
  EOT
}

To fix your provider on a particular version, add:

terraform {
  required_providers {
    myprovider = "~> 0.0.1"
  }
}

Conclusion

With just a few lines of code, we have just created a plugin provider for Terraform.

The simplicity and flexibility of the SDK really allows an excellent integration with the tool.

If you have any questions on the subject, don't hesitate to contact me by mail or on Twitter.