Categories

MinIO Appliance on GIG’s Cloud as a Service with Terraform and Ansible

Created On
Last Updated On
byAhmed Elsayed
You are here:
  • Main
  • IaC with GIG
  • MinIO Appliance on GIG's Cloud as a Service with Terraform and Ansible
< All Topics

Introduction

This tutorial is meant to show how S3 compatible MinIO storage can be easily deployed on top of GIG's Cloud-as-a-Service using tools like Terraform and Ansible.

Architecture

Steps discussed in this tutorial aim to configure the following infrastructure:

  • Cloudspace (CS)
  • Virtual Machine (VM) parameters:
    • Boot disk size: 10 GB2020-05-14 09:53:37 Thursday
    • RAM: 8 GB
    • Number of CPU's: 4
    • Data disks: 4 disks of 2TB, IOPS: 4000
    • Port forwards: 22, 80, 443
  • MinIO server is running on a local port :9091, sharing data over data disks of the VM.
  • Traefik reverse proxy is configured to forward requests to the MinIO instance. Traefik server is listening on ports :80 and :433. Secured https connection is provided by Let's Encrypt.

Architecture

Prerequisites

  • Compatible Terraform and G8 provider binaries. For this tutorial we used:
    • Terraform version: 0.11.14
    • G8 provider version: 1.0.0
  • Access to one of the G8 accounts
  • JWT token

For detailed information on how to generate JWT access token, install and use Terraform to deploy your infrastructure on a G8 please check this tutorial.

Step 1 - Deploy Virtual Machine with Terraform

See below Terraform configuration files needed to deploy a CS with a VM according to the requirements. In terraform.tfvars replace values of variables with your own data:

  • server_url - URL of the G8 you are using
  • account - name of the existing account
  • userdata - user access to the VM. Use this variable to public SSH key to the root user as shown in this example. Make sure this SSH key is loaded in the ssh-agent on the local machine that you use to trigger deployment. Use any of existent SSH keys or generate a new SSH key. Adding your SSH key for the root user is required for Terraform and Ansible to connect to the VM over SSH and proceed with deployment steps.
  • image_name - name of the boot image of the VM. Make sure that the image with provided name exists on the G8 and is accessible for your account. Note that image_name field supports regular expressions, for example provided in form "(?i).*ubuntu.*16.04" will search for all images having the combination of ubuntu and 16.04 in their names. If more than one image meets the requirements, set the the argument most_recent = true in the image data source to select the latest image only.
terraform.tfvars
    
server_url = "https://be-g8-demo.gig.tech"
account = "your_account"
cloudspace = "minio"
machine = "minio-appliance"
machine_description = "MinIO behind Traefik reverse proxy"
memory = 8192
vcpus = 4
boot_disksize = 20
data_disksize = 2000
iops = 4000
nr_data_disks = 4
image_name = "Ubuntu Server 16.04 - x86_64"
userdata = "users: [{name: root, shell: /bin/bash, ssh-authorized-keys: [YOUR SSH KEY]}]"

variables.tf contains definition of all variables used in the config.

variables.tf
    
variable "server_url" {
  description = "API server URL"
}
variable "client_jwt" {
  description = "jwt token"
}
variable "account" {
  description = "account"
}
variable "cloudspace" {
  description = "cloudspace name"
}
variable "machine" {
  description = "cloudspace name"
}
variable "machine_description" {
  description = "Description of the VM"
}
variable "image_name" {
  description = "Image name or a regex string to much image name"
}
variable "memory" {
  description = "memory provisioned for the VM"
  default     = 8192
}
variable "vcpus" {
  description = "number of CPUs provisioned for the VM"
  default     = 4
}
variable "boot_disksize" {
  description = "bootdisk size"
  default     = 20
}
variable "data_disksize" {
  description = "datadisk size"
  default     = 2000
}
variable "iops" {
  description = "IOPS of data disks"
  default     = 2000
}
variable "nr_data_disks"{
  description = "Number of data disks"
  default = 1
}
variable "userdata" {
  description = "user data"
}

main.tf defines resources of various types. G8 resources ovc_machine, ovc_disk, ovc_portforward are implemented in ovc provider. null_resource is a standard resource defined by Terraform that allows uploading files and running commands and scripts on the managed infrastructure. output block is intended to print public IP address of the deployed CS for the sake of convenience.

Number of data disks is set to 4, which is a minimum number of shards required for erasure coding, see more on erasure coding in MinIO documentation. When configuring your own infrastructure note that the number of shards should be even.

main.tf
    
provider "ovc" {
    server_url = "${var.server_url}"
    client_jwt="${var.client_jwt}"
}
resource "ovc_cloudspace" "cs" {
  account = "${var.account}"
  name = "${var.cloudspace}"
}

data "ovc_image" "ubuntu"{
    most_recent = true
    name_regex = "${var.image_name}"
}
resource "ovc_machine" "minio" {
    cloudspace_id = "${ovc_cloudspace.cs.id}"
    name          = "${var.machine}"
    image_id      = "${data.ovc_image.ubuntu.id}"
    disksize      = "${var.boot_disksize}"
    memory        = "${var.memory}"
    vcpus         = "${var.vcpus}"
    userdata      = "${var.userdata}"
    description   = "${var.machine_description}"
}
resource "ovc_port_forwarding" "ssh" {
    cloudspace_id = "${ovc_cloudspace.cs.id}"
    public_ip     = "${ovc_cloudspace.cs.external_network_ip}"
    machine_id    = "${ovc_machine.minio.id}"
    public_port   = 22
    local_port    = 22
    protocol      = "tcp"
}
resource "ovc_port_forwarding" "https" {
    cloudspace_id = "${ovc_cloudspace.cs.id}"
    public_ip     = "${ovc_cloudspace.cs.external_network_ip}"
    public_port   = 80
    machine_id    = "${ovc_machine.minio.id}"
    local_port    = 80
    protocol      = "tcp"
}
resource "ovc_port_forwarding" "ssl" {
    cloudspace_id = "${ovc_cloudspace.cs.id}"
    public_ip     = "${ovc_cloudspace.cs.external_network_ip}"
    public_port   = 443
    machine_id    = "${ovc_machine.minio.id}"
    local_port    = 443
    protocol      = "tcp"
}
resource "ovc_disk" "minio" {
  machine_id = "${ovc_machine.minio.id}"
  disk_name = "Data-Disk-${count.index + 1}"
  description = "S3Disk ${abs(count.index + 1)}"
  type = "D"
  iops = "${var.iops}"
  size = "${var.data_disksize}"
  count = "${var.nr_data_disks}"
  depends_on = ["ovc_machine.minio"]
}
resource "null_resource" "mount_disks" {
  connection {
    user     = "root"
    host     = "${ovc_port_forwarding.ssh.*.public_ip[count.index]}"
    port     = "${ovc_port_forwarding.ssh.*.public_port[count.index]}"
  }
   provisioner "remote-exec" {
    inline = [
        "mkfs -t ext4 /dev/vdb",
        "mkfs -t ext4 /dev/vdc",
        "mkfs -t ext4 /dev/vdd",
        "mkfs -t ext4 /dev/vde",
        "mkdir /mnt/vdb",
        "mkdir /mnt/vdc",
        "mkdir /mnt/vdd",
        "mkdir /mnt/vde",
        "blkid -s UUID | sed -e s/\\\"//g -e s/://g -e s/dev/mnt/g | awk '{print($2 \"\t\" $1 \"\text4\")}' | tail -n +2 >> /etc/fstab",
        "mount -a"
    ]
   }
   depends_on = ["ovc_disk.minio"]
}
output "ip" {
  value = "${ovc_cloudspace.cs.external_network_ip}"
}

The configuration files should be gathered in a directory dedicated for this specific Terraform config, for example in the project working directory

mkdir ~/minio-appliance && cd ~/minio-appliance

From inside the directory initialize and then apply the Terraform configuration:

terraform init
terraform apply -parallelism=1

Flag -parallelism=1 limits the number of parallel resource operations to 1. The flag is used here to avoid adding disks in parallel, which is not supported by G8s.

If the configuration is correct, you will see the list of requested resources and a prompt waiting for confirmation. If everything is correct type yes in the prompt to trigger the actual deployment.

Once deployment succeeded Terraform will report all added resources and print outputs, in this case the IP address of the CS.

Apply complete! Resources: 10 added, 0 changed, 0 destroyed.

Outputs:
ip = 185.15.201.98

If you want to use secured https connection, please register a domain name the IP address of your CS. This tutorial shows how to use Let's Encrypt certificate authority, so lets assume registered domain is minio-appliance-example.com.

Step 2 - Install and Configure Ansible

Ansible version used in this tutorial: 2.7.10.

If you are not familiar with Ansible, check out this video tutorial and official user guide. Installation process for various OS's is described in this Installation Guide.

Steps to install Ansible on the control machine for Ubuntu/Debian:

$ sudo apt-get update
$ sudo apt-get install software-properties-common
$ sudo apt-add-repository ppa:ansible/ansible
$ sudo apt-get update
$ sudo apt-get install ansible

IP addresses or domains of the target remote machines should be listed in the Ansible inventory. Lets create an inventory file hosts in the project directory and add domain of the MinIO appliance. Inventory can also include necessary variables:

  • ansible_python_interpreter defines path to the python binary
hosts
    
[servers]
minio-appliance-example.com ansible_user=root
[all:vars]
ansible_python_interpreter=/usr/bin/python3

At this point, Ansible is configured and we can move on to deployment of specific packages.

Step 3 - Install MinIO

To learn more about the MinIO S3 compatible object storage get familiar with MinIO Documentation.

Install ansible-playbook for MinIO:

ansible-galaxy install atosatto.minio

Define a MinIO configuration yaml file in the Ansible working directory:

minio.yaml
    
- name: "Install Minio"
  hosts: servers
  roles:
    - atosatto.minio
  vars:
    minio_server_datadirs:
      - "/mnt/vdb"
      - "/mnt/vdc"
      - "/mnt/vdd"
      - "/mnt/vde"
    minio_access_key: ""
    minio_secret_key: ""

For more configuration options see in atosatto.minio playbook repository.

Deploy a MinIO instance

ansible-playbook -i hosts minio.yaml

Once playbook execution is finished a MinIO instance is running on the port 9091 - the default port defined in the playbook.

Step 4 - Install Traefik

Install ansible-playbook for Traefik:

ansible-galaxy install fastgeert.ansible_traefik

Add the Traefik playbook configuration to the working directory:

traefik.yaml
    
- hosts: all
  roles:
     - { role: fastgeert.ansible_traefik}
  vars:
    traefik_binary_url: https://github.com/containous/traefik/releases/download/v1.7.11/traefik_linux-amd64
    traefik_static_config: |
      logLevel = "INFO"
      defaultEntryPoints = ["http", "https"]
      [traefikLog]
      [accessLog]
      [entryPoints]
        [entryPoints.http]
        address = ":80"
        [entryPoints.http.redirect]
        entryPoint = "https"
        [entryPoints.https]
        address = ":443"
          [entryPoints.https.tls]
      [acme]
      email = "example@domain.com"
      storage = "acme.json"
      onDemand = true
      caServer = "https://acme-v02.api.letsencrypt.org/directory"
      entryPoint = "https"
        [acme.httpChallenge]
        entryPoint = "http"
      [file]
      watch = true
      [frontends]
        [frontends.frontend1]
        backend = "backend1"
        passHostHeader = true
          [frontends.frontend1.routes.test_1]
          rule = "Host:minio-appliance-example.com"
      [backends]
        [backends.backend1]
          [backends.backend1.servers.server1]
          url = "http://127.0.0.1:9091"
          weight = 10

traefik_static_config allows to pass the Traefik config file. Traefik is configured as a reverse proxy to listen on ports :80 and :443 and forward calls to the MinIO server running on local port :9091, the certificates for secure https connection are issued by the Let's Encrypt certificate authority. For more configuration options see fastgeert.ansible_traefik playbook repo.

Deploy a Traefik server by providing the inventory file and Traefik configuration to the ansible-playbook command

ansible-playbook -i hosts traefik.yaml

Once the playbook execution is finished, the Traefik server is running on the ports 80 and 443.

Step 5 - Communicate with MinIO Appliance: MinIO Client

Use MinIO client to communicate with the MinIO appliance. For more information on the MinIO client see official documentation. Follow the steps described on the MinIO client download page to install a MinIO client appropriate to your system.

Add credentials of the MinIO server to the client config. In Ubuntu default config file is ~/.mc/config.json. Set accessKey and secretKey to the values chosen in Step 3.

config.json
    
{
  "version": "9",
  "hosts": {
    "minio-server": {
      "url": "https://minio-appliance-example.com",
      "accessKey": "",
      "secretKey": "",
      "api": "s3v2",
      "lookup": "auto"
    }
  }
}

As a usage example let us now create a bucket, upload an image from the local machine to the MinIO object storage and get a sharable link to the image with MinIO client.

# create bucket
$ mc mb minio-server/appliance
Bucket created successfully `minio-server/appliance`.

# upload file to MinIO
$ mc cp ~/Downloads/bionic-server-cloudimg-amd64.ova minio-server/appliance
...oudimg-amd64.ova:  96.57 MiB / 312.51 MiB ┃▓▓▓▓▓▓▓▓▓▓▓▓▓▓█░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░┃  30.90% 13.05 MiB/s 00m16s

# get sharable download link
$mc share download minio-server/appliance/bionic-server-cloudimg-amd64.ova

URL: https://minio-appliance-example.com/appliance/bionic-server-cloudimg-amd64.ova
Expire: 7 days 0 hours 0 minutes 0 seconds
Share: https://minio-appliance-example.com/appliance/bionic-server-cloudimg-amd64.ova?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=<>Expires=604800&X-Amz-SignedHeaders=host&X-Amz-Signature=<>

var copyButtonHtml = '

';
$(function () {
var counter = 0;
$.each($("div.code-block div:first-child"), function (key, value) {
var copyButton = $(copyButtonHtml)[0];
counter ++;
var id = "blockid" + counter;
$("button", copyButton).attr("data-clipboard-target", "#"+id);
$("code", value.parentElement).attr("id", id);
value.append(copyButton);
})
new ClipboardJS('.btn');
});

Destroy MinIO Appliance

The entire infrastructure can be easily destroyed with Terraform. Run the following command in Terraform configuration directory

terraform destroy -parallelism=1

Note that the flag -parallelism=1 should be provided to avoid detaching disks from the VM in parallel, which is not supported by G8s.

Table of Contents
© 2020 GIG.tech - Cloud as a Service.