MinIO Appliance on GIG’s Cloud as a Service with Terraform and Ansible
Introduction
This tutorial is meant to show how S3 compatible MinIO storage can be easily deployed on top of GIG's Cloud-as-a-Service using tools like Terraform and Ansible.
Architecture
Steps discussed in this tutorial aim to configure the following infrastructure:
- Cloudspace (CS)
- Virtual Machine (VM) parameters:
- Boot disk size: 10 GB2020-05-14 09:53:37 Thursday
- RAM: 8 GB
- Number of CPU's: 4
- Data disks: 4 disks of 2TB, IOPS: 4000
- Port forwards: 22, 80, 443
- MinIO server is running on a local port
:9091
, sharing data over data disks of the VM. - Traefik reverse proxy is configured to forward requests to the MinIO instance. Traefik server is listening on ports
:80
and:433
. Securedhttps
connection is provided by Let's Encrypt.
Prerequisites
- Compatible Terraform and G8 provider binaries. For this tutorial we used:
- Terraform version: 0.11.14
- G8 provider version: 1.0.0
- Access to one of the G8 accounts
- JWT token
For detailed information on how to generate JWT access token, install and use Terraform to deploy your infrastructure on a G8 please check this tutorial.
Step 1 - Deploy Virtual Machine with Terraform
See below Terraform configuration files needed to deploy a CS with a VM according to the requirements. In terraform.tfvars
replace values of variables with your own data:
server_url
- URL of the G8 you are usingaccount
- name of the existing accountuserdata
- user access to the VM. Use this variable to public SSH key to theroot
user as shown in this example. Make sure this SSH key is loaded in the ssh-agent on the local machine that you use to trigger deployment. Use any of existent SSH keys or generate a new SSH key. Adding your SSH key for theroot
user is required for Terraform and Ansible to connect to the VM over SSH and proceed with deployment steps.image_name
- name of the boot image of the VM. Make sure that the image with provided name exists on the G8 and is accessible for your account. Note thatimage_name
field supports regular expressions, for example provided in form"(?i).*ubuntu.*16.04"
will search for all images having the combination ofubuntu
and 16.04 in their names. If more than one image meets the requirements, set the the argumentmost_recent = true
in the image data source to select the latest image only.
server_url = "https://be-g8-demo.gig.tech"
account = "your_account"
cloudspace = "minio"
machine = "minio-appliance"
machine_description = "MinIO behind Traefik reverse proxy"
memory = 8192
vcpus = 4
boot_disksize = 20
data_disksize = 2000
iops = 4000
nr_data_disks = 4
image_name = "Ubuntu Server 16.04 - x86_64"
userdata = "users: [{name: root, shell: /bin/bash, ssh-authorized-keys: [YOUR SSH KEY]}]"
variables.tf
contains definition of all variables used in the config.
variable "server_url" {
description = "API server URL"
}
variable "client_jwt" {
description = "jwt token"
}
variable "account" {
description = "account"
}
variable "cloudspace" {
description = "cloudspace name"
}
variable "machine" {
description = "cloudspace name"
}
variable "machine_description" {
description = "Description of the VM"
}
variable "image_name" {
description = "Image name or a regex string to much image name"
}
variable "memory" {
description = "memory provisioned for the VM"
default = 8192
}
variable "vcpus" {
description = "number of CPUs provisioned for the VM"
default = 4
}
variable "boot_disksize" {
description = "bootdisk size"
default = 20
}
variable "data_disksize" {
description = "datadisk size"
default = 2000
}
variable "iops" {
description = "IOPS of data disks"
default = 2000
}
variable "nr_data_disks"{
description = "Number of data disks"
default = 1
}
variable "userdata" {
description = "user data"
}
main.tf
defines resources of various types. G8 resources ovc_machine
, ovc_disk
, ovc_portforward
are implemented in ovc provider. null_resource
is a standard resource defined by Terraform that allows uploading files and running commands and scripts on the managed infrastructure. output
block is intended to print public IP address of the deployed CS for the sake of convenience.
Number of data disks is set to 4, which is a minimum number of shards required for erasure coding, see more on erasure coding in MinIO documentation. When configuring your own infrastructure note that the number of shards should be even.
provider "ovc" {
server_url = "${var.server_url}"
client_jwt="${var.client_jwt}"
}
resource "ovc_cloudspace" "cs" {
account = "${var.account}"
name = "${var.cloudspace}"
}
data "ovc_image" "ubuntu"{
most_recent = true
name_regex = "${var.image_name}"
}
resource "ovc_machine" "minio" {
cloudspace_id = "${ovc_cloudspace.cs.id}"
name = "${var.machine}"
image_id = "${data.ovc_image.ubuntu.id}"
disksize = "${var.boot_disksize}"
memory = "${var.memory}"
vcpus = "${var.vcpus}"
userdata = "${var.userdata}"
description = "${var.machine_description}"
}
resource "ovc_port_forwarding" "ssh" {
cloudspace_id = "${ovc_cloudspace.cs.id}"
public_ip = "${ovc_cloudspace.cs.external_network_ip}"
machine_id = "${ovc_machine.minio.id}"
public_port = 22
local_port = 22
protocol = "tcp"
}
resource "ovc_port_forwarding" "https" {
cloudspace_id = "${ovc_cloudspace.cs.id}"
public_ip = "${ovc_cloudspace.cs.external_network_ip}"
public_port = 80
machine_id = "${ovc_machine.minio.id}"
local_port = 80
protocol = "tcp"
}
resource "ovc_port_forwarding" "ssl" {
cloudspace_id = "${ovc_cloudspace.cs.id}"
public_ip = "${ovc_cloudspace.cs.external_network_ip}"
public_port = 443
machine_id = "${ovc_machine.minio.id}"
local_port = 443
protocol = "tcp"
}
resource "ovc_disk" "minio" {
machine_id = "${ovc_machine.minio.id}"
disk_name = "Data-Disk-${count.index + 1}"
description = "S3Disk ${abs(count.index + 1)}"
type = "D"
iops = "${var.iops}"
size = "${var.data_disksize}"
count = "${var.nr_data_disks}"
depends_on = ["ovc_machine.minio"]
}
resource "null_resource" "mount_disks" {
connection {
user = "root"
host = "${ovc_port_forwarding.ssh.*.public_ip[count.index]}"
port = "${ovc_port_forwarding.ssh.*.public_port[count.index]}"
}
provisioner "remote-exec" {
inline = [
"mkfs -t ext4 /dev/vdb",
"mkfs -t ext4 /dev/vdc",
"mkfs -t ext4 /dev/vdd",
"mkfs -t ext4 /dev/vde",
"mkdir /mnt/vdb",
"mkdir /mnt/vdc",
"mkdir /mnt/vdd",
"mkdir /mnt/vde",
"blkid -s UUID | sed -e s/\\\"//g -e s/://g -e s/dev/mnt/g | awk '{print($2 \"\t\" $1 \"\text4\")}' | tail -n +2 >> /etc/fstab",
"mount -a"
]
}
depends_on = ["ovc_disk.minio"]
}
output "ip" {
value = "${ovc_cloudspace.cs.external_network_ip}"
}
The configuration files should be gathered in a directory dedicated for this specific Terraform config, for example in the project working directory
mkdir ~/minio-appliance && cd ~/minio-appliance
From inside the directory initialize and then apply the Terraform configuration:
terraform init
terraform apply -parallelism=1
Flag -parallelism=1
limits the number of parallel resource operations to 1. The flag is used here to avoid adding disks in parallel, which is not supported by G8s.
If the configuration is correct, you will see the list of requested resources and a prompt waiting for confirmation. If everything is correct type yes
in the prompt to trigger the actual deployment.
Once deployment has succeeded Terraform will report all added resources and print outputs, in this case the IP address of the CS.
Apply complete! Resources: 10 added, 0 changed, 0 destroyed.
Outputs:
ip = 185.15.201.98
If you want to use secured https
connection, please register a domain name, the IP address of your CS. This tutorial shows how to use Let's Encrypt certificate authority, so let's assume registered domain is minio-appliance-example.com
.
Step 2 - Install and Configure Ansible
Ansible version used in this tutorial: 2.7.10.
If you are not familiar with Ansible, check out this video tutorial and official user guide. Installation process for various OS's is described in this Installation Guide.
Steps to install Ansible on the control machine for Ubuntu/Debian:
$ sudo apt-get update
$ sudo apt-get install software-properties-common
$ sudo apt-add-repository ppa:ansible/ansible
$ sudo apt-get update
$ sudo apt-get install ansible
IP addresses or domains of the target remote machines should be listed in the Ansible inventory. Let's create an inventory file hosts
in the project directory and add domain of the MinIO appliance. Inventory can also include necessary variables:
ansible_python_interpreter
defines path to the python binary
[servers]
minio-appliance-example.com ansible_user=root
[all:vars]
ansible_python_interpreter=/usr/bin/python3
At this point, Ansible is configured and we can move on to deployment of specific packages.
Step 3 - Install MinIO
To learn more about the MinIO S3 compatible object storage get familiar with MinIO Documentation.
Install ansible-playbook for MinIO:
ansible-galaxy install atosatto.minio
Define a MinIO configuration yaml
file in the Ansible working directory:
- name: "Install Minio"
hosts: servers
roles:
- atosatto.minio
vars:
minio_server_datadirs:
- "/mnt/vdb"
- "/mnt/vdc"
- "/mnt/vdd"
- "/mnt/vde"
minio_access_key: ""
minio_secret_key: ""
For more configuration options see atosatto.minio playbook repository.
Deploy a MinIO instance
ansible-playbook -i hosts minio.yaml
Once playbook execution is finished a MinIO instance is running on the port 9091 - the default port defined in the playbook.
Step 4 - Install Traefik
Install ansible-playbook for Traefik:
ansible-galaxy install fastgeert.ansible_traefik
Add the Traefik playbook configuration to the working directory:
- hosts: all
roles:
- { role: fastgeert.ansible_traefik}
vars:
traefik_binary_url: https://github.com/containous/traefik/releases/download/v1.7.11/traefik_linux-amd64
traefik_static_config: |
logLevel = "INFO"
defaultEntryPoints = ["http", "https"]
[traefikLog]
[accessLog]
[entryPoints]
[entryPoints.http]
address = ":80"
[entryPoints.http.redirect]
entryPoint = "https"
[entryPoints.https]
address = ":443"
[entryPoints.https.tls]
[acme]
email = "example@domain.com"
storage = "acme.json"
onDemand = true
caServer = "https://acme-v02.api.letsencrypt.org/directory"
entryPoint = "https"
[acme.httpChallenge]
entryPoint = "http"
[file]
watch = true
[frontends]
[frontends.frontend1]
backend = "backend1"
passHostHeader = true
[frontends.frontend1.routes.test_1]
rule = "Host:minio-appliance-example.com"
[backends]
[backends.backend1]
[backends.backend1.servers.server1]
url = "http://127.0.0.1:9091"
weight = 10
traefik_static_config
allows to pass the Traefik config file. Traefik is configured as a reverse proxy to listen on ports :80
and :443
and forward calls to the MinIO server running on local port :9091
, the certificates for secure https
connection are issued by the Let's Encrypt certificate authority. For more configuration options see fastgeert.ansible_traefik playbook repo.
Deploy a Traefik server by providing the inventory file and Traefik configuration to the ansible-playbook
command
ansible-playbook -i hosts traefik.yaml
Once the playbook execution is finished, the Traefik server is running on the ports 80 and 443.
Step 5 - Communicate with MinIO Appliance: MinIO Client
Use MinIO client to communicate with the MinIO appliance. For more information on the MinIO client see official documentation. Follow the steps described on the MinIO client download page to install a MinIO client appropriate to your system.
Add credentials of the MinIO server to the client config. In Ubuntu default config file is ~/.mc/config.json
. Set accessKey
and secretKey
to the values chosen in Step 3.
{
"version": "9",
"hosts": {
"minio-server": {
"url": "https://minio-appliance-example.com",
"accessKey": "",
"secretKey": "",
"api": "s3v2",
"lookup": "auto"
}
}
}
As a usage example let us now create a bucket, upload an image from the local machine to the MinIO object storage and get a sharable link to the image with MinIO client.
# create bucket
$ mc mb minio-server/appliance
Bucket created successfully `minio-server/appliance`.
# upload file to MinIO
$ mc cp ~/Downloads/bionic-server-cloudimg-amd64.ova minio-server/appliance
...oudimg-amd64.ova: 96.57 MiB / 312.51 MiB ┃▓▓▓▓▓▓▓▓▓▓▓▓▓▓█░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░┃ 30.90% 13.05 MiB/s 00m16s
# get sharable download link
$mc share download minio-server/appliance/bionic-server-cloudimg-amd64.ova
URL: https://minio-appliance-example.com/appliance/bionic-server-cloudimg-amd64.ova
Expire: 7 days 0 hours 0 minutes 0 seconds
Share: https://minio-appliance-example.com/appliance/bionic-server-cloudimg-amd64.ova?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=<>Expires=604800&X-Amz-SignedHeaders=host&X-Amz-Signature=<>
var copyButtonHtml = '
';
$(function () {
var counter = 0;
$.each($("div.code-block div:first-child"), function (key, value) {
var copyButton = $(copyButtonHtml)[0];
counter ++;
var id = "blockid" + counter;
$("button", copyButton).attr("data-clipboard-target", "#"+id);
$("code", value.parentElement).attr("id", id);
value.append(copyButton);
})
new ClipboardJS('.btn');
});
Destroy MinIO Appliance
The entire infrastructure can be easily destroyed with Terraform. Run the following command in Terraform configuration directory
terraform destroy -parallelism=1
Note that the flag -parallelism=1
should be provided to avoid detaching disks from the VM in parallel, which is not supported by G8s.