Minio – The secure alternative to Amazon S3 for object storage

Shortly before the Univention Summit, I released the Minio app for the Univention App Center. The development of the app started after Univention introduced their Docker-based apps and Minio, as a single binary that does not need much external configuration, seemed like the perfect test case. As it turned out, it was.

What struck me though is how few people know Minio. During the Univention Summit I talked to various people about our new app, and even though many mentioned using apps that supported Amazon S3 for file storage, most of them had never heard of Minio. A pity, I think, because Minio has quite a few benefits for people who want to keep their data under their own control.

So, What Is Minio?

The short and simplified answer is “It’s like Amazon S3, but hosted locally.”

Minio is an object storage server that implements the same public API as Amazon S3. This means that applications that can be configured to talk to Amazon S3 can also be configured to talk to Minio. An object store, such as Minio, can then be used to store unstructured data such as photos, videos, log files, backups and container/VM images. The size of a single object can range from only a few KB to a maximum of 5TB. Files are organised in so -called “buckets”, which are logical separators for your stored data and need to be passed to your application along with your access key, secret key and the HTTP address of the Minio instance.

 

Great for Handling Big Amounts of Data

At Kopano we are used to juggling with large sets of data. So we know that when you’re running large data sets in distributed setups, you do not always want to have your attachment data (which can account for as much as 90% of your total mailbox size) stored on the same system you are running Kopano on. Also, ever-growing mailboxes tend to cause headaches, since disk storage needs to be extended and replicated to protect against disk failure.

Minio offers an easy way out of this dilemma. On the one hand by decoupling disk storage from your local machines, by making them accessible through a simple HTTP interface. And on the other hand by creating redundant data storage without the need to deal with cluster filesystems.

Easy setup

While the Minio app for UCS simply exposes a local directory through the S3 API, Minio itself can easily be configured to protect against disk failures and data corruption through distributing data between multiple disks and multiple nodes hosting Minio.

But is Minio the only S3 API implementation our there? No, certainly not. There are solutions such as OpenStack Swift or OpenIO that also implement the S3 API and therefore provide similar benefits, but being a single go binary Minio is most likely the easiest to set up.

 

How Can I Use Minio e.g. For My Kopano Data?

To use Minio with Kopano the following variables need to be set (before installing Kopano Core):

ucr set \
kopano/cfg/server/attachment_storage=s3 \
kopano/cfg/server/attachment_s3_hostname=minio.ucs.intranet \
kopano/cfg/server/attachment_s3_region=us-east-1 \
kopano/cfg/server/attachment_s3_protocol=https \
kopano/cfg/server/attachment_s3_uristyle=path \
kopano/cfg/server/attachment_s3_accesskeyid=accessKey \
kopano/cfg/server/attachment_s3_secretaccesskey=secretKey \
kopano/cfg/server/attachment_s3_bucketname=bucket \
kopano/cfg/server/attachment_path=attachments

With these set Kopano will automatically use Minio for attachments. Converting an existing installation is possible as well, but slightly more complicated. Please contact the Kopano support for an offer on how to convert your current installation from disk-based attachments to S3 attachments.

What other uses are there for S3?

Apart from functioning as a file storage backend for other applications, S3 can also be utilised for long-term storage of backup data. One such backup application that can leverage S3 for backup storage (among others) is „restic“. Restic is written in Go as well and can do file-based backups as well as backups piped to it from the stdout of e.g. mysqldump. To make backups to S3 even easier with restic Github user binarybucks created the so-called „restic-tools“. I‘ve been using the combination of Minio, restic and restic-tools for a few weeks to backup various systems, among them a Univention setup with Kopano installed. For the Univention system I am using the following pre-execution script for restic-tools:

echo "running kopano-backup"
mkdir -p /backup/bricklevel/ && cd /backup/bricklevel
kopano-backup -l INFO --skip-junk --skip-deleted -w 2 --differential
echo "backing up Kopano MySQL database"
mysqldump --defaults-extra-file=/etc/mysql/debian.cnf --single-transaction kopano | \
backup minio backup --tag mysql --stdin --stdin-filename kopano.sql

So now a simple „/usr/local/bin/backup minio local“ creates a full backup of the Univention filesystem and additionally triggers „kopano-backup“ and a MySQL dump before doing so.

Should I now migrate all my apps to use Minio/S3 for file storage?

Amazon S3 is the gold standard when it comes to backend file storage for cloud applications. Tools like Minio provide the same easy application access (simple HTTP requests) to your apps, without giving up control over your data. But does that mean you should now migrate all your applications to use S3? Probably not, but this also greatly depends on the overall architecture of your IT infrastructure. For example, when you are already using Amazon S3 for storage within your applications, then it’s quite easy to replace the external dependency on Amazon with a self-hosted Minio instance and move existing buckets to your Minio installation through the command line tools provided by Minio.

Still not sure? Just try Minio out yourself via the Univention App Center.

Test Minio