Amazon S3: configuration and management

Key scenarios of working with S3 include backups storage and distribution of static files. Moreover, automation is welcomed. Amazon S3 usage

File synchronization

For files uploading to S3, synchronization and object management you can use all existing clients, web console, plug-ins, command line utilities and REST API. In the simplest case it is sufficient to have AWS CLI for simple management of buckets and objects:

$ aws s3 sync s3://test-bucket /usr/local/test-bucket/ --metadata-directive REPLACE --expires "Wed, 7 Jun 2017 08:16:32 GMT" --cache-control "max-age=2592000"

# File synchronization with storage, caching and age

Another method — a simple bash command:

file=/path/to/file/to/upload
bucket=your-bucket
resource="/${bucket}/${file}"
contentType="application/x-compressed-tar"
dateValue=`date -R`
stringToSign="PUT\n\n${contentType}\n${dateValue}\n${resource}"
s3Key=xxxxxxxxxxxxxxxxxxxx
s3Secret=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
signature=`echo -en ${stringToSign} | openssl sha1 -hmac ${s3Secret} -binary | base64`
curl -X PUT -T "${file}" \
  -H "Host: ${bucket}.aws-region.amazonaws.com" \
  -H "Date: ${dateValue}" \
  -H "Content-Type: ${contentType}" \
  -H "Authorization: AWS ${s3Key}:${signature}" \
  https://${bucket}.aws-region.amazonaws.com/${file}

# Don’t forget to specify the path to the file, name of the bucket and AWS region

This example can be easily converted into a bash-script that will create backups of necessary files and upload them to S3:

#!/bin/bash

cd /tmp
rm -rf backup
mkdir backup
cd backup

mkdir nginx && cd nginx
cp -R /etc/nginx/sites-enabled .
cp /etc/nginx/nginx.conf .

cd ..
mkdir git && cd git
repos=`ls -1 /home/git | grep '.git$'`
for repo in $repos; do
    cp -R "/home/git/${repo}" .
done    

cd ..
date=`date +%Y%m%d`
bucket=my-bucket
for dir in git nginx; do
    file="${date}-${dir}.tar.gz"
    cd $dir && tar czf $file *
    resource="/${bucket}/${file}"
    contentType="application/x-compressed-tar"
    dateValue=`date -R`
    stringToSign="PUT\n\n${contentType}\n${dateValue}\n${resource}"
    s3Key=xxxxxxxxxxxxxxxxxxxx
    s3Secret=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
    signature=`echo -en ${stringToSign} | openssl sha1 -hmac ${s3Secret} -binary | base64`
    curl -X PUT -T "${file}" \
        -H "Host: ${bucket}.s3.amazonaws.com" \
        -H "Date: ${dateValue}" \
        -H "Content-Type: ${contentType}" \
        -H "Authorization: AWS ${s3Key}:${signature}" \
        https://${bucket}.s3.amazonaws.com/${file}
    cd ..
done

cd
rm -rf /tmp/backup

# Uploading backups of Nginx configuration file and the Git repository

PHP SDK

In order to use PHP for uploading the backups, you need to install the SDK for Amazon AWS. PHP and Composer should be installed, too:

curl -sS https://getcomposer.org/installer | php # downloading from official site


# installing latest version of SDK
php composer.phar require aws/aws-sdk-php

# Composer is recommended, but not required

And then you need to enable the autoloader:

<?php
require 'vendor/autoload.php';

# Don’t forget to add your own scripts

Optionally can be installed through Phar or ZIP-file:

 # for Phar
<?php
require '/path/to/aws.phar';
 # for zip
require '/path/to/aws-autoloader.php';

# It is necessary to extract the files and include them into scripts

First you need to configure the S3 client:

<?php

# Включить SDK при помощи Composer
require 'vendor/autoload.php';

$s3 = new Aws\S3\S3Client([
    'version' => 'latest',
    'region'  => 'us-east-1'
]);

# Credentials are better specified with environment variable or the ini-file in AWS directory

SDK allows you to use classes for applying common configs between different clients:

 # region and client version
$sharedConfig = [
    'region'  => 'us-west-2',
    'version' => 'latest'
];

# creating SDK class
$sdk = new Aws\Sdk($sharedConfig);


# creating Amazon S3 client with shared configuration
$client = $sdk->createS3();

# Shared options for all clients located in the root in the form of key/value

To perform operations it is necessary to call a method in the client's name:

 # using Aws\Sdk class 
$s3Client = $sdk->createS3();


# request and output of PutObject
$result = $s3Client->putObject([
    'Bucket' => 'my-bucket',
    'Key'    => 'my-key',
    'Body'   => 'this is the body!'
]);


# object upload
$result = $s3Client->getObject([
    'Bucket' => 'my-bucket',
    'Key'    => 'my-key'
]);


# result
echo $result['Body'];

# SDK uses a configuration file that corresponds to the correct version

Deleting the bucket in its simplest form looks like this:

$result = $client->deleteBucket([
    'Bucket' => 'test-bucket',
]);

# The bucket must be empty

A deletion of objects is as follows:

use Aws\S3\S3Client;

$s3 = S3Client::factory();

$bucket = 'test-bucket';
$keyname = 'object-key';

$result = $s3->deleteObject(array(
    'Bucket' => $bucket,
    'Key'    => $keyname
));

# Don’t forget to specify the object key

Everything else the system supports asynchronous and HTTP requests.

Given the opportunities of SDK, backups uploading to S3 is easily implemented:

putenv('AWS_ACCESS_KEY_ID=Your_key_ID');
putenv('AWS_SECRET_ACCESS_KEY=Your_access_ID');
putenv('S3_BUCKET=bucket-name');
$s3 = new Aws\S3\S3Client(['version' => '2006-03-01', 'region' => 'eu-central-1', 'signature_version' => 'v4']);
$s3->upload('backup', 'path' . date('Y_m_d') . '.gz', fopen($dump, 'rb'));

# Specifying login data through environment variables, versions and region

The lifecycle of the objects

To automatically delete objects of a certain age, the easiest way is to use web console Amazon AWS, indicating the expiration of the required files. If you need automation, you can use the SDK script or S3 console that will run periodically.

There’s also lifecycle policy. It’s a form (in the XML), a set of rules consisting of the ID, status (on/off), types of objects to which a rule can be used, transfer and expiration of objects.

It looks like this:

<LifecycleConfiguration>
    <Rule>
        <ID>sample-rule</ID>
        <Prefix></Prefix>
        <Status>Enabled</Status>
        <Transition>Disabled</Transition>    
        <Expiration>
             <Days>365</Days>
        </Expiration>
    </Rule>
</LifecycleConfiguration>

# All objects with the age of more than 365 days will be removed

This set of rules is applied to the desired bucket (recorded in the properties of the bucket):

$ aws s3api put-bucket-lifecycle  \
--bucket bucketname  \
--lifecycle-configuration filename-containing-lifecycle-configuration

# It is necessary to specify the bucket name and XML file name, that is in the local directory

AWS web console is an easy tool for creating policies for moving and deleting objects.

You can also put the rules into JSON-file:

{
    "Rules": [
        {
            "Status": "Enabled",
            "Prefix": "logs/",
            "Expiration": {
                "ExpiredObjectDeleteMarker": true
            },
            "ID": "TestOnly"
        }
    ]
}

# It will apply to all log files

You can then automatically apply the necessary rules for new objects using a simple script that contains the following command:

$ aws s3api put-bucket-lifecycle  \
--bucket bucketname  \
--lifecycle-configuration file://lifecycle.json

# The principle can be used for checking and removing bucket rules

S3 and Nginx

Nginx can distribute static files that are stored on S3. Amazon S3 and Nginx

Just edit the configuration file, including S3 in the location section:

location / {
  set $s3_bucket        “BUCKET.s3.amazonaws.com';
  set $aws_access_key   'AWSAccessKeyId=ACCESS_KEY';
  set $url_expires      'Expires=$arg_e';
  set $url_signature    'Signature=$arg_st';
  set $url_full         '$1?$aws_access_key&$url_expires&$url_signature';
  proxy_http_version    1.1;
  proxy_set_header       Host $s3_bucket;
  proxy_set_header       Authorization '';
  proxy_hide_header      x-amz-id-2;
  proxy_hide_header      x-amz-request-id;
  proxy_hide_header      Set-Cookie;
  proxy_ignore_headers   "Set-Cookie";
  proxy_buffering        off;
  proxy_intercept_errors on;
  resolver              172.16.0.23 valid=300s;
  resolver_timeout      10s;
  proxy_pass             http://$s3_bucket/$url_full;
}

# Be sure to include your buckets and access keys

The most important

Automate all upload and managing tasks of Amazon S3. The presence of the SDK, the console utility and API will allow using all the features of the repository.

Подпишитесь на Хайлоад с помощью Google аккаунта
или закройте эту хрень