Part 2: Oracle Cloud Backups to AWS S3

Posted in: Cloud, Oracle, Technical Track

Getting Started with Oracle Cloud Backups – Amazon Web Services (AWS)

This is part 2 of a 3 part series on “Getting Started with Oracle Cloud Backups”.

  • Part 1 covers setting up RMAN to backup directly to the new Oracle Cloud Database Backup Service (ODBS).
  • Part 2 covers setting up RMAN to backup directly to the cloud using Amazon Web Services (AWS) Simple Storage Service (S3).
  • Part 3 compares and contrasts the two services.

Background

As mentioned in part 1 of this series, it’s actually really easy to get started with writing Oracle backups directly to the cloud. So regardless of the motivation, it’s certainly possible to go from zero to fully backed-up to a cloud based service in just a few hours, hence meeting any requirements to implement off-site backups extremely quickly. Of course, there are several dependencies including the database size, internet upload speed (and cost), and change control formalities.

Recovery speed is most certainly going to be slow compared to recovering from local on-premise storage, but this service probably isn’t meant to replace your on-premise backup strategy, but rather compliment it or possibly replace your off-site tape-vaulting vendor service (at least for Oracle database backups). Regardless, recovery time and RTO must of course also be taken into consideration.

Also, while the Amazon Web Services metered usage costs are extremely competitive, directly integrating with RMAN does require the Oracle Secure Backup Cloud Module, which is a licensed option.

However, Amazon does also provide some additional unique features with their S3 storage: namely object versioning, object life-cycle rules, the ability to pick the storage region with more granularity (i.e. a specific region within the United States) and “Cross-Region Replication”.

This article provides a quick start for getting up and running with Oracle RMAN backups to AWS S3 using the Oracle Secure Backup (OSB) Cloud Module for users new to the service and cloud backups.

 

Backing up to Amazon Web Services (AWS)

Backing up to the AWS Simple Storage Service or S3 isn’t new. I first experimented with this in June 2011 and hadn’t really touched it since. Yet, fundamentally it hasn’t really changed at all since that time.

From a very high level the process is:

  1. Create an AWS S3 account and setup the necessary credentials.
  2. Install an AWS specific Oracle Secure Backup library into your Oracle Home.
  3. Run an RMAN backup using the SBT_TAPE device type.

Really that’s it. And the first two steps are pretty quick and easy. The 3rd is the one that will vary depending on the size or your database and your upload bandwidth.

 

Creating an AWS S3 Account

Creating the necessary AWS S3 account is almost trivially easy. If you or your organization does not already have an “Amazon Web Services account”, start by signing up for the Free Trial. The free trial gives you 5GB of storage (along with about 20 other services) for a full year.

For almost all organizations the 5GB of storage probably won’t be sufficient. However, Amazon does provide a simple to use billing calculator to try to estimate your monthly costs based on usage.

Amazon also provides other benefits, such as being able to choose what data centers or geographic regions your data is stored in. Further details can be found in their FAQ.

After filling in the necessary information and creating an account (if your organization does not already have one), the next step is to set up a security credential. In the top right corner of your S3 console navigate to “Security Credentials”:

AWS1

 

From the “Users” option select “Create New Users”:

AWS2

 

Enter a user name and do check the check box to generate an access key – this is what will be used by RMAN, not the actual username:

AWS3

 

Once added, remember to record display (and maybe temporarily record or use the “Download Credentials” button in the bottom right) the “Access Key ID” and “Secret Access Key”, as you’ll need this information during the setup. Don’t worry, if you forget or loose this information, it’s very easy to generate new security credentials for this user. Or, to delete credentials as I have for the one shown in the screenshots. (It’s conceptually similar to the “single use password” you can generate with a Google Account with their 2-factor authentication enabled.)

AWS4

 

Before the new credential can be used, it must be given permissions though a “Policy”. Just like within the Oracle database, the required permission (or “policy”) can be granted directly or through a role allowing for flexible management.

From “Users” click on the newly created username and then the “Attach Policy” button:

AWS5

 

For RMAN to write to the S3 storage, the “AmazonS3FullAccess” policy will be required. After selected choose the “Attach Policy” button in the bottom right.

At this point, we’re ready to start configuring the database and using the service.

 

Installing the “Oracle Secure Backup Cloud Module for Amazon S3”

Installing the necessary “Oracle Secure Backup Cloud Module for Amazon S3” into your Oracle home is very easy.

From OTN download an installer Java JAR file and copy and extract the zip to your database server. When run, the installer will determine the proper database version and OS platform, as well as download the appropriate library file to your Oracle home or other specified directory.

Installation requires a few mandatory arguments, namely:

  • The AWS credentials created previously
  • Your OTN username (but not your Oracle account password)
  • The location for the library file it will download. Usually use $ORACLE_HOME/lib
  • The location for the secure wallet file which stores the AWS credentials

There are a number of other optional arguments (specified in the README or by running the JAR file without arguments) such as proxy server details if necessary.

Example installation:

$ java -jar osbws_install.jar \
>    -AWSID AKI***************QA \
>    -AWSKey no/MD*******************************upxK \
>    -otnUser pane@pythian.com \
>    -walletDir $ORACLE_HOME/dbs/osbws_wallet \
>    -libDir $ORACLE_HOME/lib
Oracle Secure Backup Web Service Install Tool, build 2015-06-22
AWS credentials are valid.
S3 user already registered.
Registration ID: 17d*****-0***-4***-8***-41e******ccc
S3 Logging Bucket: oracle-log-pane-1
Validating log bucket location ...
Validating license file ...
Oracle Secure Backup Web Service wallet created in directory /u01/app/oracle/product/12.1.0/dbhome_2/dbs/osbws_wallet.
Oracle Secure Backup Web Service initialization file /u01/app/oracle/product/12.1.0/dbhome_2/dbs/osbwsCDB121.ora created.
Downloading Oracle Secure Backup Web Service Software Library from file osbws_linux64.zip.
Downloaded 27151475 bytes in 40 seconds. Transfer rate was 678786 bytes/second.
Download complete.

 

This determines the appropriate Oracle Secure Backup library file for your OS platform and database version and downloads it to the specified location (recommended $ORACLE_HOME/lib) and creates a config file and the wallet file:

$ ls -ltr $ORACLE_HOME/lib | tail -1
-rw-r--r--. 1 oracle oinstall  86629108 Sep  3 09:05 libosbws.so

$ ls -ltr $ORACLE_HOME/dbs | tail -1
-rw-r--r--. 1 oracle oinstall      204 Sep  3 09:04 osbwsCDB121.ora

$ cat $ORACLE_HOME/dbs/osbwsCDB121.ora
OSB_WS_HOST=http://s3.amazonaws.com
OSB_WS_LICENSE_ID=17d*****-0***-4***-8***-41e******ccc
OSB_WS_WALLET='location=file:/u01/app/oracle/product/12.1.0/dbhome_2/dbs/osbws_wallet CREDENTIAL_ALIAS=pane_aws'

$ ls -l $ORACLE_HOME/dbs/osbws_wallet
total 12
-rw-r--r--. 1 oracle oinstall 10228 Sep  3 09:04 cwallet.sso

 

At this point we’re ready to backup directly to the AWS S3 cloud.

 

Using with RMAN

Sending the RMAN backup pieces to the AWS S3 storage is as simple as backing via the normal RMAN commands but to the SBT_TAPE device. Of course the new OSB library file and configuration file to use must be specified. For example we can backup in a single run block without over-riding any of our existing RMAN configuration:

backup device type SBT_TAPE tablespace users;

RMAN> run {
2> allocate channel aws_s3 type sbt
3> parms='SBT_LIBRARY=libosbws.so,SBT_PARMS=(OSB_WS_PFILE=/u01/app/oracle/product/12.1.0/dbhome_2/dbs/osbwsCDB121.ora)';
4> backup tablespace users;
5> }

allocated channel: aws_s3
channel aws_s3: SID=21 device type=SBT_TAPE
channel aws_s3: Oracle Secure Backup Web Services Library VER=3.15.7.14

Starting backup at 03-SEP-15
channel aws_s3: starting full datafile backup set
channel aws_s3: specifying datafile(s) in backup set
input datafile file number=00006 name=/u01/app/oracle/oradata/CDB121/users01.dbf
channel aws_s3: starting piece 1 at 03-SEP-15
channel aws_s3: finished piece 1 at 03-SEP-15
piece handle=0aqg7f8h_1_1 tag=TAG20150903T095737 comment=API Version 2.0,MMS Version 3.15.7.14
channel aws_s3: backup set complete, elapsed time: 00:00:15
Finished backup at 03-SEP-15

Starting Control File and SPFILE Autobackup at 03-SEP-15
piece handle=c-3847224663-20150903-01 comment=API Version 2.0,MMS Version 3.15.7.14
Finished Control File and SPFILE Autobackup at 03-SEP-15
released channel: aws_s3

RMAN>

 

And to verify:

RMAN> list backup of tablespace users;

List of Backup Sets
===================

------- ---- -- ---------- ----------- ------------ ---------------
6       Full    1.34M      DISK        00:00:00     03-SEP-15
        BP Key: 6   Status: AVAILABLE  Compressed: NO  Tag: TAG20150903T094342
        Piece Name: /u01/app/oracle/product/12.1.0/dbhome_2/dbs/06qg7eee_1_1
  List of Datafiles in backup set 6
  File LV Type Ckp SCN    Ckp Time  Name
  ---- -- ---- ---------- --------- ----
  6       Full 1366827    03-SEP-15 /u01/app/oracle/oradata/CDB121/users01.dbf

BS Key  Type LV Size       Device Type Elapsed Time Completion Time
------- ---- -- ---------- ----------- ------------ ---------------
8       Full    1.50M      SBT_TAPE    00:00:02     03-SEP-15
        BP Key: 8   Status: AVAILABLE  Compressed: NO  Tag: TAG20150903T095453
        Handle: 08qg7f3d_1_1   Media: s3.amazonaws.com/oracle-data-pane-1
  List of Datafiles in backup set 8
  File LV Type Ckp SCN    Ckp Time  Name
  ---- -- ---- ---------- --------- ----
  6       Full 1367173    03-SEP-15 /u01/app/oracle/oradata/CDB121/users01.dbf

RMAN>

 

Notice that the first backup was a local one run earlier and shows a local backup piece (file). The second shows that the media was “s3.amazonaws.com”. The “oracle-data-pane-1” is the “bucket” or logical container automatically created within the Amazon S3.

If we want to make the backup command automatically use the AWS S3 SBT_TAPE it’s trivial to do using the RMAN CONFIGURE command:

RMAN> configure channel device type sbt parms='SBT_LIBRARY=libosbws.so,
2> SBT_PARMS=(OSB_WS_PFILE=/u01/app/oracle/product/12.1.0/dbhome_2/dbs/osbwsCDB121.ora)';

using target database control file instead of recovery catalog
new RMAN configuration parameters:
CONFIGURE CHANNEL DEVICE TYPE 'SBT_TAPE' PARMS  'SBT_LIBRARY=libosbws.so,SBT_PARMS=(OSB_WS_PFILE=/u01/app/oracle/product/12.1.0/dbhome_2/dbs/osbwsCDB121.ora)';
new RMAN configuration parameters are successfully stored

RMAN> backup device type sbt tablespace users;

Starting backup at 03-SEP-15
released channel: ORA_DISK_1
allocated channel: ORA_SBT_TAPE_1
channel ORA_SBT_TAPE_1: SID=16 device type=SBT_TAPE
channel ORA_SBT_TAPE_1: Oracle Secure Backup Web Services Library VER=3.15.7.14
channel ORA_SBT_TAPE_1: starting full datafile backup set
channel ORA_SBT_TAPE_1: specifying datafile(s) in backup set
input datafile file number=00006 name=/u01/app/oracle/oradata/CDB121/users01.dbf
channel ORA_SBT_TAPE_1: starting piece 1 at 03-SEP-15
channel ORA_SBT_TAPE_1: finished piece 1 at 03-SEP-15
piece handle=0eqg7ft3_1_1 tag=TAG20150903T100834 comment=API Version 2.0,MMS Version 3.15.7.14
channel ORA_SBT_TAPE_1: backup set complete, elapsed time: 00:00:15
Finished backup at 03-SEP-15

Starting Control File and SPFILE Autobackup at 03-SEP-15
piece handle=c-3847224663-20150903-03 comment=API Version 2.0,MMS Version 3.15.7.14
Finished Control File and SPFILE Autobackup at 03-SEP-15

RMAN>

 

And really that’s all there is to it. Of course you can perform more advanced RMAN commands such as allocating multiple channels, etc. And we’re free to perform any combination of local backups and/or cloud backups to the SBT_TAPE device type:

RMAN> list backup summary;

List of Backups
===============
Key     TY LV S Device Type Completion Time #Pieces #Copies Compressed Tag
------- -- -- - ----------- --------------- ------- ------- ---------- ---
1       B  F  A SBT_TAPE    01-SEP-15       1       1       NO         TAG20150901T123222
2       B  F  A SBT_TAPE    01-SEP-15       1       1       NO         TAG20150901T123222
3       B  F  A SBT_TAPE    01-SEP-15       1       1       NO         TAG20150901T123222
4       B  F  A SBT_TAPE    01-SEP-15       1       1       NO         TAG20150901T123222
5       B  F  A SBT_TAPE    01-SEP-15       1       1       NO         TAG20150901T150814
6       B  F  A DISK        03-SEP-15       1       1       NO         TAG20150903T094342
7       B  F  A DISK        03-SEP-15       1       1       NO         TAG20150903T094343
8       B  F  A SBT_TAPE    03-SEP-15       1       1       NO         TAG20150903T095453
9       B  F  A SBT_TAPE    03-SEP-15       1       1       NO         TAG20150903T095737
10      B  F  A SBT_TAPE    03-SEP-15       1       1       NO         TAG20150903T095752
11      B  F  A DISK        03-SEP-15       1       1       NO         TAG20150903T100555
12      B  F  A DISK        03-SEP-15       1       1       NO         TAG20150903T100557
13      B  F  A SBT_TAPE    03-SEP-15       1       1       NO         TAG20150903T100834
14      B  F  A SBT_TAPE    03-SEP-15       1       1       NO         TAG20150903T100850

 

And obviously restoring works exactly the same way.

IMPORTANT: Note here that none of the backups are encrypted or compressed. Not having to encrypt backups is the first major functional difference from the Oracle Database Backup Service (OSDB) where encryption in mandatory.

Either encryption and/or compression can be used. Normally, backup encryption requires the Oracle Advanced Security Option however one exemption to that is when backing up using Oracle Secure Backup. Similarly, “basic” RMAN backup compression is included with the database without any additional licenses. However the additional compression options (such as “HIGH”, “MEDIUM”, or “LOW”) usually do require the Oracle Advanced Compression option but are included as “Special-use licensing” of the Oracle Secure Backup product.

Compressing prior to backup is generally highly recommended. While this will consume local CPU cycles, it will minimize transfer time through the internet and S3 space used.

 

Advanced Configuration

Generally, the base parameters should be sufficient. However, if further customization is required there are a number of optional parameters that can be added to the configuration/initialization file (in this example “$ORACLE_HOME/dbs/osbwsCDB121.ora”). An easy way to look for the available parameters including hidden parameters is to search through the OSB Cloud Module library file. For example:

$ strings $ORACLE_HOME/lib/libosbws.so |grep OSB_
OSB_WS_PFILE
OSB_WS_HOST
OSB_WS_PROXY
OSB_WS_BUCKET
OSB_WS_LOCATION
OSB_WS_CHUNK_SIZE
OSB_WS_LICENSE_ID
OSB_WS_LICENSE_MAX_SESSIONS
OSB_WS_WALLET
OSB_WS_VIRTUAL_HOST
OSB_WS_PRIVATE_CLOUD
_OSB_WS_ALLOCBUF_DISABLE
_OSB_WS_ACCESSLOG_RETENTION_TIME
_OSB_WS_SEND_BUF_SIZE
_OSB_WS_RECV_BUF_SIZE
_OSB_WS_SDU_SIZE
_OSB_WS_TDU_SIZE
_OSB_WS_REUSE_CONNECTION
_OSB_WS_SESSION_RETENTION_TIME
_OSB_WS_CHECKER_RETENTION_TIME
_OSB_WS_PURGE_LIMIT
_OSB_WS_CLEANER
_OSB_WS_CLEANER_RETENTION_TIME
_OSB_WS_RUN_CLEANER
_OSB_WS_ACCESSLOG_CLEANER
_OSB_WS_ACCESSLOG
_OSB_WS_CONNECT_TIMEOUT
_OSB_WS_RESPONSE_TIMEOUT
_OSB_WS_SEND_TIMEOUT
_OSB_WS_RETRY_WAIT_TIME
_OSB_WS_UPLOAD_DELAY
_OSB_WS_EVENT
_OSB_WS_TRACE_LEVEL
_OSB_WS_VALIDATE_CHUNK
_OSB_WS_100_CONTINUE
_OSB_WS_DEFERRED_DELETE
_OSB_WS_AUTH_SCHEME
_OSB_WS_LOG_BUCKET
_OSB_WS_REDIRECT
_OSB_WS_BUFFER_WRITE
_OSB_WS_BUFFER_READ
_OSB_WS_USE_IPV6
_OSB_WS_NO_SSL
_OSB_WS_VALIDATE_CERT

 

Secure Transfer

Similar to how backups using the OSB cloud module do not require encryption, they also transfer the data without SSL security by default.

This is apparent from a simple Linux netstat command while the backup is running:

$ netstat | grep http
tcp        0 299300 ORALINUX.localdom:46151 s3-1-w.amazonaws.c:http ESTABLISHED

 

However we can easily remedy this by changing the URL in the configuration/initialization file to use the “https” address:

$ head -1 $ORACLE_HOME/dbs/osbwsCDB121.ora
OSB_WS_HOST=https://s3.amazonaws.com

 

After adjusting, starting a new RMAN session (critical) and re-running the same backup command now shows a secure SSL (or HTTPS) connection:

$ netstat | grep http
tcp        0 299300 ORALINUX.localdom:34048 s3-1-w.amazonaws.:https ESTABLISHED
tcp        0      0 ORALINUX.localdom:25700 s3-1-w.amazonaws.:https TIME_WAIT

 

One would think that the undocumented parameter “_OSB_WS_NO_SSL” (which is a boolean accepting only TRUE or FALSE as possible values) might also effect which connection is used though experimentation showed no effect.

 

Viewing Usage

Viewing the files stored and data usage on the AWS S3 at first is not as intuitive as one might think. From the AWS dashboard selecting S3 shows the “buckets” (logical storage containers) and allows you to drill down into each to see the actual files stored:

AWS6

 

However, properties of the bucket such as the number of files and total size is not displayed. To find this information you must navigate to the “Billing and Cost Management” account option:

AWS7

 

From here we can see the usage and the associated costs and other relevant details of the account:

AWS8

 

Conclusion

Just like with the Oracle Database Backup Service, getting going with the Oracle Secure Backup Cloud Module and Amazon Web Services is very simple. It really can be implemented quickly and easily allowing for RMAN backups to be written directly to off-site storage almost immediately.

However unlike OSDB, the AWS OSB module is not secure by default. With Oracle’s service, the data must be encrypted and the transfer is secured by default and hence they can live up to their claim that your data is secure in flight and at rest. However with the AWS OSB module, by default the reverse is true. Though of course, both HTTPS data transfer and backup encryption can be enabled if desired.

On a positive note, the ability to generate AWS ID and Key values (or “credentials”) specific for each backup configuration can be considered a security advantage.

Apart from the security differences, functionally the two services are almost identical as would be expected. After all, the OSDB module appears to simply be a clone of the OSBWS module introduced about half a decade ago.

The one big caveat of the AWS OSB module is the licensing cost. The Oracle Secure Backup Cloud module is licensed per RMAN channel. Channel based licenses can be shared among multiple databases however.

And overall the S3 storage service is more functional and mature than Oracle’s newer public cloud service. AWS includes some nice features with respect to geographic data placement and replication.

The bottom line is that the Oracle Secure Backup Cloud Module is a great technical solution for implementing off-site Oracle RMAN backups (likely to compliment your on-premise backups) and specifically putting them onto the AWS S3. Functionally it seems great. However, the additional licensing costs is likely the road block for most small to mid-sized organizations.

 

Additional References

 

Discover more about our expertise in Oracle and Cloud.

email

Interested in working with Simon? Schedule a tech call.

About the Author

Simon describes himself as a technology enthusiast who is passionate about collecting and sharing interesting database tips. If you want to see his eyes light up, let him teach you something new. Based out of Calgary, Alberta, Simon is known for his contributions to various online Oracle communities, and being very thorough in his work. A self-proclaimed stereotypical Canadian, Simon can be found watching hockey with his family in his spare time.

3 Comments. Leave new

Dear Simon Pane
thanks for sharing this S3 storage service is really good for data backup as well as new user.

Reply

Is it possible to retrieve (import) backups from S3 to local disk? (not restore db, just retrieve to local disk, so that it will be available from local disk to RMAN)

Reply

Hi John, I’m not aware of any method that permits that. It doesn’t seem possible to download the pieces from S3 nor to retrieve them through RMAN.

I was thinking that maybe you could do it with the RMAN “BACKUP BACKUPSET” command but on checking the Oracle documentation (https://docs.oracle.com/cd/E11882_01/backup.112/e10643/rcmsynta007.htm#RCMRF107) it says: “You cannot back up from tape to tape or from tape to disk: only from disk to disk or disk to tape” and remember that the OSB module essentially treats the S3 storage as a tape device.

So to the best of my knowledge it can’t be done but maybe someone else will post a suggestion.

Simon

Reply

Leave a Reply

Your email address will not be published. Required fields are marked *