Getting Started with Oracle Cloud Backups - Amazon Web Services (AWS)
This is part 2 of a 3 part series on "Getting Started with Oracle Cloud Backups".
- Part 1 covers setting up RMAN to backup directly to the new Oracle Cloud Database Backup Service (ODBS).
- Part 2 covers setting up RMAN to backup directly to the cloud using Amazon Web Services (AWS) Simple Storage Service (S3).
- Part 3 compares and contrasts the two services.
Background
As mentioned in
part 1 of this series, it's actually really easy to get started with writing Oracle backups directly to the cloud. So regardless of the motivation, it's certainly possible to go from zero to fully backed-up to a cloud based service in just a few hours, hence meeting any requirements to implement off-site backups extremely quickly. Of course, there are several dependencies including the database size, internet upload speed (and cost), and change control formalities. Recovery speed is most certainly going to be slow compared to recovering from local on-premise storage, but this service probably isn't meant to replace your on-premise backup strategy, but rather compliment it or possibly replace your off-site tape-vaulting vendor service (at least for Oracle database backups). Regardless, recovery time and RTO must of course also be taken into consideration. Also, while the Amazon Web Services metered usage costs are extremely competitive, directly integrating with RMAN does require the
Oracle Secure Backup Cloud Module, which is a licensed option. However, Amazon does also provide some additional unique features with their S3 storage: namely object versioning, object life-cycle rules, the ability to pick the storage region with more granularity (i.e. a specific region within the United States) and "Cross-Region Replication". This article provides a quick start for getting up and running with Oracle RMAN backups to AWS S3 using the Oracle Secure Backup (OSB) Cloud Module for users new to the service and cloud backups.
Backing up to Amazon Web Services (AWS)
Backing up to the AWS Simple Storage Service or S3 isn't new. I first experimented with this in June 2011 and hadn't really touched it since. Yet, fundamentally it hasn't really changed at all since that time. From a very high level the process is:
- Create an AWS S3 account and setup the necessary credentials.
- Install an AWS specific Oracle Secure Backup library into your Oracle Home.
- Run an RMAN backup using the SBT_TAPE device type.
Really that's it. And the first two steps are pretty quick and easy. The 3rd is the one that will vary depending on the size or your database and your upload bandwidth.
Creating an AWS S3 Account
Creating the necessary AWS S3 account is almost trivially easy. If you or your organization does not already have an "Amazon Web Services account", start by
signing up for the Free Trial. The free trial gives you 5GB of storage (along with about 20 other services) for a full year. For almost all organizations the 5GB of storage probably won't be sufficient. However, Amazon does provide a simple to use
billing calculator to try to estimate your monthly costs based on usage. Amazon also provides other benefits, such as being able to choose what data centers or geographic regions your data is stored in. Further details can be found in their
FAQ. After filling in the necessary information and creating an account (if your organization does not already have one), the next step is to set up a security credential. In the top right corner of your S3 console navigate to "Security Credentials":
From the "Users" option select "Create New Users":
Enter a user name and do check the check box to generate an access key - this is what will be used by RMAN, not the actual username:
Once added, remember to record display (and maybe temporarily record or use the "Download Credentials" button in the bottom right) the "Access Key ID" and "Secret Access Key", as you'll need this information during the setup. Don't worry, if you forget or loose this information, it's very easy to generate new security credentials for this user. Or, to delete credentials as I have for the one shown in the screenshots. (It's conceptually similar to the "single use password" you can generate with a Google Account with their 2-factor authentication enabled.)
Before the new credential can be used, it must be given permissions though a "Policy". Just like within the Oracle database, the required permission (or "policy") can be granted directly or through a role allowing for flexible management. From "Users" click on the newly created username and then the "Attach Policy" button:
For RMAN to write to the S3 storage, the "AmazonS3FullAccess" policy will be required. After selected choose the "Attach Policy" button in the bottom right. At this point, we're ready to start configuring the database and using the service.
Installing the "Oracle Secure Backup Cloud Module for Amazon S3"
Installing the necessary "Oracle Secure Backup Cloud Module for Amazon S3" into your Oracle home is very easy. From OTN download an installer
Java JAR file and copy and extract the zip to your database server. When run, the installer will determine the proper database version and OS platform, as well as download the appropriate library file to your Oracle home or other specified directory. Installation requires a few mandatory arguments, namely:
- The AWS credentials created previously
- Your OTN username (but not your Oracle account password)
- The location for the library file it will download. Usually use $ORACLE_HOME/lib
- The location for the secure wallet file which stores the AWS credentials
There are a number of other optional arguments (specified in the README or by running the JAR file without arguments) such as proxy server details if necessary. Example installation:
$ java -jar osbws_install.jar \
> -AWSID AKI***************QA \
> -AWSKey no/MD*******************************upxK \
> -otnUser pane@pythian.com \
> -walletDir $ORACLE_HOME/dbs/osbws_wallet \
> -libDir $ORACLE_HOME/lib
Oracle Secure Backup Web Service Install Tool, build 2015-06-22
AWS credentials are valid.
S3 user already registered.
Registration ID: 17d*****-0***-4***-8***-41e******ccc
S3 Logging Bucket: oracle-log-pane-1
Validating log bucket location ...
Validating license file ...
Oracle Secure Backup Web Service wallet created in directory /u01/app/oracle/product/12.1.0/dbhome_2/dbs/osbws_wallet.
Oracle Secure Backup Web Service initialization file /u01/app/oracle/product/12.1.0/dbhome_2/dbs/osbwsCDB121.ora created.
Downloading Oracle Secure Backup Web Service Software Library from file osbws_linux64.zip.
Downloaded 27151475 bytes in 40 seconds. Transfer rate was 678786 bytes/second.
Download complete.
This determines the appropriate Oracle Secure Backup library file for your OS platform and database version and downloads it to the specified location (recommended $ORACLE_HOME/lib) and creates a config file and the wallet file:
$ ls -ltr $ORACLE_HOME/lib | tail -1
-rw-r--r--. 1 oracle oinstall 86629108 Sep 3 09:05 libosbws.so
$ ls -ltr $ORACLE_HOME/dbs | tail -1
-rw-r--r--. 1 oracle oinstall 204 Sep 3 09:04 osbwsCDB121.ora
$ cat $ORACLE_HOME/dbs/osbwsCDB121.ora
OSB_WS_HOST=https://s3.amazonaws.com
OSB_WS_LICENSE_ID=17d*****-0***-4***-8***-41e******ccc
OSB_WS_WALLET='location=file:/u01/app/oracle/product/12.1.0/dbhome_2/dbs/osbws_wallet CREDENTIAL_ALIAS=pane_aws'
$ ls -l $ORACLE_HOME/dbs/osbws_wallet
total 12
-rw-r--r--. 1 oracle oinstall 10228 Sep 3 09:04 cwallet.sso
At this point we're ready to backup directly to the AWS S3 cloud.
Using with RMAN
Sending the RMAN backup pieces to the AWS S3 storage is as simple as backing via the normal RMAN commands but to the SBT_TAPE device. Of course the new OSB library file and configuration file to use must be specified. For example we can backup in a single run block without over-riding any of our existing RMAN configuration:
backup device type SBT_TAPE tablespace users;
RMAN> run {
2> allocate channel aws_s3 type sbt
3> parms='SBT_LIBRARY=libosbws.so,SBT_PARMS=(OSB_WS_PFILE=/u01/app/oracle/product/12.1.0/dbhome_2/dbs/osbwsCDB121.ora)';
4> backup tablespace users;
5> }
allocated channel: aws_s3
channel aws_s3: SID=21 device type=SBT_TAPE
channel aws_s3: Oracle Secure Backup Web Services Library VER=3.15.7.14
Starting backup at 03-SEP-15
channel aws_s3: starting full datafile backup set
channel aws_s3: specifying datafile(s) in backup set
input datafile file number=00006 name=/u01/app/oracle/oradata/CDB121/users01.dbf
channel aws_s3: starting piece 1 at 03-SEP-15
channel aws_s3: finished piece 1 at 03-SEP-15
piece handle=0aqg7f8h_1_1 tag=TAG20150903T095737 comment=API Version 2.0,MMS Version 3.15.7.14
channel aws_s3: backup set complete, elapsed time: 00:00:15
Finished backup at 03-SEP-15
Starting Control File and SPFILE Autobackup at 03-SEP-15
piece handle=c-3847224663-20150903-01 comment=API Version 2.0,MMS Version 3.15.7.14
Finished Control File and SPFILE Autobackup at 03-SEP-15
released channel: aws_s3
RMAN>
And to verify:
RMAN> list backup of tablespace users;
List of Backup Sets
===================
------- ---- -- ---------- ----------- ------------ ---------------
6 Full 1.34M DISK 00:00:00 03-SEP-15
BP Key: 6 Status: AVAILABLE Compressed: NO Tag: TAG20150903T094342
Piece Name: /u01/app/oracle/product/12.1.0/dbhome_2/dbs/06qg7eee_1_1
List of Datafiles in backup set 6
File LV Type Ckp SCN Ckp Time Name
---- -- ---- ---------- --------- ----
6 Full 1366827 03-SEP-15 /u01/app/oracle/oradata/CDB121/users01.dbf
BS Key Type LV Size Device Type Elapsed Time Completion Time
------- ---- -- ---------- ----------- ------------ ---------------
8 Full 1.50M SBT_TAPE 00:00:02 03-SEP-15
BP Key: 8 Status: AVAILABLE Compressed: NO Tag: TAG20150903T095453
Handle: 08qg7f3d_1_1 Media: s3.amazonaws.com/oracle-data-pane-1
List of Datafiles in backup set 8
File LV Type Ckp SCN Ckp Time Name
---- -- ---- ---------- --------- ----
6 Full 1367173 03-SEP-15 /u01/app/oracle/oradata/CDB121/users01.dbf
RMAN>
Notice that the first backup was a local one run earlier and shows a local backup piece (file). The second shows that the media was "s3.amazonaws.com". The "oracle-data-pane-1" is the "bucket" or logical container automatically created within the Amazon S3. If we want to make the backup command automatically use the AWS S3 SBT_TAPE it's trivial to do using the RMAN CONFIGURE command:
RMAN> configure channel device type sbt parms='SBT_LIBRARY=libosbws.so,
2> SBT_PARMS=(OSB_WS_PFILE=/u01/app/oracle/product/12.1.0/dbhome_2/dbs/osbwsCDB121.ora)';
using target database control file instead of recovery catalog
new RMAN configuration parameters:
CONFIGURE CHANNEL DEVICE TYPE 'SBT_TAPE' PARMS 'SBT_LIBRARY=libosbws.so,SBT_PARMS=(OSB_WS_PFILE=/u01/app/oracle/product/12.1.0/dbhome_2/dbs/osbwsCDB121.ora)';
new RMAN configuration parameters are successfully stored
RMAN> backup device type sbt tablespace users;
Starting backup at 03-SEP-15
released channel: ORA_DISK_1
allocated channel: ORA_SBT_TAPE_1
channel ORA_SBT_TAPE_1: SID=16 device type=SBT_TAPE
channel ORA_SBT_TAPE_1: Oracle Secure Backup Web Services Library VER=3.15.7.14
channel ORA_SBT_TAPE_1: starting full datafile backup set
channel ORA_SBT_TAPE_1: specifying datafile(s) in backup set
input datafile file number=00006 name=/u01/app/oracle/oradata/CDB121/users01.dbf
channel ORA_SBT_TAPE_1: starting piece 1 at 03-SEP-15
channel ORA_SBT_TAPE_1: finished piece 1 at 03-SEP-15
piece handle=0eqg7ft3_1_1 tag=TAG20150903T100834 comment=API Version 2.0,MMS Version 3.15.7.14
channel ORA_SBT_TAPE_1: backup set complete, elapsed time: 00:00:15
Finished backup at 03-SEP-15
Starting Control File and SPFILE Autobackup at 03-SEP-15
piece handle=c-3847224663-20150903-03 comment=API Version 2.0,MMS Version 3.15.7.14
Finished Control File and SPFILE Autobackup at 03-SEP-15
RMAN>
And really that's all there is to it. Of course you can perform more advanced RMAN commands such as allocating multiple channels, etc. And we're free to perform any combination of local backups and/or cloud backups to the SBT_TAPE device type:
RMAN> list backup summary;
List of Backups
===============
Key TY LV S Device Type Completion Time #Pieces #Copies Compressed Tag
------- -- -- - ----------- --------------- ------- ------- ---------- ---
1 B F A SBT_TAPE 01-SEP-15 1 1 NO TAG20150901T123222
2 B F A SBT_TAPE 01-SEP-15 1 1 NO TAG20150901T123222
3 B F A SBT_TAPE 01-SEP-15 1 1 NO TAG20150901T123222
4 B F A SBT_TAPE 01-SEP-15 1 1 NO TAG20150901T123222
5 B F A SBT_TAPE 01-SEP-15 1 1 NO TAG20150901T150814
6 B F A DISK 03-SEP-15 1 1 NO TAG20150903T094342
7 B F A DISK 03-SEP-15 1 1 NO TAG20150903T094343
8 B F A SBT_TAPE 03-SEP-15 1 1 NO TAG20150903T095453
9 B F A SBT_TAPE 03-SEP-15 1 1 NO TAG20150903T095737
10 B F A SBT_TAPE 03-SEP-15 1 1 NO TAG20150903T095752
11 B F A DISK 03-SEP-15 1 1 NO TAG20150903T100555
12 B F A DISK 03-SEP-15 1 1 NO TAG20150903T100557
13 B F A SBT_TAPE 03-SEP-15 1 1 NO TAG20150903T100834
14 B F A SBT_TAPE 03-SEP-15 1 1 NO TAG20150903T100850
And obviously restoring works exactly the same way.
IMPORTANT: Note here that none of the backups are encrypted or compressed. Not having to encrypt backups is the first major functional difference from the Oracle Database Backup Service (OSDB) where encryption in mandatory. Either encryption and/or compression can be used. Normally, backup encryption requires the Oracle Advanced Security Option however one exemption to that is when
backing up using Oracle Secure Backup. Similarly, "basic" RMAN backup compression is included with the database without any additional licenses. However the additional compression options (such as "HIGH", "MEDIUM", or "LOW") usually do require the Oracle Advanced Compression option but are included as "Special-use licensing" of the
Oracle Secure Backup product. Compressing prior to backup is generally highly recommended. While this will consume local CPU cycles, it will minimize transfer time through the internet and S3 space used.
Advanced Configuration
Generally, the base parameters should be sufficient. However, if further customization is required there are a number of optional parameters that can be added to the configuration/initialization file (in this example "$ORACLE_HOME/dbs/osbwsCDB121.ora"). An easy way to look for the available parameters including hidden parameters is to search through the OSB Cloud Module library file. For example:
$ strings $ORACLE_HOME/lib/libosbws.so |grep OSB_
OSB_WS_PFILE
OSB_WS_HOST
OSB_WS_PROXY
OSB_WS_BUCKET
OSB_WS_LOCATION
OSB_WS_CHUNK_SIZE
OSB_WS_LICENSE_ID
OSB_WS_LICENSE_MAX_SESSIONS
OSB_WS_WALLET
OSB_WS_VIRTUAL_HOST
OSB_WS_PRIVATE_CLOUD
_OSB_WS_ALLOCBUF_DISABLE
_OSB_WS_ACCESSLOG_RETENTION_TIME
_OSB_WS_SEND_BUF_SIZE
_OSB_WS_RECV_BUF_SIZE
_OSB_WS_SDU_SIZE
_OSB_WS_TDU_SIZE
_OSB_WS_REUSE_CONNECTION
_OSB_WS_SESSION_RETENTION_TIME
_OSB_WS_CHECKER_RETENTION_TIME
_OSB_WS_PURGE_LIMIT
_OSB_WS_CLEANER
_OSB_WS_CLEANER_RETENTION_TIME
_OSB_WS_RUN_CLEANER
_OSB_WS_ACCESSLOG_CLEANER
_OSB_WS_ACCESSLOG
_OSB_WS_CONNECT_TIMEOUT
_OSB_WS_RESPONSE_TIMEOUT
_OSB_WS_SEND_TIMEOUT
_OSB_WS_RETRY_WAIT_TIME
_OSB_WS_UPLOAD_DELAY
_OSB_WS_EVENT
_OSB_WS_TRACE_LEVEL
_OSB_WS_VALIDATE_CHUNK
_OSB_WS_100_CONTINUE
_OSB_WS_DEFERRED_DELETE
_OSB_WS_AUTH_SCHEME
_OSB_WS_LOG_BUCKET
_OSB_WS_REDIRECT
_OSB_WS_BUFFER_WRITE
_OSB_WS_BUFFER_READ
_OSB_WS_USE_IPV6
_OSB_WS_NO_SSL
_OSB_WS_VALIDATE_CERT
Secure Transfer
Similar to how backups using the OSB cloud module do not require encryption, they also transfer the data
without SSL security by default. This is apparent from a simple Linux netstat command while the backup is running:
$ netstat | grep http
tcp 0 299300 ORALINUX.localdom:46151 s3-1-w.amazonaws.c:http ESTABLISHED
However we can easily remedy this by changing the URL in the configuration/initialization file to use the "https" address:
$ head -1 $ORACLE_HOME/dbs/osbwsCDB121.ora
OSB_WS_HOST=https://s3.amazonaws.com
After adjusting, starting a new RMAN session (critical) and re-running the same backup command now shows a secure SSL (or HTTPS) connection:
$ netstat | grep http
tcp 0 299300 ORALINUX.localdom:34048 s3-1-w.amazonaws.:https ESTABLISHED
tcp 0 0 ORALINUX.localdom:25700 s3-1-w.amazonaws.:https TIME_WAIT
One would think that the undocumented parameter "_OSB_WS_NO_SSL" (which is a boolean accepting only TRUE or FALSE as possible values) might also effect which connection is used though experimentation showed no effect.
Viewing Usage
Viewing the files stored and data usage on the AWS S3 at first is not as intuitive as one might think. From the AWS dashboard selecting S3 shows the "buckets" (logical storage containers) and allows you to drill down into each to see the actual files stored:
However, properties of the bucket such as the number of files and total size is not displayed. To find this information you must navigate to the "Billing and Cost Management" account option:
From here we can see the usage and the associated costs and other relevant details of the account:
Conclusion
Just like with the Oracle Database Backup Service, getting going with the Oracle Secure Backup Cloud Module and Amazon Web Services is very simple. It really can be implemented quickly and easily allowing for RMAN backups to be written directly to off-site storage almost immediately. However unlike OSDB, the AWS OSB module is not secure by default. With Oracle's service, the data must be encrypted and the transfer is secured by default and hence they can live up to their claim that
your data is secure in flight and at rest. However with the AWS OSB module, by default the reverse is true. Though of course, both HTTPS data transfer and backup encryption can be enabled if desired. On a positive note, the ability to generate AWS ID and Key values (or "credentials") specific for each backup configuration can be considered a security advantage. Apart from the security differences, functionally the two services are almost identical as would be expected. After all, the OSDB module appears to simply be a clone of the OSBWS module introduced about half a decade ago. The one big caveat of the AWS OSB module is the licensing cost. The Oracle Secure Backup Cloud module is licensed per RMAN channel. Channel based licenses can be shared among multiple databases however. And overall the S3 storage service is more functional and mature than Oracle's newer public cloud service. AWS includes some nice features with respect to geographic data placement and replication. The bottom line is that the Oracle Secure Backup Cloud Module is a great technical solution for implementing off-site Oracle RMAN backups (likely to compliment your on-premise backups) and specifically putting them onto the AWS S3. Functionally it seems great. However, the additional licensing costs is likely the road block for most small to mid-sized organizations.
Additional References
Discover more about our expertise in Oracle and Cloud.
