Troubleshooting Oracle Cloud Infrastructure Automatic Backups

Posted in: Cloud, Oracle, Technical Track

I could start this blog the same way I started this one because everything is supposed to be easy and at a 1 click distance in any cloud, so it should be the same for the database backups in Oracle Cloud!

If is it indeed truly easy to set up automatic databases backups in a few clicks:

(if you haven’t manually recreated the database, don’t do it, I did it for you already :))

. . . then troubleshooting is another story . . .

Indeed, you may find this kind of output in your console one day:

And here you are “at 1 click distance” to . . . the documentation. You have no way of opening the backup logfiles in your browser like I would have expected, and the documentation shows how to manually troubleshoot the backups (like on any non-cloud database **but** with the cloud software system – let’s look at how this works!). I can now confirm that vi, grep and friends have not been killed by the cloud :)

First of all, you need to check the databases which are running there. You are supposed to already know, but double-checking with dbcli ensures that your system is healthy from a cloud configuration perspective and that you could go further:

[root@oci_server ~]# dbcli list-databases

ID                                       DB Name    DB Type  DB Version           CDB        Class    Shape    Storage    Status        DbHomeID                                
---------------------------------------- ---------- -------- -------------------- ---------- -------- -------- ---------- ------------ ----------------------------------------
d5b748a3-c9ca-4ad0-87b4-a6228b4a3e57     MYDB122    Si       12.2.0.1.190115      true       Oltp              ASM        Configured   72371d3f-74ee-4925-b0c1-232b5a4c9145    
[root@oci_server ~]#

Okay, now we can have the list of jobs which occurred on this database (I tailed it as this command will list all the jobs, feel free to “grep Failure” if you wish):

[root@oci_server ~]# dbcli list-jobs | grep -i MYDB122 | tail -20
4c3860ac-8cf1-4987-9ef1-fc2e2442ce7a     Create Regular-L0 Backup with TAG-DBTRegular-L01569481521878j3k for Db:MYDB122 in OSS:bYmzpVkMq1BLgxHxZSwc September 26, 2019 7:06:38 AM UTC   Failure   
6f604571-d73c-4c0f-b172-836628fc386d     Create Regular-L0 Backup with TAG-DBTRegular-L01569567793191oB6 for Db:MYDB122 in OSS:bYmzpVkMq1BLgxHxZSwc September 27, 2019 7:05:12 AM UTC   Failure   
be730740-b599-4c9b-8bfe-50d1c3ec50ab     Create Regular-L0 Backup with TAG-DBTRegular-L01569654323853e8o for Db:MYDB122 in OSS:bYmzpVkMq1BLgxHxZSwc September 28, 2019 7:06:54 AM UTC   Failure   
ca020c2f-12d3-435d-aa78-22b3e8718d1d     Create Regular-L0 Backup with TAG-DBTRegular-L0156974066756851B for Db:MYDB122 in OSS:bYmzpVkMq1BLgxHxZSwc September 29, 2019 7:20:54 AM UTC   Failure   
2d00f5a5-5ede-4fd6-b196-b460750e367b     Create Regular-L0 Backup with TAG-DBTRegular-L01569827110982sy7 for Db:MYDB122 in OSS:bYmzpVkMq1BLgxHxZSwc September 30, 2019 7:07:07 AM UTC   Failure   
c8d910ef-1cd3-4310-9636-5bb6ba15cbb1     Create Regular-L0 Backup with TAG-DBTRegular-L01569913536096G7c for Db:MYDB122 in OSS:bYmzpVkMq1BLgxHxZSwc October 1, 2019 7:07:18 AM UTC      Failure   
5b9c15b8-8938-43ba-8e16-db25e8b94407     Create Regular-L0 Backup with TAG-DBTRegular-L01569999888129pmL for Db:MYDB122 in OSS:bYmzpVkMq1BLgxHxZSwc October 2, 2019 7:05:54 AM UTC      Failure   
010909de-bae0-4511-bda9-81d2955cfd75     Create Regular-L0 Backup with TAG-DBTRegular-L01570086276758tz0 for Db:MYDB122 in OSS:bYmzpVkMq1BLgxHxZSwc October 3, 2019 7:21:21 AM UTC      Failure   
5aae3578-7451-4c7c-9071-c643e7f01417     Create Regular-L0 Backup with TAG-DBTRegular-L01570172676836JSm for Db:MYDB122 in OSS:bYmzpVkMq1BLgxHxZSwc October 4, 2019 7:05:40 AM UTC      Success   
ea9c92b4-299e-4a15-93aa-c3f40f290a9b     Delete Backup for Database name: MYDB122_phx1bf                             October 4, 2019 7:18:26 AM UTC      Success   
76d0531a-3599-41ba-b440-4261db78c292     Create Regular-L1 Backup with TAG-DBTRegular-L11570259048404mCf for Db:MYDB122 in OSS:bYmzpVkMq1BLgxHxZSwc October 5, 2019 7:05:49 AM UTC      Success   
4055c4bc-9594-491f-b18c-82c15081cf5c     Delete Backup for Database name: MYDB122_phx1bf                             October 5, 2019 7:54:06 AM UTC      Success   
18424327-f775-40e5-bbdb-f73e03453c0a     Create Regular-L0 Backup with TAG-DBTRegular-L01570345462290fs3 for Db:MYDB122 in OSS:bYmzpVkMq1BLgxHxZSwc October 6, 2019 7:05:28 AM UTC      Success   
5a05e4c7-d857-46f0-9030-5f33f6722860     Delete Backup for Database name: MYDB122_phx1bf                             October 6, 2019 7:49:22 AM UTC      Success   
a7a51004-2da4-452f-a02e-1b6f11afc87a     Create Regular-L1 Backup with TAG-DBTRegular-L11570431880603Lmh for Db:MYDB122 in OSS:bYmzpVkMq1BLgxHxZSwc October 7, 2019 7:15:23 AM UTC      Success   
f0404baf-84cd-4c58-a4cd-5ccbf3bd3ccd     Delete Backup for Database name: MYDB122_phx1bf                             October 7, 2019 8:18:46 AM UTC      Success   
249b81d9-1efd-4494-bf6f-0bfae8acaf76     Create Regular-L1 Backup with TAG-DBTRegular-L11570518241132W91 for Db:MYDB122 in OSS:bYmzpVkMq1BLgxHxZSwc October 8, 2019 7:05:23 AM UTC      Success   
2b019e8a-a466-4084-87d3-03984c037754     Delete Backup for Database name: MYDB122_phx1bf                             October 8, 2019 8:19:44 AM UTC      Success   
93303906-4af4-499e-98b9-067010e05134     Create Regular-L1 Backup with TAG-DBTRegular-L11570604660275VZv for Db:MYDB122 in OSS:bYmzpVkMq1BLgxHxZSwc October 9, 2019 7:05:56 AM UTC      Success   
2f04d01b-1912-43cb-91b0-2dd231013030     Delete Backup for Database name: MYDB122_phx1bf                             October 9, 2019 8:00:09 AM UTC      Success   
[root@oci_server ~]# 

We can see here that some backups are successful and some are not. Let’s check the latest one that failed:

[root@oci_server ~]# dbcli describe-job -i 010909de-bae0-4511-bda9-81d2955cfd75 -j
{
  "jobId" : "010909de-bae0-4511-bda9-81d2955cfd75",
  "status" : "Failure",
  "message" : "DCS-10001:Internal error encountered: Failed to run RMAN command. Please refer log at location : oci_server: /opt/oracle/dcs/log/oci_server/rman/bkup/MYDB122_phx1bf/rman_backup_DBTRegular-L01570086276758tz0_2019-10-03_07-22-07-6121233765598704321.log.",
  "reports" : [ {
    "taskId" : "TaskZLockWrapper_7698",
    "taskName" : "task:TaskZLockWrapper_7698",
    "taskResult" : "DCS-10001:Internal error encountered: Failed to run RMAN command. Please refer log at location : oci_server: /opt/oracle/dcs/log/oci_server/rman/bkup/MYDB122_phx1bf/rman_backup_DBTRegular-L01570086276758tz0_2019-10-03_07-22-07-6121233765598704321.log.",
    "startTime" : "October 03, 2019 07:21:24 AM UTC",
    "endTime" : "October 03, 2019 07:22:21 AM UTC",
    "status" : "Failure",
    "taskDescription" : null,
    "parentTaskId" : "TaskServiceRequest_7697",
    "jobId" : "010909de-bae0-4511-bda9-81d2955cfd75",
    "tags" : [ ],
    "reportLevel" : "Error",
    "updatedTime" : "October 03, 2019 07:22:21 AM UTC"
  }, {
    "taskId" : "TaskSequential_7699",
    "taskName" : "task:TaskSequential_7699",
    "taskResult" : "DCS-10001:Internal error encountered: Failed to run RMAN command. Please refer log at location : oci_server: /opt/oracle/dcs/log/oci_server/rman/bkup/MYDB122_phx1bf/rman_backup_DBTRegular-L01570086276758tz0_2019-10-03_07-22-07-6121233765598704321.log.",
    "startTime" : "October 03, 2019 07:21:24 AM UTC",
    "endTime" : "October 03, 2019 07:22:21 AM UTC",
    "status" : "Failure",
    "taskDescription" : null,
    "parentTaskId" : "TaskZLockWrapper_7698",
    "jobId" : "010909de-bae0-4511-bda9-81d2955cfd75",
    "tags" : [ ],
    "reportLevel" : "Error",
    "updatedTime" : "October 03, 2019 07:22:21 AM UTC"
  }, {
    "taskId" : "TaskParallel_7700",
    "taskName" : "Validate backup config",
    "taskResult" : "",
    "startTime" : "October 03, 2019 07:21:24 AM UTC",
    "endTime" : "October 03, 2019 07:21:24 AM UTC",
    "status" : "Success",
    "taskDescription" : null,
    "parentTaskId" : "TaskSequential_7699",
    "jobId" : "010909de-bae0-4511-bda9-81d2955cfd75",
    "tags" : [ ],
    "reportLevel" : "Info",
    "updatedTime" : "October 03, 2019 07:21:24 AM UTC"
  }, {
    "taskId" : "TaskZJsonRpcExt_7703",
    "taskName" : "Validate opc_pfile configuration",
    "taskResult" : "successfully validated the opc_config",
    "startTime" : "October 03, 2019 07:21:24 AM UTC",
    "endTime" : "October 03, 2019 07:21:24 AM UTC",
    "status" : "Success",
    "taskDescription" : null,
    "parentTaskId" : "TaskSequential_7699",
    "jobId" : "010909de-bae0-4511-bda9-81d2955cfd75",
    "tags" : [ ],
    "reportLevel" : "Info",
    "updatedTime" : "October 03, 2019 07:21:24 AM UTC"
  }, {
    "taskId" : "TaskParallel_7705",
    "taskName" : "Database container validation",
    "taskResult" : "",
    "startTime" : "October 03, 2019 07:21:24 AM UTC",
    "endTime" : "October 03, 2019 07:21:25 AM UTC",
    "status" : "Success",
    "taskDescription" : null,
    "parentTaskId" : "TaskSequential_7699",
    "jobId" : "010909de-bae0-4511-bda9-81d2955cfd75",
    "tags" : [ ],
    "reportLevel" : "Info",
    "updatedTime" : "October 03, 2019 07:21:25 AM UTC"
  }, {
    "taskId" : "TaskParallel_7708",
    "taskName" : "libopc existence check",
    "taskResult" : "",
    "startTime" : "October 03, 2019 07:21:25 AM UTC",
    "endTime" : "October 03, 2019 07:21:25 AM UTC",
    "status" : "Success",
    "taskDescription" : null,
    "parentTaskId" : "TaskSequential_7699",
    "jobId" : "010909de-bae0-4511-bda9-81d2955cfd75",
    "tags" : [ ],
    "reportLevel" : "Info",
    "updatedTime" : "October 03, 2019 07:21:25 AM UTC"
  }, {
    "taskId" : "TaskZJsonRpcExt_7711",
    "taskName" : "Backup Validations",
    "taskResult" : "Successfully validated for database backup",
    "startTime" : "October 03, 2019 07:21:25 AM UTC",
    "endTime" : "October 03, 2019 07:21:46 AM UTC",
    "status" : "Success",
    "taskDescription" : null,
    "parentTaskId" : "TaskSequential_7699",
    "jobId" : "010909de-bae0-4511-bda9-81d2955cfd75",
    "tags" : [ ],
    "reportLevel" : "Info",
    "updatedTime" : "October 03, 2019 07:21:46 AM UTC"
  }, {
    "taskId" : "TaskZJsonRpcExt_7713",
    "taskName" : "Recovery Window validation",
    "taskResult" : "successfully validated recovery window",
    "startTime" : "October 03, 2019 07:21:46 AM UTC",
    "endTime" : "October 03, 2019 07:21:53 AM UTC",
    "status" : "Success",
    "taskDescription" : null,
    "parentTaskId" : "TaskSequential_7699",
    "jobId" : "010909de-bae0-4511-bda9-81d2955cfd75",
    "tags" : [ ],
    "reportLevel" : "Info",
    "updatedTime" : "October 03, 2019 07:21:53 AM UTC"
  }, {
    "taskId" : "TaskZJsonRpcExt_7716",
    "taskName" : "Archivelog deletion policy configuration",
    "taskResult" : "Skipping the Archivelog deletion policy configuration",
    "startTime" : "October 03, 2019 07:21:53 AM UTC",
    "endTime" : "October 03, 2019 07:22:01 AM UTC",
    "status" : "Success",
    "taskDescription" : null,
    "parentTaskId" : "TaskParallel_7715",
    "jobId" : "010909de-bae0-4511-bda9-81d2955cfd75",
    "tags" : [ ],
    "reportLevel" : "Info",
    "updatedTime" : "October 03, 2019 07:22:01 AM UTC"
  }, {
    "taskId" : "TaskZJsonRpcExt_7718",
    "taskName" : "Database backup",
    "taskResult" : "DCS-10001:Internal error encountered: Failed to run RMAN command. Please refer log at location : oci_server: /opt/oracle/dcs/log/oci_server/rman/bkup/MYDB122_phx1bf/rman_backup_DBTRegular-L01570086276758tz0_2019-10-03_07-22-07-6121233765598704321.log.",
    "startTime" : "October 03, 2019 07:22:01 AM UTC",
    "endTime" : "October 03, 2019 07:22:21 AM UTC",
    "status" : "Failure",
    "taskDescription" : null,
    "parentTaskId" : "TaskSequential_7699",
    "jobId" : "010909de-bae0-4511-bda9-81d2955cfd75",
    "tags" : [ ],
    "reportLevel" : "Error",
    "updatedTime" : "October 03, 2019 07:22:21 AM UTC"
  } ],
  "createTimestamp" : "October 03, 2019 07:21:21 AM UTC",
  "resourceList" : [ ],
  "description" : "Create Regular-L0 Backup with TAG-DBTRegular-L01570086276758tz0 for Db:MYDB122 in OSS:bYmzpVkMq1BLgxHxZSwc",
  "updatedTime" : "October 03, 2019 07:22:21 AM UTC"
}
[root@oci_server ~]# 

I am not sure why the Oracle documentation points to use the “-j” option which is to have a JSON output. Indeed, a regular TEXT output looks clearer to me. Maybe because the JSON outputs usually have more information than the TEXT outputs. But for this particular case, the TEXT output has the log file I am looking for which is enough for now:

[root@oci_server ~]# dbcli describe-job -i 010909de-bae0-4511-bda9-81d2955cfd75M   

Job details                                                      
----------------------------------------------------------------
                     ID:  010909de-bae0-4511-bda9-81d2955cfd75
            Description:  Create Regular-L0 Backup with TAG-DBTRegular-L01570086276758tz0 for Db:MYDB122 in OSS:bYmzpVkMq1BLgxHxZSwc
                 Status:  Failure
                Created:  October 3, 2019 7:21:21 AM UTC
                Message:  DCS-10001:Internal error encountered: Failed to run RMAN command. Please refer log at location : oci_server: /opt/oracle/dcs/log/oci_server/rman/bkup/MYDB122_phx1bf/rman_backup_DBTRegular-L01570086276758tz0_2019-10-03_07-22-07-6121233765598704321.log.

Task Name                                Start Time                          End Time                            Status    
---------------------------------------- ----------------------------------- ----------------------------------- ----------
task:TaskZLockWrapper_7698               October 3, 2019 7:21:24 AM UTC      October 3, 2019 7:22:21 AM UTC      Failure   
task:TaskSequential_7699                 October 3, 2019 7:21:24 AM UTC      October 3, 2019 7:22:21 AM UTC      Failure   
Validate backup config                   October 3, 2019 7:21:24 AM UTC      October 3, 2019 7:21:24 AM UTC      Success   
Validate opc_pfile configuration         October 3, 2019 7:21:24 AM UTC      October 3, 2019 7:21:24 AM UTC      Success   
Database container validation            October 3, 2019 7:21:24 AM UTC      October 3, 2019 7:21:25 AM UTC      Success   
libopc existence check                   October 3, 2019 7:21:25 AM UTC      October 3, 2019 7:21:25 AM UTC      Success   
Backup Validations                       October 3, 2019 7:21:25 AM UTC      October 3, 2019 7:21:46 AM UTC      Success   
Recovery Window validation               October 3, 2019 7:21:46 AM UTC      October 3, 2019 7:21:53 AM UTC      Success   
Archivelog deletion policy configuration October 3, 2019 7:21:53 AM UTC      October 3, 2019 7:22:01 AM UTC      Success   
Database backup                          October 3, 2019 7:22:01 AM UTC      October 3, 2019 7:22:21 AM UTC      Failure   

[root@oci_server ~]# 

The above output makes it clear that we’ll find the problem in the /opt/oracle/dcs/log/oci_server/rman/bkup/MYDB122_phx1bf/rman_backup_DBTRegular-L01570086276758tz0_2019-10-03_07-22-07-6121233765598704321.log file.

In this log file, you’ll find all the RMAN logs of the backup including the RMAN code which is used by the automatic backups:

RMAN> set echo on;
2> set command id to "010909de-bae0-4511-bda9-81d2955c";
3> report schema;
4> show all;
5> list incarnation of database;
6> set echo on;
7> set encryption on;
8> backup force device type sbt as compressed backupset incremental level 0 database tag 'DBTRegular-L01570086276758tz0' format 'DBTRegular-L01570086276758tz0_df_%d_%I_%U_%T_%t_set%s' plus archivelog not backed up tag 'DBTRegular-L01570086276758tz0' format 'DBTRegular-L01570086276758tz0_arc_%d_%I_%U_%T_%t_set%s' ;
9> backup device type sbt as compressed backupset current controlfile tag 'DBTRegular-L01570086276758tz0' format 'DBTRegular-L01570086276758tz0_cf_%d_%I_%U_%T_%t_set%s' spfile tag 'DBTRegular-L01570086276758tz0' format 'DBTRegular-L01570086276758tz0_spf_%d_%I_%U_%T_%t_set%s' ;
10> delete force noprompt obsolete;
11> set encryption off;
12>

And the error which was, in this case, due to an FRA full:

Starting backup at 2019/10/03 07:22:16
ORACLE error from target database:
ORA-16038: log 3 sequence# 2319 cannot be archived
ORA-19809: limit exceeded for recovery files
ORA-00312: online log 3 thread 1: '+RECO/MYDB122_PHX1BF/ONLINELOG/group_3.259.1008210757'

You can now fix your FRA full issue and the next backup should be successful!

Let’s explore a bit more and have a look at a successful backup:

[root@oci_server ~]# dbcli describe-job -i 93303906-4af4-499e-98b9-067010e05134 

Job details                                                      
----------------------------------------------------------------
                     ID:  93303906-4af4-499e-98b9-067010e05134
            Description:  Create Regular-L1 Backup with TAG-DBTRegular-L11570604660275VZv for Db:MYDB122 in OSS:bYmzpVkMq1BLgxHxZSwc
                 Status:  Success
                Created:  October 9, 2019 7:05:56 AM UTC
                Message:  

Task Name                                Start Time                          End Time                            Status    
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Validate backup config                   October 9, 2019 7:06:00 AM UTC      October 9, 2019 7:06:00 AM UTC      Success   
Validate opc_pfile configuration         October 9, 2019 7:06:00 AM UTC      October 9, 2019 7:06:00 AM UTC      Success   
Database container validation            October 9, 2019 7:06:00 AM UTC      October 9, 2019 7:06:00 AM UTC      Success   
libopc existence check                   October 9, 2019 7:06:00 AM UTC      October 9, 2019 7:06:00 AM UTC      Success   
Backup Validations                       October 9, 2019 7:06:00 AM UTC      October 9, 2019 7:06:22 AM UTC      Success   
Recovery Window validation               October 9, 2019 7:06:22 AM UTC      October 9, 2019 7:06:28 AM UTC      Success   
Archivelog deletion policy configuration October 9, 2019 7:06:28 AM UTC      October 9, 2019 7:06:36 AM UTC      Success   
crosscheck database backup               October 9, 2019 7:06:36 AM UTC      October 9, 2019 7:06:57 AM UTC      Success   
Database backup                          October 9, 2019 7:06:57 AM UTC      October 9, 2019 7:59:14 AM UTC      Success   
Restore preview validate database        October 9, 2019 7:59:14 AM UTC      October 9, 2019 7:59:29 AM UTC      Success   
Backup TDE Wallet files to ObjectStore   October 9, 2019 7:59:29 AM UTC      October 9, 2019 7:59:29 AM UTC      Success   

[root@oci_server ~]# 

Unfortunately, there’s no log file information here. The JSON output neither has the log file name when a backup is successful (I won’t paste it here as it is a very long output to show … nothing interesting at this point).

So where can I find the log files of the backups (successful or not)?

    • There’s no log file for the successful backups (this is not a bug, this is a feature as confirmed by Oracle Support):
    • The unsuccessful backup logs can be found here:
/opt/oracle/dcs/log/`hostname -s`/rman/bkup/DB_UNIQUE_NAME/rman_backup_UNIQUE_BACKUP_TAG_A_TIMESTAMP.log

Note that you can also directly check / grep the /opt/oracle/dcs/log/dcs-agent.log file which contains some JSON outputs of some backups as well as a few different kinds of DEBUG information. I say some JSON outputs because we can see in the below example information about the October 3rd failed backup and about the October 9th successful backup, but nothing in between but DEBUG information:

    "parentTaskId" : "TaskSequential_7699",
    "jobId" : "010909de-bae0-4511-bda9-81d2955cfd75",
    "tags" : [ ],
    "reportLevel" : "Error"
  } ],
  "createTimestamp" : "Thu Oct 03, 2019 07:21:21.018 (UTC) [1570087281018]",
  "resourceList" : [ ],
  "description" : "Create Regular-L0 Backup with TAG-DBTRegular-L01570086276758tz0 for Db:MYDB122 in OSS:bYmzpVkMq1BLgxHxZSwc"
}
2019-10-09 23:39:19,456 INFO [dw-494] [] c.o.d.c.t.CustomSslContextFactory: entering SSLEngine
2019-10-09 23:39:20,571 INFO [dw-53] [] c.o.d.c.t.CustomSslContextFactory: entering SSLEngine
2019-10-09 23:39:20,602 INFO [dw-513 - GET /jobs?count=100] [] c.o.d.a.r.JobsApi: Received GET request to getJobsList on JobsApi
2019-10-09 23:39:20,602 INFO [dw-513 - GET /jobs?count=100] [] c.o.d.a.r.JobsApi: where clause:
2019-10-09 23:39:48,764 INFO [dw-72] [] c.o.d.c.t.CustomSslContextFactory: entering SSLEngine
2019-10-09 23:39:49,742 INFO [dw-53] [] c.o.d.c.t.CustomSslContextFactory: entering SSLEngine
2019-10-09 23:39:49,814 INFO [dw-41 - GET /jobs/93303906-4af4-499e-98b9-067010e05134] [] c.o.d.a.r.JobsApi: Received GET request to getJobDetail on JobsApi with jobid = 93303906-4af4-499e-98b9-067010e05134
2019-10-09 23:39:49,818 DEBUG [dw-41 - GET /jobs/93303906-4af4-499e-98b9-067010e05134] [] c.o.d.c.t.r.ServiceJobReport: Add TaskReport r.id=93303906-4af4-499e-98b9-067010e05134 jid=93303906-4af4-499e-98b9-067010e05134 status=Success
2019-10-09 23:39:49,818 DEBUG [dw-41 - GET /jobs/93303906-4af4-499e-98b9-067010e05134] [] c.o.d.c.t.r.ServiceJobReport: Add TaskReport r.id=TaskParallel_10120 jid=93303906-4af4-499e-98b9-067010e05134 status=Success
2019-10-09 23:39:49,818 DEBUG [dw-41 - GET /jobs/93303906-4af4-499e-98b9-067010e05134] [] c.o.d.c.t.r.ServiceJobReport: Add TaskReport r.id=TaskZJsonRpcExt_10123 jid=93303906-4af4-499e-98b9-067010e05134 status=Success
2019-10-09 23:39:49,818 DEBUG [dw-41 - GET /jobs/93303906-4af4-499e-98b9-067010e05134] [] c.o.d.c.t.r.ServiceJobReport: Add TaskReport r.id=TaskParallel_10125 jid=93303906-4af4-499e-98b9-067010e05134 status=Success
2019-10-09 23:39:49,818 DEBUG [dw-41 - GET /jobs/93303906-4af4-499e-98b9-067010e05134] [] c.o.d.c.t.r.ServiceJobReport: Add TaskReport r.id=TaskParallel_10128 jid=93303906-4af4-499e-98b9-067010e05134 status=Success
2019-10-09 23:39:49,818 DEBUG [dw-41 - GET /jobs/93303906-4af4-499e-98b9-067010e05134] [] c.o.d.c.t.r.ServiceJobReport: Add TaskReport r.id=TaskZJsonRpcExt_10131 jid=93303906-4af4-499e-98b9-067010e05134 status=Success
2019-10-09 23:39:49,818 DEBUG [dw-41 - GET /jobs/93303906-4af4-499e-98b9-067010e05134] [] c.o.d.c.t.r.ServiceJobReport: Add TaskReport r.id=TaskZJsonRpcExt_10133 jid=93303906-4af4-499e-98b9-067010e05134 status=Success
2019-10-09 23:39:49,818 DEBUG [dw-41 - GET /jobs/93303906-4af4-499e-98b9-067010e05134] [] c.o.d.c.t.r.ServiceJobReport: Add TaskReport r.id=TaskZJsonRpcExt_10136 jid=93303906-4af4-499e-98b9-067010e05134 status=Success
2019-10-09 23:39:49,818 DEBUG [dw-41 - GET /jobs/93303906-4af4-499e-98b9-067010e05134] [] c.o.d.c.t.r.ServiceJobReport: Add TaskReport r.id=TaskZJsonRpcExt_10138 jid=93303906-4af4-499e-98b9-067010e05134 status=Success
2019-10-09 23:39:49,818 DEBUG [dw-41 - GET /jobs/93303906-4af4-499e-98b9-067010e05134] [] c.o.d.c.t.r.ServiceJobReport: Add TaskReport r.id=TaskZJsonRpcExt_10140 jid=93303906-4af4-499e-98b9-067010e05134 status=Success
2019-10-09 23:39:49,818 DEBUG [dw-41 - GET /jobs/93303906-4af4-499e-98b9-067010e05134] [] c.o.d.c.t.r.ServiceJobReport: Add TaskReport r.id=TaskZJsonRpcExt_10142 jid=93303906-4af4-499e-98b9-067010e05134 status=Success
2019-10-09 23:39:49,818 DEBUG [dw-41 - GET /jobs/93303906-4af4-499e-98b9-067010e05134] [] c.o.d.c.t.r.ServiceJobReport: Add TaskReport r.id=TaskZJsonRpcExt_10144 jid=93303906-4af4-499e-98b9-067010e05134 status=Success
2019-10-09 23:39:49,820 DEBUG [dw-41 - GET /jobs/93303906-4af4-499e-98b9-067010e05134] [] c.o.d.c.t.r.ReportApi: Job Report:
{
  "updatedTime" : "Wed Oct 09, 2019 07:59:29.518 (UTC) [1570607969518]",
  "jobId" : "93303906-4af4-499e-98b9-067010e05134",
  "status" : "Success",
  "message" : null,
  "reports" : [ {

Indeed:

[root@oci_server ~]# grep 010909de-bae0-4511-bda9-81d2955cfd75 /opt/oracle/dcs/log/dcs-agent.log | wc -l  # October 3rd failed backup
48
[root@oci_server ~]# grep 76d0531a-3599-41ba-b440-4261db78c292 /opt/oracle/dcs/log/dcs-agent.log | wc -l  # October 5th successful backup
0
[root@oci_server ~]# grep a7a51004-2da4-452f-a02e-1b6f11afc87a /opt/oracle/dcs/log/dcs-agent.log | wc -l  # October 7th successful backup
0
[root@oci_server ~]# grep 93303906-4af4-499e-98b9-067010e05134 /opt/oracle/dcs/log/dcs-agent.log | wc -l  # October 9th successful backup
5505
[root@oci_server ~]# 

Oracle says “If you require more information, review the /opt/oracle/dcs/log/dcs-agent.log file” so let’s take it like this, as it seems that we don’t have all the information we expect in this file. By the way, it is interesting to note that this “DCS agent” was already here with Oracle Database Appliance (ODA) and is used to “allow you to manage your database through the cloud platform”. It then obviously needs to be improved as we are unable to see a log file through the console (which may come later on).

Thanks for reading and enjoy the Oracle Cloud (OCI) Automatic Backups Troubleshooting!

email

Interested in working with Fred? Schedule a tech call.

No comments

Leave a Reply

Your email address will not be published. Required fields are marked *