SLOB on Steroids – Part 1

SLOB on steroids v0.1: Use it at your own risk. If you don’t know what SLOB is, read up on it here.

Love

First of all, let me state that I’m falling in love with SLOB :). It isn’t because it allows you to easily Benchmark your Oracle platform’s components. In fact, for some components there are other tools that allow to reach the same results more easily (see “My First Experience Running SLOB – ORION vs SLOB – 2:0“). SLOB’s biggest value for me is learning how Oracle Instance behaves under different type of load and how to tune it to get the best results.

Just to illustrate, I would like to mention two things I learned during my resent SLOB testing: 1. Oracle 11GR2 sometimes uses “db file sequential read” optimization and converts it to “db file parallel read” – How cool is this? I learned it while assessing SLOB testing results. 2. In RAC configuration you sometimes need to adjust gcs_server_processes init.ora parameter to ensure that an interconnect is utilized effectively (details here). If you are like me, a curious Oracle DBA, then SLOB is an excellent tool to explore, use, and learn. At this stage, I am quite sure that in any testing the results are not that important. The most important is what we learned from the results.

SLOB related warnings

I should warn you about several things I learned from my SLOB testing experience:

• Before relying and publishing sky-high world record numbers you received using SLOB tests, you should analyse the results carefully and make sure that the tests you executed corresponds to the application’s behavior you are going to run on the platform. From my experience, it is quite easy to get high numbers that wouldn’t be useful in real life. I published some of my recommendations on what you should be careful with over here.
• The time you have planned for SLOB testing most likely isn’t enough, and you should plan more. Someone may think that since SLOB is simple, it will take them 2 afternoon hours to complete all the testing. Well, you may manage to make a first run, but I bet that to get a good result you will need to adjust, run, analyse, adjust, run… All of this many times before you confirm that the results are good enough. Just to give you an example: To understand and get good and reliable results for 3 different scenarios, I ended up running 140 different tests. It took me ~4 calendar weeks, ~20-30 personal time hours.
• Do not take results for granted. Think about what you see and assess it. For example, if you see that in “db file sequential read” an average response time in an AWR report is 1ms for a storage test (with spinning HDDS), ensure that you haven’t tested raw storage performance that your application may or may not benefit from. The IO requests were returned from the storage’s cache, and the storage cache behaves very differently depending on the application load a system currently executing. After any testing, you should assess results and question them yourself before someone else does it for you and questions your competence.

Anyway, enough talking about general stuff. Let’s get practical.

SLOB on steroids

If you used SLOB, the observations below probably make sense to you. However, if you didn’t have a chance to run SLOB and are just preparing to do so, you have two options from here:

1. Install and run the default SLOB scripts provided by Kevin Closson. See how it goes, and get back to this blog post after running it several times. (Recommended!)
2. Install the default SLOB. Download and copy a set of adjusted scripts to SLOB’s home directory from the link on top of this page. Run tests using the instructions at the end of this blog post.

If you have been using SLOB for some time now, you probably want some improvements to be implemented to make your experience with SLOB easier and testing more efficient. Have a look at the following observations. The set of scripts provided on the top of this blog post addresses those issues:

• RUN SLOB X MINUTES ONLY: It’s kind of difficult to predict how much time a default SLOB test will take to execute for a given test. It may take 5 seconds or 2 hours, depending on what you would like to test and the platform performance characteristics. It is a bit annoying to wait another hour or two to get results. You probably adjusted the “FOR i IN 1..5000 LOOP” part of the reader.sql script to make SLOB running a longer or slightly shorter period of time. However, what we really want to do is run a test 2, 5, 10, or x number of minutes and evaluate LIO/PIO and other statistics per second. I spent way too much time trying to adjust FOR … LOOP for different type of tests before I implemented a solution that allowed me to run SLOB X number of minutes only.
• MAKE WRITERS RUN AT THE SAME TIME AS READERS:  To get consistent AWR report in write/read tests, writers and readers should finish running the tests at the same time. This exercise is even more difficult to complete when adjusting the “FOR i IN 1..5000 LOOP” line in writer/reader scripts.
• SYNCHRONIZE SLOB RUNS IN RAC ENVIRONMENT: Let’s take it to the next level and try to test things in the RAC environment. It is even more challenging to get all readers/writers processes complete at the same time. While it isn’t that difficult to start a test at the some time (You just need to run the runit.sh script in different terminals in a very short period of time, 1-2 seconds difference doesn’t impact anything), to make sure that tests complete at the some time is a challenge.
• AWR.TXT: Each time we run a test the default SLOB instruction suggests us to make a copy awr.txt file to a file with a different name. I did it for a couple of time and later just let runit.sh script do it for me (as I am a lazy person).
• HOW TO DOCUMENT: As time went, I started modifying SLOB scripts, and it became almost impossible to remember what kind of adjustments I made in SLOB scripts  for a test represented by awr_____.txt file. In the end, I added a simple set of lines to the runit.sh file to backup all involved scripts in a uniquely named directory each time we run SLOB. I effectively made SLOB scripts part of SLOB runs reports/documentation.
• HOW TO STOP A RUN: If you made a mistake in the test parameters or for a reason or another would like to stop a run, there is no easy way of doing it but to restart an instance. While it is a solution, it isn’t the best one. I would prefer to have an option to say to SLOB processes okay enough, go away, and generate AWR report for the current run time.

I know that Kevin designed SLOB to be Silly/Simple. However I and most probably some other folks out there would like it to be a Silly/Simple/Convenient to use :). This is why I am sharing my version of “SLOB on steroids” with the world. Well, “steroids” sounds too big for the modifications I made. But it sounds cool. So there you go! :)

NOTE: Please feel free to adjust, improve and publish your versions of the scripts.  I will be happy to see your versions. I am sure that Kevin feels the same way.

Details

At this stage, I would encourage you to download the set of scripts from the link on the top of this blog post, carefully review the modifications I made, copy scripts in the SLOB directory, and give it a try. Or alternatively, you can borrow my ideas and possibly code and implement those in your version of SLOB.

HOW TO RUN IT?

There is just one additional optional parameter I have introduced to the runit.sh file. The third parameter value represents the number of seconds the SLOB processes should run. Depending on your testing requirements, you may set it to start from a few minutes to several hours. I used 20 seconds just for demonstration purposes.

[[email protected] SLOB]$ v_r=12; v_w=0; v_c=20
[[email protected] SLOB]$ nohup time bash -x runit.sh ${v_w} ${v_r} ${v_c} 2>&1 > ./runit.sh_${v_w}_${v_r}c1t.`date +%Y%m%d_%H%M%S`.log ; egrep "Elapsed:  |Logical reads:   |Redo size:   |Block changes:    "  `ls -trp awr_* | tail -1` ; ls -trp awr_* | tail -1
   Elapsed:                0.38 (mins)
       Redo size:           96,243.3           91,110.3
   Logical reads:        5,711,825.1        5,407,194.5
   Block changes:              308.9              292.4
awr_0w_12r.20120708_171129.txt
[[email protected] SLOB]$

WHAT YOU GET AFTER A SLOB RUN?

• You get a directory bck_< date > _ < time > that contains all the SQL and SH files used in the execution, plus LOG file and AWR report.
• There the AWR report’s name file have the following format awr_< writers count >w_< readers count > r.< date > _ < time >.txt. There are AWR report’s copies in SLOB home and in the backup directories.

HOW TO STOP A SLOB RUN?

You execute the following script any time during your run.

sqlplus -s ‘/ as sysdba’ @sem_end.sql 0

WHAT COULD YOU EXPECT IN PART 2?

Well, I must admit that I didn’t document the changes I made well enough. In the next post, I may describe changes and add comments in the scripts. However, with this post, I wanted to share my ideas and possibly make your SLOB experience more enjoyable. Enjoy your SLOB testing, folks!

Yury