Vdbench Changelog

What's new in Vdbench 5.02

• File system testing (FSD/FWD):
• Complete Data Validation and Journaling for File system testing.
• format= parameter enhancements, including ‘format=restart’.
• new openflags= options, including fsync and directio.
• openflags= now available on FSD, FWD, and RD.
• fwd=format special FWD to set xfersize=, threads= , and openflags= for format runs.
• The format has changed from first creating all files as empty and then filling them with the
• requested amount of file size, to now create and then immediately fill.
• ‘flatfile.html’ now also created for file system tests.
• ‘./vdbench compare’ can now also be used for output of file system tests.
• Vdbench now has skew control just like it has always had for raw workloads: if you want 50/50 read/write, Vdbench will make sure that those objectives are met and no longer will let faster workloads dominate the slower ones.
• If requested, Vdbench now will create mountpoint directories and mount file systems, and
• will even change mount options in between runs.
• File system workloads now report a response time histogram. These histograms are reported for each FSD, each FWD, and each RD.
• There are now detail reports for each FSD and FWD instead of only the totals.
• Just like with ‘sd=’, ‘fsd=’ now also supports the ‘count=(start,###)’ parameter to quickly define multiple FSDs.
• ‘fsd=xxx,shared=yes’ parameter to allow multiple clients and/or JVMs to use the same FSD instead of requiring a unique FSD for each host or JVM.
• ‘fsd=xxx,workingsetsize=’ (or wss=) parameter to force Vdbench to use only a specific subset of the total amount of file space that has been created.
• ‘fsd=xxx,distribution=all’ parameter forcing Vdbench to create files under each directory instead of only in the lowest level directory.
• ‘fwd=xxx,stopafter=’ parameter now allows you to specify a percentage, e.g. stopafter=20%, where, during a random I/O workload, Vdbench will switch to a new file after processing 20% of the file’s data.
• ‘fwd=xxx,stopafter=’ can be used to simulate an ‘append’ to disk files.
• ‘fsd=xxx,size=(100m,0)’ A new option that instead of creating fixed file sizes it will create random file sizes, with an average of the requested file size.
• Vdbench now keeps track of the file structure that it has created or is creating. If for instance you created 10 million files in a previous execution it would waste a lot of time in the next execution verifying ‘which file exists already, how long is each file? Etc’. Now Vdbench just keeps the info available in file ‘vdb_control.file’ placed directly in the FSD anchor.
• Changes to file system reporting: the ‘ops’ column has been renamed to the ‘ReqstdOps’ column. I also added a ‘total mb/sec’ column. The ‘ReqstdOps’ column reports only the requested operations. For instance if you ask for reads to be done this column contains only the read statistics though the ‘open’ and ‘close’ statistics are still reported separately in their
• own columns.
• The anchor directory will now always be created if it is not there yet. The parent directories only will be created however when you add the ‘create_anchors=yes’ parameter.
• New parameter to allow a sequential file to be deleted before it is rewritten. ‘fwd=xxx,… ,fileio=(sequential,delete)’.
• New ‘fwd=xxx,…..,rdpct=xx’ parameter to allow reads and writes to a dataset to be mixed. This does not make much sense for sequential I/O (read block1, write block2, read block3, write block4, etc) but has use for random I/O.
• Raw I/O testing (SD/WD):
• When Vdbench finds disk files that do not exists, all files will be created during one single format run instead of one format run per file, improving throughput.
• If Vdbench finds a disk file that is too small for the size= values used it will expand that file, also just in one single format run. This equates almost to the new ‘format=restart’ option of file system testing.
• A new formatxfersize= parameter to control the data transfer size used during above format runs.
• openflags= now available on SD, WD and RD.
• On Windows any error code returned from the GetLastError() function will now be properly translated into text using the getWindowsErrorText() API call.
• new openflags= options, including fsync() and directio()
• Garbage in the /dev/rdsk/ directory could cause Vdbench to abort because of a non-zero return code from the ‘ls’ command. Fixed.
• The ‘L’ from the ‘ls –lL /dev/rdsk’ command has been removed. This could cause temporary hangs if there was a device that was in an error state. This hang would occur at the start of a run where Vdbench (on Solaris) was interrogating the storage configuration.
• Vdbench now no longer creates a report for each separate SD, once for each host and once for each slave. Running 200 SDs against 8 slaves (on one host) now has 1800 less report files. Override this using the ‘report=slave’ or ‘report=host’ parameter.
• I finally bit the bullet. On Linux when opening devices starting with ‘/dev/’, you are now required to specify ‘openflags=o_direct’ to force raw I/O. It you really need to run against a /dev/ device while not doing raw I/O, I coded in a trick, let me know if you need it.
• When more than one Workload Definition (WD) is specified, Vdbench will now generate a new Workload Report.
• You now can specify specific target I/O rates for each Workload Definition.
• You now can specify an I/O priority for each Workload Definition.
• Vdbench will no longer create a new file that starts with /dev/.
• You can now specify a complete workload without ever specifying a Workload Definition by entering (most of) the parameters under the Run Definition.
• ./vdbench sds now also works for Linux and Windows
• General changes:
• Journal recovery now is aware of those write operations that were pending at the time of the Vdbench or system shutdown and will analyze the status of those outstanding I/Os to see whether those blocks contain the old contents (from before the write) or the new contents, though at this time Vdbench is not able to completely verify the contents if half of the block contains ‘new’ data, with the other half containing ‘old’ data. This scenario will happen when
• a write is interrupted, for instance due to a power outage.
• All platforms: a check has been added in the JNI read and write functions to make sure that IF an I/O error is returned the ‘errno’ variable contains a non-zero value.
• Java 1.5 is now required. It already was required, but the check was not done.
• Added a new ‘print’ function to be used for collecting diagnostics data after a Data Validation error.
• After a Data Validation error discovered during the pre-read of a write, the block will not be overwritten (this was a bug).
• ‘host=(host_name,host_label’ has been replaced by ‘hd=host_label,system=host_name’ (HD=Host Definition). This was done to keep syntax in line with all other SD/WD/FSD/FWD/RD parameters. However, if you use the older format it will still be accepted for a while.
• A new ‘include=filename’ parameter, allowing Vdbench to dynamically insert multiple parameter files. This in addition to the already existing ‘./vdbench –f file1 file2 ….. filen’ execution parameters
• To assist with the creation of complex parameter files, any time that Vdbench finds ‘$host’ in a line of parameters, that whole line is repeated once for each defined Host Definition, repeating ‘$host’ each time with the host label defined in the HD.
• New ‘vdbench jstack’ function. When running from a java JDK, Vdbench will print out the current Java execution stacks for all currently running JVMs. Great for debugging Vdbench hangs when you see any.
• Vdbench compare now allows you to compare output directories where the amount of runs completed does not match.
• Small enhancement to using the ‘-s’ (simulate) execution parameter. This now displays the run names and ‘forxxx’ parameters for each run that you have requested.
• All Data Validation and Journaling options now can be specified inside of the parameter file instead of only on the command line.
• A new ‘warmup=’ parameter giving you control over the amount of reporting intervals that will be excluded from run totals.
• Introduced ‘forxfersize=(512k-1k,d)’, the reverse of ‘double the previous value’. Can also be used for other forxx parameters.
• The data pattern generation for Data Validation has improved. Instead of using only the lba and the DV key as a seed value I now also add the SD or FD name to the seed. This allows a greater data pattern diversity when using multiple SDs.
• Journals can now be done on raw devices to make sure that possible file system issues cannot compromise the integrity of the journal file. Add ‘journal=/dev/xxxx’ to the SD or FSD. The ‘map’ file (backup copy of the raw DV map) will in this case be written to the current directory.
• The journal file format has been rewritten to allow the integrity of the journal file to be checked.
• Resolved a loop when Journal recovery was done on a system with a single cpu. They still exist.
• ‘Journal recovery only’. Allows you to recover journals and validate all data without executing a workload.
• The ‘patterns=’ option has been changed to only support a ‘default’ pattern.
• To accommodate synchronous journal writes, the journal file is now opened using the ‘O_SYNC’ parameter to make sure that a file’s metadata also is updated.
• A Data Validation Post-processing GUI. The amount of data reported in the errorlog.html file after a DV error can be so overwhelming that it becomes difficult to filter out useful information.