Whats New Qube 6.0

1. Realtime Performance Charts………………………………………………… 1

2. “User/Job” Automated Render Wrangling………………………………….. 3

2.1. Job ETA and Average Frame Times……………………………………………………… 3

2.2. Frame timeouts……………………………………………………………………………………… 3

2.3. Basic Image validation……………………………………………………………………………. 4

3. “Supervisor” Automated Render Wrangling……………………………….. 4

4. UI Filtering and Streamlining…………………………………………………… 6

4.1. Hide unused layouts and menus…………………………………………………………….. 6

4.2. Submission dialog group collapsing and filtering advanced parameters…… 7

4.3. Submission dialog “account” choices and “kind” parameter………………………. 8

4.4. Clearer naming for common terms………………………………………………………….. 9

4.5. Reorganized and regrouped advanced Qube submission parameters…. 10

5. UI Enhancements………………………………………………………………. 11

5.1. Added panel for global resource and farm summary…………………………….. 11

5.2. Added highlights panel for showing parsed info from stdout/stderr………… 11

5.3. Added commandline options to set initial search filter…………………………….. 12

5.4. Added commandline ability to resubmit job with parameter overrides…… 12

5.5. Multiple external viewer support………………………………………………………….. 12

6. Common action accelerators……………………………………………….. 13

6.1. Integrated Quicktime Generation…………………………………………………………… 13

6.2. Shotgun Version Submission………………………………………………………………. 14

7. Enhanced Submission Framework………………………………………….. 15

7.1. SimpleCmd Stdout/Stderr parsing framework……………………………………… 15

7.2. Redirect Stderr -> Stdout………………………………………………………………………. 17

8. API………………………………………………………………………………… 17

1. Realtime Performance Charts

Ever been asked these questions?

• Do we need more machines on our renderfarm?
• Do we need more software licenses?
• When is the farm most used?
• What is running on the farm?

A charting framework has been added to the QubeGUI to visually display historical and current information about the renderfarm and the render jobs. It gives you insight on how your hardware and software are used — and what is needed to meet your rendering needs. It will give you concrete information instead of having you rely on guesses to best manage your studio resources and infrastructure.

This information can display totals along with drilling down be filtered and categorized by individual users, job types, accounts/projects, clusters, and job names (using wildcards). The “Top 5″ allows one to see the most influential individual items in a list at a glance without the clutter of all items.

It will keep this historic data and can be plotted across the last 24 hours all the way to years back. One can use the date range presets for today, yesterday, last 24/36 hours, last 3/7/30 days, or put in any custom range. It will provide chart totals along with drilling down to individual items like

There are currently 4 charts initially deployed within the charting framework.

• Running Frames – display how many frames (or tasks) are currently running on the renderfarm. Specifying on a per-user, per-cluster, or per-account, basis lets one see the most influential individual items.
• Pending Load – display number of frames (or tasks) in the queue and when they were submitted. See patterns as to when the most frames are submitted or when there are lulls.
• Farm Utilization – display renderfarm Worker slots currently running versus total available. See if the farm is fully utilized.
• Farm Size – display separate data lines for Worker slots that are active, locked slots, down. See how large the farm currently is and how large it can be.

These charts can be printed or saved as images for inclusion into reports.

2. “User/Job” Automated Render Wrangling

There are a number of manual tasks typically performed when overseeing jobs on the renderfarm and the renderfarm itself. The header “auto-wrangling” is meant to automate some of those more manual tasks and calculations. Some of these render wrangling tasks can now be done on a per-job basis by the user.

2.1. Job ETA and Average Frame Times

The Job Layout has new columns for “Frame Average” (Avg) and “ETA” to automatically calculate the estimated time it will take to complete running jobs based on previously rendered frames.

It performs a simple calculation of determining the average time per frame, multiplying that by the number of frames remaining, and then dividing by the number of subjob processes running.
It calculates this “on the fly” in the QubeGUI when the frame information is present, so selecting the desired job is necessary to have it perform the calculation. It will display the ETA if there are running subjob processes or left blank if not (since it can only move toward completion if there are running subjob processes).

2.2. Frame timeouts

To complement the “subjob process timeouts”, frame timeouts have been added in the Submit Dialog that will allow the user when submitting the job to specify the maximum time a frame should take before reporting it as failed. If “Retry Frame/Work” is specified, then it will retry the frame.

This addresses the occasions when there are possible “runaway” frames that take way longer than expected due to renderer hangs, system memory thrashing, or other causes.

2.3. Basic Image validation

An optional basic file size check has been added to the cmdline/cmdrange simplecmd framework. If an image is identified from parsing the stdout or stderr from a cmdline/cmdrange job (or any job using the simplecmd framework), it can optionally have rudimentary validation automatically performed on it to make sure that it is valid.

For this to work, the regex_outputPaths submission parameter needs to be specified for it to parse the stdout/stderr of the job for images, or manually specifying the job frame’s (in python of the jobFrame is a frame in the job, then represented by jobFrame['resultPackage']['outputPaths']).

3. “Supervisor” Automated Render Wrangling

There are a number of manual tasks typically performed when overseeing jobs on the renderfarm and the renderfarm itself. The header “auto-wrangling” is meant to cover automating some of those more manual tasks and calculations.

Some of the most common render wrangling tasks can now be handled automatically by the Supervisor through a set of global parameters. Qube has added built-in logic that detects faulty jobs and workers. These parameters can be configured by modifying the Supervisor qb.conf directly or through the “QubeGUI -> Administration -> Config (local)” dialog.

Automated Actions

• Automatically blocks faulty jobs and sends email to the submission user.
• Automatically locks faulty workers and sends email to the Qube Administrator.

To enable:

• Make sure that the “supervisor_language_flags” includes “auto_wrangling” (it’s ON by default, if “supervisor_language_flags” isn’t found in qb.conf).

• To globally turn on auto-wrangling for all submitted jobs (recommended), include “auto_wrangling” in the “supervisor_job_flags” configuration parameter in the supervisor’s qb.conf. Restart the supervisor if qb.conf settings were changed.

• Auto-wrangling also may be enabled on a job-by-job basis, by NOT including “auto_wrangling” in “supervisor_job_flags”, and explicitly setting the “auto_wrangling” flag for jobs.

• Auto-wrangling also may be enabled on a client-to-client basis, by NOT including “auto_wrangling” in “supervisor_job_flags”, but including it in the “client_job_flags” on desired submission machines. This will enable auto-wrangling on all jobs submitted from those machines.

To disable:

• Remove “auto_wrangling” from the “supervisor_language_flags”, in the supervisor’s qb.conf file. Restart the supervisor.

Auto-Wrangling-specific qb.conf parameters:

• aw_activation_work_count : This positive integer specifies the number of agenda-items that should fail on a given worker before the auto-wrangling logic is activated. (default: 5)

• aw_job_migrate_max : This positive integer specifies the maximum number of automatic migration the auto-wrangling subsystem will perform on a job, before it decides that the job is faulty and blocks it, and notifies the submitting user. In other words, this effectively determines the number of workers on which a job will be tried, before the system “gives up”. (default: 3)

Other related qb.conf parameters:

• “supervisor_language_flags”, which lists the enabled callback “languages”, must include “auto_wrangling” to enable auto-wrangling.

• “supervisor_job_flags” may include “auto_wrangling” which will globally turn on auto_wrangling for all submitted jobs (i.e., by automatically turning on the “auto_wrangling” flag for every job when submitted).

• The email address specified in the “mail_administrator” parameter will receive email messages when auto-wrangling determines that a worker is faulty and locks it.

Job parameters:

• A job’s “email address” field, if set, is used as the recepient address of email notifications sent out by the auto-wrangling subsystem.

Email

The Auto-Wrangling system will send notification emails when it takes certain actions.

• When a job is blocked because auto-wrangling determined that it’s malfunctioning, an email notification is sent to the user.
• When workers are locked because auto-wrangling decided that the worker is faulty, then the qube admin is notified via email.

Auto-wrangling Logic

Currently, the built-in auto-wrangling logic is basically an implementation of the following pseudocode:

if(a worker has failed more than aw_activation_work_count agenda-items) {

if(the worker hasn’t completed any frame it has been assigned so far) {

if(other workers are completing frames from this job) {

* Worker is faulty, so lock it.

* Change the status of failed frames back to “pending” so other workers may retry them.

* Notify admin that the worker is faulty and has been locked

} else { // no other worker is completing frames.

if(there is at least another worker running this job) {

if(all the other workers are also failing frames) {

* Job is faulty, so block it.

* Notify submitter.

} else {

// There are other workers running this job, but there

// are no complete frames. All the failed frames were

// processed by this worker.

* No conclusion can be drawn at this time. Do nothing.

} } else {

// this is the only worker that has run this job.

// no frames are completing

if(job has not been migrated to aw_job_migrate_max workers) {

* Migrate the job to another worker

} else {

// job has been migrated aw_job_migrate_max times

* Job is faulty, so block it.

* Notify user. } } } } }

4. UI Filtering and Streamlining

4.1. Hide unused layouts and menus

The QubeGUI is designed to allow both artists and administrators to manage their part of the renderfarm. There are some panels and interfaces that do not need to be exposed to all users. In the Preference Dialog, one can optionally hide the following menu items and panels:

• Administration menu
• Admin Layout
• Charts Layout

One can also specify the Submit menu to only display the used submission interfaces instead of all of them. Under “Submit Interfaces” in the Preferences dialog, select only the desired interfaces from the list. The rest will be hidden. If <ALL> is checked, then all submission interfaces will be displayed.

4.2. Submission dialog group collapsing and filtering advanced parameters

The Submission dialogs have a number of enhancements. One can now collapse parameter groups to quickly show or hide only the sets of fields you want to see. This is particularly helpful when there are hundreds of renderer options exposed.

There is also an Expert parameter view toggle that will filter out all default-valued advanced parameters so that only the basic ones or ones that have been edited are shown. can can now be collapsed hiding non-default-value submission fields.

4.3. Submission dialog “account” choices and “kind” parameter

The user specified “account” parameter now has an optional dropdown menu for easy selection of the project or department that a job is submitted for. The choices are specified in the QubeGUI Preferences dialog. There is also a parameter to allow users to specify custom account values or require them to only use available account choices.

The user specified “kind” parameter has also been exposed as well to allow users to have more control over reservations and restriction rules on where jobs can be run.

4.4. Clearer naming for common terms

Some renaming of terms in the QubeGUI have been made to clear up ambiguities or overloaded meanings. In particular, the CPUs term has been overloaded in Qube to mean the number of “Subjob Processes” requested for jobs and not the more common meaning of number of hardware CPU cores.

Submit Dialog

CPUs -> Subjob Processes

Job List Columns

CPUs -> Processes

CPUs Running -> Processes Running

CPUs Requested -> Processes Requested

Tasks -> Frames

Host List Columns

Processors -> Slots

A few of the layout panels have also been renamed to make them clearer and provide future expansion.

Renamed Panels

Host/Worker Layout -> Farm Layout

User Layout -> Admin Layout

Timeline -> Job Timeline

Histogram -> Frame Times

Host/Workers -> Subjob Processes

4.5. Reorganized and regrouped advanced Qube submission parameters

The growing list of “Detailed Qube” parameters are regrouped into more logical and collapsible groups. The usually untouched “Command Template” parameter (present for all SimpleCmds) has been moved to the bottom of the list so that the commonly edited parameters are first.

5. UI Enhancements

5.1. Added panel for global resource and farm summary

Under the Farm Layout, there is a new panel on the right called “Resource Summary” that presents a summary of the Farm Worker states, Farm active slots, and global/license resources. Having the global and license resources exposed in the GUI provides more general access to this functionality instead of just through the commandline “qbadmin” command.

5.2. Added highlights panel for showing parsed info from stdout/stderr

A new Highlights Panel helps prevent the need to manually scan through potentially hundreds of thousands of lines in the stdout and stderr.

For cmdline, cmdrange, and all SimpleCmd jobs, the “Highlights” Job Panel has been added to the Job Layout that stores a few lines of information on each frame directly in the MySQL database for easy loading. This panel is populated with information collected from the regex_* submission parameters. For common simplecmd jobtypes like for Maya BatchRender, these parameters are already specified.

5.3. Added commandline options to set initial search filter

A commandline options -jobfilter <string> and -farmfilter <string> allows one to launch the QubeGUI from the commandline and specify the search filters on startup.

5.4. Added commandline ability to resubmit job with parameter overrides

The commandline options -resubmit <jobid> and -submitDict <string> can now be used together. This allows one from the commandline to resubmit a job and override some of the parameters.

Example: qube.exe –resubmit 280 –submitDict ‘{“name”:”another job name”}’

5.5. Multiple external viewer support

There is now multiple viewer support in Qube. You can have different viewers open up depending on the file type. This works for images (including exr), movies (like Quicktime), or any other file type.

The setup is done in the Preferences Dialog of the QubeGUI. Specify a wildcard filename (*.mov for example) and then an explicit viewer or leave blank to have it use the viewer associated by the filesystem.

6. Common action accelerators

6.1. Integrated Quicktime Generation

A common action is to render out individual images and then compile time into a Quicktime or some other movie format. This can all be done on the Qube renderfarm. A new generateMovie submission parameter has been added to all submission dialogs to start making that process easier and more automatic.

If the generateMovie checkbox is checked, it will bring up a second submission dialog for the qube_imagesToMovie Simplecmd. This will then call the images_to_movie.py python script with the specified parameters. The script will gather the images list from the jobid and then generate the movie from the specified transcoder and options.

Note: This function will only work on jobtypes that retrieve the output image filenames into Qube (Maya, 3dsMax, XSI, AfterEffects, etc).

6.2. Shotgun Version Submission

Shotgun Version Submission has been integrated into Qube to allow studios that use Shotgun to easily and automatically create Shot/Asset versions when submitting a render job through Qube.
To enable Shotgun version submission, set the “Shotgun URL”, “Script Name”, and “Script Key” in the QubeGUI preferences dialog.

When submitting a job through the submission dialogs, the Version Name and description can be flexibly constructed with direct text or from existing metadata fields from Qube and Shotgun by using standard python formatting in the form %(shotgun.*)s and %(job.*)s.

7. Enhanced Submission Framework

7.1. SimpleCmd Stdout/Stderr parsing framework

A framework has been added to the cmdline/cmdrange jobtypes that the SimpleCmd framework leverages. It adds regular expression parsing of the stdout and stderr for interesting highlights (errors and warnings), messages that indicate fatal errors, and also output paths for rendered images.

There are a number of new regex_* fields added to all simplecmd submission dialogs to give it instructions on how to parse the stdout and stderr for the job. Multiple regular expressions can be added as separate lines in the submission dialog. The results can be viewed in the Highlights Panel for each job, frame, or subjob process.

• regex_highlights: specify for non-fatal errors, warnings, and info that should be highlighted. Example: Total Elapsed Time
• regex_errors: specify expressions that if parsed should mark the frame or subjob process as failed. Example: Error:
• regex_outputPaths: used to gather the output images, movies, or files from the frame or subjob process. The path should be marked with parenthesis (). Example: Finished Rendering (.*)\.

Technical Note: The results are stored in the frame or subjob process resultspackage directly in the MySQL database.

7.2. Redirect Stderr -> Stdout

The stderr stream can now be redirected to appear with the stdout stream for applications that send some information to each place, but that should really be stored together. It can be set when submitting a job by checking the Stderr -> Stdout checkbox under “Qube Advanced Job Control” in the submission dialogs. The output can all be seen then in the Stdout Panel for each job.

8. API

There have been a few new functions exposed in the python API for the 6.0 release.

The qb.Job class has the following additional parameters that can be referenced by qb.Job[<keyword>]:

• `agenda_timelimit` : maximum time in seconds a frame is allowed to run before it’s forcefully failed (int)
• `agenda_timeout` : maximum time in seconds a frame is allowed to run before a frame timeout event is generated (int)

These can also be referenced by their function qb.Job.agenda_timelimit() and qb.Job.agenda_timeout().
referenced by their function qb.Job.agenda_timelimit() and qb.Job.agenda_timeout().