PipelineFX was putting together some tip tweets for December on “10 Lessons When Changing Render Managers” and since they are actually great ideas for anyone to use, we at CGSociety thought they would be great advice for anyone heading into that storm.

1. Set-up a test farm

2. Roll out over time.

3. Don’t forget to train your artists too.

4. Realize everything won’t be exactly the same.

5. Do the integration work BEFORE a short deadline project.

6. Train your admins.

7. Take advantage of new capabilities.

8. Get help making the change (consulting and training and services).

9. Emphasize the new capabilities.

10. Everyone hates change.

• Many 3rd-party applications have issues running on multiple operating systems:
  • not all plugins are available on all OS’s
  • fonts may render differently across OS’s.
  • bugs may exist in the same app on one OS and not the other
• File permissions can be tricky to resolve across different OS’s.
• User’s identities may not be consistent between OS’s.

2. Solve cross-platform issues first.

• Each operating system accesses files in its own, unique way. If an artist submits a job from one operating system, but the facility has chosen to render on a second OS, the paths to the source material may be different. For example, in an OS X environment, an artist may use a file located at /Volumes/fileserver/project/myproject.aep. If the workers are running Windows, they would see the same file as either \\fileserver\project\myproject.aep or Z:\project\myproject.aep.

These types of issues can be solved using Qube’s path translation mechanism or they can be solved at an operating system level using symbolic links. The latter solution provides greater transparency and is less prone to paths that are outside of Qube’s control – like those defined within the source material itself.

3. Centralize your storage.

• Artists’ work should not be kept on local disks. In order to render on all machines in a farm, each machine needs to be able to access all the files. Artists used to working on local disk often will add a “publish” step where they will copy their work to a central location when it’s time to render, but this is very error-prone and assets, textures, and external references often get left behind during the publish step. However, if artists get in the habit of working on shared central file storage, they will eliminate the necessity to publish. The other benefit is that their work can be easily transferred to other artists, and the risk of loss due to hard disk failure is minimized; central file storage is also usually configured to be much more reliable and is backed up more often than workstations’ local disks.

4. Open up file system permissions.

• If your environment allows for a more lenient security policy, it may be in your best interest to set up the access permissions on areas of your file server where the artists are working so that all users have read/write access, instead of having to manage permissions on a more granular basis. Many sites work off the assumption that if you’re logged in, you should have access to almost everything on the server. Instead, adopt an “opt-out” model; grant full access by default, then remove permissions from areas of the file system that require additional security. This is much easier to maintain than an “opt-in” model, where the default access permissions are “no one accesses anything,” and permissions are granted on a case-by-case basis.

5. Centralised staging of worker/artists workstation for guaranteed consistency

• Every computer that renders needs to have the same functionality as those that create the files to be rendered. They must have all the same applications, plugins, licensing, and OS settings. To guarantee consistency, we recommend basing all machines off a single, central machine image.

6. Use revision based workflow ( saved revisions of working files )

• Rendering is an iterative process.hi there.Using revision based workflow allows an artist to quickly move from one iteration to another, using the exact same settings as the approved version.

7. Ensure all dependent media is located on shared storage.

• Artists should be instructed to actively work off of shared storage, instead of trying to move assets, textures, etc to shared storage when it’s time to render. The slower response some perceive when working off networked storage is many times less than the time lost to trying to get things up to shared storage at render time, and it avoids lost hours of re-rendering due to missed textures, etc. If your central storage is slow enough that the increase in load times experienced by the artists in their day-to-day work is significant, that storage will be crushed by the load imposed on it by a render farm when all machines start on a job at the same time.

8. Single centralised licensing server

• Many applications use a central license server & depend on the availability of that server to determine whether or not they can start and run. Using a single license server allows for better accounting of license usage and is an easy place to track down problems if/when they arise.
• Coincidentally, a Qube! supervisor will happily coexist on the same hardware as your license servers (assuming the server is up to spec).

9. Use Time-based cleanup scripts

• Visual effects can be grueling work. Hundreds of iterations may be used on a single shot. Iterations that are many days and/or weeks old, or even older, are of little use; often ending up more as a burden than an aid. Time-based cleanup scripts can eliminate the waste. Qube! provides API calls to help determine which jobs were submitted or completed more than X days ago to make the creation of these types of scripts that much easier.

10. Use commercial off the shelf products every place you can.

• Don’t write what you can buy. The flip side of this is: “Don’t buy what you can’t customize.” A 3rd-party product that allows you to customize it to your workflow can provide the best of all possible worlds.
  • It’s easy to underestimate the cost and time involved in developing your own in-house solution; chances are someone else has solved the issue in a commercial product.
  • On-going maintenance, bug fixes, feature requests, etc, are never-ending. Chances are that in-house expert who’s writing your custom solution isn’t cheap, and you hired them to work on something else.
  • Knowledge sharing is pretty rare for custom solutions; what happens when (not if) the developer who wrote your in-house custom tool leaves’
  • Most custom solutions are written under time pressure, and may not be the easiest code to maintain or understand, especially if the maintainer is not the original author. (see the previous point about employees leaving).
  • When something is written in-house, the tendency is to continually add features to it (these are free, right?), when these features might not be strictly necessary from a business standpoint. Your company gets paid to deliver work to clients, and these “nice to have” features cost money and time to produce, but often don’t pay for themselves in increased productivity. Users will learn to adopt and adapt to off-the-shelf products, and can often produce the same amount of work for the same effort with them.

The supervisor is the dealer, and the pack of cards is the list of frames (the agenda).

The number of subjobs determines how many seats are present at the table. The players in the seats are the workers; actually, if a single worker has 8 slots (worker_cpus=8) and the job is set to reserve host.processors=1, then up to 8 seats at the table can be occupied by that single worker.

Cards are dealt one at at time to each player in a seat; frames get dispatched to subjobs. Once each player processes the card he’s been dealt, he asks the supervisor for another card to play (another frame to render).

You can increase the number of seats at a table while the game is in progress (while the job is running); you modify the job’s “subjob process” count (was called “cpus” before Qube! 6.0), and more empty seats appear at the table. If there are players milling about who aren’t doing anything, and they are capable of meeting the requirements for playing at this particular table (they have the correct app installed, enough RAM, whatever is specified by the job’s reservations and requirements), then they will take a seat and ask the dealer for a card. Now you have more players working through the deck of cards and the entire deck is processed in a shorter time.

Once all the cards have been dealt out, when a player finishes the card he’s been working on he’s told to leave the table, He goes back into the crowd, perhaps to sit at an open seat at another table if one exists.

So if you set the job’s “subprocess count” to 2, only 2 job slots on the farm will be working on the job at the same time; if the job only reserves a single processor (host.processor=1), then both those job slots could very well be on the same machine. If you set the job’s reservations to reserve the entire machine (host.processors=1+, which means “I don’t know how many cores you’ve got, but reserve them all”), then the 2 subjobs will run on 2 different machines. Each subjob will process 1 frame at a time, but you will have 2 frames being processed at the same time, 1 on each subjob.

It’s normal with Maya’s network render licenses for mental ray to see “denied” messages. It has to do with the unusual way mental images has implemented counting the floating mental ray network rendering licenses.

An Autodesk customer is entitled to 5 mental ray network render licenses for every floating license of Maya. Autodesk gives you an 1 R1 license, 1 R2 license, 1 R3, 1 R4, and 1 R5 license.

So if you own 7 floating copies of Maya, you will have 7 R1′s, 7 R2′s, etc.
Then, when a machine attempts to check out a license, it first asks for an R1. If all R1′s are in use, a “denied” message appears in the license server logs, and then client asks for an R2. If all R2′s are checked out, the client asks for an R3, etc, all the way up to R5′s. Only if the client is refused an R5 does (or should it) give up and decide it can’t get a license. So, the only ~actual~ “denied” message is when you see a denied message for the 85537MAYAMMR5_F licenses.

If you follow the logs, you will see a single machine initially ask for and get denied an R1, then work its way up the ladder until it either succeeds in getting a license or is denied an R5.

As for the license consumption, Autodesk explicitly does not allow sharing of network rendering licenses on a render node; they say this is to prevent “application servers” with the new machines with high processor core counts.

So if you are running 3 instances of the mental ray render on a single machine, you will see 3 licenses consumed.

With the way that mental ray multi-threads up to around 8 threads, it’s very close to as efficient to render a single 8-threaded render on one machine (and consume a single license) as it is to render 8 separate single-threaded renders (but consume 8 licenses). And the multi-threaded render only consumes a bit more than 1/8 the memory than the 8 single-threaded renders.

To limit the number of concurrent mental ray jobs, define a global resource on your Supervisor and then specify in the job reservation to reserve one of these global resources. Details on how to do this are here.

Richard Lewis and Michael Cooper

I recently had the pleasure of meeting Michael Cooper of Silverdraft Studios just prior to their official launch of MobilViz, the “first super-computer-powered digital and visual effects (VFX) studios-on-wheels.” Silverdraft is a Boise, Idaho-based company but their first MobilViz product has been parked in Hollywood behind previz experts The Third Floor for several weeks now in preparation for their launch event.

The vision of MobileViz is to provide a portable VFX studio that is self-contained with a week’s worth of fuel in a generator and every video and audio spigot there is coming out of the truck. Silverdraft designed the truck to contain best-in-class products and technologies in order to support near-real time previz workflows. Silverdraft selected Qube! as their render manager and are running over 1,500 CPU cores in a portable environment!

A recent IndieFilm3D article notes “Software and hardware found within the Mobileviz system include Autodesk MotionBuilder, Maya and 3DS Max; Mental Images’ Mental Ray renderer; Chaos Group’s Vray; Qube! render management from PipelineFX; Apple Final Cut Pro and Avid editing; plus on-set dailies and full color management capabilities. It also has 20TB of Micron solid-state storage.

Look for exciting developments as Silverdraft builds out their fleet of ever more powerful Mobile previz units.

(L-R) Richard Lewis, Phil Peterson, Troy Brooks, Michel Besner

The creation of this Board of Advisors marks a second important milestone in the strategy of PipelineFX in taking render management to the next level. Last year, the company launched their “Smart Farming” technology; a business intelligence system analyzing the power of production pipeline and providing integrated charting and reporting, and automating labor intensive render farm processes. The concept was developed after an extensive product validation process with customers across digital media verticals in major markets around the world. The Board of Advisors will help guide this “Smart Farming” initiative as well as other innovative ideas to deliver never seen before business intelligence solutions for production pipelines.

“We are extremely happy to welcome Michel, Troy and Phil to our Board of Advisors,”  said Richard Lewis, President and CEO of PipelineFX. “Insights and guidance provided by these industry veterans will help PipelineFX extend its leadership in render farm management solutions for digital media creation.”

Michel Besner is currently a business consultant and Executive Chairman of Artfox. He has been successful for more than 15 years in bringing to market innovative products and solutions for mobile, media and entertainment, including OZ Communications, the leading mobile messaging solution provider, acquired by Nokia in 2008. Before OZ, Michel was Senior Director of Product Management at Autodesk M&E. Michel also co-founded and was President and CEO of Kaydara, a pioneer in real-time 3D animation systems for entertainment, acquired by Alias Systems in 2004.

Troy Brooks is currently Head of Studio at Digital Domain Vancouver. He has more than 15 years experience in leading teams in digital media and has served as CEO of PipelineFX, Production Systems Supervisor at Square USA, Pipeline Supervisor at Vanguard Animation and Manager of Production Technology at Mainframe Entertainment.  Troy has also worked as a CG supervisor at Electronic Arts and Head of Engineering for Vertigo Software.

Phil Peterson is currently a consultant in the entertainment industry, where he has more than 15 years of experience advancing technology for animation and visual effects production at leading studios. Phil has held several key technology positions with the Lucas companies including overseeing R&D for Industrial Light & Magic, heading technology for Lucasfilm Animation in the US and Singapore, and as Principal Engineer at ILM. After nearly a decade with Lucasfilm, he joined Digital Domain, where he served as Chief Technology Officer. Most recently, he was SVP and CTO at ImageMovers Digital (a division of the The Walt Disney Studios). Phil began his studio career at Mainframe Entertainment and holds an M.Sc. in Computing Science.

Integrated Pipelines Tour with Tweak Software and Shotgun Software at Xing Xing in Beijing

PipelineFX has many customers already in China but I had not met BaseFX before. We visited their studio, met Chris Bremble and his team, and re-connected with some old friends from the Square USA days.

Chris is running quite the operation and is expanding rapidly. He has several hundred people and has his own training program to insure quality work. He shared some of their challenges and their current workflow and expressed a desire to raise the efficiency and feature set of many of their tools. They were about to forge into a major Houdini-based project that would require more rendering than anything they had previously attempted. As I unpacked my laptop and iPad for the demo Chris said “I wish someone would connect a render manager to Roambi so I can see what my render pipeline is doing the way I can see my businesses financial data on my iPhone.” I smiled as I turned on the iPad and showed him version 6 of Qube!, and a host of render farm data displayed via the Roambi interface. Things went well from there.

Chris visited our booth at Siggraph between meetings (Chris is one of the busiest people I have ever met). As we finalized our deal for several hundred licenses he commented “You had me at Roambi.” In the end, he purchased Qube! for render management, Shotgun for production tracking and RV for preview/playback. BaseFX’s production pipeline just took a huge leap forward. Chris graciously took the time to talk about his decision in a recent press release.

“We work pretty hard to maximize our resources, and when we looked at our render needs and the options available to us, we realized that only Qube! and PipelineFX had the tools and support to solve the issues we were facing,” said Christopher Bremble, President and CEO of Base FX. “As we’ve taken on more and more complexity in our work, and increased our render times, our needs for additional capacity took us a bit by surprise. We’d relied on internal tools for the last five years, which just couldn’t keep up with our workflow.  We knew we needed to bullet-proof our render management and sought a vendor that not only had an out of the box solution for all the applications we currently use – Nuke and Houdini – but also has integration with our next generation of pipeline tools, RV and Shotgun. Qube!’s new web-based reporting feature with Roambi was also a big plus for us.  I’m data dog, and want to see exactly what’s happening, and the Roambi integration had me smiling in seconds.”

“We’re also really excited about the API, and our ability to take the great platform Qube! offers and make adjustments quickly and easily to match our pipeline.  We have a lot of younger artists in the company, so we’ve scripted checking tools into the pipeline to make sure we’re being smart and efficient.  Qube! has made that process very straightforward.  The support has been amazing.”

Base FX is currently working on projects for all the major studios, including Soul Surfer, Conan the Barbarian, Roadkill, and Boardwalk Empire. The company recently won an Emmy for their work on HBO’s hit series The Pacific, and provided a large number of shots for HBO’s currently running series Boardwalk Empire.

“We are excited to be working with such a fast-growing studio in China,” said Richard Lewis, CEO of PipelineFX. “Base FX has done the work to build a talented team and an efficient pipeline that is paying off in terms of speed of shot turn-around and maximizing artist productivity.”

We also took an afternoon and went to the Forbidden City which was very cool. Unknown to us it was a Chinese national holiday so there were about 100,000 Chinese tourists assaulting the palace that day. Endlesss streams of elementary school children went by smiling and waving at what seemed like the only two Americans on the grounds. Then we walked over to Tienanmen Square. Lot of history there. All in all we had a great time, met a lot of nice people building CG businesses and ate a lot of good food, even if some was quite unusual (solid duck blood, etc…).

In 1986, while finishing a Bachelor of Environmental Design in Architecture at North Carolina State University, I attempted an independent study using a fancy mini computer, a digitizing table and some alpha solid modeling software donated to the school by a local engineering firm. My premise was that computers and specifically 3D animation and visualization would allow non-design professionals to understand the possible consequences of zoning laws on the built environment. In short, when lawmakers say how dense and how high buildings can be, twenty years later you have certain kind of city. What that city looks like and to some extent “feels” like could be anticipated via computer modeling and simulation.

So focused armed with the energy of youth and a clip board and measuring tape I proceeded to walk 36 blocks of downtown Raleigh N.C., stepping off the foot print of every building, noting materials and colors and guesstimating their heights. Ten class volunteers helped me and after two days of this and 60 pages of notes I returned to the Simulation Lab and my seat at a Cadmus mini-computer and began digitizing the site plan of those same 36 blocks as a footprint for the “as built” version of my Raleigh model. The OS was Unix, the software very simple, and everything was commandline. But the display was a 19” 24-bit color Rastergraphics and after a very slow re-draw the model was coming to life.

First I focused on creating the as-built model, then defining different zoning laws that might be implemented and were reasonable but would clearly result in a different looking capitol city. One was a “high-density on the capitol mall only” set of laws and the other was a “nine-square” rule that spread high buildings out across all of downtown. Then I designed the future cities to follow these rules, as well as take into account real world factors such as existing empty lots, the age of existing buildings and the likelihood of them being preserved for the next twenty years or being replaced.

Now to create these models I needed to “render” still images from different camera viewpoints, checking the colors and geometry of each building and road. In the first month of a four month project the screen re-drew in two to ten minutes. By month three, with all models complete, I was faced with thirty minute screen re-draws, or ‘renders.” Now the deliverable of this independent study was a slide show moving through these different models, allowing the viewer to experience the city of today and two possible cities of tomorrow. With 30 minute renders per camera location, and a desire to have many images to “dissolve” together to simulate animation and movement I realized I would need to almost move into the sim lab and sleep next the system, waking every hour or two to re-position the camera and start another screen render, sleep, wake up, take a photo (yes, 35mm slide film, I know, I know…), move the camera, back to sleep….for two months!

Even to show my professor my work each week was too render intensive to prepare much. I’d have a render done to discuss then move the camera once and we’d talk for half an hour while the screen painted line by line. Even then it was clear to me that once we begin using computers and rendered images as a critical part of a design project, the time to render is THE bottleneck in the review, critique, and re-work cycle. The amount of critical feedback from faculty, peers and even myself was limited by the number of times I could make decisions, implement those decisions in the model, and re-render the necessary images in the time allotted. Any increase in rendering speed would directly benefit the project by providing more opportunity for review, thought and change, resulting in a better overall project and more compelling presentation.

Even given the challenges of early computing hardware and software I was excited about the possibilities of visual simulation in design.  The accuracy, ease of perspective display, and just hi-tech look of the imagery foretold a useful place in any designers toolkit. Today design professionals, post production houses, broadcast television stations, visual effect studios and film makers around the world use computers and software to create, advertise, entertain, inform and amaze their clients and audiences every day.

In the education field computer graphics systems are not relegated to the graduate area of a simulation lab with limited student access. Labs are filled with multi-core desktops running Maya and Nuke, whole Mac labs running AfterEffects, editing labs with Avid and FinalCut, and even dedicated render farms are being built as a shared resource across an entire digital media program. Schools are deploying render management solutions to maximize the use of their computing resources and simplify the use and administration of the rendering process. Advanced render management systems like Qube! allow labs to share resources yet retain priority for their own users, making sure every CPU is processing someone’s project, reducing that creative loop and increasing the overall educational experience of every student.

Another recent challenge is transcoding. Students are beginning to shoot film on digital cameras like RED. To ingest the RED footage can take days on a single machine, and though there are special accelerator solutions the sheer amount of data can still delay time begin editing and review for many hours if not days. Distributing the transcoding across many machines reduces that delay dramatically. Advanced render managers manage the transcoding process as a job the same as any 3D render of 2D composite and divide the work across many computers. Students can begin editing much sooner and can re-shoot parts that require it much earlier.

Render times were the bottleneck to my design project in 1986, and in my opinion remain the primary bottleneck to improving the digital media education experience of students today. HD formats and stereoscopic rendering have recently exacerbated this problem. The good news is that by deploying a smart render manager and harnessing existing computing resources the iterative cycle can be reduced significantly with little expense. Once a program has all of it’s computers working at high utilization rates (which you should know from the reports your render manager provides) then budgets can be planned for a dedicated but shared render farm. It is interesting to note that as of this writing I am finding that China leads the way in dedicated render farms for digital media education, commonly deploying fifty to one hundred or more dedicated rendering servers per school. The Chinese government funds these schools and they run classes nearly 24 hours a day. they have recognized the importance of rendering in the feedback loop of their teachers and students and devote budget and staff to making sure that loop is as short as possible.

As I visit schools around the world this message is well received and changes embraced to improve the iterative cycle and maximize existing investments in infrastructure and computing power. Simply deploying smart render management is a huge step in the right direction.

A potential customer calls our sales department and asks “Can I just install Qube! on our workstations and turn them into a render farm?  We don’t have a file server, we all work off of local disks.”

If your site is running Linux or OSX and you have a limited number of machines (say, less than 10), then you could possibly have each workstation function as a file server.  Each machine would then have to mount all other machines’ filesystems, and the path to any scene or scene component would have to include the machine name.  This gets a bit tricky when the file lives on your own machine; you have to treat it as if it lived on a remote filesystem.  Your machine would have to be configured to mount its own exported filesystem the same way that other machines would mount it.  This is known as a loopback file system, and setting this up is beyond the scope of this post.

I’m going to say that I strongly recommend that you don’t try and run without a central file server if the job content resides on machines running a Windows desktop OS.

Qube! doesn’t have a file transfer agent that would copy files around and you wouldn’t want to add that load to your network anyway.  It relies on the networking setup present at your site.  All workers have to have visiblity into the shared filesystem where the files resides.

While you could set all paths in the project to UNC and include that machine name of the host where the files reside (simulating a loopback filesystem), which would allow all machines to access files on that host’s local disk, Windows desktop operating systems do not support more than 5 connections from remote hosts doing CIFS access (remote machines accessing the local host as a file server).  They’re only meant to share disks with a limited number of machines under very light load; if you’re trying to act as a file server, Microsoft wants you to buy a server OS.

When more than 5 hosts try and access a Windows desktop OS as a file server, the connections from the remote machines are dropped in an unpredictable manner, and are not reinstated.  It will work if you only have 1 or 2 subjobs running at the same time accessing the job content, but as soon as you try and run more than 5 or 6 subjobs at the same time, you’ll get nothing but errors and hung jobs, since the remote workers will either be able to only intermittently connect to the machine where the files reside, or they’ll get a connection that might be dropped in the middle of a read or write request.

Very Bad Things will happen…so just don’t do it.

“Qube! came highly recommended to us and its features matched up with the demands we had when researching a render manager,”? said Curtis Linstead, Systems Administrator at Rodeo FX. Rodeo FX chose Qube! and initially customized the artist submission interface for Shake down to a two click process. For peak periods, Rodeo FX pushes render out to high-end desktops to help get the work done. Qube! is installed on all systems in the studio.

Qube! is great for render management, and I’ve also found other powerful uses for it as a system administrator,” said Linstead. “Since Qube! is installed on all the workstations and all the dedicated render systems I can push out all my software updates and installs to the users without interrupting their workflow. This saves me time and is less intrusive to the artist.”

The studio has recently added Nuke for compositing and now the artists enjoy the same ease of submission they enjoyed with Shake.

“With the seamless integration of Qube! with Nuke, Shake, and XSI our render pipeline is really streamlined,” added Linstead. “The in-application submission helps speed things by automatically filling in many of the dialogue entries. Cross platform capability with Nuke has been really helpful. We can send renders from a Nuke-based PC system to a Mac OSX workstation for rendering and vice-versa. No compatibility issues. Jobs just run.”

Rodeo FX has made use of Qube!’s internal database and callback functionality. In computer programming, a “callback” is executable code that is passed as an argument to other code. It allows a lower-level software layer to call a subroutine (or function) defined in a higher-level layer.

“We are now leveraging the ability to create callbacks,” Linstead continued. “We render a job, then create a quicktime movie, then create a daily with a slate and finally move it to preview location. This level of render pipeline automation allows us to maximize our infrastructure and automate repetitive processes like dailies.”

“PipelineFX has been very quick to respond to all of my support requests,” concluded Linstead. “Support has been phenomenal. Cases get immediately resolved and we benefit from the deep render pipeline knowledge held by the support team at PipelineFX. I look forward to working together as we grow our studio.”

Rodeo FX is a good example of a smaller studio with lots of experience leveraging the best in technology to their competitive advantage. Since our companys’ birth out of the Square Pictures studio in 2002, PipelineFX is proud to be able to deliver large studio features and functionality to digital media professionals around the globe.