Building a render farm seems rather simple on the surface.  After all, it doesn’t take a rocket scientist to grasp the basic concept that assembling a cluster of computers to render a series of frames will complete the task faster than a single computer, providing you have the proper software for the job. However, to do this efficiently requires an understanding of the cluster’s expectations and the environment in which it will operate. Regardless of what solution you buy, there will be hidden costs associated with it. This article attempts to uncover those hidden costs so you can make the best possible decision for your work flow.

Note: This article addresses the costs associated with high computation render farms. We plan to do this same cost analysis on a low computation render farm in the coming weeks so stay tuned :)

farm

There are many things to consider when building a render farm. It’s temping to assume that the more nodes and the faster they are, the better farm will be. From a computational standpoint, that is true. From a practical standpoint there are many more variables you need to think about that can add to, or subtract from, the utility of your farm.

We ran a cost analysis on several different high computation farm configurations. We took into account equipment costs, infrastructure costs, power consumption and cooling costs over three years. We realize there may be other contributing factors like equipment depreciation but for comparison’s sake, we will not get into those factors. Since the Option 4 system is the fastest, we configured the other options to match the computation power of Option 4 and analyzed the costs associated with each . Here are the results of our analysis… We used our render server product line for these examples but the concepts are universal and would apply to any similarly configured systems.

Option 1 – RenderStream Sub-Light Speed render server (34.4 required to equal Option 4)
Option 2 – RenderStream Light Speed render server (26.1 required to equal Option 4)
Option 3 – RenderStream Ridiculous Speed render farm (7 required to equal Option 4)
Option 4 – Renderstream Ludicrous Speed render farm

Year 0 represents equipment and infrastructure costs only.

Year 0 represents equipment and infrastructure costs only.

Option 1 is the cheapest right out of the box but once you plug the system in that will soon change.

Year 1 costs reflecting total equipment and power consumption to date.

Year 1 costs reflecting total equipment and power consumption to date.

After year one we start to see some of the effects of power consumption from running the farm and power consumption from cooling the farm. Option 3 and Option 4 have virtually the same power costs while Option 1 has the most expensive power costs. Overall Option 2 is the most expensive and Option 3 is the least expensive after one year.

Year 2 costs reflect total equipment and power consumption to date.

Year 2 costs reflect total equipment and power consumption to date.

After year two we see that Option 3 is still the least expensive with Option 4 not far behind. The power inefficiencies of Option 1 are starting to become clear yet Option 2 ended up being the most expensive overall.

Year 3 costs reflecting total equipment and power consumption to date.

Year 3 costs reflecting total equipment and power consumption to date.

After year 3 we see that Option 1 becomes the most expensive overall due to the power and cooling needs of all those servers. Option 3 is once again the best deal on paper but Option 4 is the easiest to manage. The price difference between Options 3 and 4 could be a wash if you consider the time spent maintaining each system.

Before we get into the details of our analysis we’ll provide a brief conclusion:
In order to stay competitive in the computer graphics industry, hardware upgrades are necessary but they don’t necessarily have to be expensive. In this article we observed how choosing the wrong system for your computation needs can lead to unnecessary costs long after the initial purchase your hardware. By taking into account the hidden costs discussed above, you will ensure that your next hardware purchase generates a nice ROI.

Appendix: Now let’s talk about the details and go over some examples…

For this article we used render times from the CPU performance benchmarks by Tom’s Hardware found here: http://www.tomshardware.com/charts/2009-desktop-cpu-charts/3DS-Max-2009,1380.html. The graph above represents those render times. Please note that there was no render time for the Nehalem 2.4 GHz so we made an estimate.

GHz benchmark

Our work shows that a double quad-core Nehalem (Xeon X5550) is slightly more than twice the speed of an i7-920. On our benchmark the i7-920 ran in 11 m 24 s and the dual quad core ran the same render with the same version of 3DS Max in 5 m 37 s. (http://www.3dspeedmachine.com/?page=3&scene=33)  Exactly half is 5 m 42 s and we believe the faster system is due to the larger GT/s of the Xeon which is 6.4 GT/s versus 4.8 GT/s for the 920.

Some of the key variables and assumptions related to this article are:
1. Monthly render farm utilization for this discussion will be 50% (This covers items 4, 5, and 6 for this discussion)
2. Time to solution for each computational computer in the farm (Taken from Tom’s Hardware Benchmarking table or extrapolated for processors not in the table using a best fit regressions analysis)
3. We assume for arguments sake that computational efficiency is the same whether we are using 1 computer or 100 computers (this assumption is of course not true since more computers will as an aggregate be faster but on average per unit will have poorer efficiency than a few computers. This is due to communication across the farm with the master. We typically experience in a cluster with proper backplane management and sufficient bandwidth to be better than 85% efficient and typically in the mid 90% range.)
4. The number of computational units that are needed to meet demand on average across a month. (covered by item 1)
5. The number of computational units that are needed to meet peak demand. (covered by item 1)
6. The number of times peak demand is needed per month. (covered by item 1)
7. The number of layer 2 or even layer 3 switches are needed (at 33W per switch).
8. The number of 42u racks used.
9. The amount of Un-interruptable Power Supplies used (not accounted for in this blog but certainly important).
10. Power consumption for running the render farm.
11. Power consumption needed to control the temperature of the farm environment at a desired set point temperature accounting for all equipment, lighting, human occupancy, windows and their facing direction and the room area for the render farm.
12. The number of master (or non-farm) computers in the server room.
13. For this article we assume that there are no cooling needs in the server room while farm isn’t in use.
14. To calculate the power consumption costs we assume the machines will run 12 hours a day for 30 days and assume 28 cents/kilowatt hour.

For all examples in this article we will assume the render farm room is 10 feet x 10 feet (3.048 m x 3.048 m) with no windows and two ceiling fluorescent lights each with two 40 watt T12 bulbs. We will assume a 5% monthly utilization for the lights and a 5% human occupancy rate for the room over 1 month. (If someone enters the room they will need to turn on the lights) For our air conditioning calculations we used a 12,000 BTU (one ton) unit drawing 1,170 watts.

Lets take a look at some example configurations and weigh the pros and cons with these assumptions…

option1

Option 1: The RenderStream Sub-Light render server is a single AMD Phenom II X4 945 based system.
Price: $1,075 (with 2 hard drives) after discounts
Specs:
Server Chassis : 4U Rack Mount Chassis with 1 internal hard drive
System Board : RenderStream AMD motherboard Asus M4A78-EM
Processor : AMD Phenom II X4 945 (Deneb 4c)
Operating System : Microsoft Windows Vista Business 64-Bit
Memory : 4GB (2 x 2GB) 240-Pin DDR2 SDRAM DDR2 800 (PC2 6400) Dual Channel
RAID Controller : Embedded SATA II 2 channel RAID storage controller with RAID 0, 1 support
RAID Configuration : RAID Level 0: Striped Disk Array without Fault Tolerance; requires 2 hard drives
Hard Drive : 250 GB SATA II Hard Drive 7200RPM 32 MB Cache
Optical Drive : Sony Optiarc DDU1681S 18x DVD-ROM Dual Layer Drive SATA Interface (OEM) – Black
Network Interface : Single 10/100/1000 Gigabit Ethernet Adapters Integrated
Slide Rails : Yes
Power Supply: 400W but operating at 280W

Equipment cost to equal computational power of Option 2: $1,397 with discounts. (1.3 required)
Equipment cost to equal computational power of Option 3: $5,267 with discounts. (4.9 required)
Equipment cost to equal computational power of Option 4: $36,980 with discounts. (34.4 required)

Example costs associated with Option 1  system:
Price to power 34.4 Option 1 render servers: $971/month for 12 months or $11,651/year
Price to cool server room using 34.4 Option 1 servers: $355/month or $4,264.83/year

Note: It will take 34.4 of the Option 1 nodes to equal the same computational power of one Option 4 system (14 dual quad core blades in a 7u enclosure).

Pros: From a price per node standpoint, Option 1 is an attractive option. The Phenom chip used in this server gets respectable render times compared to Intel’s i7 line (http://www.tomshardware.com/charts/2009-desktop-cpu-charts/3DS-Max-2009,1380.html)
Cons: If your workflow requires high computational power, this system becomes inefficient rather quickly. Regarding power consumption, if you run the Option 4 system for 12 hours a day for 30 days and assume 28 cents/kilowatt hour, you will spend about $459/month to operate that system. In contrast, if you run 34.4 of the Phenom nodes over the same time frame, at the same rate, you will spend about $971/month to operate all of those nodes!

Let’s look at it another way. If the Option 1 server costs $1075 and we need 34.4 to equal the compute power of Option 4, that amounts to $36,980. Add infrastructure costs like switches and cables and racks, the cost comes to about $39,740. The Option 4 system costs $45,553 including infrastructure. So the total cost over one year is as follows:

Option 1 -  $39,740 for all the nodes plus $15,916/year to power the servers and cool the room = $55,656

Compare that to an Option 4 system (14 dual quad core blades in a 7u enclousure)….

Option 4 – $45,553 for the system plus $7,706.05/year to power the severs and cool the room  = $53,259

Bottom Line: The total costs for equipment, infrastructure and power during the first year for Option 1 and Option 4 are fairly similar. Yearly power consumption for Option 1 is nearly double that of Option 4. By year three, Option 4 is $14,686 cheaper to operate due it’s more efficient power consumption. If you have short term goals for your render farm, Option 1 might make sense but overall, Option 4 is a more efficient and easier to manage system.

Recommended configuration for Option 1:
Number of Servers: 4

option2
Option2: The RenderStream Light Speed system is an i7 920 based render server.
Price: $1,880 (with 2 hard drives) after discounts
Specs:
Server Chassis : 1U Rack Mount Chassis with 2 internal hard drives
System Board : RenderStream i7 motherboard w/integrated graphics card
Processors : Intel Core i7 920 2.66 GHz L3 8 MB (Quad-Core)
Operating System : Microsoft Windows Vista Business 64-Bit
Memory : 3 x 2GB DDR3 PC3-10600 1333MHz
RAID Controller : Embedded SATA II 2 channel RAID storage controller with RAID 0, 1 support
RAID Configuration : RAID Level 0: Striped Disk Array without Fault Tolerance; requires 2 hard drives
Hard Drives : 2 x 250 GB SATA II Hard Drive 7200RPM 32 MB Cache
Optical Drives : Sony Optiarc DDU1681S 18x DVD-ROM Dual Layer Drive SATA Interface
Network Interface : Dual 10/100/1000 Gigabit Ethernet Adapters Integrated
Power Supply: 280W but operating at 238W
Slide Rails : Yes

Equipment cost to equal computational power of Option 3: $6,956 with discounts (3.7 required)
Equipment cost to equal computational power of Option 4: $49,074 with discounts (26.1 required)Again we will compare Option 2 (single i7 920 based render server) to Option 4 (14 dual quad core blades in a 7u enclosure) Remember, to get the equivalent computational power you will need 26.1 Option 1 servers to equal one Option 4 system.

Example costs associated with Option 2 system:
Price to power 26.1 Option 2 render servers: $626.15/month or $7,513.79/year
Price to cool server room using 26.1 Option 2 servers: $242/month or $2,906/year
Compared to…
Price to power an Option 4 system: $458.75/month or $5,505.03/year
Price to cool server room with an Option 4 system: $183/month or $2,201/year

Pros: The Intel i7 920 processor was shown to be 53 seconds faster than the AMD Phenom 945 used in the Option 1 server according to the Tom’s Hardware article: http://www.tomshardware.com/charts/2009-desktop-cpu-charts/3DS-Max-2009,1380.html

Cons: After year one, total cost for Option 2 is $55,656 compared to $53,259 for Option 4.

Bottom Line:
High infrastructure costs and high power demands make Option 2 a less desirable solution for a high computation farm. Option 2 is cost effective with a much lower number of servers in use.

Recommended Number of  Servers: 3

option3
Option 3: The RenderStream Ridiculous Speed render server is really a small render farm that contains two mother boards each with Intel Xeon E5530 2.4 GHz dual quad-core processors and 6GB of memory in a 1U box.
Price: $6,007.50 after discounts
Specs:
Server Chassis : RenderStream 1U Rack Mount Dual Node Chassis with 2 Hot-swap hard drive bay each node
System Board : Intel X8 Render System Boards (2 Boards per server)
Processors : DUAL Intel Quad-Core Xeon E5530 2.4 GHz 1066 MHz L2 8 MB 5.86 GT/s QPI w/HT & TB per board ridiculous
Operating System : Microsoft Windows Vista Business 64-Bit
Memory : 6GB DDR-3 PC3-10600 1333MHz Registered ECC per board
RAID Controller : Embedded SATA II 2 channel RAID storage controller with RAID 0, 1 support
RAID Configuration : RAID Level 0: Striped Disk Array without Fault Tolerance; requires 2 hard drives
Hard Drives : 2 x 250 GB SATA II Hard Drive 7200RPM 32 MB Cache per board
Optical Drives : 20x DVD+/-RW External Drive
Network Interface : Dual 10/100/1000 Gigabit Ethernet Adapters Integrated
Slide Rails : Yes
Power Supply: 1200W running at 325W/computer. Total 650W/server.
Equipment cost to equal computational power of Option 4: $42,052.50 with discounts (7 required)

It will take seven Option 3 systems to equal the computational power of 1 Option 4 system.

Example costs associated with an Option 3 system:
Price to power seven Option 3 systems: $458.92/month or $5,507/year
Price to cool server room using seven Option 3 servers: $183/month or $2,200/year
Compared to:
Price to power an Option 4 system: $458.75/month or $5,505.03/year
Price to cool server room with an Option 4 system: $183/month or $2,201/yearPros: With 4 Xeon E5530 2.4 GHz cores in a 4U space, Option 3 provides an incredible amount of computational power in a very small form factor.
Cons: Beign a rack mounted system, Option 3 is not as easy to manage as Option 4 which consists of 14 dual qual-core blades in a 7U enclosure.

Bottom Line:
Option 3 is an excellent choice for its computation capabilities and its efficient energy consumption.  After three years of power consumption, Option 3 is cheaper than Option 4 by a narrow margin.

Recommended Configuration:
Number of Servers: 7

option4
Option 4: The RenderStream Ludicrous Speed render server is a full render farm with fourteen dual quad core blades each with 6GB of memory in a 7U box. Price: $45,553 with discounts

Specs:
Server Chassis : RS Blade 7U Rack Mount Chassis with Management Module, includes KVM-over-IP/LAN
System Board : Intel X8 5500 Render Processor Blades
Processors : DUAL Intel Quad-Core Xeon E5530 2.4 GHz 1066 MHz L2 8 MB 5.86 GT/s QPI w/HT & TB 7Uenclosure
Operating System : Microsoft Windows Vista Business 64-Bit
Memory : 6GB DDR-3 PC3-10600 1333MHz Registered ECC (6 x 1GB)
RAID Controller : Embedded Intel SATA-II RAID Storage Controller with RAID 0, 1, 5 Support
RAID Configuration : RAID Level 0: Striped Disk Array without Fault Tolerance; requires 2 hard drives
Hard Drives : 2 x 160 GB SATA 3.0Gbs Hard Drive 7200RPM 32 MB Cache
Optical Drives : 20x DVD+/-RW External Drive
Network Interface : Port Gigabit Ethernet Switch Plug-in Module
Slide Rails : Yes

Example costs associated with an Option 4 system:Price to power system: $458.75/month or $5,505.03/year
Price to cool server room: $183/month or $2,201/year
Total energy costs: $7,706/yearPros: Extremely powerful render farm system. 112 hyperthreaded cores in a 7U space running at an energy efficient 325W/blade.
Cons: High upfront cost.
Bottom Line: If you have the computational needs, Option 4 is an great choice. As we saw in the charts above, Option 3 is slightly cheaper after 3 years. That price difference could be a wash if you take into account the time spent maintaining the systems.

References:
Tom’s Hardware CPU comparison chart: http://www.tomshardware.com/charts/2009-desktop-cpu-charts/3DS-Max-2009,1380.html
Calculating heat load: http://www.tombling.com/cooling/heat-load-calculations.htm

For more information on RenderStream renderfarm solutions please visit http://www.renderstream.com/renderservers.html

RenderStreamLogo