Remove Advertisements

7 GPU SLI??

Hardware, software, geek stuff

Moderators: laterna, Fridmarr, Worldie, Aergis, Sabindeus, PsiVen

7 GPU SLI??

Postby Aergis » Wed Mar 23, 2011 11:39 am

What would happen if you put 7 GTX580's into this mb?
http://www.newegg.com/Product/Product.a ... 6813131390


You'd have to first water cool them to make them single width instead of dual, and modify the back panels to make them fit in a single slot instead of double, but after that? You'd be running 6 cards at x8 and one at x16...

Would you be able to SLI 7 gpu? Is that even possible?
Image
User avatar
Aergis
Site Admin
 
Posts: 1144
Joined: Tue Feb 06, 2007 3:37 pm

Re: 7 GPU SLI??

Postby laterna » Wed Mar 23, 2011 12:51 pm

No drivers over 4 way afaik

Heat will still be an issue because the pump will not be able to force the flow between 7 cards . dual pump won't help since you can't really force a break between a cards unless you get lucky with a T junction between 2 cards...

You might be able to water cool it using 2 different loops but again, the space between the cards is limited so dunno.

Also, you might work on 6x8 but you won't have enough bandwidth for the data. The bus is said to start being bandwidth limited at 3-4 cards, so 7 will be hard on your data busses.

Other than that, go ahead, if you overcome heat issues (you can try using a dry-air cooler (you'll need to buy an industrial one, never seen one for pc's) and use it to pump 0C air into the case to cool them with smaller 1 width size fans (which will have to be custom made since I've never heard of such a thing for a GTX580.

All in all, why no 6990 ? 14 gpus sound mighty awesome in relation to 7 :p
laterna
Moderator
 
Posts: 325
Joined: Tue Mar 10, 2009 7:00 pm

Re: 7 GPU SLI??

Postby Aergis » Wed Mar 23, 2011 1:17 pm

Yeah, I was thinking of ideas for the heat.
Particularly fluorinert. Google it. Might work... Mostly I'm just toying with the idea though. 7 580s would run me $3,500 minimum... :shock:

And I'm not worried about the drivers, I don't think. I'm using it for data processing rather than video output. I *think* the drivers would be fine. So technically my SLI title is misleading. I don't them to function together in SLI at all. Just be available for computation.


As for why not the 6990, the processing I'm using requires CUDA, not openCL. Though lets see what Nvidia announces tomorrow with the GTX 590...
Image
User avatar
Aergis
Site Admin
 
Posts: 1144
Joined: Tue Feb 06, 2007 3:37 pm

Re: 7 GPU SLI??

Postby cerwillis » Wed Mar 23, 2011 1:27 pm

Yes, but would it fit in your case?

Aergis wrote:Image
Mistawillis - Cerwillis - Turbowillis - Evilan
<Silent Resolve> Durotan/SW:TOR Shii-Cho
Miniwillis <PK> SC
User avatar
cerwillis
Maintankadonor
 
Posts: 1306
Joined: Fri Oct 17, 2008 8:00 am

Re: 7 GPU SLI??

Postby laterna » Wed Mar 23, 2011 1:41 pm

I would be weary of dropping so much GPU's into one system. You'll run into other issues mostly in other areas...

CPU will start giving you a bottlneck. Given, its x58 platform so you can go for a 6core 990x , but I still feel it might give you some issues in handling data.

If you want my advice on things, if you do this, give it a good RAID array to provide enough flow of data to allow you to provide enough read/write MB's for 7 cores to work.

Also, you'll need atleast 2 or 3 PSU's.

I'd suggest the SR-2. It has 8 full PCI-E lanes, you can slot 24GB of RAM and 2 Xeons to overcome the CPU bottlneck. The only issue again, is the amount of bandwidth it offers. Also if you're gonna do something so big, call up eVGA. They'll be willing to give you on-hands support as they are really intrested in such things. Also, I'm sure they know more about the bandwidth limitation on the PCI-E lanes.

The liquid you showed me... its 600$ a litre for new, and 300 for processed. Are we seriously talking for upwards of 6-8k for a single system ? just to house 7 gpus? you can do more than that by splitting it to 2-3 GPU's per system, and house it each on its own.

It sounds nice as a concept, but 7 GPU's working under constant load will need just as much CPU power to feed them the data, enough motherboard bandwidth to allow the data to flow, and enough read/write MB's on your drives to allow you to load/unload data into your system
laterna
Moderator
 
Posts: 325
Joined: Tue Mar 10, 2009 7:00 pm

Re: 7 GPU SLI??

Postby Aergis » Wed Mar 23, 2011 2:04 pm

Yeah, the liquid is expensive, and I was considering building a custom case to reduce/eliminate open space. If you think about the empty air space that normally gets wasted in cases, you could considerably reduce the volume of liquid required by eliminating much of the dead space. And I've seen sources for $400 per gallon, not $600 per liter. It still likely beats the $110 per waterblock, plus rads plus pumps, and that's IF it would even work, like you said.

As for cpu bottleneck, again I think that's specific to the 3d rendering application I'm planning to use this for. It requires high video ram because it loads everything onto it. While it's rendering the cpu is close to idle. The drawback is I'm limited to 3Gb of vram ( if I go with the 580 ), but I can work with that.

When looking at splitting it up to 2 or 3 machines, that unfortunately doesn't work for this application, it requires the gpu in the current system to be used... At least thus far.

And the only other option is a Cubix expander box, which is $2,000 just for the BOX, no cards and it only holds 2... There's a $2,500 4 gpu version, again no cards.


Ultimately I have a project coming up in a few weeks that's going to require intense rendering times. I normally could use a traditional rendering farm if I was using a traditional render engine. But the specific one I am looking at using provides exceptional results for less work. Though it's capped in the GPU rather than network rendering across the farm.

Ideally yeah, I would just plop 1 or 2 new GPUs in each of the machines I already run for a render farm, but this application doesn't recognize the network in that way, at least not yet... And I won't have time for them to release v1.2 in the next 6 weeks...


Of course I could just go with the 4 gpu setup if I have to, but was curious to see the possibilities.
Image
User avatar
Aergis
Site Admin
 
Posts: 1144
Joined: Tue Feb 06, 2007 3:37 pm

Re: 7 GPU SLI??

Postby laterna » Wed Mar 23, 2011 2:41 pm

Well, the board you listed would work then, but still, call them up to make sure it can actually handle all of this and doesn't buckle under stress. Ask them about the bandwidth on the PCI-E lanes. Thats my biggest question right now
laterna
Moderator
 
Posts: 325
Joined: Tue Mar 10, 2009 7:00 pm

Re: 7 GPU SLI??

Postby Aergis » Wed Mar 23, 2011 2:53 pm

Yeah, good call.
Maybe they will even supply the parts for a PR story! lol
If it works, that is...
Image
User avatar
Aergis
Site Admin
 
Posts: 1144
Joined: Tue Feb 06, 2007 3:37 pm

Re: 7 GPU SLI??

Postby gibborim » Wed Mar 23, 2011 5:42 pm

I have a hard time imagining your PCI bus not becoming saturated with 5+ cards going at full bore. I imagine the feasibility of this setup being dependent on how much data your application actually needs to pass in and out of the cards to operate.

I guess I am having a hard time grasping the nature and constraints of your project.

What you need as I understand it:
>You need to use a specific rendering engine.
>Said engine does not allow cluster computing.
>Rendering time/power is a significant issue.
>Cost seems to be a secondary concern.


What I don't understand is why you seem married to this specific engine. Does your whole setup need to be mobile?
gibborim
 
Posts: 282
Joined: Sun Nov 02, 2008 12:13 am

Re: 7 GPU SLI??

Postby Aergis » Wed Mar 23, 2011 6:20 pm

Does your whole setup need to be mobile?


It's not mobile at all, so I'm not following there.

It's a 3D project in which I have to create as close to photo-real results as possible, with a large number of variations. Think of a condo company that lets you choose materials for your unit before you purchase it, ie floors can be hardwood with various types or tile, counters in the kitchen can be marble, granite, steel, solid wood, etc. I have to create each permutation... I am going to cheat as much as possible with transparencies, but some will need independant images due to reflections. If you change the fridge it will be visible in the reflections on the hardwood, for example.

The engine I'm currently planning to use can get great results quite fast, and pretty damn close to real. It's a fully interactive engine which lets me see updates and make corrections incredibly faster than the traditional "guess and check" method during texturing.

For example, I was able to get this out of it in a weekend of my free time... I was a bit overzealous with the post color correction and dof blur, but like I said I haven't gotten paid yet. It was just a test of the render engine.

Image


Compare that to the other engine I'm used to using. Much flater, less quality in the materials from the annoying process of test rendering every change, etc:
Image


For this type of work I think the new engine is FAR better suited.

Cost is a semi-secondary concern, I could just purchase a couple Cubix Expander boxes but it's considerably more expensive to do so. I'd like to get the same power for less money, or more power for the same money.
Image
User avatar
Aergis
Site Admin
 
Posts: 1144
Joined: Tue Feb 06, 2007 3:37 pm

Re: 7 GPU SLI??

Postby Aergis » Wed Mar 23, 2011 8:21 pm

Hah!
Got a contact back from the company who makes the renderer, they have tested this before and started to see dramatic diminishing returns after 5 cards. Perhaps it was for the bus problems you guys were mentioning...

Good to know, I'll stick to the standard 4way SLI and save myself some money, maybe figure out a batch script for multiple files rendering overnight...
Image
User avatar
Aergis
Site Admin
 
Posts: 1144
Joined: Tue Feb 06, 2007 3:37 pm

Re: 7 GPU SLI??

Postby gibborim » Wed Mar 23, 2011 8:36 pm

Ok, so it is really an issue of your preferred working environment.

As far as fitting a ton of high-end cards onto a single mobo, you should take a look at flexible PCIe extender riser cards. I don't know if those would produce any performance issues, but they would certainly alleviate the space issues from the double width cards. You would need to get a bit creative with your casing and cooling, but if you are really motivated(and assuming there are no overarching technical issues with the whole setup), you should be able to make it work. Maybe even with air cooling.
gibborim
 
Posts: 282
Joined: Sun Nov 02, 2008 12:13 am

Re: 7 GPU SLI??

Postby gibborim » Wed Mar 23, 2011 9:35 pm

Hah!
Got a contact back from the company who makes the renderer, they have tested this before and started to see dramatic diminishing returns after 5 cards. Perhaps it was for the bus problems you guys were mentioning...


Yea, if the renderer does not really interact with your CPU, then your PCIe bus will be your bottleneck. Later this year, there should be motherboards coming out with roughly double the PCIe bus bandwidth if you want to revisit the monster workstation idea later.

Good to know, I'll stick to the standard 4way SLI and save myself some money, maybe figure out a batch script for multiple files rendering overnight...


Maybe a two computer setup with a KVM switch? Work on one scene while you monitor the rendering of the other? I have no idea how complicated that would be though.
gibborim
 
Posts: 282
Joined: Sun Nov 02, 2008 12:13 am

Re: 7 GPU SLI??

Postby gibborim » Sat Apr 02, 2011 10:40 pm

What is the name of the rendering program? I would like to submit this question to the Tekzilla web show since they have graphics card expert coming on in a week or two and are soliciting questions.
gibborim
 
Posts: 282
Joined: Sun Nov 02, 2008 12:13 am

Re: 7 GPU SLI??

Postby knaughty » Sun Apr 03, 2011 3:34 am

Bah, whiny Mr McWhineyPants, back in my day we paid half a million for our desk-side visualisers, and we liked it!

One of these:
Image

Had one Geometry Engine:

Image

And up to four of these (we had four, we were building a flight simulator):
Image

And then finally you had one of these to generate your actual raster (analogue) images. This is the DAC on a modern card...

Image

Each of those cards is 34 cm by 36 cm (a bit over a foot each way if you're from Burma, Liberia or the USA, since you're the last three countries still using the average width of a man's thumb as a unit of measurement).

The main bus is VME Class D, but there was a subsidiary high-speed buss for sync and data transfer (think SLI connector and you'll be close enough).
This isn't the "Offtankadin" forum. My MoP FAQ: http://tinyurl.com/FAQ-5-0
- Knaughty.
User avatar
knaughty
Maintankadonor
 
Posts: 3558
Joined: Mon Dec 17, 2007 10:06 pm
Location: Sydney, plotting my next diatribe against the forces of ignorance!

Next

Return to Tech Stuff

Who is online

Users browsing this forum: No registered users and 1 guest


Remove Advertisements

Who is online

In total there is 1 user online :: 0 registered, 0 hidden and 1 guest (based on users active over the past 5 minutes)
Most users ever online was 380 on Tue Oct 14, 2008 6:28 pm

Users browsing this forum: No registered users and 1 guest
?php } else { ?