I have a dream - clustering miggys...

[Paul_s]

I want to blow top off Sysinfo speed tests and see how many MIPS we could manage by combining 10 A1200's with PPC's installed

Possible?
Necessary?
Cool?
Too Cool?
Pointless?
Are you daft?

etc

[rkauer]

Where is the poll?

[chiark]

One of the most practical applications of this would be in rendering, and I did have my miggy successfully controlling 2 pcs IIRC with Lightwave

[Paul_s]

I plan to...

'Run Windows 7'
Emulate Mac OSX
Emulate OS4

All at the same time

[Jope]

I hope this thread takes a turn to the serious after this post.

[DoogUK]

It just did with your post Jope

[chiark]

Paul's been on the happy pills a little early today, I feel...

[TheCyberDruid]

Originally Posted by chiark

Paul's been on the happy pills a little early today, I feel...

[Paul_s]

I've been feeling Amiga fever lately.

Anyway, seriously how would one cluster Amiga's together... kind of like a Google server?

hmmm

[chiark]

Er, you really are madder than a treeful of frogs. I do seem to remember some form of Occam being available on the amiga, but that was for a local transputer not a cluster.

You want to parallel the processors up to increase processing power...

To cluster like this, merely write a parallel programming language (or use parallel extensions to C), create an appropriate protocol (or use an existing one), write the algorithm you want in a parallel aware way, network the amigas, distribute the software then manage the execution.

Makes me wonder why no-one's done it before!

[rkauer]

They did...

[Retro-Nerd]

Skynet?

[rkauer]

No. The guys from Babylon5. They use 5 clustered Amigas to render the space ships.

[alexh]

Does clustering actually work for anything other than encryption breaking, mass SPAM emailing or detecting little green men??

The only implementation I've ever seen that works was the client-server batch data processing system. Where a client asks for a set of data, goes away and processes the data (where processing takes a long time compared to send/receive of data), and returns the result before asking for another set of data.

The concept of clustering was great, the implementations all rubbish. If programmers cannot even use the 4-cores in my CPU linked by incredibly fast buses, how on earth are they going to use ones which are connected by dog-slow ethernet?

Originally Posted by rkauer

The guys from Babylon5. They use 5 clustered Amigas to render the space ships.

Only in the pilot episode did they use Amiga's to render. They must have been dog slow.

[chiark]

we use clustering on websphere for high availability... It isn't just for getting more MIPS, but I guess you know that.

Using Lightwave screamernet was "fun", for a given value of fun. I really can't remember now if I had the 3000 controlling the 2 windows PCs or whether I had to copy the scene and stuff to the one PC and use that as the master... I *think* I controlled it from the amiga, but that's going back years...

[Thorham]

Originally Posted by alexh

Does clustering actually work for anything other than encryption breaking, mass SPAM emailing or detecting little green men??

Isn't it as simple as dividing up the frames over the individual cpus? Example: You have an animation of a 1000 frames to render and ten cpus. Just chop the animation into ten parts of a 100 frames each and let each cpu render a 100 frames. Or better, let each cpu render every tenth frame plus the cpu number. Cpu 1 renders frames 1, 11, 21,... Cpu 2 renders frames 2, 12, 22,... and so forth.

I might have gotten this wrong, but shouldn't it work like that in 3D rendering?

[Rabbit80]

Originally Posted by Thorham

Isn't it as simple as dividing up the frames over the individual cpus? Example: You have an animation of a 1000 frames to render and ten cpus. Just chop the animation into ten parts of a 100 frames each and let each cpu render a 100 frames. Or better, let each cpu render every tenth frame plus the cpu number. Cpu 1 renders frames 1, 11, 21,... Cpu 2 renders frames 2, 12, 22,... and so forth.

I might have gotten this wrong, but shouldn't it work like that in 3D rendering?

Or it could work like current SLi or xFire configs which can do split frame / alternate frame or even tile based rendering... Remember that not all processors will be able to do the same amount of work and that some frames may be way more complicated than others!

[Thorham]

Originally Posted by Rabbit80

Or it could work like current SLi or xFire configs which can do split frame / alternate frame or even tile based rendering... Remember that not all processors will be able to do the same amount of work and that some frames may be way more complicated than others!

Of course, but it would be more complicated that way. If you have cpus render one frame each, you can divide the remaining frames over the cpus which are done first. Or better, have a frame pool from wich free cpus select a frame to render until all frames are done. Much easier to implement, and less overhead when running

[xc8]

I think clustering *may* be possible, if you run netBSD, but we need network cards that 'support' clustering and of course the suitable s/ware etc etc...

[chiark]

Taking this back tothe original question:

You couldn't use Sysinfo in a clustered environment like this, even if you managed to cluster the miggys. You'd need to rewrite the benchmarking to be cluster aware.

For clustering to work, you need to quantise work in a way that can be split across distinct CPUs. This works nicely for rendering frames (a frame is a work unit), web servers (session is the work unit), but for raw MIPS testing you'd need to write a parallel aware benchmark.

[Predabot]

That doesn't exactly sound like something Paul can mock up in an hour, if I may be frank...

Anybody know who the guys that did the Babylon 5 pilot are? I think there's lots to be learned by e-mailing them and asking for tricks. They're probably closet amiga-freaks, so they could be delighted to know there's interest in their work from back in the day.

[Paul_s]

Originally Posted by Predabot

That doesn't exactly sound like something Paul can mock up in an hour, if I may be frank...

Anybody know who the guys that did the Babylon 5 pilot are? I think there's lots to be learned by e-mailing them and asking for tricks. They're probably closet amiga-freaks, so they could be delighted to know there's interest in their work from back in the day.

make that a few thousand hours!

Hmm I remember watching the B5 pilot all those years ago and being blown away by it all... very cool. Would be awesome to do something like this... yeah, ok it maybe primitive in this day and age but what's cooler? Some ghey IBM blade rack with pretty lights and a fancy UPS or some old skool retro shizzle kicking some ass

[Predabot]

Hmm, you know... if you ever get around to doing this in some way... some rendering or whatnot, you'll probably need at the very least 10-15 A1200's and pretty damn "tricked out" and modded ones too...

Have you started hording them Ami's, mate?

[Paul_s]

I could use the ones in my sig that would give some weird output

[cosmicfrog]

me want PIC`s of

4 x A500 all 1mb/GVP A530 Turbo (2MB/170MB SCSI)/68030EC) `s

Complete with wiring diagram and muckky finger prints

Plz

[rkauer]

Also we need your street address and time of the day when no one is at home. Can you give the alarm code too?

[DDNI]

Who needs hardware!
Why not run multiple instances of Winuae on several networked PCs?

Maybe we should go the whole hog and set up the Amiga version of Folding@home !!!

[Paul_s]

Originally Posted by cosmicfrog

me want PIC`s of

4 x A500 all 1mb/GVP A530 Turbo (2MB/170MB SCSI)/68030EC) `s

Complete with wiring diagram and muckky finger prints

Plz

I only have the 1 GVP 530 - I ain't won the lotto yet

About clustering...

IMO: it is the way to go. We get insane computation power for many tasks by using clustered CPUs. CPUs can then be local (MP CPU's second core that AOS is not able to use otherwise or another CPU on a PCI bus) or remote (typically somewhere in the LAN or WAN).

I think several examples were mentioned where clustering is already used. Another is compiling large projects in clustered environment. @work we use computer clusters to compile our embedded SW. The compiling takes hours if done on a single computer, by using a cluster it happens in about twenty minutes. There are a lot of ways to avoid bottlenecs of the system. Powerful cluster network can be build already by using 100mbit ethernet for interconnection, if done cleverly.
@home I noticed that open source DVD/MP4 SW seem to have clustering support with them as default. I rip the film, deside how I want it processed and send the job to the cluster. One such SW is DVD::rip.

etc...

Clustering is something the community can do to get POWER to the people NOW rather than in some future decade.

One example that I have not seen done anywhere: HD video playing by routing the MP4 data via a networked CPU (or several) to get it into a form that some low power Amiga client can show it.

It would be cool if we had a state-of-the art clustering tools & engines & etc available from Aminet to enable heterogenous clustering for every modern Amigalike-system owner


ALSO: IMO one example of a unigue way of using cluster (kind of) was PowerUP/WarpUP. Too bad that we had only one PPC. What if someone developes ClusterUP library that uses 100/1000mbit ethernet or 133Mbit PCI for the connection and more than just one "co-processor"...

[Mick_AKA]

I have seen a cluster of A4000's rendering in lightwave, my recollection is a bit iffy but I do remember them using some sort of custom interconnects with external ribbon cables.

Last time I did anything myself with clustering it was a project at my former employers using gigabit ethernet and 4 dual opteron workstations that were dead stock, at the time (pre dual core CPUs) the company wanted to find a way of squeezing 2 tyan quad processor boards into a single desk side unit and connecting the two board without purchasing very expensive custom interconnects which linked both the boards through PCIe.