Cyclone 20 - idea for homemade USB-based floppy controller (Catweasel-ish)

[alewis]

Originally Posted by chiark

Just not Silica, according to the confessions thread!

No, i was always *given* XCopy and Cyclone, following reviewing SCA.

Had a nice result that show. In the afternoon I persuaded a young lad on the Silica stand to sell me a 14MHz 68000 board for a tenner, and an ICD FFV for a fiver :-)

[alewis]

For the life of me I can't remember who it was I used to meet, and knew me... I'm sure I've mentioned his name in a thread elsewhere, will have a gander.

bearing in minds its 15+ years ago, my memory - or impression - is that of someone with glasses. thats pretty distinct. Less distinct is dark hair and maybe a beard or goatee? "tall", but thats relative as everyone is taller than me...

Originally Posted by RichAplin

Hang on... let me see... hmm... two arms, two legs, a head.. is that you? :-)
No seriously, goodness knows, maybe; I can barely remember last week.
Claus did all the legwork on selling Cyclone; I imagine you met him if you dealt with any Cachet stuff.
...very hard-working guy, and refreshingly straightforward (i.e. paid cash) in his business dealings. He even took me to a crazy German fraternity initiation ritual that I don't think I'll ever forget. Thanks Claus! ;-)

[RichAplin]

Yay am finding some time and getting some stuff done and things are working without too much complaint.

"Normally clocked" timer resolution is 41ns, overclocked is 20.8ns. Due to the need to use USB those are then only two options, both of them function for me on both the boards I have, and both are 'better' than the fabled Trace Machine (although not a very relevent measurement). The nice thing is that with such good resolution we can dither the LSB to get any track length without introducing significant jitter (20ns=only 0.5% of a 4us '10' stream).

Have now got the ARM board controlling disks, pulling off the MFM, finding 4489's etc.
With any luck I'll have it reading and writing raw disk images from the PC today, and I'll post something promise.

BTW in terms of direct WinUAE control, it looks quite promising so far in terms of it all coming together. I'm assuming Toni will be more than happy to hack on that; likely using C# for the client end stuff is not remotely going to be suitable for WinUAE, but writing a UAE plugin talking to the same Microsoft WinUSB drivers from C is pretty straightforward and there's lots of sample code. Remains to be seen how we'll sort out the latency issue but it might not be a real problem unless running very low-level read+write apps like disk copiers.I expect the latency to be tens to (maybe) hundreds of milliseconds depending on the PC itself.

[hit]

i guess the geeks here are simply speechless, so I do add a _wow_
since you are discussing 3.5" disks at the moment, will we have support for 5.25" too, "just" by adding this to the software? i'm thinking to c64 disks.

[RichAplin]

Yes, should be just software.
...well, you'd need a 5.25" drive too unless you want to take a pair of scissors to your disks.

Just to briefly cover C64 disks
a) Can almost certainly read/write any C64 disks by connecting a regular 5.25" PC drive as found on ebay (not a 1541); only expected problem is the well-known issue of "Flippy Discs and the Index Hole" (which has got to be the sequel to "Hedwig and the Angry Inch").
b) Can probably emulate a 1541 drive to a real C64 without too much pain but not alas with fast loader/disk protection support (anything that attempts to reprogram the 'drive'). Full 1541 drive emulation (much more work) has been done on an FPGA (and separately in DOS) has already been done by various crazy people: http://commodore-gg.hobby.nl/innovat...1kaart_eng.htm


Someone's also done C64 cassette emulation, I think 'just because it was there'...
http://commodore-gg.hobby.nl/innovatie_DC2N_eng.htm ;-)
certainly that's easy to add to Cyclone20, again someone just needs to pop up + do it. ;-)


Regarding the idea of running a real 3.5" drive with WinUAE; it's worth pointing out that the best way for this to actually work in technical terms is for you to insert the disk and it to just read the whole thing onto the PC in one go.
This is 'boring' because it's essentially the same thing as emulating off IPF files,
a) you have to wait on disk insertion (pause winuae) while the whole disk is sucked to the PC.
b) you won't hear the floppy drive stepping around as the disk is being read, which is a cute retro thing (I know uae has a sound fx for this)

actually there's a bunch of fun solutions for this including dynamically building a cache of the real disk tracks on the fly in an opportunistic way, but let me stop typing waffle and get back to the code..

[hit]

had to search for hedwig on wikipedia, funny movie (or should i say weird?) it seems, so "obstacles" might occure with 5.25. but its good to hear, its somehow something possible
i saw the website of ultimate 1541. thats some impressive device, no question. didnt knew about the datasette, thats also remarkable .

[RichAplin]

Originally Posted by hit

had to search for hedwig on wikipedia, funny movie (or should i say weird?)

It's an awesome movie, rent it tonight. You'll be humming the songs for weeks.

[hit]

not tonight, but will for sure watch it, if i can catch it somewhere

[whiteb]

Originally Posted by hit

i guess the geeks here are simply speechless, so I do add a _wow_
since you are discussing 3.5" disks at the moment, will we have support for 5.25" too, "just" by adding this to the software? i'm thinking to c64 disks.

Are you kidding me, every time I come on, there is more Hardware Pron to drool over, Rich is making VAST strides with this project.

[hit]

i second this. Richard does some awesome job. thats something really impressing me; and the crowd here too, it seems

[RichAplin]

Thanks!
Before my head swells enough that I can no longer lift it, here's some stuff from the header file listing the command set:

// Cyclone specific requests
#define C20_CMD_BASE 0x28 //all cmds are 0x28xx where xx is ... (xx bit 7=set means set data direction of ep0 to c20->pc for rest of cmd packet)
#define C20_GET_DESCRIPTOR 0x80 //get version string
#define C20_RESET 0x01 //reset
#define C20_GET_STATUS 0x82 //get status
#define C20_SELECT_DRIVE 0x03 //motor on/off (=value)
#define C20_SEEK_TRACK 0x04 //seek where value= (track*2)+side
#define C20_SET_STREAMS 0x05 //value = set enabled streams
#define C20_FRICKIN_LASER_BEAMS 0x06 //value = ( 0x01=1mw red, 0x02=20mw green, 0x04=900mw blue, 0x80=10w Co2)

//Streams are just torrents of bits that go over EP1 and EP2.
#define C20_STREAM_ENA_WRITE (1<<0) //stream of write data over EP2 host->c20
#define C20_STREAM_ENA_READ (1<<1) //stream of read data over EP1 c20->host
#define C20_STREAM_ENA_INDEX (1<<2) //disk index pulse sent over command channel (EP0) (~5hz)


//these bytes appear in a raw stream (both to and from controller)
//pulse values shorter than about 24 clocks are impossible, so we use them as control codes in the RAW stream
#define C20_RAW_EXTEND10 0x00 //00-0xf are the MSB of an interval value (lsb follows), allowing intervals up to 0xfff clocks, which at 20ns CLK is 20us
#define C20_RAW_ENDSTREAM 0x10 //end of stream
#define C20_RAW_INDEXPULSE 0x11 //(read stream only) followed by time from last index pulse as a 24-bit value in clks
#define C20_RAW_WRITETOGGLE 0x12 //(write stream only) toggle write gate (start writing/stop writing)
#define C20_RAW_EXTEND24 0x13 //following 3 bytes are a 24-bit interval count (much more than one disk revolution at 20ns clk) usually used for delay-before-write
#define C20_RAW_WAITINDEX 0x14 //in a write stream this causes wait until index pulse
#define C20_RAW_OVERFLOW 0x15 //(read stream only) holy shineola there was a buffer overflow waiting for the PC to catch up - you missed a bit! This is bad to find in a read stream.
#define C20_RAW_LASTCOMMAND 0x17 //values above this are 8-bit intervals

...basically you send commands to start motor, seek to track, etc, and then enable a "stream" of read (or write) data that consists of intervals between disk flux transitions and the occasional embedded command (C20_RAW...). Hence the index marker noted as a specific command in the bitstream.

When reading a track you just position the head, start the read stream, and gulp data into the PC until you're happy, then you just turn the stream off and decode what you've read (you can do it on the fly). Currently I read 100k transitions which is typically about 1.5revolutions of the disk (depending on data). On the PC you then turn the interval timings into bits (basically a digital PLL; about 5 lines of code), find the 4489 sync marks and decode the track, if that's what you're after.

For WinUAE emulation Toni should be able to the raw Cyclone data and feed it straight into the emulated Amiga. As mentioned before it probably makes sense to do this with a bit of C++ as a DLL rather than my C# test app.

When writing a track to the cyclone you just generate (or read from an IPF file) an appropriate bitstream and pour it into the cyclone via USB, inserting "C20_RAW_WRITETOGGLE" at the points in the stream you want the write gate turned on + off, and end the stream with a C20_RAW_ENDSTREAM.

When copying a track you read it, process the bitstream to find the 'splice point' (i.e. where the drive that wrote the disk stopped recording), chop out the right section of the bitstream to get exactly one disk revolution, add "WRITETOGGLE" commands to the stream, and then write it.

The magic of making a really good disk copier is in this software stage where you process the read track and work out how exactly you're going to write it. You could do this on the ARM but why make life hard.
Specifically you need to work out;
a) where to start/stop writing because you'll unavoidably screw up a few bits of the track at that point.
b) you need to detect any sections of AGC noise (an artifact of the drive electronics turning up the gain; such bits are not actually present on the disk) and replace them with 'write nothing' (if you write back the AGC noise that you read, it'll probably be noticable to any copy protection looking for weak bits),
c) optionally whether to shift any of the bits slightly to add write precompensation,
d) "time stretching" the entire track slightly to make it fit precisely on the currently connected floppy drive that might be running slightly fast or slow. As mentioned before it seems there's a 'lot' of speed wobble on the drive I have at the moment so there is probably a fair bit of in-depth hax0ring that can be done to write a track juuuuuuust right even as the drive speed varies minutely from revolution to revolution.

....As usual for disk copiers, this is either some code that makes intelligent guesses based on the data, or in hard cases you just give it a parameter manually. I'd go as far as to say that there's absolutely nothing you can write on a floppy disk that this can't copy as long as it can get the parameters right. In most cases it'll be a piece of cake I think.

The SPS people (and Jens with Catweasel) have been tweaking this kind of raw analysis software for a long time I think, the SPS people have their own trace-like language which maybe they'll release in a tool for Cyclone20 users.


So anyway, this is the most basic (and most useful) mode where the Cyclone isn't doing that much except reading + buffering bit intervals over USB. All the thinking is on the PC side, which makes life easier. We can certainly do MFM decoding etc on the ARM, but I'm just about to go on holiday for 3 weeks so I thought I'd get the basics going first.

More when I can grab a second from my 'real job'. Meh.

[Crackersixx]

Are you going to put any code on sourceforge before you leave?

I've bought the same Olimex dev board that you bought and am anxiously awaiting some starter code

Thanks for all your hard work!
-C6

[mr.vince]

Looks good to me. Where will we do the diffusion to stretch or shorten the data we have, depending on the speed the drive has? Do we preformat the data on the PC side? I wonder if this will work for turning off the write gate, as we want to push it to the limit and keep the real gap ("splice point") as short as possible.

Best,
Chris

[RichAplin]

Ok really geeky.

This is the USB log when reading 120k bytes per track off disk + sending to the PC... (each line = one track)

USB has hardware handshaking and is polled by the host device, so basically our ARM chip is buffering up disk data until the PC comes along (about once per millisecond I think) and says "you got any data?". At this point the PC reads as much as it wants to in 64-byte packets.

Ok so, buffering statistics...

Bus reset
Set:0 : read:120603 usbpkts:1878 = 120192 bufmax:493 (maxto:4) : overruns:0
Set:0 : read:120687 usbpkts:1880 = 120320 bufmax:602 (maxto:5) : overruns:0
Set:0 : read:120920 usbpkts:1880 = 120320 bufmax:761 (maxto:6) : overruns:0
Set:0 : read:120915 usbpkts:1880 = 120320 bufmax:768 (maxto:6) : overruns:0
Set:0 : read:120837 usbpkts:1880 = 120320 bufmax:660 (maxto:6) : overruns:0
Set:0 : read:120886 usbpkts:1880 = 120320 bufmax:669 (maxto:6) : overruns:0

Ok so basically 'read:' = number of bytes off disk, usbpkts=# 64-byte USB bulk packets sent (=number of bytes) [note that more bytes were read than sent because the PC turns off the stream when there's always more waiting to go out]

The 'interesting' bit is bufmax: which is the 'high water mark' of the FIFO used to send data to the PC. I set the buffer at 32k bytes (i.e. can read up to 32k off disk before we need the PC to get around to reading it) , here we're seeing that the fullest the buffer ever gets (i.e. data off disk waiting to be read by the PC over USB) is about 800 bytes. This is pretty great.

The 'maxto' number is the # of which packet (of the 1880 sent per track) had the fullest buffer, and we see that it's around packet #6 (of 1880) which is entirely logical.
I've not looked at it in detail but I assume that the PC USB controller pings the device semi-regularly until it says it has data (at this point we get a few hundred bytes backed up), but as soon as there is data to be had, the PC just continually reads packets almost non-stop until there are no more or the PC needs to pause for breath (which it doesn't).


Note that plugging your Cyclone20 into a busy USB hub will alter this behavior, although will still be fine I'm sure. I have mine plugged direct into the PC. The app will have a basic "USB throughput" thing to show you if there are any problems.

Also, if I make my PC busy, the high-water mark can go way up (as the PC latency increases), but it's nice to see that (in near ideal conditions) the PC sucks USB data as fast as you can spit it out and the buffer sizes you can get away with are tiny. Still, I'm sure we'll use around 32k in practice because that allows for a lot (maybe 100ms) of USB polling latency.

In any cases a buffer over/underflow is no biggie and will just silently retry.

The reason (in passing) the first track read shown above is sending slightly less data (1878 packets not 1880) is simply the variability of the delay between the PC code reading its required 120k and sending the "ok, stop sending me data" command and the board receiving it and actually stopping. This is all fine, just pointing it out FYI.


Yay geekin'

[RichAplin]

Originally Posted by Crackersixx

Are you going to put any code on sourceforge before you leave?

I've bought the same Olimex dev board that you bought and am anxiously awaiting some starter code

Thanks for all your hard work!
-C6

YAY! Yes I will. I'm trying to squeeze in time here + there (between powerpoint docs) to fix it up, make it work reliably, remove dead code, etc.
Both C and C# bits are real junkpiles of bits of code found all over the place and duct-taped together to get it all basically working. About the only clean bit is the command defs I posted above.

I might just post a zip of the source (and binaries) rather than checking it into sourceforge as it's really all over the place atm and I'd rather tidy it a bit before checking it into subversion. I expect we'll pull stuff more into order and post code on sourceforge in January.


Did you get your board yet? ...Did you solder your wires? If so... show us some PCB Pr0n.

As far as I know only Mr Vince has been bold enough... ;-)

[RichAplin]

Originally Posted by mr.vince

Looks good to me. Where will we do the diffusion to stretch or shorten the data we have, depending on the speed the drive has? Do we preformat the data on the PC side? I wonder if this will work for turning off the write gate, as we want to push it to the limit and keep the real gap ("splice point") as short as possible.

Best,
Chris

The PC can do it; you could do it on the ARM but there's no big advantage.

The PC gets the drive index pulse interval from the current floppy drive, then looks at the track it's about to write, and stretches the track bits to fit (including dithering; a.k.a. diffusion).

In terms of getting the splice as good as possible; the ARM timing is essentially perfect, (30ppm clock error at worst) so it's capable of hitting the write gate timing right on the nail; the speed wobble of the drive is going to make things much harder to get right (i.e. you're using a precise timer to know when the end of the disk track is supposed to arrive, but in practice the disk itself changes speed all the time so.. we'll see.. If it's super critical you can just keep trying (write - read it back and check -write again) until you get it good enough. )

[Crackersixx]

Originally Posted by RichAplin

Did you get your board yet? ...Did you solder your wires? If so... show us some PCB Pr0n.

As far as I know only Mr Vince has been bold enough... ;-)

I haven't done the soldering yet, so it sits here mocking me.

[mr.vince]

Rrrright! Wanna see some p0rn! Can you beat Jackson Pollock or Mona Lisa's smile? ;-)

Rich... I think it would be ok to have a .zipped package first. We can then fiddle around and have an online meeting or similar in January and contribute our experiences back to you, and then put this into subversion... Regarding finding the splice point... this trial and error is what I had in mind. First prepare the track, then stretch it, write it, repeat until you overlap, then slightly reduce... Might need some more finetuning, but I guess trying would be ok. It's the same the first Cyclone did to sync the drives and I think you just have see if you're lucky.

Best,
Chris

[RichAplin]

Drivers
http://cyclone20.wiki.sourceforge.ne...SB+Drivers.zip

Unzip, plug in the ARM board (without floppy attached), when Windows asks for drivers point it at the unzipped folder.

(If windows doesn't ask for drivers and little pretty LEDs flash on your board instead, you need to reload the bootloader on the board (short TEST jumper, power on 10 secs, remove TEST, done))

Drivers for x86, AMD and IA64 are included.

Note that it grabs the USB ID used by the Atmel bootloader, so the Atmel flashing tool (previously dissed) won't work unless you uninstall this one.
You shouldn't need to do that.


Very broken super early alpha test executable V0.1 (now third (working hopefully) version)

http://cyclone20.wiki.sourceforge.ne...-12-23-08..zip (fxied again)

There's no point downloading this unless you have a board, and even if you do, it barely does anything useful to man nor beast, but it's a start.
Readme file in the main zip.

It comes with firmware that it downloads into the board RAM, which is what the delay is when you hit 'connect' (find the board, connect to the atmel loader, send new firmware, run it, then make windows disconnect the USB device and reattach it (ugh) in order to make the device re-enumerate ok). At this point the board is running my code out of ram and can read tracks.

If/when things hang up, you'll have noticed there's a reset button on the board. Hit that and hit 'connect' again in the windows tool. Unless that's hung too, of course. ;-)

If anyone wants board debugging output, plug a serial cable into to the serial port on the board next to the USB connector and look at a 57600 8N1 serial stream on your PC, which is printf output of the board firmware. (We can send this over the USB control channel also, to save hassle later on). I recommend "RealTerm" for such a purpose, which is great.

To avoid the mindless repetition of what is fast becoming my catchphrase, I'm going to spice it up a bit and say it backwards... ok .. ready...
"!YAY"

...hm.
Rich.



NOTE - link update to second version. This one reads disks to ADF files (which work, handily enough)

Temporary instructions:
Read readme file to install drivers etc.
Run program, hit connect. If it connects to the board ok, cool. (you'll know). Note that you don't need to do this again (hit connect) until you reset the board even if you restart the windows app.
Remember to set the lower slider to 79.1 to read a whole disk otherwise you get a very small .ADF
Stick a normal Amigados format disk of your chosing in the drive (you remembered the drive? good.)
Set filename of output ADF file in nasty text box.
Hit 'USBTest' to read disk (slowly for now; reads each track 4 times)
It should read disk, suck it in raw format to the PC, decode the Amgiados sectors (and complain if it can't read any) and output an ADF file. Run it on WinUAE. Bingo.

But apart from that it's still ugly and buggy. Sounds like there's people eager to sort it out, so I'll up the source before I go, and y'all can fix it.

Notes on the ADF files it makes: I tried a few (unprotected of course) disks that Mr. Vince sent me and they work great on WinUAE. Only regular Amigados right now. Other formats (IPF) etc will work fine when we put them in, and let you image protected disk. Writing disks also coming up Real Soon Now.
With any ADF files you make with these junk versions of the app, I recommend you don't keep them around as 'real' copies of your disks because the app currently it plays a bit fast+loose with the error checking and if it finds errors it'll go ahead and generate you an image with missing or broken sectors (although it does tell you).

[gizmomelb]

many thanks Rich! I hope you enjoy your holiday and we manage to come up with something great software wise by the time you return.

I'm so tempted to order the board now I can get my hands on it in a few weeks, rather than waiting until Feb when it's my birthday (and the finances are better under control).