CD32-tools

[jotd]

Thanks Patrik but it does not work.

I have added -allow-lowercase to the mkisofs command as follows:
mkisofs -quiet -V JOTD_GAMES -copyright JOTD_2016 -publisher JOTD -o %2.raw -allow-lowercase -relaxed-filenames -d -input-charset ASCII -output-charset ASCII -iso-level 3 -A "" -sysid CDTV %1

your script seems to perform properly, the CD boots (which doesn't work if I don't patch the .ISO) but it then cannot lock "C" directory, doesn't find "setpatch" ...

[patrik]

Originally Posted by jotd

Without it, it would work, but the UPPERCASE names are ugly, and I have a problem with my menu starter that scans files ending with ".start" in lowercase (that's lame!)

Noticed another issue with dots in filenames and mkisofs. You have to specify -ldots to have leading dots, else it will convert ".start" to "_start". Also, to be able to use dots anywhere inside the middle of the filename you have to specify -allow-multidot. Noticed this on "kick34005.a500.rtb" which turned into "kick34005_a500.rtb".

So now I am using the following commandline (which seems identical to yours except the dot-stuff added):
mkisofs -quiet -D -allow-lowercase -ldots -allow-multidot -V cd32comp -copyright none -publisher none -o cd32.raw -relaxed-filenames -d -input-charset ASCII -output-charset ASCII -iso-level 3 -A "" -sysid CDTV rootdir

Followed by the make_cd32_iso.py script and ending the process with pathTableUtil.py uppercase.

Strange that I am getting an iso that does work. Can there be some ordering issue or something that is different on windows? How does the order of the path table look at your end compared to when you create an iso with ISOCD?

[patrik]

Tested the iso on real hardware and it works fine there too. However, it seems like mkisofs orders the files in a very bad way for the CD32 or something as the boot time is really bad.

Have added a simple boot benchmark to the menu we are working on - when it has loaded everything and shows its screen, it records the number of ticks the system is at with dos.library/DateStamp(). As the CD32 has no RTC it will always be set to 0 ticks during reset, so can be used as a comparison to see how changes affect boot time.

It is ofcourse quite a bit higher than the total time your own code has spent booting as it is zeroed very early, several seconds before it even shows the boot anim, I approximated the zeroing by stopwatch to atleast 4 seconds before the CD starts booting, but as a comparison it is great.

Anyhow, enough blabbering, here are the results:
boot time with mkisofs iso: 16.54s
boot time with ISOCD iso (100% standard): 12.96s
boot time with ISOCD iso (prioritization from buildiso.sh): 11.96s

[jotd]

@Patrik: I'll check my path table. In the meanwhile, with all the problems induced by mkisofs (speed, compatibility), I guess I can still use it for my tests, and use ISOCD when producing the release for people to enjoy fast boot times.

Edit: start of my path table:

PathTable size: 1578
00000-00009(1):
00010-00019(1): C
00020-00031(1): DEVS
00032-00045(1): GAMES
00046-00057(1): LIBS
00058-00073(1): MENUDATA
00074-00083(1): S
00084-00099(2): DEVS/KEYMAPS
00100-00117(2): DEVS/KICKSTARTS
00118-00137(2): GAMES/ANOTHERWORLD
00138-00151(2): GAMES/APIDYA
00152-00167(2): GAMES/AWESOME
00168-00187(2): GAMES/BMXSIMULATOR
00188-00203(2): GAMES/CHASEHQ

Edit2: I had a hunch, and renamed all my dirs in uppercase (no problem with that). The cd went a little further.
Then I noticed that all the files starting with an uppercase letter were found, whereas the others (only lowercase) did not. I've created a small ISO, do you want me to zone it so you can analyse it?
The workaround would be to go deeper in your conversion: convert tables to first letter uppercase, and also all the filename references (that, you're not doing right now).

[jotd]

Patrik, I just checked, and you can add several root directories to mkisofs. Of course, you would have to create a rootdir with only priority items (C, S, DEVS??) and a rootdir with the games. Maybe that would be an equivalent for the optimization you perform with the isocd layout.

[patrik]

Originally Posted by jotd

Then I noticed that all the files starting with an uppercase letter were found, whereas the others (only lowercase) did not. I've created a small ISO, do you want me to zone it so you can analyse it?

Please zone it. I can take a look at it tomorrow, away right now.

[jotd]

zoned as "prince.zip". If you boot it you'll see that files like "Freeanim" are found whereas files like "cd32load" (starting with lowercase) are not found (same for C directory, worked for "s" strangely)

[patrik]

Originally Posted by jotd

zoned as "prince.zip". If you boot it you'll see that files like "Freeanim" are found whereas files like "cd32load" (starting with lowercase) are not found (same for C directory, worked for "s" strangely)

I currently have no simple solution ready for this issue, I just wanted to inform you that the issue is the ordering of directory entries containing the file and directory names.

The CD32 expects them to be sorted case insensitive, which is the way ISOCD orders them, while mkisofs orders them case sensitive.

Take a look at the order difference of your /C directory between mkisofs and ISOCD:

$ isoinfo -f -i prince.iso | grep ^/C/
/C/Assign;1
/C/Avail;1
/C/CDSpeed;1
/C/Copy;1
/C/Date;1
/C/Delete;1
/C/Dir;1
/C/Ed;1
/C/Edit;1
/C/Execute;1
/C/Freeanim;1
/C/Info;1
/C/MakeDir;1
/C/Mount;1
/C/Rename;1
/C/SetPatch;1
/C/Status;1
/C/Wait;1
/C/cd32load;1
/C/cdtest;1
/C/hardreset;1

$ isoinfo -f -i prince_isocd.iso | grep ^/C/
/C/Assign;1
/C/Avail;1
/C/cd32load;1
/C/CDSpeed;1
/C/cdtest;1
/C/Copy;1
/C/Date;1
/C/Delete;1
/C/Dir;1
/C/Ed;1
/C/Edit;1
/C/Execute;1
/C/Freeanim;1
/C/hardreset;1
/C/Info;1
/C/MakeDir;1
/C/Mount;1
/C/Rename;1
/C/SetPatch;1
/C/Status;1
/C/Wait;1

So the reason it worked for the S directory was that the only file there was lowercase. As long as the case is the same in a directory the ordering is consistent, so a temporary and ugly solution would be to use only lowercase or only uppercase names.

[jotd]

I understand that the search algorithm is not linear (rather bisect-like) and relies on the sorting of names.
Thanks for the info, I'll figure something out.

Edit: found that you can generate a sort file for mkisofs:

http://reboot.pro/topic/7032-need-he...-mkisofs-sort/

That would help right?

[patrik]

It would have been a neat solution to create a sort file with all the paths in the right order, but the docs say:

Note: this option does not sort the order of the file names that appear
in the ISO9660 directory. It sorts the order in which the file data is
written to the CD image.

I have tried several times now and it seems to be true, cannot get it to change the file/dir order.

However, it could be used for optimizing boot time and file loading in general.

[patrik]

I retract my earlier statement about the path table being ok, of course it is also sorted case insensitive amongst entries in the same directory. Mkisofs @#€%!.

[patrik]

Updated the pathTableUtil.py script so that it sorts the directory entries case-insesitive/uppercase order plus it also fixes the sorting of path table.

The directory entries reordering stuff got a bit ugly, but it works as far as I have tested and does generate a working prince cd .

[jotd]

I'll test that ASAP. Thanks! your python stuff is great.

Edit: it works fine. Thanks! your python stuff is great

BTW maybe you could host the CD32.TM/CDTV.TM python script from SysX in your github. I'm sure he'll be OK with that.
I'll create a all-in-one python tool with proper arg parsing to chain all scripts. BTW defining a "doit" method and nesting all the code would be a nice way to use your scripts as modules, whilst retaining the standalone functionality with if __name__ == "__main__". I've done that with the generateLayout.py script and it worked great.

Edit2: it's not working so good with another image I've got. Some files are not listed anymore. I'll try to create some reproducer of some sort. Damn this problem is tricky.
Some guy made a modification for case insensitive sorting in the link I found. Maybe it would be worth it to rebuild mkisofs with those changes. A lot of fun to build stuff on windows but that I can do.

[patrik]

Realized I forgot to update the parent numbers when resorting the path table. Pushed an updated version of the script, please retry if that was the only issue.

[jotd]

it wasn't.

I have zoned a .zip with 2 ISOs. First one works (uppercase) and the other (exact same files) does not when it looks for DEVS:Kickstarts/kickstart file

(as a bonus, the OK CD is a fully functional Another World CD32 release )

BTW Patrik when you're building an exe (launcher) for a CD, I create a CD configuration with added a harddrive where I copy the exe and access it with the test CD startup seq.
That way I don't have to re-create the ISO each time I update the exe, until the final release.

[patrik]

Originally Posted by jotd

it wasn't.

I have zoned a .zip with 2 ISOs. First one works (uppercase) and the other (exact same files) does not when it looks for DEVS:Kickstarts/kickstart file

(as a bonus, the OK CD is a fully functional Another World CD32 release )

BTW Patrik when you're building an exe (launcher) for a CD, I create a CD configuration with added a harddrive where I copy the exe and access it with the test CD startup seq.
That way I don't have to re-create the ISO each time I update the exe, until the final release.

Cheers, will take a look!

Regarding the btw, that is a good solution. However, if we just get the iso creation fast enough, you won't care. The < 10second ISOCD build for our 281MB iso already is borderline won't care. A mkisofs solution has even greater potential .

[patrik]

The problem was that I sorted on directory entry names including the last ";1" part of it. That made kick34005.A500 and kick34005.A500.RTB appear in the opposite order. Pushed a quick update.

[jotd]

it works now! thanks (that was worth)

[patrik]

I have cleaned up the code a bit and moved most of the directory entry functionality into its own class. Haven't looked at the if __name__ == "__main__" yet, but sounds like a good idea.

[jotd]

Tell me when it's in the git repo and I'll adapt it. Now we have a full windows native toochain, not up to your standards in terms of boot speed, but still very convenient.