Issues with non-RDB HDFs & full drive/RDB mode

[mark_k]

I'm having some issues getting non-RDB HDFs to work. The specific HDF is of a MO disk with 1273011 1024-byte sectors. I'm using manual geometry 24961/1/51 (C/H/S) in default.geo. A single FFS partition occupies the entire disk. I've enabled "Full drive/RDB mode" so WinUAE shouldn't be adding any fake RDB sectors.


The issues differ depending on whether the HDF is connected to uaehf or emulated GVP SCSI. There was also a change between 3.6.1b5 and 3.6.1b6 which affected things. (From winuaechangelog.txt: - Virtual RDB non-512 byte block size support. Last update also broke it if filesystem field was empty.)


When attached to uaehf:
1. With "Do not mount" checked, after booting Workbench I can double-click a mount file then access the disk's contents. So that works OK.

2. With "Do not mount" not checked (and bootable not checked), after booting to Workbench C:Info reports Not a DOS disk.
There is this log output (with both 3.6.1b5 and 3.6.1b6):

FS: mounted HDF unit MO0 (0000-4db2cc00, D:\00401B_B\00401B_B.bin)
Partition 'MO0' Dostype=444F5301 (DOS\1) Flags: 00000000
BlockSize: 1024, Surfaces: 1, SectorsPerBlock 1
SectorsPerTrack: 51, Reserved: 0, LowCyl 0, HighCyl 24960, Size 1243M
Buffers: 50, BufMemType: 00000001, MaxTransfer: 7fffffff, Mask: ffffffff, BootPri: -128
Total blocks: 1273011, Total disk blocks: 1273011
First block 0 dostype: 444F5301 (DOS\1)
RDB: fakefilesys, trying to load 'Z:\Shared\FastFileSystem_45.14_hacked', dostype 0x444F5301 (DOS\1)
RDB: 444F5301 (DOS\1) in FileSystem.resource version 40.1
RDB: faked RDB filesystem 444F5301 (DOS\1 45.14) loaded. ADD2FS=0
Filesys 444f5301 entry=40018f14

I guess something about the way WinUAE fakes the RDB isn't correct. uaehf doesn't support HD_SCSICMD/READ CAPACITY so I didn't investigate that any further.

When attached to GVP SCSI (Series I, gvpscsi.device 3.15):
1. There are no "Do not mount" or Bootable check boxes in Hardfile Settings for the HDF. Even though Full drive/RDB mode is enabled, WinUAE creates a fake RDB and increases drive size.
On 3.6.1b5: System boots to Workbench. Info shows the HDF is not mounted. There is this log output:

Adding SCSI HD 'gvp1' unit 1 ('D:\00401B_B\00401B_B.bin')
hfd attempting to open: 'D:\00401B_B\00401B_B.bin'
HDF 'D:\00401B_B\00401B_B.bin' 076D09C8 opened, size=1273011K mode=1 empty=0
hfd attempting to open: 'D:\00401B_B\00401B_B.bin'
HDF 'D:\00401B_B\00401B_B.bin' 0031BF00 opened, size=1273011K mode=1 empty=0
Geometry file '.\Configurations\default.geo' section '1303563264' found
Creating virtual RDB

So it seems WinUAE uses a virtual/fake RDB even though full drive/RDB mode is enabled (that button is ghosted out in Hardfile Settings). But something wrong about that causes the HDF to not be mounted. I can't access the HDF using the same mount file as before. The mount file has LowCyl=0 and I guess the fake RDB messes that up.

2. With WinUAE 3.6.1b6 (with entries in FileSys: and Device: fields in Hardfile Settings), trying to boot to Workbench I get "Not a DOS disk in device MO0" requester at boot. The requester causes Workbench screen to open before CLI/Shell window appears. Repeatedly click Cancel and eventually CLI opens. Can't do anything because of bogus/invalid boot drive though? It seems WinUAE ignores the Full drive/RDB mode setting and always adds fake RDB so partition is mounted and booted from. But the geometry is wrong (due to added fake RDB sectors), hence the Not a DOS disk requesters.


I used SCSIQuery to print some info about the (emulated) drive in that case on WinUAE 3.6.1b5. READ CAPACITY reports capacity 1273291 sectors, which is 280 more than the HDF size, and note the additional amount is not a multiple of 51. But the correct non-increased number of cylinders (24961) was in the rigid disk parameter page data.

After SCSIQuery printed
Device capacity:

Capacity in blocks : 1273291.
Block length : 1024.
Length in the middle of the medium in blocks: 38.
There was a requester saying:
gvpscsi.device: unit 001
Unexpected Status $49/$45
(That may not be a problem with WinUAE.) Click OK and requester reappears several times. Then SCSIQuery printed:
Primary defect list:
Read Defect List Header returned error :
Good.

Grown defect list:
Read Defect List Header returned error :
Good.

Archive containing config, mount files and test HDF:

http://www.media!fire.com/file/thk31gx3hjmejhb/MO_HDF_test.zip

[Toni Wilen]

Virtual RDB is added if:
- sectors and surfaces are zero (=RDB mode)
and
- first 4 bytes are non-zero and not RDSK

Geometry file is not included. Perhaps it modifies sectors or surfaces values and disables RDB mode?

[mark_k]

My default.geo contains this entry:

; 2.6GB 5.25" MO disk
[1303563264]
Surfaces=1
BlocksPerTrack=51
BlockSize=1024
FileSystem=Z:\Shared\FastFileSystem_45.14_hacked

Does the FileSystem= line cause WinUAE to always use a fake RDB, even when Full drive/RDB mode is enabled?

By the way... when WinUAE is using fake RDB, the number of extra sectors should (I think?) be a multiple of the number of sectors/track in order for the partition to be mountable. I saw WinUAE adding 280 extra sectors which isn't a multiple of 51. (Maybe a coincidence, but 280 1024-byte sectors is the same size as 560 512-byte sectors, and 561 is a multiple of 51...)

[Toni Wilen]

As I said, RDB mode = sectors and surfaces == 0. You override it back to non-RDB mode. Do you want some option to always disable virtual RDB?

Virtual RDB size probably is more correct now if block size > 512.

[mark_k]

When "Full drive/RDB mode" is enabled, I don't think WinUAE should add a fake RDB. Once you enable Full drive/RDB mode in hardfile settings, the button is ghosted out so you can't disable it, without removing that entry in the list and re-adding it.

With today's winuae.exe (2018-06-05 19:01): Connected via uaehf.device, Full drive/RDB mode enabled. Bootable box not checked, Do not mount not checked either. So WinUAE seems to be trying to mount the partition. This is in the log:

Mounting uaehf.device:0 1 (0):
FS: mounted HDF unit MO0 (0000-4db2cc00, D:\00401B_B\00401B_B.bin)
Partition 'MO0' Dostype=444F5301 (DOS\1) Flags: 00000000
BlockSize: 1024, Surfaces: 1, SectorsPerBlock 1
SectorsPerTrack: 51, Reserved: 0, LowCyl 0, HighCyl 24960, Size 1243M
Buffers: 50, BufMemType: 00000001, MaxTransfer: 7fffffff, Mask: ffffffff, BootPri: -128
Total blocks: 1273011, Total disk blocks: 1273011
First block 0 dostype: 444F5301 (DOS\1)
RDB: fakefilesys, trying to load 'Z:\Shared\FastFileSystem_45.14_hacked', dostype 0x444F5301 (DOS\1)
RDB: 444F5301 (DOS\1) in FileSystem.resource version 40.1
RDB: faked RDB filesystem 444F5301 (DOS\1 45.14) loaded. ADD2FS=0
Filesys 444f5301 entry=40018f1c

However after booting to Workbench C:Info says Not a DOS disk. Running SCSIQuery (which in the latest WinUAE now works on uaehf units) reports capacity as the correct no-added-sectors 1273011 sectors. [If WinUAE is not increasing the capacity, where is the fake RDB?]

With the HDF connected to GVP SCSI, WinUAE attempts to boot from it. (There's no Bootable checkbox to change that.) SCSIQuery reports capacity 1273291 sectors as before.

As I mentioned, connected to GVP SCSI WinUAE always seems to create a fake RDB and so the GVP ROM code tries to mount the partition, no way to stop it doing that. (And since the amount of extra sectors isn't a whole number of cylinders, it fails to mount.)

[Toni Wilen]

How many times I have to repeat it: you OVERRIDE RDB mode by setting surfaces/sectors count to non-zero. Ignore what the GUI button tells because it is not going to become inactive. RDB mode = no logical geometry!

EDIT: I also have problems understanding what are you trying to do because non-RDB hardfiles don't exists in real world and you still try to use it with emulated real world controller. They are always "rdb/full drive". So far it seems you are trying something unsupported.

EDIT2: Virtual RDB size is always divisible by surfaces*sectors*512. So it should be always cylinder aligned.

[mark_k]

By non-RDB hardfile, you mean a file which is an image of a single partition? Those do exist in the real world. Users of removable media (MO disk, DVD-RAM maybe, possibly others like Zip, Jaz, SyQuest) may use mount files to access those disks on real hardware. [That works fine with my GVP controller, and changing disks works fine too, which may not be the case if you install RDBs on your disks.]

I want to use image files like that (of a single partition covering the entire disk), like on real hardware. So not having WinUAE add any fake RDB, and I use a mount file to access the drive.

But it seems having an entry in default.geo forces WinUAE to always add a fake RDB? I renamed default.geo so WinUAE doesn't load it. Now WinUAE doesn't try to use a fake RDB. I can use a mount file to mount after booting Workbench when connected to GVP SCSI. Great.


Anyway. Let's suppose I have a single-partition image file, and do want WinUAE to automatically mount it by adding a fake RDB. There are some problems there, hopefully you can reproduce the issues from what's below.

Open Hardfile Settings. There's no way to disable full drive mode.
So I remove the HDF from the list on the CD & Hard drives setting page. Then click Add Hardfile... and add it back, with Read/write: unchecked, Bootable: unchecked, Do not mount: unchecked.

I have to manually enter 1024 for block size (since WinUAE defaults to 512 in Hardfile Settings). On booting WB, C:Info reports "Not a DOS disk" for MO0:.
WinUAE uses default/guessed geometry of surfaces 1, sectors/track 32. Which doesn't cover the whole partition. SCSIQuery reports drive capacity 1273267 sectors.

Then I changed the sectors/track number in Hardfile Settings to 51 (which is correct). WinUAE then says:
24961/1/51, 1273011/1273011 blocks, 1243.2MB/1243.2MB
So far so good.

However on booting to WB, C:Info reports "Unreadable disk" for MO0:.
This time SCSIQuery reports (via READ CAPACITY) drive capacity as 1273291 sectors, sector size 1024 bytes. That's an extra 280 sectors, not a multiple of 51. So the partition isn't mounted properly.

I used Scout to show info about the MO0: DOS device:
- SizeBlock = 128. It should be 256 (corresponding to 1024 bytes/sector).
- BlockSize = 1024
- SecsPerBlk = 2. Should be 1. I guess SizeBlock & SecsPerBlk would be OK if drive sector size were 512 bytes. But it's 1024 bytes.
- BlocksPerTrack = 51
- LowCyl = 11, HighCyl = 24971. That would be the correct number of cylinders (24972-11 = 24961).

So WinUAE should be adding 11 extra cylinders for RDB area. That's 561 (1024-byte) sectors. But it's only actually emulating drive capacity of +280 sectors (= +560 512-byte sectors). Maybe an off-by-one error somewhere?

[Toni Wilen]

Originally Posted by mark_k

By non-RDB hardfile, you mean a file which is an image of a single partition? Those do exist in the real world. Users of removable media (MO disk, DVD-RAM maybe, possibly others like Zip, Jaz, SyQuest) may use mount files to access those disks on real hardware. [That works fine with my GVP controller, and changing disks works fine too, which may not be the case if you install RDBs on your disks.]

I want to use image files like that (of a single partition covering the entire disk), like on real hardware. So not having WinUAE add any fake RDB, and I use a mount file to access the drive.

Yes, you have image file of amigados partition but when people are talking about UAE and RDB mode vs partition hardfile mode the important part is mounting method. Contents of image file are irrelevant. Mounting as a partition hardfile means using external metadata (geometry, filesystem etc. in config file) to simulate RDB automounting or adding virtual RDB. It is basically autobooting real world mountlist mounting method.

[Toni Wilen]

Geometry problem seems to happen because virtual rdb assumes 512 physical block size (it was designed for HDs only) but MO is 1024 byte.

So there needs to be 2 block size options.. Filesystem block size and physical block size to support (magneto)optical disks..

EDIT: and it is too late now. Later..

[mark_k]

Here's how I'd like it to work.

In full drive mode: If there's a geometry entry for the HDF, use the specified sector size(*) & C/H/S for emulated drive geometry (of course almost no programs use that, but it appears in fixed disk parameter mode page data).

In fake-RDB mode: If there's a geometry entry, use the #heads, sector size(*) & sectors/track for for the faked drive, adding some cylinders to give enough space in RDB area. Then the partition can be specified correctly in the fake RDB.

If there is no geometry entry, you could maybe also sanity-check the used geometry. E.g. if DOS\1 in first four bytes, check for root block where it would be if partition occupies entire file. Then adjust the auto-generated geometry to match.

The (*) because it's not quite that easy.
Some emulated-hardware SCSI controller drivers have problems with non-512-byte sectors. For those and all IDE controllers [assuming you're not emulating an ATAPI removable drive...] you can adjust the fake drive's geometry. If I remember correctly FFS earlier than v40 doesn't support SectorsPerBlock, but you don't need to use that.

Suppose geometry in .geo is 24961×1×51 1024-byte sectors. Then have the fake 512-bytes/sector drive size be (24961+n)×2×51 sectors. Mount parameters in the PartitionBlock can still have SizeBlock=256 (i.e. 1024 bytes) and specify Surfaces=1, BlockSize=1024.

[mark_k]

I took a look at hardfile.cpp/create_virtual_rdb(). Here are some suggested changes to make the with-fake-RDB drive always have 512-byte sectors (except if partition blocksize is 256). Obviously untested but maybe food for thought.

You have:

int blocksize = hfd->ci.blocksize;
int minblocksize = blocksize >= 512 ? 512 : blocksize;
int cyl = hfd->ci.surfaces * hfd->ci.sectors;
int cyls = (262144 + (cyl * blocksize) - 1) / (cyl * blocksize);
int size = cyl * cyls * blocksize;

Let's plug in the example geometry from my disk image, 24961×1×51:
blocksize = 1024
minblocksize = 512
cyl = 1 * 51 = 51
cyls = (262144 + (51 * 1024) - 1) / (51*1024) = 314367 / 52224 = 6.019... rounds down to 6.
size = 51 * 6 * 1024 = 313344

Building the fake RDSK sector later there is

pl(rdb, 4, blocksize); // blockbytes	// See below
...
pl(rdb, 16, hfd->ci.highcyl);		// ??? Should be hfd->ci.highcyl + cyls - 1 (as in PART block)???
pl(rdb, 17, hfd->ci.sectors);
pl(rdb, 18, hfd->ci.surfaces);		// See below
pl(rdb, 19, hfd->ci.interleave);
...
pl(rdb, 33, cyl * cyls); // rdbblockshi
pl(rdb, 36, cyl); // cylblocks		// See below

Suggested changes:

pl(rdb, 4, minblocksize);		// 512, except if filesystem block size is 256
pl(rdb, 16, hfd->ci.highcyl + cyls - 1);
// For rdb_Heads, multiply by (hfd->ci.blocksize / 512) for blocksize >= 512
pl(rdb, 18, hfd->ci.surfaces * (hfd->ci.blocksize / minblocksize));
pl(rdb, 33, cyl * (hfd->ci.blocksize / minblocksize) * cyls - 1);	// rdb_RDBBlocksHi - Note the -1
pl(rdb, 36, cyl * (hfd->ci.blocksize / minblocksize));	// rdb_CylBlocks

// Building the fake PART sector
pl(denv, 4, 1);				// SectorsPerBlock field

(I think the couple of changes in bold could apply to the current source as-is since they seem to be tiny bugs?)


Edit to add: I missed a few things... in each of the lines below change blocksize to minblocksize:

part = rdb + blocksize * idx;
fs = rdb + blocksize * idx;
uae_u8 *lseg = rdb + blocksize * idx;
if (lseg + blocksize > rdb + size)

[Toni Wilen]

RDSK block values are mostly irrelevant, only used by partitioning tools (but obvious bugs fixed).

Not using minblocksize is by design. I am quite sure there are drivers that expect 512 and nothing else, especially when looking for LSEG blocks. Even if filesystem uses 256 byte blocks.

This feature is designed to be safe with normal HD use case, not to be flexible with weird things that no on else will ever use

[mark_k]

Originally Posted by Toni Wilen

Not using minblocksize is by design. I am quite sure there are drivers that expect 512 and nothing else, especially when looking for LSEG blocks. Even if filesystem uses 256 byte blocks.

A filesystem with 256-byte blocks can't work on a drive with larger sectors, at least not without some intermediate driver to handle the conversion. The filesystem would issue CMD_READ/WRITE with IO_LENGTH = 256 or IO_OFFSET not being sector-aligned which should be rejected by the driver. So I think using minblocksize (which is only ever not 512 when fileystem block size is 256) is safe.

Edit: Think I misunderstood, you were talking about the LSEG data size. Are there drivers which do work with 256-byte sectors, but whose LSEG filesystem loader code doesn't?

Edit 2: Just use 512 for the LSEG blocks then?

Edit 3: Thinking about it some more, any drivers which can't handle LSEG blocks being 256 bytes (on 256 bytes/sector drive) probably will be hard-coded to assume 512-byte sectors anyway, requiring IO_OFFSET and IO_LENGTH be multiples of 512. They couldn't possibly work with a 256-bytes/sector filesystem. And "good" drivers expect LSEG data to match the drive sector size. (I just checked gvpscsi.device and scsi.device, both issue CMD_READ with IO_LENGTH = sector size.)

[mark_k]

In create_virtual_rdb() where you fill in the PartitionBlock, there is
pl(denv, 1, 512 / 4);
to set the de_SizeBlock field.

That's hard-coded for 512-byte filesystem blocks. Should it instead be
pl(denv, 1, blocksize / 4);
or (equivalently I guess)
pl(denv, 1, hfd->ci.blocksize / 4);

[Toni Wilen]

I think it should be working now. Main problem was IDE and SCSI emulation taking partition HDF blocksize value as physical blocksize. UAE controller worked because it mounts the partition "directly", no virtual RDB exists.

Now IDE and SCSI always uses 512 byte blocks if virtual RDB is detected (There is no easy way to extend support to 256 bytes, later if really needed..)

Your test image now boots in A1200 IDE.

[mark_k]

Thanks, I'll check that.

It would be nice to have some way to control the emulated drive sector size. Probably no IDE drivers support anything other than 512, and some SCSI drivers assume 512-byte sectors too. But others don't, and I've been meaning to experiment with 2TB+ FFS partitions. If that works at all it would probably need >512-byte drive sectors. [A SCSI-direct FFS might have issues because it couldn't use READ/WRITE(10) commands, READ CAPACITY (10) max. LBA would/should be $FFFFFFFF, and TD_GETGEOMETRY dg_TotalSectors would overflow.]

[Toni Wilen]

RDB/Full drive mode + ticking "Manual geometry" checkbox and adjusting block size should change drive's "physical" block size value. (=if it is 1024, SCSI READ would return 1024 bytes if number of blocks is 1 and so on..)

If RDB mode and emulated hardware controller: it adjusts "real" drive geometry/block size. If non-RDB: it adjusts partition's geometry and block size. (If non-RDB + emulated harddrive controller: broke >512 filesystem block size partitions because it previously adjusted both).

[mark_k]

Looking at the 4.0.0b14 source now.

In the RDSK block, physical drive characteristics:
pl(rdb, 18, hfd->ci.surfaces);

Should probably change that to reflect the possibly-larger filesystem sector size, otherwise physical drive characteristics in RDB won't match emulated drive size. (I think at least one driver may use those values to limit/sanity-check IO_OFFSET values.) So:
pl(rdb, 18, hfd->ci.surfaces * fsblocksize / hardblocksize);

Similarly instead of
pl(rdb, 33, cyl * cyls); // rdbblockshi
pl(rdb, 36, cyl); // cylblocks
maybe have
pl(rdb, 33, cyl * cyls * fsblocksize / hardblocksize); // rdbblockshi
pl(rdb, 36, cyl * fsblocksize / hardblocksize ); // cylblocks

Also, is the rdb_HighRDSKBlock field supposed to be the highest LBA used by all RDB sectors (RDSK, PART, FSHD, LSEG)? It should probably be higher than 2 then?


In the PART block:
pl(part, 3, 7); // hostid
Maybe change to reflect user-selected controller ID (for those controllers which support that)?

[mark_k]

Still on the subject of hardfiles (but not really related to the subject of this thread)...

If emulating a drive with more than 2^32 sectors (i.e. >2TB for 512-byte sectors), the max. LBA field returned in READ CAPACITY (10) data should be 0xFFFFFFFF.

In hardfile.cpp there is

case 0x25: /* READ CAPACITY */
		{
			...
			uae_u32 blocks;
			blocks = (uae_u32)(hfd->virtsize / hfd->ci.blocksize);
			if (hfd->ci.max_lba)
				blocks = hfd->ci.max_lba;
			...
			wl (r, blocks - 1);
			...

I think that means that with a >2TB 512-byte sector drive, blocks is set to the size modulo 2TB, so the max. LBA field won't be 0xFFFFFFFF?

[Toni Wilen]

Originally Posted by mark_k

If emulating a drive with more than 2^32 sectors (i.e. >2TB for 512-byte sectors), the max. LBA field returned in READ CAPACITY (10) data should be 0xFFFFFFFF.

I never bothered to handle it. Fixed but it probably isn't the only place that need fixing.

Also, is the rdb_HighRDSKBlock field supposed to be the highest LBA used by all RDB sectors (RDSK, PART, FSHD, LSEG)? It should probably be higher than 2 then?

Yes, apparently it should contain last block used by RDB. (But again, only partition programs care)

In the PART block:
pl(part, 3, 7); // hostid
Maybe change to reflect user-selected controller ID (for those controllers which support that)?

I am quite sure this is yet another field that means nothing. RDB should be transferable between different controllers and I have only seen 7 in that position.

Other changes done.