Getting started with WHDLoad development

[paraj]

Everything is handled by kick13 kickemu ($HDBASE/WHDLoad/Src/sources/whdload/kick13.s). If you compare the example2 source with the generic kick13 example ($HDBASE/WHDLoad/Src/slave-examples/kick13.s) you will see that all I did was copy that file and tweak the settings at the top.

1. Kickemu will - after patching - run the normal boot sequence of kickstart, including reading the boot sector, validating it, and jumping to it. The boot sector is not read explicitly by the slave. It's all done by the code in the ROM image, though of course the disk loading stuff has been patched (all handled in sources/.../kick13.s).

2. "blitz" is a standard WHDLoad macro ($HDBASE/WHDLoad/Include/whdmacros.i) that flashes the screen and waits for LMB. But notice again, that the code is not present in the slave, since "BOOTEARLY" is not defined.

3. Yes, this is "normal", and the meat of what goes into making a slave. WHDLoad needs to stay in control of the system (otherwise you can't quit or load from HD etc.), so the emulated program cannot be allowed to take over. You need to patch out all the "bad stuff" it potentially does (like modifying the VBR). Trap instructions may or may not be a problem, so WHDLoad has a flag to allow them (WHDL_EmulTrap). You need to analyze the program to see what it does, and whether the trap should be allowed (or patched around). It's often used to switch to supervisor mode as part of a system takeover.
If it is doing something problematic around the trap instruction, the "hooks" that can be enabled at the top of the file will come in handy. BTW the SNOOP options are useful for finding "raw" disk access, and other potential issues: http://www.whdload.de/docs/en/opt.html#Snoop

[field3d]

@paraj with your latest script. I could make boot and put in the right memory position all it seems ok. However, after click the mouse to move forwarded I have the error “ Exception "TRAP #0" ($80) at $7306A (Task 'exec.library') occurred.”, besides the snoop I note something that is not behaving as real amiga because in a real image the floppies work or in the winaue works 100% the disk images you have but not in the source code you sent. Something is not making to behave as a real amiga. In Winaue to run the images I usually use CPU 68000, ‘ECS Agnus” checked, ROM Kcik 1.3, RAM Chip 512KB or 1MB, Slow memory 0, Adv Chipset checked ‘Compatible Settings’ and done. What could be?

[jotd]

TRAP must be emulated using WHDLF_EmulTrap as paraj stated earlier.

But to create a proper whdload slave, it's better to know what the various executables are doing, by disassembling them and partially reverse the interesting parts.

[field3d]

I attach a photo of the part of the code but the code is not wrong as runs perfect in amiga and winaue is just whdload that has the issues. The last soruce code @paraj is perfect but is missing something to emulate real the amiga. I used to pu NOP in the trap and skips the error but it continues giving more errors similar in other parts of the code. Meaning Task 'exec.library' execution is causing problems. Maybe is possible in the soruce code enable or is missing to add to enable that support? As a library or something? But in the photo the line 73060 is giving error tryin to put in $80 the contents of $7306A. But I repeat although you skip this more and more errors of that exec.library. something is missing.

[jotd]

this is just to warp to supervisor mode. Then you need WHDLF_EmulTrap. Then I suppose that the game takes over the system and uses custom floppy routines, the next problem you'll have

BTW you should use winuae debugger, hrtmon is just not powerful enough.

[field3d]

Also I tried the suggested command line arguments to try when launch: NOCACHE,NOVBRMOVE and NOAUTOVEC. Not work, entering that arguments the trap error disappears no more errors but a pretty black screen. And as the images run perfectly in a real amiga or winaue, batocera, definitely is something to add in your latest slave source code @paraj. You can see what I mean with the disk images. Is really crazy this one

[copse]

This is probably an easy thing for someone with more recent m68k experience to point out. What is going on here?

vasmm68k_mot -pic -x -IC:\Data\Programs\whdload\/includes/NDK_3.9/Include/include_i -IC:\Data\Programs\whdload\/WHDLoad/Include  -IC:\Data\Programs\whdload\/WHDLoad/Src/sources -nosym -Fhunkexe -o ../PoolOfRadiance13_whd.slave PoolOfRadiance13_whd.s
vasm 1.9f (c) in 2002-2023 Volker Barthelmann
vasm M68k/CPU32/ColdFire cpu backend 2.6c (c) 2002-2023 Frank Wille
vasm motorola syntax module 3.18 (c) 2002-2023 Frank Wille
vasm hunk format output module 2.14c (c) 2002-2022 Frank Wille

error 35 in line 390 of "PoolOfRadiance13_whd.s": relocation not allowed
>               lea     (graphicslibra_MSG).l,a1

error 35 in line 396 of "PoolOfRadiance13_whd.s": relocation not allowed
>               lea     (intuitionlibr_MSG).l,a1

error 35 in line 402 of "PoolOfRadiance13_whd.s": relocation not allowed
>               move.l  (_gfxbase).l,a6

error 35 in line 421 of "PoolOfRadiance13_whd.s": relocation not allowed
>               movea.l (_intuibase).l,a6

error 35 in line 424 of "PoolOfRadiance13_whd.s": relocation not allowed
>               move.l  (_intuibase).l,a1
***maximum number of errors reached!***

[a/b]

Looks like a combination of using -pic to produce a fully pc-relative code, and having relocs (which are absolute and not pc relative).

[copse]

Originally Posted by a/b

Looks like a combination of using -pic to produce a fully pc-relative code, and having relocs (which are absolute and not pc relative).

Thanks, the messages were ambiguous to me about which way it went. Fixed and slave compiled. Ready for some experimentation..

[jotd]

vasm doesn't add ,pc if you don't specify optimization options. I usually don't use optimization on slaves. I wasn't aware of this -pic option which is super useful. Will add it on my toolchains.

You fixed it but yes, you need something like

lea     graphicslibra_MSG(pc),a1

Plus you cannot assign to a variable directly:

move.l  d0,flags   ; not possible in 68000 as position independent

you'd need

  lea flags(pc),a0
  move.l  d0,(a0)

[jotd]

BTW you'll be happy to hear that I'm in the process of refactoring my makefiles in my repository

There will be only a .mk file to configure (I may add env. variables too...) then all makefiles will look like this

#added by python script
include ../../options.mk
PROGNAME = AnotherWorld
WHDLOADER = $(PROGNAME).slave
SOURCE = $(PROGNAME)HD.s

all :  $(WHDLOADER)

$(WHDLOADER) : $(SOURCE)
	$(WDATE)
	$(VASM) -o $(WHDLOADER) $(SOURCE)

This is going to be a long process of refactoring all makefiles. I'll create a script for that but of course scripts often fail

[paraj]

Since this is a tutorial thread, I've given it a look, and I think it'll serve as an instructive example of how you'd get started. Unfortunately I couldn't find these exact images in TOSEC, but I've attached the first track since that contains everything we'll need here. Handily WinUAE accepts just a single track as a valid disk image.

For disk images you always want to start at the beginning: the bootblock. Start WinUAE and select the A500 (most common) quick start. Make sure the Compatibility slider is all the way to the left (Best compatibility), and insert "track0.bin" in DF0:.
Start emulating and immediately enter the debugger (shift+f12). At the prompt, type "f" and press enter. This will run until we're no longer executing from ROM, i.e. the system has finished initializing. Now use the "d" command to disassemble from the current PC.

00001564 2c79 0000 0004           movea.l $00000004 [00c00276],a6
Next PC: 0000156a
>d
0000156a 2a7c 0007 3040           movea.l #$00073040,a5
00001570 337c 0002 001c           move.w #$0002,(a1,$001c) == $00c014fe [0002]
00001576 234d 0028                move.l a5,(a1,$0028) == $00c0150a [00001558]
0000157a 237c 0000 0200 002c      move.l #$00000200,(a1,$002c) == $00c0150e [00000000]
00001582 237c 0000 1200 0024      move.l #$00001200,(a1,$0024) == $00c01506 [00000400]
0000158a 4eae fe38                jsr (a6,-$01c8) == $00c000ae
0000158e 4e71                     nop
00001590 4e71                     nop
00001592 4e71                     nop
00001594 43fa 0136                lea.l (pc,$0136) == $000016cc,a1
>d
00001598 7000                     moveq #$00,d0
0000159a 4eae fdd8                jsr (a6,-$0228) == $00c0004e

Now you need to know that when the bootblock is called A1 points to an IOStdReq that can be used to load data from the disk we're booting from.
I hope I don't have to mention that address 4 of course contains a pointer to ExecBase (and in fact A6 always contains this value when the boot block is started though that isn't documented).

To figure out what fields the offsets correspond to there is this handy reference: Structure.offs. Be sure to always have a LVO reference handy (like this) if you haven't memorized every one.
Translating the above into what you'd write in an assembly file (mentally or actually), we end up with:

	MOVEA.L	AbsExecBase,A6		
	MOVEA.L	#$00073040,A5		
	MOVE.W	#$0002,io_Command(A1)	
	MOVE.L	A5,io_Data(A1)		
	MOVE.L	#$00000200,io_Offset(A1)
	MOVE.L	#$00001200,io_Length(A1)
	JSR	_LVODoIO(A6)

From $HDBASE/includes/NDK_3.9/Include/include_i/exec/io.i we can see that command 2 is CMD_READ, and that the code is reading $1200 bytes from offset $200 on the disk to address $73040.

The next call is to OpenLibrary with version=0 and libName:

>m 000016cc
000016CC 6772 6170 6869 6373 2E6C 6962 7261 7279  graphics.library

This process can be continued to figure out what the rest of the code exactly does, and it's a good exercise, but it is not too interesting until this part:

0000161e 0839 0006 00bf e001      btst.b #$0006,$00bfe001
00001626 66f6                     bne.b #$f6 == $0000161e (T)
00001628 4ef9 0007 3040           jmp $00073040

So the code uses the system to load some code to $73040 and jumps there after displaying a message. Now you could put a breakpoint there "f 73040" and "g" to run to there, but let's switch gears instead.
We are interested in the code at offset $200 in the file, so let's extract that (I use python here, but substitute your favorite language if you like):

def read_file(filename):
    with open(filename, mode='rb') as file:
        return file.read()
disk = read_file('track0.bin')
offset = 0x200
length = 0x1200
with open('stage2.bin', 'wb') as f:
    f.write(disk[offset:offset+length])

In this case extraction is easy, but the data sometimes needs to be massaged a bit (or decrypted/decrunched) before being useful, so I like to do it a proper programming language. There is also often a "file list" with track/sector offsets that can be parsed and used for batch extraction of game assets.

Disassemble with IRA "ira -M68000 -BINARY -A -OFFSET=$73040 stage2.bin" and you'll get a stage2.asm. IRA is a whole topic onto itself, so look there for further info.
Looking at that we see code matching the screenshot:

SECSTRT_0:
	LEA	EXT_0007,A7		;73040: 4ff900080000
	LEA	SECSTRT_0(PC),A0	;73046: 41fafff8
	LEA	SECSTRT_0,A1		;7304a: 43f900073040
	MOVE.W	#$00f9,D0		;73050: 303c00f9
LAB_0001:
	MOVE.L	(A0)+,(A1)+		;73054: 22d8
	DBF	D0,LAB_0001		;73056: 51c8fffc
	JMP	LAB_0002		;7305a: 4ef900073060
LAB_0002:
	MOVE.L	#$0007306a,TRAP_01.W	;73060: 21fc0007306a0080
	TRAP	#0			;73068: 4e40
	MOVE	#$2700,SR		;7306a: 46fc2700
	MOVEM.L	LAB_0007(PC),D0-D5	;7306e: 4cfa003f00cc
	MOVEM.L	D0-D5,AUTO_INT1.W	;73074: 48f8003f0064
	LEA	CIAA_PRA,A4		;7307a: 49f900bfe001
	LEA	CIAB_PRA,A5		;73080: 4bf900bfd000
	LEA	HARDBASE,A6		;73086: 4df900dff000

The top seems to be a vestige from the uncracked game that relocates itself to the address it's already located at and then continues fro $73060 (LAB_0002). One thing of note is that it only copies 1000 bytes, so we can conclude that we probably only need to consider the code in the range $73040-$73428).

Next we see a common system takeover pattern: installing a trap handler to a close-by instruction and then executing a trap. This is just done to get to supervisor mode without using exec.library/Supervisor. There a multiple variations of this trick, involving using an illegal instruction, dividing by zero, etc.

NOTE. Below are spoilers, so if you want to look at this with an unspoiled mind, now is the time to look at the disassembly yourself. I highly encourage doing so before continuing.

OK, so loading CIAA_PRA and CIAB_PRA into registers is a very strong clue that the code is going to use a custom trackloader, and indeed it does. This post is already long enough, so I won't go into much more detail about that part. With experience you will start to notice the tell-tale signs of a custom trackloader.
In this case there is not much code, so I've gone ahead and annotated it all (spoiler.s in the attached archive). There are many variations to how track loaders are structured, but this is a fairly standard one, with a common enough layout. Though not that it's sub-optimal in that it always turns the motor on/off for reads and seeks to track 0 before seeking to the target track (and changes sides while doing so!).

Usually there is no need fully disassemble/understand the trackloader. What you want to look for is the high-level routine(s)., in this case what I've called Disk_ReadTracks (another common variation reads sector ranges rather than full tracks).
Your number one job as a WHDLoad author is finding and replacing routines like this. In this case, we can see that on entry D7.w contains the number of tracks to load, D0.w the starting track and A0 the destination address for the decoded data.
So in we will need a replacement:

stage2_readtracks
                movem.l d0-a6,-(sp)
                move.w  #$1600,d1               ; Number of bytes in a track = NUMSECS*TD_SECTOR = 11*512
                mulu.w  d1,d0                   ; Adjust offset to bytes
                mulu.w  d7,d1                   ; And the length
                moveq   #1,d2                   ; From disk 1
                move.l  _resload(pc),a2
                jsr     resload_DiskLoad(a2)
                movem.l (sp)+,d0-a6
                rts

We can also see that kickemu isn't really needed, and that an "old-school" slave will suffice, since the system is only used to display a message and load "stage2". We also don't need to do the "TRAP #0" dance since an old-school slave starts executing in supervisor mode. Thus:
As matter of professional pride, I will only use kickemu when necessary, and only enable WHDLF_EmulTrap when I have to. This is optional though, and there is nothing wrong with starting and sticking with the generic 1.3 (or 3.1) slaves.

Continuing a bit with the code, we see that it loads and display a splash screen, before loading one more track full of code starting from track 8:

	MOVEQ.L	#8,D0			;073130: 7008
	MOVEQ.L	#1,D7			;073132: 7e01
	LEA.L	$00000430.W,A0		;073134: 41f80430
	BSR.B	Disk_ReadTracks		;073138: 611c
	JMP	$00000434.W		;07313a: 4ef80434

We are now at the end of what we can do with just track 0. Sometimes patching a stage 2 track loader is (mostly) sufficient if it stays resident and the rest of the game/demo will keep reusing it. In that case further patching is needed (spoiler: it is in this case).
So we end up with:

STAGE2ADDR=$73040
start           A0 = resident loader
                move.l  a0,a2
                lea	_resload(pc),a0
                move.l	a2,(a0)

                move.l  #$200,d0        ; offset (in bytes)
                move.l  #1000,d1        ; size (in bytes)
                moveq   #1,d2           ; disk number
                move.l  #STAGE2ADDR,a0  ; destination address
                jsr     resload_DiskLoad(a2)

                lea     pl_stage2(pc),a0
                move.l  #STAGE2ADDR,a1
                jsr     resload_Patch(a2)

                jmp     $07306e ; Jump in after TRAP #0 stuff

pl_stage2       PL_START
                PL_P    $073138-STAGE2ADDR,stage2_done
                PL_P    $073156-STAGE2ADDR,stage2_readtracks
                PL_END

stage2_done
                bsr     stage2_readtracks
                blitz
                jmp     $434.w

Now say that for e.g. nostalgic reasons you wanted to preserve the boot text, then you need to stick with kickemu. The trap #0 can be worked around, or as already mentioned WHDLF_EmulTrap can be added. The bootloader also needs to be patched, so comment in BOOTBLOCK, and do the patching:

_bootblock	movem.l d0-a6,-(sp)
                lea     pl_boot(pc),a0
                move.l  a4,a1
                move.l  _resload(pc),a2
                jsr     resload_Patch(a2)
                movem.l (sp)+,d0-a6
		jmp	(12,a4)

pl_boot         PL_START
                PL_P    $d0,patch_stage2
                PL_END

Where patch_stage2 would look very similar to the above, except we would want to jump to the original entry point:

patch_stage2
                movem.l d0-a6,-(sp)
                lea     pl_stage2(pc),a0
                move.l  #STAGE2ADDR,a1
                move.l  _resload(pc),a2
                jsr     resload_Patch(a2)
                movem.l (sp)+,d0-a6
                ; Original code
                JMP	$00073040		;2000d0: 4ef900073040

[field3d]

Oh @paraj I see it was more complex than it looked. I will try in these days to check a normal slave with that patches and see if works. Just two questions. The python is ok but where you ran the python to read the .bin? From PC? I use python but how you made that the python recognizes the disk image to download the segment of track 0? Suppose I have the .py where I execute PC? But to recognize the disk image? And the second question the .bin you attach is just the extraction to illustrate purposes only ok? I don’t need to rewrite back to the disk image ok? From what I see I see your patches suggestion and I will try to enter in a slave source ok?

[paraj]

Originally Posted by field3d

Oh @paraj I see it was more complex than it looked. I will try in these days to check a normal slave with that patches and see if works. Just two questions. The python is ok but where you ran the python to read the .bin? From PC? I use python but how you made that the python recognizes the disk image to download the segment of track 0? Suppose I have the .py where I execute PC? But to recognize the disk image? And the second question the .bin you attach is just the extraction to illustrate purposes only ok? I don’t need to rewrite back to the disk image ok? From what I see I see your patches suggestion and I will try to enter in a slave source ok?

Yes, save the snippet to e.g. extract.py and run it with "python extract.py" on the PC side from the same directory that the binary file is in. I just extracted and included track0.bin here so others can follow along with the example.


You can continue extracting parts for examination by changing the filename to "disk.1" and adjust offset and length appropriately.

[jotd]

note that resload functions systematically trash registers D0/D1/A0/A1 (or possibly return a value in D0) but preserve others. So if you load resload in A2 you can just save the minimum number of registers.

patch_stage2
                movem.l d0-d1/a0-a2,-(sp)
                lea     pl_stage2(pc),a0
                move.l  #STAGE2ADDR,a1
                move.l  _resload(pc),a2
                jsr     resload_Patch(a2)
                movem.l (sp)+,d0-d1/a0-a2
                ; Original code
                JMP	$00073040		;2000d0: 4ef900073040

Sometimes it helps as game original stack is really low and you didn't relocate it.

[paraj]

Originally Posted by jotd

note that resload functions systematically trash registers D0/D1/A0/A1 (or possibly return a value in D0) but preserve others. So if you load resload in A2 you can just save the minimum number of registers.
Sometimes it helps as game original stack is really low and you didn't relocate it.

Oh yeah, that's a good point. Just did the d0-a6 out of habit (usually only trim it down late in development after being bitten by premature optimization one time too many).


And sounds good about you're include.mk changes I added "-pic" to command line after I search vasm docs for such an option after being annoyed at having yet again run accidentally afoul of the no relocs rule.

[jotd]

-pic was already there (and is the main motivation behind the refactoring). I created VASM and VASM_NO_PIC command prefixes. NO_PIC is useful for RawDIC slaves where relocation is allowed.

There's some history about the directory tree. It may seem strange that the tracked files are in "usr" subdirs, when the main folders are empty. That's because I use to copy the dev files here, and only copy them back in usr/source when I create the lha package. The files weren't tracked by any VCS until a few years back.

It is convenient as I can leave slaves in work for months but the repository won't contain changes that don't work. Only working / released source files are in the commits. That explains the "../../" in the makefile (not ../../../..). I think I'll rename the makefile_windows.mak at some point. There were an old makefile for barfly, I'll clobber it as I consider Barfly+developing on the emu or original machine obsolete now (even if it was really good and I did that for 15+ years)

[paraj]

Originally Posted by jotd

-pic was already there (and is the main motivation behind the refactoring). I created VASM and VASM_NO_PIC command prefixes. NO_PIC is useful for RawDIC slaves where relocation is allowed.

Yes I know, just meant I started using it for that reason

Originally Posted by jotd

There's some history about the directory tree. It may seem strange that the tracked files are in "usr" subdirs, when the main folders are empty. That's because I use to copy the dev files here, and only copy them back in usr/source when I create the lha package. The files weren't tracked by any VCS until a few years back.

It is convenient as I can leave slaves in work for months but the repository won't contain changes that don't work. Only working / released source files are in the commits. That explains the "../../" in the makefile (not ../../../..). I think I'll rename the makefile_windows.mak at some point. There were an old makefile for barfly, I'll clobber it as I consider Barfly+developing on the emu or original machine obsolete now (even if it was really good and I did that for 15+ years)

When I've worked with your slaves, I've done so in a separate working directory with a local git repo where I go nuts (commit when I feel like it, weird test branches, etc.), only syncing once I think everything is right. No reason to publicly air your dirty laundry


BTW in options.mk maybe you could do:

HDBASE ?= K:\jff\AmigaHD\amiga39_JFF_OS\include

, i.e. add question mark before the equal. This way HDBASE is only set if not already defined (from environment variable).

[jotd]

that's a good idea.

[copse]

Originally Posted by jotd

vasm doesn't add ,pc if you don't specify optimization options. I usually don't use optimization on slaves. I wasn't aware of this -pic option which is super useful. Will add it on my toolchains.

You fixed it but yes, you need something like

lea     graphicslibra_MSG(pc),a1

Appreciate the tips. Yes, that is what I did. If you assume I removed the -pic, then that would make me a sloppier programmer than what I imagine myself to be

This is psygore's Pool of Radiance slave code. It must have worked at some point and likely still does I guess in the form of the PoR slave that's out there for download.