asm source checker

[jotd]

Well, I'm not sure this thread belongs here but

After several years of attempting to fix bad code with lousy "tools", I have finally written a tool (running in Python 2.5) which performs a certain number of static checks on a ressourced asm file (using IRA). That's an understatement to say that the use of Python eased the task.

I use this tool on a daily basis now to fix 68000-only / A500-only code / more

- checks if the program is tampering with VBR/CACR registers (not good for WHDLoad)
- checks for move SR,xxx (crashes 68010+ in user mode)
- checks for RESET instructions in the program (crashes the amiga even under whdload)

well, ATM you'd say a couple of good "grep"s could do the trick but wait:

- detects cpu-dependent/infinite loops (empty dbfs, that's easy, but has an heuristic algorithm to detect a lot more constructions, by trying to evaluate if the loop has a relevant side effect or not)
- detects self-modifying code. Once again this is heuristic.

I have finetuned the tool using several disassembled games and even this very early version gives very good results.

- found C-style loops in It Came From The Desert whdload slave, thus fixing sound issues and game speed. The bug has been here for 5 years!
- found other flavours of loops in Neuromancer & SMC code.
- Syndicate asm source also revealed cpu-dependent loop, inducing a too fast gameplay (TBC)
- The tool was able to detect loops in Hollywood Poker Pro that took me some time to find a few years ago (by running in A500 mode and breaking, but I was lucky because the waits were long)

Well, it's free, and if you need it it's here m68kchecker.zip

How to use (tested on Windows, but should work any other platform maybe even Amiga since AFAIR python is available on it):

- First, disassemble your executable with IRA: ira -a -m68020 myexe
- Then, run "m68kchecker.py -i myexe.asm" (redirect the output in a file if needed)
- Sort out the false alarms

Of course I'm craving for your feedback and new ideas of what could be checked.

regards

[phx]

Nice idea. There are probably many possibilities to extend its functionality.

I just gave it a try under NetBSD, and here are some notes:

- /bin/env is a bad path. All BSD and Linux system I tried use /usr/bin/env.

- When no input files are given it runs into an exception. Maybe a friendly error message would be better?

- A single run with "ira -a" will probably not disassemble all code correctly. For best results you will have to use a config file and find the data areas manually.

- What is asm_parsing.py for? It doesn't work at all here:
Traceback (most recent call last):
File "asm_parsing.py", line 588, in <module>
selftest();
File "asm_parsing.py", line 567, in selftest
addresses = a.linedict.addresses()
File "asm_parsing.py", line 206, in addresses
s = set(self.label.keys()).union(self.constant.keys(),self.section.keys(),self.instruction.keys(),self.other.keys())
TypeError: union() takes exactly one argument (4 given)

I'm using Python 2.5.4.
Apart from that m68kchecker is working fine!

[jotd]

Hi Frank!!

(BTW I saw your replies on my post on IRA and did not reply to them but I'll take the advice and migrate to IRA 2.05 ASAP, thx for the information)

- About the magic number #!/bin/env, you're right. Will change it (on windows it doesn't matter unless you're under MinGW)
- To get a friendly message, just set debug_mode = False at the bottom of the file
- ira -a always gives good results, and the risk of mistaking data for code is handled in the tool (of course it's better to have a good code/data layout). I'm planning to add some heuristic code/data detection. I've got some ideas, that's a very interesting project. If some other ppl are interested I may put that on my top priority list
- asm_parsing.py is the generic object used to parse the asm code. The checker relies on it, but it could be used for other applications (an assembler for instance ) or the data/code detector I was talking about.

Thanks for your feedback. Python rules

[SyX]

I think that it's a great idea jotd!!!

I am proud of my sourcecodes passing the test of lint, pylint and similar tools. I already have added your tool at my sdk

In the first pass, it only has found a pair of false alarms

m68kchecker.py: Processing uwol/appack.s
VIOLATION:appack.s:22:probable CPU-dependent/infinite loop for label '.copy_byte'
.copy_byte:             move.b  (a0)+,(a1)+
.next_sequence_init:    moveq   #0,d1           ; Initialize LWM
.next_sequence:         bsr.b   .get_bit
                        bcc.b   .copy_byte      ; if bit sequence is %0..., then copy next byte
VIOLATION:appack.s:98:probable CPU-dependent/infinite loop for label '.loop_do_copy'
.loop_do_copy:          move.b  (a2)+,(a1)+
                        dbf     d2,.loop_do_copy

Python rules

YEAAAHH!!!

[jotd]

Oh, as the tool is intended for disassembled code, it does not handle label + instruction on the same line properly (yet)
hence the false alarms

[SyX]

Well, a tool that let you detect "stupid" logic errors , it's useful not only for disassembly code

[jotd]

wow did you find something fishy in your code?
Any suggestions of features that I could add?

[SyX]

No, nothing fishy with my code... surely i have been lucky

But when i use your tools for test a lot of sourcecodes that i have in my hard disk, i have found a lot of "VIOLATION: Probable CPU-dependent/infinite loop for label"

And the problems (the most cosmetics) that i have found are:
1.- When a source code reference at label in other file, the parser stop and i get the next error:

Traceback (most recent call last):
  File "m68kchecker.py", line 28, in init_from_sys_args
    self.__do_init()
  File "m68kchecker.py", line 51, in __do_init
    self.__doit()
  File "m68kchecker.py", line 369, in __doit
    self.__process_file(f)
  File "m68kchecker.py", line 362, in __process_file
    self.__check_visible_self_modifying_code(asm_file)
  File "m68kchecker.py", line 150, in __check_visible_self_modifying_code
    label_address = asm_file.label_address[label_without_offset]
KeyError: 'game_persondata'

I always can join all the files before parse, but of course catch this situation and print a meaningful error for not python literate people.

2.- Don't worry about this, but the syntax of the old assemblers used for these sources usually gave me a few exceptions with things how using '*' for comments and similar. How your tool support the syntax of IRA and VASM (i have used the assembly output of VBCC for test, too), i think that implement the syntax of "ye olde and wacky" assemblers is lose the time

3.- Don't support label names with less of 3 char (it's common when you are using local labels), you get an exception with this:

xx
  bne xx

And the correct "VIOLATION" if you call the label "xxx".

4.- The most annoying feature is that i can use wildcards to parse more that one file. I use this code in my scripts:

from optparse import make_option, OptionParser
import glob

def process_command_line(command_line):
    """
    Return a tuple with: (options, file_list).
    `command_line` is an argument list, or `None` for ``sys.argv[1:]``.
    """
    if command_line is None:
        command_line = sys.argv[1:]

    program_version = "%prog v0.6"
    program_usage = "usage: %prog [options] img1.png img2.png ... imgX.png"
    program_description = "%prog convert images in png format to binary data for Amiga."

    # Define the options that it will support in the command line
    program_options_list = [
        make_option("-m", "--map", action="store_true", dest="map", default=False, help="Generate map"),
        make_option("-p", "--palette", action="store_true", dest="palette", default=False, help="Generate palettes"),
        make_option("-s", "--sprite", action="store_true", dest="sprite", default=False, help="Generate sprites"),
        make_option("-t", "--tile", action="store_true", dest="tile", default=False, help="Generate tiles"),
        make_option("-X", "--map_width", action="store", type="int", dest="ancho_casilla", default=1, help="Tilemap width"),
        make_option("-Y", "--map_height", action="store", type="int", dest="alto_casilla", default=1, help="Tilemap height"),
        make_option("-x", "--tile_width", action="store", type="int", dest="ancho_patron", default=8, help="Tile width"),
        make_option("-y", "--tile_height", action="store", type="int", dest="alto_patron", default=8, help="Tile height")
    ]
        
    parser = OptionParser(usage=program_usage, description=program_description,
        version=program_version, option_list=program_options_list)
    
    # Get the options and file list passed to the program
    (options, file_list_tmp) = parser.parse_args(command_line)

    # Get the file list
    if (not file_list_tmp):
        parser.error("No files to process.")
    else:
        /* Fix for get wildcard expansion in windows */
        file_list = []
        for i in file_list_tmp:
            file_list += glob.glob(i)

    # Verify for invalid options
    if (options.ancho_patron < 1):
        parser.error("Tile width invalid.")
    if (options.alto_patron < 1):
        parser.error("Tile height invalid.")

    return options, file_list

def main(command_line=None):
    """
    Main function
    """
    options, file_list = process_comand_line(command_line)

    for image_name in file_list:
        .
        .
        .

With respect to possible suggestions, i have been looking for texts with sane recomendations for assembly coding, with the idea of adding more tests a your tool, but i only have found the olds How to Code and Action's Guide to AGA-Fixing!, and i'm sure that you already know them.

[jotd]

Thanks for this feedback. Shows that you used the tool a lot. I'm still convinced that you won't find it very useful. If you use self-modifying code or cpu-dependent loops, you surely remember you did.

About wildcards, it's supported. Try m68kchecker.py -i *.asm

(using glob.glob too here )

but that does not solve the "multi-file" issue with labels. For that you have to concatenate all the files, which is not good. I keep your suggestions in store for now.

bye

[SyX]

Originally Posted by jotd

Thanks for this feedback. Shows that you used the tool a lot. I'm still convinced that you won't find it very useful. If you use self-modifying code or cpu-dependent loops, you surely remember you did.

Yes, of course... but tell that to phx, i was helping to test all the optimizations of vasm and i was trying to muyltiply by -1 with mulu instead of muls (my excuse is that i'm really tired )

[jotd]

argh. Yeah that opens many doors (not on disassembled code)

the other thing I'm thinking off is the basm bug

move d0,$8000.W

no such thing as 8000.W so it clips the value and moves to 0

Anyway don't expect too much of my checker because it does not evaluate expressions. The checker is kind of limited to disassembled code. I wrote it for no other purpose