Blitzlibs/Blitz library tokenisation

BippyM · 04 September 2019, 19:06

Does anyone have any info on how Blitz stores token information? I really don't want to reinvent the wheel if someone already knows?

Daedalus · 05 September 2019, 10:57

I don't have anything specific, other than it being a binary representation of the library and token number, so two bytes, which is why source gets mangled if you load into an installation with different library numbers. I assume there's a magic byte as well to differentiate the token from text in a source file, but would need to verify that with a hex editor. I'll check it out when I have a chance.

BippyM · 05 September 2019, 11:02

I've already started to figure it out, but wondered if there was any other info already out there

idrougge · 05 September 2019, 12:07

I think internal commands are tokenised in a different way, but don't ask me for evidence.

Here is some information, including an ARexx program to exercise the Blitz tokeniser and a map of all known commands (in the test directory): https://github.com/grovdata/pyblitz2

BippyM · 05 September 2019, 19:39

Yeah, I've had a look at this and it is pretty useless to be fair!!

Quote:

Originally Posted by idrougge

I think internal commands are tokenised in a different way, but don't ask me for evidence.

Here is some information, including an ARexx program to exercise the Blitz tokeniser and a map of all known commands (in the test directory): https://github.com/grovdata/pyblitz2

BippyM · 08 September 2019, 19:36

So I have figured out how the libraries are identified within blitz. What I am struggling with a little is identifying how Blitz knows how many tokens there are within the library. There is nothing obvious stating an offset, or number of commands. The only thing I can see is the following marker which is common to each command

Code:

$FF FF 00 00 00 00 00 00

I am unsure if this is before the command, or after hmmm..

idrougge · 09 September 2019, 14:46

The FF FF 00 sequence seems to be part of the command name assembly macro as well as part of the "end library" signature.

BippyM · 22 November 2019, 21:33

So after a bit of research I have no identified how the blitz libraries are stored. I can now start to code something to extract all commands from each library, deflibs etc.

Next is to determine how source tokenisation works, and then chuck a mock gui together.

I'll continue this in the correct thread

04 September 2019, 19:06	#1
BippyM Global Moderator Join Date: Nov 2001 Location: Derby, UK Age: 48 Posts: 9,355	Blitzlibs/Blitz library tokenisation Does anyone have any info on how Blitz stores token information? I really don't want to reinvent the wheel if someone already knows?

08 September 2019, 19:36	#6
BippyM Global Moderator Join Date: Nov 2001 Location: Derby, UK Age: 48 Posts: 9,355	So I have figured out how the libraries are identified within blitz. What I am struggling with a little is identifying how Blitz knows how many tokens there are within the library. There is nothing obvious stating an offset, or number of commands. The only thing I can see is the following marker which is common to each command Code: $FF FF 00 00 00 00 00 00 I am unsure if this is before the command, or after hmmm..

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
[blitz basic] How much amiga-blitz friendly is this?	saimon69	Coders. Blitz Basic	105	21 April 2022 19:45
Blitz & Reqtools Library	BippyM	Coders. Blitz Basic	5	14 December 2018 06:37
CDXL library for Blitz?	earok	Coders. Blitz Basic	4	12 October 2016 23:22
How to do AGA copperbars in Blitz using display library	Coagulus	Coders. Blitz Basic	9	25 January 2016 15:53
HELP: Opus5, KingCON, SysInfo.library (Sysmon.library) triangle	Michael	support.Apps	6	10 March 2014 14:08

05 September 2019, 10:57	#2
Daedalus Registered User Join Date: Jun 2009 Location: Dublin, then Glasgow Posts: 6,335	I don't have anything specific, other than it being a binary representation of the library and token number, so two bytes, which is why source gets mangled if you load into an installation with different library numbers. I assume there's a magic byte as well to differentiate the token from text in a source file, but would need to verify that with a hex editor. I'll check it out when I have a chance.

05 September 2019, 11:02	#3
BippyM Global Moderator Join Date: Nov 2001 Location: Derby, UK Age: 48 Posts: 9,355	I've already started to figure it out, but wondered if there was any other info already out there

05 September 2019, 12:07	#4
idrougge Registered User Join Date: Sep 2007 Location: Stockholm Posts: 4,332	I think internal commands are tokenised in a different way, but don't ask me for evidence. Here is some information, including an ARexx program to exercise the Blitz tokeniser and a map of all known commands (in the test directory): https://github.com/grovdata/pyblitz2

09 September 2019, 14:46	#7
idrougge Registered User Join Date: Sep 2007 Location: Stockholm Posts: 4,332	The FF FF 00 sequence seems to be part of the command name assembly macro as well as part of the "end library" signature.

22 November 2019, 21:33	#8
BippyM Global Moderator Join Date: Nov 2001 Location: Derby, UK Age: 48 Posts: 9,355	So after a bit of research I have no identified how the blitz libraries are stored. I can now start to code something to extract all commands from each library, deflibs etc. Next is to determine how source tokenisation works, and then chuck a mock gui together. I'll continue this in the correct thread

Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)