New WHDLoad Download Tool

[mrv2k]

So, I've been writing a tool to help manage and download Retroplay's WHDLoad archives. It is very much the same as Turran's version except it is a lot more flexible and can do some really useful stuff.

Main Features

• Uses Retroplay's dat files for it's database, so is always up to date.

• Flexible on where files are saved to.

• Supports the beta WHDLoad set

• Over 20 filters available including all supported languages.

• Create your own curated lists that can be shared.

• Scan your existing set and remove unneeded files.

• Only downloads what you are missing.

***Warning***

IMPORTANT! Before you do anything to your existing WHDLoad sets, MAKE A BACKUP!. If you delete everything, TOUGH! Post what happened and I can try to fix the problem.

To leave feedback, bugs and suggestions, only use this thread. Please try to make sure that any feedback is reasonable and constructive.

Posts on the Commodore Amiga Facebook group and the r/Amiga Reddit will be ignored as they are a PITA to keep on top of.

Version 0.2a is available

Fixes / Updates

• Updated Turran FTP info in docs.
• Changed donate link to correct one.
• Fixed path gadgets actions when loading prefs files.

Version 0.3a is available

The connection errors have been fixed! Thanks to Conrad Fenech and Brian Lloyd for the help debugging and testing!

Fixes / Updates

• Load / Save Prefs now only show prefs files in the file requester.
• Changed FTP connections to PBEx_FTP DLL which has fixed the connection errors.
• Fixed a typo in the demo download paths. Demos are now downloaded to the right subfolder if the option is enabled.
• Fixed bug in list creator/editor that ignored highlighted files when updating main list.
• Changed the download window to a console window and added escape key to abort the download at any time.
• Blocked close console button

Version v0.4a is available.

Fixes / Updates

• Added support for WHDLoad based disk magazines.
• Added a check to FTP download to catch if a file is created but no data is downloaded. The error is logged in the ftp.log file and the empty file is deleted.
• Fixed a crash if the required DLL files are missing by preventing the program loading.
• Added minimize to main window.
• Switched dat file download to a console and added ftp log support.
• The edit window now work with the filters. You can now filter and add/remove files to the same list.
• Added Chip and Fast memory files to the filter.
• Preference files have been updated for new filters.
• Re-worked the filter panel to make room for the new filters by moving game, demo and magazine filters next to their path boxes.
• All FTP settings and FTP paths are now stored in the preferences files and can be edited.
• Updated the help window to reflect all the new changes.

Version v0.5a is available.

Fixes / Updates

• Added magazines to the 'Clean Files' procedure.
• Added 0kb file check to 'Clean Files' procedure.
• Added temporary fix to FTP download for 0kb files via DOS FTP command.
• Added a few more donation notifications to the download procedure.

Version 0.6a is available

Fixes / Updates

• Fixed escaping FTP download crashing the download window.
• Added separate download folder for Beta games.
• Added preview window to show download file list.
• Fixed Port gadget overlap on main window.
• Fixed disabled gadgets when loading a prefs file.
• Added switch to change between default PB or alternate FTP procedures.
• Updated help to reflect changes.
• Adapted filter to show beta files separate to games

Version 0.7a is available

This release should hopefully fix the remaining FTP download and connection issues. Make sure that you allow the program through your firewall when Windows asks.

You may get a false AV notification. This is due to how Purebasic wraps Window API commands. Just whitelist the executable if you want to use the program. I've included the source code in the zip file.

Fixes / Updates

• Moved FTP procedures to Windows API functions.
• Fixed FTP fallback related console and UI crashes.
• Fixed non unicode characters in download name saving as empty files.
• Fixed open beta folder button.
• Removed requirement for PB_FTPEx libraries
• Clean files now only lists lha & lzx files.
• Added experimental support for Linux Ubuntu 20+ & MacOS (Source available on GitHub. Executables will be released in 0.8a).
• Added check for lists folder. Creates if not found.
• Added donate button to main window.
• Added version information to executables

Version 0.8a is available

There's two versions in this release. A stable release based on the v0.7a FTP code and an unstable version based on the v0.6a FTPEx library download code that crashes the GUI sometimes. The reason is, there were still a few users who had connection issues with the new code. Having both versions available, should hopefully mean that everyone can use the tool.

Only one major addition. There is now a 'Make Folder' button which is there to create new folders out of your existing downloads. For instance, if you have a full set but only want to copy ECS/OCS games to an A500/600/2000/3000, just untick AGA and CD32 in the filter, press the 'Make Folder' button, select a new folder, decide whether to keep the folder structure and press go.

Fixes / Updates

• Fixed old dat files not being deleted in Scan FTP procedure.
• Cleaned up unneeded variables and lists from the FTP download procedures.
• Fixed console actions on Download FTP procedure.
• Tweaked FTP procedures to use real FTP variables rather than 1 & 0.
• Minor speed up when processing strings.
• Added button that makes a new folder of archives based on your filter selections.
• Make Folder opens the output folder window when complete.

Version 0.8b is available

This is a quick update that fixes the Turran FTP address as it has changed. You can either delete any prefs files and re-create them or change the ftp address line in any existing prefs file to ftp.grandis.nu with a text editor.


Version 0.9a (Test Version) is available

This is a test release which has HTTP support for downloading from the Turran server. Hopefully this will eliminate the connection issues that have been reported in previous versions. FTP is still supported but is slower due to waiting for connections and folder changes.

Fixes / Updates

• Fixed a bug in subfolder creation.
• Added ability to download to folder structures either by category or alphabetically.
• Changed 'Save Prefs' and 'Load Prefs' to be disabled until the database is loaded.
• Centered console on main window.
• Added HTTP based downloads.
• HTTP & FTP downloading now selectable fron the main window. HTTP is now the default, but the setting is saved in the prefs files if the user
• wants to change it.
• Added HTTP path to main window and prefs.
• Removed 'Use Subfolders' gadget and replaced with selectable combo gadgets for sorting downloads.
• Improved the download preview window procedure.
• Added a selectable 255 file limit to the download folders for the A500 mini home computer.
• All FTP and download gadgets are enabled by default.
• The FTP procedure has been updated to support the 0-Z and category downloads.
• Removed 'Make Folder' until I can make a better implementation of it.
• Fixed a small bug in scrape data procedure.
• Updated help file to reflect this versions changes.
• Improved the filter speed.
• Clear data now keeps the server and filter settings.
• Fixed a couple of bugs in the save/load prefs procedure. All gadgets now update properly.
• The clean files process is a bit dumber in the sense that it just looks for the right filename and deletes any lha or lzx file that doesn't match. It does still look for zero kilobyte files.
• Added a 'refresh list' keyboard shortcut (F5) to update the available files if the user deletes any files manually.

Just a quick update to the full release. I've added several suggested filters and squished a couple more bugs. There is also an x86 version in the archive again.

Version 1.1 is available

Fixes / Updates

• Added 28 new filters.
• Fixed bug in scrape procedure that forgot to add beta games to Amiga system.
• Added fix for x86 compilation
• Fixed download number in download screen

Version 1.11 is available

Fixes / Updates

• Fixed beta path gadget updating wrong path
• Fixed compilation version info
• Added paypal.me link to the about window if you think I deserve a coffee

Version v1.12 Released

Fixes / Updates

• Fixed a bug where the default.prefs file was loaded at startup but the gadgets didn't update.

This version adds a couple of feature requests. Namely downloading by category with 0-Z sorting and also a facility to make a new folder from an existing downloaded set of files with different sorting and filtering. There's also a couple of squished bugs as well.

Version v1.2 Released

Fixes / Updates

• Added alphabetical sorting to category download.
• Fixed the sort gadgets not updating when the default.prefs was loaded at startup.
• Added an improved 'Make Folder' procedure back. It doesn't support A500 mini 255 file limit though.
• Fixed sort gadget logic.
• Updated the help window to reflect the new changes.
• Increased the window height to fit new button in and space out the filter a bit better.

Version v1.21 Released

Fixes / Updates

• Fixed a bug when the FTP and HTTP settings weren't applied to the relevant gadgets when loading the default prefs.

Version v1.3 Released

This version adds CRC32 checking and support for the new Beta - Demos folder in retrogame's folder on the Turran server.

Fixes / Updates

• Added CRC32 checking to the update procedure.
• Added support for the new Beta Demo folder on the FTP. All beta files are still saved to the selected beta folder.
• Users are now prompted to download the dat files when a prefs file is loaded and no data has been downloaded.
• All download files are size checked against the dat files and removed it they are wrong.
• HTTP download errors are logged in a file called error_log.txt.
• You can now toggle the file tree in the download preview.
• Downloads are a tiny bit slower as the files are size and crc checked once received.
• You can now save the download list as a text file.
• Fixed status bar spacing.
• Changing prefs files will prompt for clearing the database and will rescan the WHDLoad folders.
• Save Prefs button is now active all the time.
• The file list now shows any errors in your files by highlighting them with red for missing and blue for bad CRC32.
• Moved all donate links to a new window.

Version v1.3 Released

Fixes / Updates

• Fixed list filter not updating for games and beta files
• Fixed Make Folder not finding any games.
• Clear data now resets all settings and filters including sort and download paths.
• You can now save the download list as a text file from the download window.

Here's the link...

https://github.com/MrV2K/WHDLoad-Download-Tool

[Tarzin]

Thanks, looks great and usefull!

[Valken]

Awesome!

[AMIGASYSTEM]

Very interesting mrv2k, it will be multiplatform? there will be an AROS version?

[mrv2k]

Originally Posted by AMIGASYSTEM

Very interesting mrv2k, it will be multiplatform? there will be an AROS version?

No AROS unfortunately as there is no version of Purebasic that supports it. Planned versions are... Windows (obviously), Linux and Raspberry Pi OS. (Maybe Mac)

[Viceroy]

Nice one ;-)

[AMIGASYSTEM]

Originally Posted by mrv2k

No AROS unfortunately as there is no version of Purebasic that supports it. Planned versions are... Windows (obviously), Linux and Raspberry Pi OS. (Maybe Mac)

Thanks anyway

[Valken]

I'm going to ask for a backup function JUST IN CASE the latest slave does not work.

It can perhaps just "move" the old slave to backup dir, or ZIP or 7zip it up into a backup file, with directory structure intact to "restore" if possible?

Thank you!

[polomint]

Extremely nice, can't wait to try it if you can add me to your list (fingers crossed)

[outsiderz]

great!!!
download link please

[mrv2k]

Alpha test links in first post!

[enigma776]

Getting can not load ftp folder when clicking load data. am I doing something wrong?

[Turran]

Very nice. Let me know if you want it hosted on the FTP with full access to its directory.

Noticed you called it EAB FTP in the program. Its called TURRAN FTP as we try to distance ourselves from EAB (by request).

The donate button (appreciate that) goes to donate towards this forum. I do have a link there are well, but you might donate to the wrong thing if you just set a sum and click donate. The correct link is
https://www.paypal.com/cgi-bin/websc....gif:NonHosted

[mrv2k]

Originally Posted by enigma776

Getting can not load ftp folder when clicking load data. am I doing something wrong?

Not sure. Only got that error when I couldn't get onto the FTP when it was down or I had another FTP connection running on the same account. Can you get onto the FTP via a program like filezilla?

[enigma776]

Originally Posted by mrv2k

Not sure. Only got that error when I couldn't get onto the FTP when it was down or I had another FTP connection running on the same account. Can you get onto the FTP via a program like filezilla?

I can yes. I can even use the old EAB-WHDLoad_Pack_Downloader_x64 tool. The help file says something about a "Dats" folder is that suppose to be local or on the FTP? If its the FTP there is no Dats folder.

[Turran]

Originally Posted by enigma776

Getting can not load ftp folder when clicking load data. am I doing something wrong?

It does not give proper error messages returned from the FTP it seems. Make sure you are not already logged in and that the username and password is correct (default ftp/amiga should work, but only for one connection per IP)

[enigma776]

Originally Posted by Turran

It does not give proper error messages returned from the FTP it seems. Make sure you are not already logged in and that the username and password is correct (default ftp/amiga should work, but only for one connection per FTP)

I had thought of this, same error even when using the default account.

[mrv2k]

Originally Posted by enigma776

I can yes. I can even use the old EAB-WHDLoad_Pack_Downloader_x64 tool. The help file says something about a "Dats" folder is that suppose to be local or on the FTP? If its the FTP there is no Dats folder.

The Dats folder is created locally in the main program folder when the program can connect and download the dat files from the FTP. It won't affect the connection to the FTP.

Are you using your own account or the ftp/amiga one?

[mrv2k]

Originally Posted by Turran

It does not give proper error messages returned from the FTP it seems. Make sure you are not already logged in and that the username and password is correct (default ftp/amiga should work, but only for one connection per FTP)

Purebasic is a bit crap at getting FTP errors unfortunately. It either gives you a 1 - Connected or -1 Not Connected. I got this error yesterday when the RAID was being changed and later on it cleared up. The standard FTP account connected but wouldn't let me download as it said my IP was already connected. I could only see this when I connected via Filezilla and tried to download a file and it gave me the error.

I'm not sure whether it's fixable without writing a custom FTP procedure which is a bit beyond me at the moment.

[Turran]

I see. Yeah, that sounds like a pain. I was lucky with my FTP routine in my version in that it grabs the entire login text, so easy to see errors..


I edited my post a few posts up as I found some "wrong" things =)

4 anonymous users downloading from the Retr0play packs right now, so seems to work fine for others (and me).

Originally Posted by mrv2k

Purebasic is a bit crap at getting FTP errors unfortunately..