Issue with large archive HDD causes WinUAE to lock itself and Windows up

Foebane · 16 February 2019, 22:52

I have recently taken many single-file demos and zipped them up and used them as archived hard drives in WinUAE rather than ordinary folders with files, and I find that performance degrades fairly quickly. The largest such zip file is 349Mb of data compressed to something like 211Mb, with a few dozen demo files in it.

The problem is that when I run files from that zip file, or others like it with far smaller and fewer contents, when I reset the Amiga, the screen contents remain on screen for a moment after I've done so, and it then restarts normally after the dark grey blank screen. BUT, eventually, the dark grey screen stays and WinUAE locks up, and I can't even get the Windows Task Manager on the screen and my only option is to Ctrl-Alt-Del to restart Windows.

It's hard to get logs for this as I don't know for sure when the freeze will happen, but it seems to have started since I decided to zip up folders into archive hard drives. I think my only option is to restore them to folders.

DamienD · 16 February 2019, 23:18

Think about it Foebane. When you are adding in large archives WinUAE is having to mount them...

This takes time and will "freeze" WinUAE if you have many.

...and you went down this silly route just to save a few MBs

I would definitely reverse your decision on this one.

Foebane · 16 February 2019, 23:45

Quote:

Originally Posted by DamienD

Think about it Foebane. When you are adding in large archives WinUAE is having to mount them...

This takes time and will "freeze" WinUAE if you have many.

...and you went down this silly route just to save a few MBs

I would definitely reverse your decision on this one.

I will, now. I actually thought it was an ingenious idea to zip them up in the first place. I mean, I access single demos in archive HDDs, but this was the first time I tried many in one go.

I had thought of taking the entire Foebane folder I use and archiving that to save space instead of zipping up smaller parts of that folder, but I am worried about losing the lot if the archive gets corrupted.

What is my best option for archiving a huge amount of data (2.4Gb+) and being able to recover most of it if corruption occurs? RAR? 7Z? I know ZIP wasn't designed with data recovery in mind.

Foebane · 17 February 2019, 01:39

I realise I'm going OT on this, but I really need advice on this matter, please.

haps · 17 February 2019, 05:12

2.4GB+ is huge?

I know I could go down the post office and buy a 16GiB USB stick for 5 quid.

All this space saving you carry on about really makes me shake my head.

Toni Wilen · 17 February 2019, 09:36

If archive is mounted as a harddrive, whole archive needs to be unpacked and converted to internal directory tree. It can take long time and use lots of RAM and unless you use 64-bit WinUAE, there may be only few hundred megabytes of RAM left (32-bit processes usually have 1G to 3G of RAM free where everything must fit, including DLLs, winuae.exe etc.., depending on Windows version), especially if your config has lots of Z3 RAM.

rare_j · 17 February 2019, 12:37

Use an NTFS compressed folder? Transparent compression.
Compression rates not too bad.

DamienD · 17 February 2019, 13:23

Quote:

Originally Posted by Foebane

I actually thought it was an ingenious idea to zip them up in the first place.

Harebrained idea more like; all for the sake of saving a little space

Quote:

Originally Posted by Foebane

If I could put a few thousand files into one archive, it would make it easier to copy them from a recordable DVD (yes, I use those, I don't trust USB sticks).

You do realise that it's actually the reverse? Don't trust burned CDs or DVDs and they have a short life expectancy...

Quote:

Originally Posted by haps

All this space saving you carry on about really makes me shake my head.

You and me both...

I should really just ignore these threads.

Tomislav · 17 February 2019, 13:50

You can use hard disk file with NTFS compression. It would be faster access and you gain some space. Use zip for archiving only or small things like ADFs.

Foebane · 17 February 2019, 14:14

Quote:

Originally Posted by DamienD

Harebrained idea more like; all for the sake of saving a little space

You do realise that it's actually the reverse? Don't trust burned CDs or DVDs and they have a short life expectancy...

You and me both...

I should really just ignore these threads.

I'm going to go back to the way everything was, believe me, but I'm not going to give up on optical media. I've always used it, the data is actually visible and definitely read-only, and nothing can delete it, unlike a USB stick left around for someone to plug in and delete files permanently.

It's not just the archive/HDD aspect that I'm going back on, but the fact that I had other problems with the files once they were zipped. But since that's over with now, it is no longer a problem.

The thing I find odd is only now I'm seeing the other half of the argument. Last week when I was zipping everything up, the people I was with were "go for it", but now the sane half have spoken, and I'm going back to uncompressed.

It was only a measly six hundred or so Mb's saved anyway.

mark_k · 17 February 2019, 21:20

Could the lock-up issue indicate a bug? If on resetting WinUAE unpacks the archive contents to memory again, without freeing the already-unpacked data, the amount of memory used will gradually increase each reset until you run out.

haps · 17 February 2019, 22:35

Quote:

Originally Posted by Foebane

unlike a USB stick left around for someone to plug in and delete files permanently.

Sigh.....

Another non-issue that has an easy and elegant solution. Lock the bloody thing away in a drawer!

Foebane · 18 February 2019, 11:59

I've put the files back to their non-zipped state. It takes up more space, but it makes things easier generally.

Hewitson · 18 February 2019, 14:41

Quote:

Originally Posted by Foebane

What is my best option for archiving a huge amount of data (2.4Gb+)

There's these new things called DVD's that can store almost double that.

Foebane · 18 February 2019, 19:08

Quote:

Originally Posted by Hewitson

There's these new things called DVD's that can store almost double that.

Oh, didn't I mention? I burn on DVD+/-Rs already (not sure which is better), from Verbatim, with the AZO logo on them. I hear they're very good, and they certainly cost enough.

I create the ISO images for the Foebane folder with ImgBurn, and burn those onto disc. I also tested to see if I could rebuild the ISO if I lost the file, and it turns out that using ImgBurn to create the ISO from disc (rather than copying the folder from disc via Explorer) is much faster.

I like the read-only aspect of optical media, although if I discover something that needs to be fixed, it irks me as I'll have to fix it and then burn the disc again. Patches don't really satisfy me.

haps · 19 February 2019, 00:24

I'm sure that if you worked as a ditch digger Foebane you would insist on doing the job with a teaspoon.

AMIGASYSTEM · 19 February 2019, 03:00

I would avoid using compressors LHA, ZIP, RAR etc..
The best solution is to compress all demos individually with PowerPacker or CrunchMania. Once this is done you can copy all the file self-extracting to a HardFile from 2GB/4GB to keep the double or triple copies.
Compressing these "full" HardFiles would be useless because the content is already compressed

Foebane · 19 February 2019, 07:22

Quote:

Originally Posted by haps

I'm sure that if you worked as a ditch digger Foebane you would insist on doing the job with a teaspoon.

If you mean I would put in more effort than needed for a simple collection of files, then yes, I would. It's my only hobby, and once that's complete, I'll have nothing else to do in my free time.

chip · 19 February 2019, 08:18

But when you will reach the end Foebane ?

I strongly guess this hobby, luckily, will take to you much more time than you think

I don't know what kind of demos are you considering, but if you consider them all

You know, we have something in common

I strongly guess i will reach "the end" at least between 10 years

Foebane · 19 February 2019, 09:19

There will always be more demos I can add on top of my 640, but finding them will take time.

Also, I don't just collect any old demos, but ones I like personally.

16 February 2019, 22:52	#1
Foebane Banned Join Date: Sep 2011 Location: Cardiff, UK Age: 51 Posts: 2,871	Issue with large archive HDD causes WinUAE to lock itself and Windows up I have recently taken many single-file demos and zipped them up and used them as archived hard drives in WinUAE rather than ordinary folders with files, and I find that performance degrades fairly quickly. The largest such zip file is 349Mb of data compressed to something like 211Mb, with a few dozen demo files in it. The problem is that when I run files from that zip file, or others like it with far smaller and fewer contents, when I reset the Amiga, the screen contents remain on screen for a moment after I've done so, and it then restarts normally after the dark grey blank screen. BUT, eventually, the dark grey screen stays and WinUAE locks up, and I can't even get the Windows Task Manager on the screen and my only option is to Ctrl-Alt-Del to restart Windows. It's hard to get logs for this as I don't know for sure when the freeze will happen, but it seems to have started since I decided to zip up folders into archive hard drives. I think my only option is to restore them to folders.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
ACA1221EC + Large HDD/CF	ItsTheSmell	support.Hardware	8	06 July 2018 15:26
Large HDD capacity formatting woes in WinUAE (fitzsteve)	WeiXing3D	support.WinUAE	3	06 April 2016 08:43
How do I create a large HDD ?	fpmpaolo	project.ClassicWB	4	29 February 2016 08:23
Best up to date way of installing ClassicWB and WHDLoad on a large HDD	KTC	project.ClassicWB	9	03 August 2008 14:34
Driver patches for large HDD's	ORSM T	support.Other	0	11 June 2007 05:57

16 February 2019, 23:18	#2
DamienD Banned Join Date: Aug 2005 Location: London / Sydney Age: 47 Posts: 20,420	Think about it Foebane. When you are adding in large archives WinUAE is having to mount them... This takes time and will "freeze" WinUAE if you have many. ...and you went down this silly route just to save a few MBs I would definitely reverse your decision on this one.

17 February 2019, 01:39	#4
Foebane Banned Join Date: Sep 2011 Location: Cardiff, UK Age: 51 Posts: 2,871	I realise I'm going OT on this, but I really need advice on this matter, please.

17 February 2019, 05:12	#5
haps Rumpig Join Date: Aug 2006 Location: The bottom of the bottle Age: 92 Posts: 243	2.4GB+ is huge? I know I could go down the post office and buy a 16GiB USB stick for 5 quid. All this space saving you carry on about really makes me shake my head.

17 February 2019, 09:36	#6
Toni Wilen WinUAE developer Join Date: Aug 2001 Location: Hämeenlinna/Finland Age: 49 Posts: 26,505	If archive is mounted as a harddrive, whole archive needs to be unpacked and converted to internal directory tree. It can take long time and use lots of RAM and unless you use 64-bit WinUAE, there may be only few hundred megabytes of RAM left (32-bit processes usually have 1G to 3G of RAM free where everything must fit, including DLLs, winuae.exe etc.., depending on Windows version), especially if your config has lots of Z3 RAM.

17 February 2019, 12:37	#7
rare_j Zone Friend Join Date: Apr 2005 Location: London Posts: 1,176	Use an NTFS compressed folder? Transparent compression. Compression rates not too bad.

17 February 2019, 13:50	#9
Tomislav Registered User Join Date: Aug 2014 Location: Zagreb / Croatia Posts: 302	You can use hard disk file with NTFS compression. It would be faster access and you gain some space. Use zip for archiving only or small things like ADFs.

17 February 2019, 21:20	#11
mark_k Registered User Join Date: Aug 2004 Location: Posts: 3,335	Could the lock-up issue indicate a bug? If on resetting WinUAE unpacks the archive contents to memory again, without freeing the already-unpacked data, the amount of memory used will gradually increase each reset until you run out.

18 February 2019, 11:59	#13
Foebane Banned Join Date: Sep 2011 Location: Cardiff, UK Age: 51 Posts: 2,871	I've put the files back to their non-zipped state. It takes up more space, but it makes things easier generally.

19 February 2019, 00:24	#16
haps Rumpig Join Date: Aug 2006 Location: The bottom of the bottle Age: 92 Posts: 243	I'm sure that if you worked as a ditch digger Foebane you would insist on doing the job with a teaspoon.

19 February 2019, 03:00	#17
AMIGASYSTEM Registered User Join Date: Aug 2014 Location: Brindisi (Italy) Age: 70 Posts: 8,248	I would avoid using compressors LHA, ZIP, RAR etc.. The best solution is to compress all demos individually with PowerPacker or CrunchMania. Once this is done you can copy all the file self-extracting to a HardFile from 2GB/4GB to keep the double or triple copies. Compressing these "full" HardFiles would be useless because the content is already compressed

19 February 2019, 08:18	#19
chip Registered User Join Date: Oct 2012 Location: Italy Age: 49 Posts: 2,942	But when you will reach the end Foebane ? I strongly guess this hobby, luckily, will take to you much more time than you think I don't know what kind of demos are you considering, but if you consider them all You know, we have something in common I strongly guess i will reach "the end" at least between 10 years

19 February 2019, 09:19	#20
Foebane Banned Join Date: Sep 2011 Location: Cardiff, UK Age: 51 Posts: 2,871	There will always be more demos I can add on top of my 640, but finding them will take time. Also, I don't just collect any old demos, but ones I like personally.

Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)