Preventing FTP timeouts

[andyhants]

Hi

I'm using AmiFTP under WinUAE AmiKit to connect to the EAB FTP server but I get timeouts when selecting large Directories. For example /Non TOSEC Games/Disk/ - which has a size of 163840. AmiFTP is noticeably slower than say Opus FTP but I strongly prefer AmiFTP as I can control the location of the Window.

Is there anything I can do to prevent a timeout at all ?

If not can anyone recommend another FTP client - aside from the one embedded in Opus ?

Thanks

[andyhants]

Originally Posted by andyhants

Hi

I'm using AmiFTP under WinUAE AmiKit to connect to the EAB FTP server but I get timeouts when selecting large Directories. For example /Non TOSEC Games/Disk/ - which has a size of 163840. AmiFTP is noticeably slower than say Opus FTP but I strongly prefer AmiFTP as I can control the location of the Window.

Is there anything I can do to prevent a timeout at all ?

If not can anyone recommend another FTP client - aside from the one embedded in Opus ?

Thanks

bump

Can any one help plse?

Have also been trying AmiTrade Centre as well as AmiFTP. Both really good in allowing me to configure the sise & location of the Lister but both end up timing out the EAB FTP server when reading v long directories - like TOSEC/Applications/ADF.

I can't be the only one having this frustration when trying to FTP to the EAB server in an Amiga Envioronment. Can anyone suggest anything? Surely the EAB server is more Amiga friendly than this ?

[Turran]

For the record, its TURRAN FTP, not EAB FTP for quite some time. It is not really connected to EAB anymore, other than having EAB forums members being able to register upload accounts on it.

I don't FTP from the Amiga so can't really help there. I can only suggest trying ftp2.grandis.nu instead as its a lot faster. Otherwise its probably just to many files in there for the poor FTP client to handle.

Or you can use a web browser to http://ftp2.grandis.nu/turran and download files that way. I try to keep the website as simple as possible so it still works on Amigas but I don't really verify that after every change I make so.. might be things that is not looking 100%

[Antti]

I’m using RNOXfer without problems on A4000. It sure takes a long time to get listing from some directories but so far I have not encountered any timeouts.

[andyhants]

Thanks for your responses guys.

Can't get RNOXfer to run - it keeps reporting AmiSSL is missing for some reason depite me d/ling & installing it. I'm going to be a rebuild of my AmiKit environment so that might fix it.

In the meantime though if you have the time plse could you have a go at FTP-ing onto the TOSEC/Games/Commodore Amiga - Games - [ADF] Directory to see if it doesn't time out - would be very interested to hear your results.

As far as I can tell its by far the largest directory on the ftp2.grandis.nu site - something like 32k + files. I can only FTP to it using Opus FTP which doesn't work for me aethetically as I can't control the location of the FT'P Lister when you launch it.

Thanks again for your help.

[Turran]

If you want to see where your limit is, here is a sorted list of folders with the most files inside.

61585 ./TOSEC - Non-Amiga/Commodore C64/Games - Alternatives
58370 ./TOSEC - Non-Amiga/Commodore C64/Games - [D64]
46702 ./TOSEC - Non-Amiga/Commodore C64/Demos - [D64]
41770 ./TOSEC - Non-Amiga/Commodore C64/Games - [T64]
31458 ./TOSEC/Games/Commodore Amiga - Games - [ADF]
22408 ./TOSEC - Non-Amiga/Commodore C64/Demos - [T64]
21490 ./TOSEC - Non-Amiga/Commodore C64/Games - [PRG]
19062 ./TOSEC - Non-Amiga/Sinclair ZX Spectrum/Games - [TAP]
18313 ./TOSEC - Non-Amiga/Sinclair ZX Spectrum/Games - [TZX]
14907 ./TOSEC - Non-Amiga/Commodore C64/Demos - [PRG]
12897 ./TOSEC - Non-Amiga/Commodore C64/Music - [D64]
12194 ./TOSEC - Non-Amiga/Nint... & Entertainment System/Games - [NES]
11773 ./TOSEC - Non-Amiga/Commodore C64/Cracktros - [PRG]
11250 ./TOSEC - Non-Amiga/Commodore C64/Diskmags - Scene - [D64]
11195 ./TOSEC - Non-Amiga/Sinclair ZX Spectrum/Games - [Z80]
11091 ./TOSEC - Non-Amiga/Commodore C64/Games - [P00]
10852 ./TOSEC - Non-Amiga/Commodore C64/Games - [TAP]
10517 ./TOSEC - Non-Amiga/Commodore C64/Collections - BugJam
9537 ./TOSEC - Non-Amiga/Sinclair ZX Spectrum/Games - [SCL]
8890 ./TOSEC/Demos/Commodore Amiga - Demos - Packs
7557 ./TOSEC - Non-Amiga/Atari ST/Compilations - Games - [ST]
7501 ./TOSEC - Non-Amiga/Commodore C64/Music - [PRG]
7228 ./TOSEC - Non-Amiga/Commodore C64/Music - [T64]
7148 ./TOSEC - Non-Amiga/Commodore C64/Applications - [D64]
6786 ./TOSEC - Non-Amiga/Commo.../Collections - Derbyshire Ram - [D64]
6674 ./TOSEC - Non-Amiga/Sega Mega Drive & Genesis/Games - [BIN]
6284 ./TOSEC - Non-Amiga/Commodore C64/Diskmags - Commercial - [D64]
6049 ./TOSEC - Non-Amiga/Atari ST/Games - [ST]
5712 ./TOSEC - Non-Amiga/Atari 8bit/Games - [ATR]
5638 ./TOSEC - Non-Amiga/Commodore C64/Games - [G64]
5545 ./TOSEC - Non-Amiga/Commodore C64/Graphics - [PRG]
5121 ./Collection/Music/Mod/Basty's Modules/Protracker/- unknown
5114 ./TOSEC - Non-Amiga/Commodore C64/Collections - Jazzcat
4992 ./TOSEC/Games/Commodore Amiga - Games - Public Domain - [ADF]
4944 ./TOSEC - Non-Amiga/Commodore C64/Public Domain - [D64]
4939 ./TOSEC - Non-Amiga/Commodore C64/Educational - [D64]
4756 ./TOSEC/Applications/Commodore Amiga - Applications - [ADF]
4710 ./TOSEC/Demos/Commodore Amiga - Demos - Various - [ADF]
4695 ./TOSEC - Non-Amiga/Commodore C64/Games - Previews - [D64]
4537 ./TOSEC - Non-Amiga/Commodore C64/Collections - Earthshaker
4343 ./Collection/Compilation/SSC
4320 ./TOSEC - Non-Amiga/Acorn BBC/Games - [SSD]
4233 ./TOSEC - Non-Amiga/Commodore C64/Collections - C64CD'96
4170 ./TOSEC - Non-Amiga/Nintendo Game Boy/Games
4095 ./TOSEC/Demos/Commodore Amiga - Demos - Music
3887 ./TOSEC - Non-Amiga/Ninte... & Super Entertainment System - Games
3694 ./Non TOSEC IPFs - Official
3656 ./Collection/Scans/Manuals/Games
3649 ./TOSEC - Non-Amiga/Sinclair ZX Spectrum/Applications - [TAP]
3647 ./TOSEC - Non-Amiga/Commodore C64/Applications - [PRG]

[Antti]

Originally Posted by andyhants

In the meantime though if you have the time plse could you have a go at FTP-ing onto the TOSEC/Games/Commodore Amiga - Games - [ADF] Directory to see if it doesn't time out - would be very interested to hear your results.

Ok so I tried RNOXfer on that directory. Did not get timeout but didn’t get directory listing either. To be sure I waited for an hour doing something else while waiting.

After RNOXfer had failed I gave the ftp client from Roadshow a try. That one starts listing the directory to shell window but timeouts before completed. Watching the files scroll in shell window it came to my mind that maybe the problem was that the printing is too slow. So I gave it another try but this time sent the shell window behind the workbench window. And it worked! So it is possible to get listing from those directories but unfortunately not with a client that has graphical ui

[jPV]

Originally Posted by andyhants

Can't get RNOXfer to run - it keeps reporting AmiSSL is missing for some reason depite me d/ling & installing it.

From the requirements section in the RNOXfer documentation: "- AmiSSL v4 (http://aminet.net/search?query=amissl)"

So, the latest AmiSSL v4 is 4.12 currently, do you have that installed? It doesn't work if you only have v5 installed, but both branches can coexist. It'd would be the best to install v4 first and then v5 next to it, but if you do it other way around, you'll have to copy the latest amisslmaster.library from v5 manually.

Originally Posted by Antti

Ok so I tried RNOXfer on that directory. Did not get timeout but didn’t get directory listing either. To be sure I waited for an hour doing something else while waiting.

It seems that parsing the directory listing is too heavy for this huge directories. It takes a minute to parse it on my Mac mini G4 too, so probably would take hours on 68k as you experienced

I'm using Hollywood/Lua pattern matching functions to parse listings and that's an easy and reliable way to do it, but probably a bit too demanding for this large data. I rather wouldn't touch it, but maybe I'll do some tests if I could make it faster in some other way, and decide then...

But generally speaking, I think these directories are unreasonable large at the FTP site, and it would be better for the server admin to think splitting them in several directories. I have TOSEC collections from other sources where people had split the ADF directory into several sub dirs, maybe for these kinds of reasons.

[andyhants]

Originally Posted by jPV

From the requirements section in the RNOXfer documentation: "- AmiSSL v4 (http://aminet.net/search?query=amissl)"

So, the latest AmiSSL v4 is 4.12 currently, do you have that installed? It doesn't work if you only have v5 installed, but both branches can coexist. It'd would be the best to install v4 first and then v5 next to it, but if you do it other way around, you'll have to copy the latest amisslmaster.library from v5 manually.

Ah ok - so presumably the amisslmaster.library file would go in SYS:Libs/ ?



It seems that parsing the directory listing is too heavy for this huge directories. It takes a minute to parse it on my Mac mini G4 too, so probably would take hours on 68k as you experienced

I'm using Hollywood/Lua pattern matching functions to parse listings and that's an easy and reliable way to do it, but probably a bit too demanding for this large data. I rather wouldn't touch it, but maybe I'll do some tests if I could make it faster in some other way, and decide then...

But generally speaking, I think these directories are unreasonable large at the FTP site, and it would be better for the server admin to think splitting them in several directories. I have TOSEC collections from other sources where people had split the ADF directory into several sub dirs, maybe for these kinds of reasons.

I think splitting the vv large TOSEC Games ADF directory is a great idea - http://eab.abime.net/member.php?u=38788 would that be possible please ? (sorry can't seem to do the @Turran thing)

[Turran]

CodyJarret is in charge of all /TOSEC folders. TCD is also currently doing a cleanup. I don't want to split it up myself because I'm not sure that'll change the structure as it comes from TOSEC thus making updates in the future a pain in the arse.

Perhaps check with TCD about his thoughts but considering there are a lot of folders with lots of files in them, there would be a lot of splitting to do so I would not get my hopes up.

https://eab.abime.net/showthread.php?t=112584

[TCD]

If we knew the amount of files per folder that are still okay it would be easier to decide if it's viable to split certain folders.

I see the same problem as Turran with TOSEC though. Each update would have to be manually split into subfolders as TOSEC doesn't have those in their DAT files.

[jPV]

Originally Posted by jPV

It seems that parsing the directory listing is too heavy for this huge directories. It takes a minute to parse it on my Mac mini G4 too, so probably would take hours on 68k as you experienced

I'm using Hollywood/Lua pattern matching functions to parse listings and that's an easy and reliable way to do it, but probably a bit too demanding for this large data. I rather wouldn't touch it, but maybe I'll do some tests if I could make it faster in some other way, and decide then...

I was able to optimize RNOXfer quite much with this after all. I changed tables to "optimized" ones and disabled garbage collector in critical parts, and the result with G4 is that the actual parsing time dropped from 56 seconds to 1 second Sorting files takes more time now, but I was able to optimize it a bit too... but only couple seconds and it takes 3-4 secs now. MUI listview updating takes couple of seconds too and can't do anything for it.

So, it's quite reasonable for NG machines now, but I haven't tried it on 68k yet, probably still quite slow to list the 31458 files in the ADF dir. And downloading the file list before even getting to process it may take some time too, because it's almost 4MB...

In any case RNOXfer is now way faster than DOpusFTP... I had a quick try with DOpusFTP and fed up waiting after a minute or so... so it isn't quite optimal for large directories either.

[jPV]

Originally Posted by andyhants

Ah ok - so presumably the amisslmaster.library file would go in SYS:Libs/ ?

Sorry I missed this question, because it was inside my quote in the post. No, it won't go into the generic libs diretory, but it goes into AmiSSL:libs/ instead.

BTW. according to Antti's tests on his 68k setup, my new changes in the RNOXfer code makes the largest directories to list in about 10 mins, instead the hour+ that it was previously It still timeouts with the three largest directories (C64 related), but the rest are working now.

[Turran]

Based on my list with the folders with the most files in it, where would you say the limit is for when it becomes a problem/annoyance on a real Amiga?

Based on that number, I could decide if its worth to split it into A-Z folders. If there are too many folders, its not worth it IMO.

[Antti]

Hi Turran.

From your list, the first 3 directories did not work with the new RNOXfer beta. The first to give directory listing was
41770 ./TOSEC - Non-Amiga/Commodore C64/Games - [T64]

When RNOXfer fails to get directory listing I see these messages in the log window:

< 421 Timeout (120 seconds): closing control connection.
* We got a 421 - timeout!
* control connection looks dead
* Closing connection 0

So is it possible to increase the control connection timeout value in server? Maybe 240 or 300 seconds?

[Turran]

I've changed it from 120 to 300 on ftp2.grandis.nu for registered users. Give it a go (not changed on ftp.grandis.nu)

Originally Posted by Antti

Hi Turran.

From your list, the first 3 directories did not work with the new RNOXfer beta. The first to give directory listing was
41770 ./TOSEC - Non-Amiga/Commodore C64/Games - [T64]

When RNOXfer fails to get directory listing I see these messages in the log window:

< 421 Timeout (120 seconds): closing control connection.
* We got a 421 - timeout!
* control connection looks dead
* Closing connection 0

So is it possible to increase the control connection timeout value in server? Maybe 240 or 300 seconds?

[Turran]

Is there a way in RNOXfer to send raw commands to the FTP?

Using a registered account, you can send the command "site idle" to see your current idle timeout value. It should be set to 0 or 300 for the default (300 on ftp2.grandis.nu. 120 on ftp.grandis.nu). This is only valid for the current session if done like that (site idle <new timeout>) but I can also change it permanently for a specific user..

Reason I'm asking is because I do not want to leave the default timeout at 300 seconds really, so if there is a way to raise it high above 300, we can allow users to do that if needed.

You can also see your current Idle time set permanently with "site user <your username>" (And a bunch of other stuff). It should be set to "disabled" for the default timeout value.

If you are using the anonymous "ftp" account, there is not much I can do because I really do not want to raise the idle timeout for that one...

[jPV]

Originally Posted by Turran

Is there a way in RNOXfer to send raw commands to the FTP?

Yes, there's a "Custom command" option in the Log window (you can enter raw commands from the string field and see the output in the log).

I just tried that at least "site idle" and "site idle <value>" do work fine.

[Turran]

Goodie. If it works better on ftp2.grandis.nu with 300 timeout now, we have a plan.

[Antti]

With 300 timeout it works better. Only the directory with most files fails (61585 ./TOSEC - Non-Amiga/Commodore C64/Games - Alternatives), but error is now

* Recv failure: Unknown error code
* Closing connection 3

I think it took more than 5 minutes (or 300 seconds) to get this error so I'm not sure what is happening. RNOXfer status was still 'Downloading'.

So RNOXfer now succeeds to get listing from most of the directories. With my A4000 060-cs1 setup it is painstakingly slow on bigger directories so I would not recommend using it on those unless there's no other option.