English Amiga Board


Go Back   English Amiga Board > Other Projects > project.Amiga File Server

 
 
Thread Tools
Old 09 May 2022, 20:33   #2981
enigma776
Bringer of Death

enigma776's Avatar
 
Join Date: Apr 2017
Location: Peterborough, UK
Age: 38
Posts: 364
Quote:
Originally Posted by Hache View Post
Thanks for your quick response!! I'm using Filezilla. Regarding to "allow 1 connection", I think this is you are you talking about?

Attachment 75552
Correct, but also in site manager if you have it saved there also as it has its own setting under Transfer Settings
enigma776 is offline  
Old 09 May 2022, 20:43   #2982
Hache
Registered User
Hache's Avatar
 
Join Date: Jan 2013
Location: Buenos Aires, Argentina
Posts: 215
SOLVED!!!

Thank you so much!!!
Hache is offline  
Old 11 May 2022, 16:01   #2983
Turran
Moderator

Turran's Avatar
 
Join Date: May 2012
Location: Stockholm / Sweden
Age: 47
Posts: 1,360
Experimenting with the web download from https://grandis.nu/turran/FTP.

The download limit of 30 GiB and 500 Files currently does not count towards the following file extensions: jpg gif png css htm html cgi txt

I've now added so that the limit does not count towards the following folders as well.
Pimiga, Retroplay WHDLoad Packs, DamienD's WinUAE Collection

Meaning that if you use https://grandis.nu/turran/FTP, you can grab anything from those folders without the limit header counting up any files or GiB.

This should help but I'm not sure yet if it'll be permanent. If its permanent, it should help to write "WHDLoad games downloaders" programs as they no longer have to deal with FTP.
Turran is offline  
Old 15 May 2022, 14:58   #2984
reno
Registered User

 
Join Date: Jul 2019
Location: London
Posts: 34
Quote:
Originally Posted by Turran View Post
Experimenting with the web download from https://grandis.nu/turran/FTP.

This should help but I'm not sure yet if it'll be permanent. If its permanent, it should help to write "WHDLoad games downloaders" programs as they no longer have to deal with FTP.
Hi,

Firstly : thanks for all the hard work, it's an invaluable resource for Amigans !

Have you considered making it available over rsync in addition to ftp/http ?

I think this could make everybody happy :
- Allow people to maintain their local mirrors (which has perfectly legitimate use cases, and is the right thing to do for preservation of such an important resource in the community anyway)
- Keep your bandwidth reasonable low by giving people a way to keep up to date without a full download.

In fact, I was doing just that with the Google Drive mirror : I downloaded the full Retroplay WHDLoad Packs just once a long time ago, then I wrote a script using "rclone" (=rsync for cloud storage) to keep my NAS (and my Amigas, and now A500 mini) in sync with the latest releases.

If you don't want to set rsync up on grandis.nu and are looking for another cloud provider to replace GDrive, it would be really nice if it was one that's supported by rclone (https://rclone.org/#providers)

Cheers
reno is offline  
Old 15 May 2022, 15:24   #2985
Turran
Moderator

Turran's Avatar
 
Join Date: May 2012
Location: Stockholm / Sweden
Age: 47
Posts: 1,360
@reno

I did use rsync when we had an official mirror a while back. I'd be happy to do so again if someone wants to set a public mirror up. Just opening up rsync to everyone is not on the table as we have no use for another private copy on someones USB disk in a closet somewhere.

As for rclone, I did switch the backup to JottaCloud (see link in first post). It supports rclone just fine. In fact, its what I use to sync to JottaCloud every night

Not sure what would be needed for you to sync from it as well. I can look that up but probably not this week as its going to get busy for me right now at work.. I have a feeling you already know more about it than I do.

I have no problems if you write a guide for how to download the whole thing from JottaCloud using rclone. I'm sure all the anonymous ftp users would love it so they are not tempbanned from the ftp and website over and over for trying to leech the whole thing, heh.

Last edited by Turran; 15 May 2022 at 15:31.
Turran is offline  
Old 15 May 2022, 17:49   #2986
reno
Registered User

 
Join Date: Jul 2019
Location: London
Posts: 34
Quote:
Originally Posted by Turran View Post
@reno

I did use rsync when we had an official mirror a while back. I'd be happy to do so again if someone wants to set a public mirror up. Just opening up rsync to everyone is not on the table as we have no use for another private copy on someones USB disk in a closet somewhere.
I suppose one thing you'd get from that, is to make it much more likely that multiple people will have full, always up to date offsite backups in case disaster strikes on your end, and can come to the rescue. As Linus Torvalds put it : "Backups are for wimps. Real men upload their data to an FTP site and have everyone else mirror it."

Bandwidth costs aside, is there another reason you wouldn't want to make it *too* easy to download the full packs ?

Quote:
Originally Posted by Turran View Post
As for rclone, I did switch the backup to JottaCloud (see link in first post). It supports rclone just fine. In fact, its what I use to sync to JottaCloud every night
Yes I saw that, but also that you were looking for another solution because of the 1000 files limitation ?

Besides, it looks like rclone doesn't support anonymous syncing from Jottacloud. It was the same with GDrive, but nearly everyone already has a Google account. I don't know if people would be able to create a Jottacloud account for free then access a shared folder like you could do with GDrive ?

Quote:
I have no problems if you write a guide for how to download the whole thing from JottaCloud using rclone. I'm sure all the anonymous ftp users would love it so they are not tempbanned from the ftp and website over and over for trying to leech the whole thing, heh.
If you're going to settle on JottaCloud and we find a way to solve the accounts/shared folder issue, sure !

Right now, I think the http plugin for rclone is actually a better solution. I did manage to update my Retroplay WHDLoad packs from your website using it. Definitely slower than GDrive but it was just 30 files or so since I last synced, so no big deal. I only had an issue with this demo : the "å" in the filename didn't get transcribed properly after downloading (it turned into "?E5"), and that crashed the HstWB "build_install_entries" Python script that I run in the end. I think the root cause is that your Apache server uses charset=ISO-8859-1 for text/html, instead of UTF-8.
reno is offline  
Old 15 May 2022, 18:57   #2987
Turran
Moderator

Turran's Avatar
 
Join Date: May 2012
Location: Stockholm / Sweden
Age: 47
Posts: 1,360
Quote:
Originally Posted by reno View Post
Bandwidth costs aside, is there another reason you wouldn't want to make it *too* easy to download the full packs ?

Well, yes, I have a reason.

EDIT: Just to be clear, I have no problem with anyone wanting to download a whole "pack" like the WHDLoad games or DamienDs pack. That is why I made it free on the website. I'm talking about the whole FTP here.

There are simply to many people that wants to download the whole thing without having any intention of sharing it. In case of a disaster, I have double backups here and also the JottaCloud. We don't need any more really. If someone said they want to host a backup, I'd be happy to oblige.


But, as it stands right now, the bandwidth is increasing by a lot day by day. More and more anonymous users try to download the whole FTP for no other reason than "they want it", I guess. Sure, there might be people who are serious about downloading it all for preservation purposes, but those guys can do it the "right" way. Show you have something to give to community by making 50 useful posts on the EAB forums and create your own account. Then you can download the whole thing.


To prove the point, since 2018-05-17, when I enabled logging, we have 218TiB downloaded. That is the entire FTP downloaded over 54 times. How many of those "backups" can you access today? None I bet. How many would come to the rescue with "I have it all backed up and will make it public" if I crash? No idea. =)
For preservation purposes, as you say, should everything crash here, we should already have 50 full backups out there, but nobody knows where.


There are simply to many people thinking the same way and I would have 10-20 anonymous users online 24/7 leeching, had it not been for the temporary bans I throw out for trying exactly this.


As for the downloads increasing by a lot:
In 2021, the total download was 52.11 TiB. Thats the entire FTP 13 times.
In 2022, SO FAR, we have 80 TiB downloaded. Entire FTP 20 times and we're not even halfway through 2022.



I am not hosting the FTP for people to download the whole thing. There are official ways to do that if you feel the need to do so (contact me for a public mirror or register an upload account once you pass the requirements).


I want the files to be available to everyone. I don't want the "Server is full" message when someone wants a WHDLoad game or similar. That is what will happen if I open the flood gates and was actually happening a lot before I implemented the limits on the anonymous account a while back.



Quote:
Originally Posted by reno View Post
Yes I saw that, but also that you were looking for another solution because of the 1000 files limitation ?
Yeah, but I've yet to find a better one. JottaCloud have been great in responding to my requests and I got one bug fixed in a couple of days after reporting it. Its in their development pipeline to have a look at the 1000 files limit, so I'm trusting that will be fixed some day.

Last edited by Turran; 15 May 2022 at 19:03.
Turran is offline  
Old 16 May 2022, 08:02   #2988
XPD
Geek Kiwi

XPD's Avatar
 
Join Date: Aug 2014
Location: Auckland, New Zealand
Age: 45
Posts: 147
Can I get my password reset please ?

Thanks!
XPD is offline  
Old 16 May 2022, 08:32   #2989
XPD
Geek Kiwi

XPD's Avatar
 
Join Date: Aug 2014
Location: Auckland, New Zealand
Age: 45
Posts: 147
If I had the storage space, I'd offer to host a mirror - but at the moment, I'm almost outta space.... need to sell another kidney to get another drive to add to the server
XPD is offline  
 


Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)
 
Thread Tools

Similar Threads
Thread Thread Starter Forum Replies Last Post
While EAB file server is down... Retroplay project.Amiga File Server 24 18 March 2020 11:34
SWOS '96-'97 (EAB File Server) Specksynder support.Games 8 27 September 2014 14:08
Secondary file server DannyBoy project.Amiga File Server 12 12 August 2013 11:33
Is EAB file server deleted ? Ribdevil project.Amiga File Server 13 03 February 2012 19:28
'Syncing' with the file server exoticaga project.Amiga File Server 5 20 August 2010 00:36

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +2. The time now is 19:34.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2022, vBulletin Solutions Inc.
Page generated in 0.18007 seconds with 15 queries