English Amiga Board


Go Back   English Amiga Board > Other Projects > project.EAB File Server

 
 
Thread Tools
Old 29 October 2017, 08:20   #1
AeroMaxx
 
Posts: n/a
EAB Mirror

Hi Everyone,

I realise you probably don't really need it, but I thought I'd make a mirror of the EAB file server.

This should be relatively easy to do it, but as some of the files have invalid and/or incomplete multibyte or wide characters in the filenames some of the transfers are failing.

Transferring file `PoWeR/PoWeR_15/Bonus/PÚlda.src.lzx'
mirror: /Commodore_Amiga/Collection/Coverdisk/PoWeR/PoWeR_15/Bonus/PÚlda.src.lzx: Invalid or incomplete multibyte or wide character

I was hoping to use lftp on debian to easily mirror the file server once every 3-4 days, I don't think it changes that often.

Thanks,
 
Old 29 October 2017, 10:37   #2
DannyBoy
Registered User
 
Join Date: Jul 2013
Location: Helsinki / Finland
Age: 37
Posts: 90
I dunno if this is such a hot idea. EAB files are available at three different places now. It's getting silly
DannyBoy is offline  
Old 29 October 2017, 11:40   #3
AeroMaxx
 
Posts: n/a
Quote:
Originally Posted by DannyBoy View Post
I dunno if this is such a hot idea. EAB files are available at three different places now. It's getting silly
Sure I realise that hence why I said you probably don't really need it, but if I did still want to make a mirror do you have any recommendations as to how to keep it up to date?

How do you keep your mirror up to date? I noticed you use vsftp, whilst the official one uses glFTPd.
 
Old 29 October 2017, 12:18   #4
DannyBoy
Registered User
 
Join Date: Jul 2013
Location: Helsinki / Finland
Age: 37
Posts: 90
Quote:
Originally Posted by AeroMaxx View Post
Sure I realise that hence why I said you probably don't really need it, but if I did still want to make a mirror do you have any recommendations as to how to keep it up to date?

How do you keep your mirror up to date? I noticed you use vsftp, whilst the official one uses glFTPd.
Main FTP syncs daily to mirror with rsync. You can do this with lftp too, but I don't think it's the tool for the job.

I think you should reconsider this. Have a look at the main thread (last few pages) as there is some discussion about the problems that leeching the whole filetree is causing.
DannyBoy is offline  
Old 29 October 2017, 14:03   #5
AeroMaxx
 
Posts: n/a
Quote:
Originally Posted by DannyBoy View Post
I think you should reconsider this. Have a look at the main thread (last few pages) as there is some discussion about the problems that leeching the whole filetree is causing.
Thanks for the information, but I will go and have a read of that thread to see what you mean.

I actually already have about 80% of the FTP, but the FTP isn't as tidy as i'd like it to be personally.
 
Old 29 October 2017, 14:51   #6
AeroMaxx
 
Posts: n/a
Quote:
Originally Posted by DannyBoy View Post
Main FTP syncs daily to mirror with rsync. You can do this with lftp too, but I don't think it's the tool for the job.
I didn't think rsync worked with ftp, but I checked and read the man pages and it seems I am correct, so unless your using sftp instead somehow I can't see how rsync is working for you.

as previously mentioned lftp doesn't work fully due to the multibyte characters in some filenames.
 
Old 29 October 2017, 15:10   #7
DannyBoy
Registered User
 
Join Date: Jul 2013
Location: Helsinki / Finland
Age: 37
Posts: 90
Quote:
Originally Posted by AeroMaxx View Post
I didn't think rsync worked with ftp, but I checked and read the man pages and it seems I am correct, so unless your using sftp instead somehow I can't see how rsync is working for you.

as previously mentioned lftp doesn't work fully due to the multibyte characters in some filenames.
Ugh.. They are two different things. rsync has been working very well for us like four years. Feel free to grab files you actually want, but I recommend dropping the mirroring plans.
DannyBoy is offline  
Old 29 October 2017, 15:19   #8
ptyerman
Registered User

ptyerman's Avatar
 
Join Date: Jun 2012
Location: Worksop/UK
Age: 54
Posts: 1,173
Have you even asked for permission to mirror it? You can't just jump in and leech the whole ftp, people have done this and it's caused untold problems, this is the reason for the new restrictions!
ptyerman is offline  
Old 29 October 2017, 21:08   #9
AeroMaxx
 
Posts: n/a
Quote:
Originally Posted by DannyBoy View Post
Ugh.. They are two different things. rsync has been working very well for us like four years. Feel free to grab files you actually want, but I recommend dropping the mirroring plans.
What are two different things?

Anyway I tried rsync with a folder that I know has multibyte and/or wide characters and as such fails to transfer using lftp, but the connection just times out.

rsync --recursive --times --compress --verbose --progress ftp://ftp@grandis.nu:/Commodore_Amig..._1_REPULSE.ori "/home/ftp/Commodore_Amiga/App/Archive/SFX_4_1_REPULSE.ori/";
 
Old 29 October 2017, 21:28   #10
Turran
Moderator

Turran's Avatar
 
Join Date: May 2012
Location: Stockholm / Sweden
Age: 44
Posts: 1,063
You can not use rsync. Me and DannyBoy have a rsync relationship set up so I can run rsync to the mirror from the main FTP. That is done every night and is not something you can do yourself. Both parties needs to be involved and it has nothing to do with FTP.

As DannyBoy said, we do not really need a 3rd mirror right now. We already have 2 and a google drive backup open to the public (also synced from me every night). Appreciate the thought though. Opening one anyway will only get you alienated from EAB. Especially if you plan to resort it (although I agree that it needs a big cleanup).

I see you registered today, both to the forums and the FTP. I have no problems with the 15GB you have leeched already today, but be advised that you are limited to 50GB/Week so you won't get it all in one go.

Regarding the files that you fail to download, I'm guessing the filesystem you are downloading to can not handle it, while ours can. Or its your client that can not handle it. Not much we can do there for you I'm afraid.

Last edited by Turran; 29 October 2017 at 21:48.
Turran is offline  
Old 29 October 2017, 22:02   #11
DannyBoy
Registered User
 
Join Date: Jul 2013
Location: Helsinki / Finland
Age: 37
Posts: 90
Quote:
Originally Posted by AeroMaxx View Post
What are two different things?
Turran explained this throghly, but I'll have a word on this yet. FTP is for downloading stuff (and sometimes uploading as with the primary server). Rsync is for synchronizing both servers. Rsync does not speak FTP, you need a service for it. Read the manpage again

I agree with you on some things though. Directory hierarchy could be more tidy, and filenames ought to be plain ascii for sure

Last edited by DannyBoy; 29 October 2017 at 23:04. Reason: Detail
DannyBoy is offline  
Old 29 October 2017, 22:37   #12
DamienD
Global Moderator

DamienD's Avatar
 
Join Date: Aug 2005
Location: London / Sydney
Age: 41
Posts: 12,351
@AeroMaxx; how many times do you need to be told from the 2 people hosting both the main FTP and the mirror (for many years now without issue) that we don't need another backup (if that's really the main reason why you're grabbing everything anyway, which I doubt)???

Seriously; please stop your leeching of our resource otherwise action will be taken; I guarantee it!!!
DamienD is offline  
Old 06 November 2017, 00:27   #13
nzlpoa
Registered User
 
Join Date: Nov 2017
Location: asaz
Posts: 12
Quote:
Originally Posted by DannyBoy View Post
I dunno if this is such a hot idea. EAB files are available at three different places now. It's getting silly
and two places isn't ?

Three (if u count Google drive)

Just saying
Quote:
Originally Posted by DamienD View Post
@AeroMaxx; how many times do you need to be told from the 2 people hosting both the main FTP and the mirror (for many years now without issue) that we don't need another backup (if that's really the main reason why you're grabbing everything anyway, which I doubt)???

Seriously; please stop your leeching of our resource otherwise action will be taken; I guarantee it!!!
Google drive is there. so really the main problem i see is no-one likes Google own limitations as the main reason for this.

Last edited by nzlpoa; 06 November 2017 at 00:34.
nzlpoa is offline  
Old 06 November 2017, 00:59   #14
DamienD
Global Moderator

DamienD's Avatar
 
Join Date: Aug 2005
Location: London / Sydney
Age: 41
Posts: 12,351
Quote:
Originally Posted by nzlpoa View Post
Google drive is there. so really the main problem i see is no-one likes Google own limitations as the main reason for this.
Well then, do the following:

... Join up to EAB (which you have already done).
... Be part of the EAB community and at least create 15 posts (yeah I know, very taxing indeed, I do double that amount in one day).
... Sign up here afterwards to gain proper access to the "EAB File Server".

If the above is all too much for you to handle then alternatively:

a) Use the mirror.
b) Use the GoogleDrive.
c) Use main FTP (but with limitations).

...I mean; seriously though... you've only just joined EAB today + made 4 posts; yet you feel you have the right to complain???

F$%k me, what's this world coming to???
DamienD is offline  
Old 06 November 2017, 02:11   #15
DamienD
Global Moderator

DamienD's Avatar
 
Join Date: Aug 2005
Location: London / Sydney
Age: 41
Posts: 12,351
@Turran; been thinking after this new member nzlpoa's crazy post above my two in this thread...

15 posts is way too easy for any old leechers to achieve and then have full access / speed at the cost of "real" EAB members

Could you please increase the limit on registering new accounts to 100 posts (which is really still peanuts for anyone seriously wanting to join our community and contribute in a meaningful manner)?

Many thanks in advance my friend

Once done; I'll update all notifications in the relevant threads
DamienD is offline  
Old 06 November 2017, 11:55   #16
Turran
Moderator

Turran's Avatar
 
Join Date: May 2012
Location: Stockholm / Sweden
Age: 44
Posts: 1,063
You da boss =)

Done
Turran is offline  
Old 06 November 2017, 12:28   #17
DamienD
Global Moderator

DamienD's Avatar
 
Join Date: Aug 2005
Location: London / Sydney
Age: 41
Posts: 12,351
Cheers Turran; have now updated the first post in the EAB File Server thread
DamienD is offline  
 


Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)
 
Thread Tools

Similar Threads
Thread Thread Starter Forum Replies Last Post
Keep mirror? DannyBoy project.EAB File Server 11 11 October 2017 02:37
UKSciFi's Aminet mirror back MethodGit Amiga websites reviews 4 21 July 2009 13:25
Mirror's Edge 2D Shoonay Nostalgia & memories 10 14 March 2009 17:33
BBoAH Mirror online novamann News 13 12 August 2008 00:31
Aminet US Mirror Back Soon Paul News 3 07 November 2004 15:23

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +2. The time now is 01:32.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2018, vBulletin Solutions Inc.
Page generated in 0.13218 seconds with 15 queries