Amiga File Server

[DamienD]

Originally Posted by cmsj

I am hoping to get a local copy of the Commodore_Amiga tree, of which I have 781GB so far, so I think I'm pretty close to done.

That's insane; no wonder the EAB file server is slow at times and getting hammered

[Turran]

Originally Posted by DamienD

That's insane; no wonder the EAB file server is slow at times and getting hammered

My thoughts exactly. When I accepted to host it, I figured it would be an .adf here, a .zip there but its constantly at full capacity.

I've been thinking of how to get around it. One way would be to limit the ftp (anonymous) account to a very low KB/Sec rate and force people to register accounts with full speed but a few gigs a day limit, but I don't know.

[Turran]

Originally Posted by tomse

TOSEC ISO updates uploaded to uploads/tomse, can someone please move them to the correct place?

EDIT: it seems there were errors uploading, so not all were sent..
Commodore Amiga is on it's wait.. so please wait until tomorrow to move it.

Moved to Commodore_Amiga\TOSEC Updates\TOSEC Release (v2016-07-02) updates. Thanks.

[dirkies]

Originally Posted by Turran

My thoughts exactly. When I accepted to host it, I figured it would be an .adf here, a .zip there but its constantly at full capacity.

I've been thinking of how to get around it. One way would be to limit the ftp (anonymous) account to a very low KB/Sec rate and force people to register accounts with full speed but a few gigs a day limit, but I don't know.

Daily limit for an ip address sounds reasonable. After all it's not a cyberlocker!

[prowler]

Originally Posted by dirkies

Daily limit for an ip address sounds reasonable. After all it's not a cyberlocker!

That sounds like the best solution to give all genuine users a fair chance.

[cmsj]

Originally Posted by DamienD

That's insane; no wonder the EAB file server is slow at times and getting hammered

Ok, so clearly I'm already violating my stated aim of not annoying you guys.

I'd be very happy to donate to either/both of the file server hosts. I'd also be happy to not use an anonymous login and be rate/transfer limited.

The good news here is that my initial sync is now finished. I would like to keep it up to date, but I will not do that unless we can agree a frequency for me to do that, which won't make you angry (monthly?)

I want to make sure I have access to this data for the remaining (hopefully) decades of my life. I don't mean to imply that you guys can't keep the servers up that long, but since a 1TB disk is pretty affordable these days, it seems sensible to me, to keep a copy in case you both die before me

Perhaps there is some scope here for doing this all differently? Some kind of p2p network where it's easier for people to contribute disk/bandwidth to the whole body of users? A subscription-only rsync server with sensible policies?

I'm sorry for the inconvenience. Please let me know if I can contribute to the hosting costs in some way.

[modrobert]

How about ratio on bytes? That should take care of the leechers.

Jokes aside, I know some tricks which prevents directly hotlinking files on FTP (or HTTP) servers, in effect making direct file links via URL posted in forums/blogs/whatever time out after a while, not sure if that's a problem in this case, but hotlinking looks possible after a quick test using the results in the PHP search interface. Turran, if you want help with that, send me a pm.

[cmsj]

Originally Posted by modrobert

How about ratio on bytes? That should take care of the leechers.

Jokes aside, I know some tricks which prevents directly hotlinking files on FTP (or HTTP) servers, in effect making direct file links via URL posted in forums/blogs/whatever time out after a while, not sure if that's a problem in this case, but hotlinking looks possible after a quick test using the results in the PHP search interface. Turran, if you want help with that, send me a pm.

IMHO, expiring hotlinks would be awful, there are already enough posts on the various Amiga forums that contain dead pictures, or links to websites that have long since disappeared.

I continue to think that the EABFS has a place on archive.org, at least for archival and hotlinking purposes.

[Turran]

Originally Posted by dirkies

Daily limit for an ip address sounds reasonable. After all it's not a cyberlocker!

Hehe yeah. The easiest solution is what I can accomplish through the FTP server directly and that does not include daily limits per IP, but it does per user. Thats why I suggested a low speed limit for the anonymous account and a daily limit per registered user, with no speed limit.

But I'm going to leave it as is over the summer in either case. If we do it what way, people are going to get around it by creating multiple accounts etc. Perhaps its just best to leave it as is and get used to 250k/sec at most most days.

Originally Posted by cmsj

Ok, so clearly I'm already violating my stated aim of not annoying you guys.
The good news here is that my initial sync is now finished. I would like to keep it up to date, but I will not do that unless we can agree a frequency for me to do that, which won't make you angry (monthly?)
I'm sorry for the inconvenience. Please let me know if I can contribute to the hosting costs in some way.

You clearly stated your intentions of grabbing the whole FTP and I fully agree with it it. That/you is not the problem at all. Its the 10th of other connections doing the same thing, all of the time, that is. A bit exaggerated perhaps, but you get the idea =)
Keeping it up to date now will only be a minimal hit to the bandwidth. Feel free to do it as often as you like. If you can use the filelist.txt for it to only grab new files instead of a sitewide directory indexing, great. Otherwise I don't think that would be a problem anyway.

As for hosting, I'm in Sweden, so it does not cost me much. Donation are still not required although I appreciate the thought.

There are calm moments on the site as well where people can get good speeds. Now is one such time with only 6 people on it. However, this is what annoys me. Why leech at that speed for that long if you're not trying to get the whole FTP?

[DamienD]

Originally Posted by cmsj

Ok, so clearly I'm already violating my stated aim of not annoying you guys.

<snip>

The good news here is that my initial sync is now finished. I would like to keep it up to date, but I will not do that unless we can agree a frequency for me to do that, which won't make you angry (monthly?)

<snip>

Sorry for the mega rart everyone...

Firstly Chris, let me say that I'm not trying to pick a fight with you or cause offence. I'll just be honest with my opinion on this matter.

The EAB File Server has been going for many years now and is an invuluable resource for the community. I can't express how much we all really appreciate this. Thanks as always must go to people like Turran, DannyBoy and Magix for providing this hosting service free of charge with no gain to themselves; and not to mention various others for undertaking administration tasks etc. like TCD

A lot of long term EAB members / others have contributed loads of rare goodies over the years to get this resource as good as what it is today. Some examples off the top of my head:

... Belgarath; the all GameBase Amiga files.
... killergorilla & Retroplay; WHDLoad files.
... Myself; contributed many ISO files and also the latest TOSEC set.
... numerous others; obviously I won't list everyone.

__________________________________________________


The latest TOSEC 2016 set:

I've give you the background on this (though you can read more here: http://eab.abime.net/showthread.php?t=81787) as it helps get my perspective across. Previous to what you now see, there was an old, outdated set from 2012 hosted on the EAB File Server.

I had an old set from 2007 so went through the laborious process of updating to the 2016 set. Now, I could have I just thought to myself; "It's going to be a lot of hard work, hassle and time in order to get my 2007 updated to even the 2012 set let alone the 2016 set. I know, first of all I'll just leech the complete 2012 set, then grab all the update packs etc.". Of course I didn't take this easy road, what I did was as follows:

a) Download the few newer "packs" from the EAB File Server.
b) Combine my ancient 2007 set with the above packs.
c) Spend weeks running everything through ClrMamePro dats and generating "missing" lists.
d) Manually download any "missing" files one at a time (which was really a small amount in terms of data size).
e) Run through ClrMamePro dats again etc. until I had a complete 2016 set.

On to my next issue. I wanted to save people going through the same pain as what I'd just undertaken so thought; "How can I share this with my fellow EAB members?". Now again, I could have taken the easy road and just started uploading all 32GB to the server but I didn't want hog resources for the rest of the community (even though in this case I'm giving not taking). So, what I then did was as follows:

a) Contacted Turran, Magix, DannyBoy with an idea.
b) Purchase numerous 64GB USB sticks.
c) Copy the complete, updated TOSEC 2016 set over to these.
d) Post to the above people (who are all located outside of the UK) so that they can just copy all content over, hence saving bandwidth.

At the time I also sort of guessed that once this resource finally made it onto the EAB File Server *some* people may be greedy and just leech everything. To try to minimise this impact on the resource; I also offered to purchase more USB sticks and post to anyone on EAB. I think in the end I must have sent 10+ around to world to different EAB members.

__________________________________________________


So now we come to you... First of all you ask questions about backups, then go down the route of mentioning that you would like to help (which is commendable) just in case, all of a sudden, out-of-the-blue the resource mysteriously disappears. Turran informs you that there is no need as he and DannyBoy have everything under control... Even though you've been informed of the previous, I guess you next step is to start leeching everything... Don't you think that if Turran or DannyBoy did ever decide to pull the resource then they would somehow pass the baton onto the next kind soul (just like Magix did)?

Next, you haven't even been a member of EAB for even 6 months as yet and then casually announce in this thread you are currently leeching over 800GB of data for you personal / local copy plus would like to keep it "sync'd" every so often. I mean, I was concerened about uploading 32GB for others yet you seem quite happy to leech this amount of data in a very short space of time. Also, don't you think that others (myself included) would love a personal copy of everything but there's no way I would ever do this... Imagine the strain on the resource if even 10% of EAB members decided like youself, that it's cool to leech 1TB of data?!? Personally, I think it's a quite selfish and disrespectful.

Anyway, I'll conclude there. At the end of the day it's Turran's / DannyBoy's decision on how to handle this. Personally in order to stop reckless leeching I would:

a) Allow only registered users access (which let's be honest, is a minimal task for people to register if they want access).
b) Cap all registered users to a 2GB limit daily (which is more than fair IMHO).

[dlfrsilver]

My very personal opinion is that the FTP is our ressource, and should be only opened for eab members. It's purpose is not to be an 'all multi download plateform'.

It's our 'toys chest' in a way

[Turran]

I will have further discussions with DannyBoy in a week or two and we'll come up with a better solution that will make everyone happy I hope.

Personally, it is not the bandwidth on the FTP that bothers me that much. If it was, I could just lower available bandwidth for the whole FTP. What bothers me is that it hurts other people getting shit speeds because of it.

Considering cmsj... The only difference between him and the other people doing it is that he was open about it, which is commendable. While I do not understand the reasons for grabbing such a large chunk of the FTP, I did say that he could do as he please. He should not get more shit about it then the others. Only problem is that we do not know who they are (part from their IP. hehe).

But we'll solve it in a short while. No biggie. No need to make it into a huge problem, cause its not =)

Or as my girlfriend always says when I whine; "The problem is not the problem. Your attitude towards the problem is the problem."

[modrobert]

Originally Posted by cmsj

IMHO, expiring hotlinks would be awful, there are already enough posts on the various Amiga forums that contain dead pictures, or links to websites that have long since disappeared.

The idea is that only people who actually login/access the FTP server (or the EAB PHP search index) directly can download, not people visiting other websites where the files are linked without any kind of permission.

For example, after a quick search and I found this site indexing the EAB file server mirror (eab.t28.net):

http://filemare.com/en-th/browse/80..../files@@@60/12

Filemare is an ftp search service which also have a premium option to pay for ad free services and unlimited downloads, and this is a site which doesn't even host the files on their own servers, just hotlinking the files (the EAB mirror in this case).

Do you think this is fair? Other websites potentially making money from hotlinking the EAB file servers.

[DamienD]

I've seen something similar before too modrobert where random Google searches for Amiga games listed <gamename> with the EAB File Server links

Then there's also other forums that post the main EAB login details:

... http://saku.bbs.fi/discus/messages/2...tml?1374724774
... http://www.amiga-dresden.de/index.ph...d&threadID=261
... http://www.commodore.gen.tr/forum/in...ic=7751.0;wap2

[cmsj]

Hey

Originally Posted by modrobert

Do you think this is fair? Other websites potentially making money from hotlinking the EAB file servers.

Well, there are different ways to look at this, right? Google makes billions from hotlinking to all the free content on the web, but we tolerate it because the value that they add, by being able to search for stuff, is really high.

EABFS does have a web search interface, so Filemare is adding fairly limited utility, except that it's exposing the full tree as web pages, so it's indexable by search engines, which the grandis.nu search isn't (AFAICS).

Perhaps I have made a fundamental mistake by thinking of EABFS as a public resource - a "as definitive as we can make it" repository of all the abandoned Amiga software, for the benefit of all remaining/future Amiga users. If it is thought of by its maintainers as a "toy box" for the EAB club, then yes, I would have to agree that the hotlinking is bad.

[tomse]

@Turran thanks for moving the ISO's :-)


@other topic:
removing bandwidth usage, you might want to sort out peoples upload dirs :-)

[Turran]

Yeah, its a mess in there. I tried doing it the drastic way a while back but that got shot down pretty fast. I have no idea where most of the stuff should be. We have no archive role on the FTP sadly. Don't know who'd want to do it =)

[DamienD]

Originally Posted by Turran

Yeah, its a mess in there. I tried doing it the drastic way a while back but that got shot down pretty fast. I have no idea where most of the stuff should be. We have no archive role on the FTP sadly. Don't know who'd want to do it =)

I can try if you like Turran (and then I can get shot down in flames)?

[Turran]

Access granted =)

Avoid renaming large folders if not necessary as it will need to be replicated to the mirror. Replication is not that smart that it understands renaming so it will delete / re-upload.

[DamienD]

Ok, to be honest I haven't even looked under each user's directory as yet... I can imagine it's a mess; well mine isn't

Definitely need a plan before even attempting anything. I think I'll do something like:

a) Check the user's directory and see what's there.
b) Think about where different files / folders can be moved to.
c) Send a PM to the user detailing what changes I believe are best.
d) After hearing back from the user, agree on the desired changes to be made.
e) Make the agreed changes.

Does that sound good to all?