English Amiga Board


Go Back   English Amiga Board > Other Projects > project.EAB File Server

 
 
Thread Tools
Old 27 July 2019, 14:17   #1
peo
Registered User
 
Join Date: Dec 2008
Location: Ursviken
Posts: 70
Timestamps for files on FTP / Gdrive

Is it possible to add a timestamp to each of the files in filelist.txt on the FTP that gets mirrored over to the gdrive ?

An alternative to not disturb those who parses the filelist.txt (and are not prepared for a simple change), add another filelist with the timestamps in the second column (before size)..

It's painfully slow to run the stat command on the remote files..

To be more clear, I would like the output from stat -c "%Y" to be added (%Y = time of last data modification, seconds since Epoch)

[or even md5 checksums for the files, but that's painfully slow on large files even on the local drive, could be built around, storing the md5s in a database for each file and use the stored value when timestamp has not changed]

Last edited by peo; 27 July 2019 at 14:34.
peo is offline  
Old 28 July 2019, 20:12   #2
peo
Registered User
 
Join Date: Dec 2008
Location: Ursviken
Posts: 70
More as a proof of concept.. might be faster if coded directly as a shell script (I partially did that first, and that code is included too).

Written in PHP, this creates a filelist of the current directory (recursively) with the same format as EAB FTP filelist.txt but with the added fields for time (seconds since Epoch) and md5 (between date and size)

I ran this for benchmarking on a smaller set of 179 files with the total size of 235MB. Benchmarking done on a DS1517+, with ordinary 3.5" spinning disks.
Initial run (which checksums all the files) took 2.92 seconds. Running every second time took 0.12 seconds (output was redirected to a file).

Note:
1. relies on some unix commands as "md5sum" (can be implemented in PHP using md5(), but will be slower)
2. set the "md5path" to a path outside the files to be checksummed, otherwise this would be included in the filelist and the md5-summing on the next run, doubling the files in the list (and so on - doubled every time). Preferrably the md5path should be set to an absolute location.
3. change the current directory to the one you want to create a file list from. Run on a smaller set of files the first times to check that it does what's expected (can be run on any directory)
4. there's no cleanup routine added to the code (could be implemented by moving the md5path folder to another place, and then for each file in the current filelist.txt scan and move back the md5 files into place). Or simply just re-checksum all the 2.5TB and nearly a million of files every year or so.

Now.. only for someone to implement it on the filelist at EAB....
Attached Files
File Type: zip eab-build-filelist.zip (1.1 KB, 11 views)

Last edited by peo; 28 July 2019 at 20:25. Reason: note about cleanup added
peo is offline  
 


Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)
 
Thread Tools

Similar Threads
Thread Thread Starter Forum Replies Last Post
How many timestamps does AmigaOS use? Foebane support.Apps 3 18 March 2019 10:48
Password protected files on EAB FTP AeroMaxx project.EAB File Server 1 08 December 2017 07:57
File timestamps lose precision in shared folders mark_k support.WinUAE 9 06 February 2013 16:25
FTP Access to Files Leandro Jardim project.EAB File Server 7 07 December 2010 13:57
New FTP-mirrorsite: ftp.fi.amigascne.org Sallinen Amiga scene 1 07 May 2007 16:29

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +2. The time now is 17:00.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2019, vBulletin Solutions Inc.
Page generated in 0.07784 seconds with 17 queries