27 July 2019, 14:17 | #1 |
Registered User
Join Date: Dec 2008
Location: Ursviken
Posts: 137
|
Timestamps for files on FTP / Gdrive
Is it possible to add a timestamp to each of the files in filelist.txt on the FTP that gets mirrored over to the gdrive ?
An alternative to not disturb those who parses the filelist.txt (and are not prepared for a simple change), add another filelist with the timestamps in the second column (before size).. It's painfully slow to run the stat command on the remote files.. To be more clear, I would like the output from stat -c "%Y" to be added (%Y = time of last data modification, seconds since Epoch) [or even md5 checksums for the files, but that's painfully slow on large files even on the local drive, could be built around, storing the md5s in a database for each file and use the stored value when timestamp has not changed] Last edited by peo; 27 July 2019 at 14:34. |
28 July 2019, 20:12 | #2 |
Registered User
Join Date: Dec 2008
Location: Ursviken
Posts: 137
|
More as a proof of concept.. might be faster if coded directly as a shell script (I partially did that first, and that code is included too).
Written in PHP, this creates a filelist of the current directory (recursively) with the same format as EAB FTP filelist.txt but with the added fields for time (seconds since Epoch) and md5 (between date and size) I ran this for benchmarking on a smaller set of 179 files with the total size of 235MB. Benchmarking done on a DS1517+, with ordinary 3.5" spinning disks. Initial run (which checksums all the files) took 2.92 seconds. Running every second time took 0.12 seconds (output was redirected to a file). Note: 1. relies on some unix commands as "md5sum" (can be implemented in PHP using md5(), but will be slower) 2. set the "md5path" to a path outside the files to be checksummed, otherwise this would be included in the filelist and the md5-summing on the next run, doubling the files in the list (and so on - doubled every time). Preferrably the md5path should be set to an absolute location. 3. change the current directory to the one you want to create a file list from. Run on a smaller set of files the first times to check that it does what's expected (can be run on any directory) 4. there's no cleanup routine added to the code (could be implemented by moving the md5path folder to another place, and then for each file in the current filelist.txt scan and move back the md5 files into place). Or simply just re-checksum all the 2.5TB and nearly a million of files every year or so. Now.. only for someone to implement it on the filelist at EAB.... Last edited by peo; 28 July 2019 at 20:25. Reason: note about cleanup added |
Currently Active Users Viewing This Thread: 1 (0 members and 1 guests) | |
Thread Tools | |
Similar Threads | ||||
Thread | Thread Starter | Forum | Replies | Last Post |
How many timestamps does AmigaOS use? | Foebane | support.Apps | 3 | 18 March 2019 10:48 |
Password protected files on EAB FTP | AeroMaxx | project.Amiga File Server | 1 | 08 December 2017 07:57 |
File timestamps lose precision in shared folders | mark_k | support.WinUAE | 9 | 06 February 2013 16:25 |
FTP Access to Files | Leandro Jardim | project.Amiga File Server | 7 | 07 December 2010 13:57 |
New FTP-mirrorsite: ftp.fi.amigascne.org | Sallinen | Amiga scene | 1 | 07 May 2007 16:29 |
|
|