HELP!!! Grapevine Diskmag Articles: Volunteers Wanted

[h0ffman]

System.IO.FileNotFoundException: Could not find file "/home/[my-user-name]/Documents/programming/GrapevineArticles#1-"

This is the error. For some reason the code can't find the file. I would suggest renaming that file because it appears that last character might be causing the issue here. It doesn't appear on the console output. In fact I think they used some weird characters for the hypens on the Amiga side.

[Dunny]

is "GrapevineArticles#1-" the actual name of the datafile you're trying to extract?

[Geordie-Jedi]

@Dunny

The name of the file is -

"--GrapevineArticles#1--" = On an actual Amiga / Amiga emulator

"GrapevineArticle#1-" = On my Linux box
Linux seems to truncate or strip away some of the hyphens in the name of the file.


@H0ffman

I originally got the Grapevine files from PiMiga 1.5 MF edition
(IIRC comes from a TOSEC archive).

I copied the GV file over to my SSD, and it's currently copied to a few different locations, including -

A couple of different dirs on my HD
Actually on an Amiga emulated HDD within FS-UAE
A copy in my programming dir

It appears to be a hard link - see below

[user-name]@computer-name:~/Documents/programming$ ls -l /home/[user-name]/Documents/programming/
total 588
-rw-r--r-- 1 [user-name] 580262 Sep 13  1990 GrapevineArticles#1-
-rwxrwxr-x 1 [user-name] 4253 Nov 14 09:25 Program.cs
-rwxrwxr-x 1 [user-name] 6656 Nov 21 23:27 Program.exe
-rw-rw-r-- 1 [user-name] 1236 Nov 20 21:58 Program.zip

So unless Linux is doing something very strange with hard and soft links
I'm a bit confused.

I did also try renaming the file to -
GrapevineArticles#1 (no hypen at the end)

Re-ran the program, and got the same error message

[Geordie-Jedi]

SUCCESS !!

I decided to (perma)-delete the file GrapevineArticles#1-

I got a brand new copy of the file, copied it to the programming dir,
then renamed it to "GrapevineArticles1"

Then I re-ran your script/app and it worked flawlessly !

I was then able to copy the 80 extracted files into a dir on my emulated Amiga.
Load Powerpacker, de-cruch the files and view them in ASCII.
After that I was able to save them out as ASCII .txt files

This has worked amazing well

Thank you to EVERYONE who has responded, with help and advice.
Esp to H0ffman for his cracking work on this little project.

Thank you - You's are all Legends !

[h0ffman]

Glad it's working. Happy ripping.

[Dunny]

Awesome but how are you gonna play a bad rendition of Pink Floyd songs in a PDF while we read?

[Geordie-Jedi]

@Dunny

Ha ha ha - ALOL

1. I'll buy some C90 cassette tapes - record their songs by humming them in a very mediocre fashion.
Then send them to you.

You can then post the cassettes onto the next person who wants to read the articles / PDFs.
Let's bring back mail trading lads !

2. Also - How VERY DARE you !
The Grapevine choons were incredible.

[Geordie-Jedi]

Reet - I've made a bit of progress......then hit a wall.

So far I have been able to -

1. Use H0ffmans script to extract the individual articles from the Amiga Grapevine article binary's.
(from about x3 issues of GV)

2. Use another Linux bash script to take the 1st level extracted files
and convert them to ASCII text.

for i in *.pp20 ; do ancient decompress "$i" "${i%%.pp20}.txt" ; done

Massive Thanks to Holger from the Ubuntu forums for that little belta.


I was pootling along nicely, and have done the two stage extraction for issues -
10-13 (about x3 disks for each issue, approx).

However as I attempted GV issue #14, things fell apart.
It appears that the LSD crew have password protected or encrypted the article files.

In order to try and unpack / access the files, so far I have tried -

10. Using Powerpacker on an emulated Amiga = Crashed with a Guru meditation.

11. Using DirOpus on emulated Amiga = Crashed with a Guru mediation.

12. Installed ppcrack on my Linux box, then running ppcrack = Fails with the following error code:

Decrypting with keys from 00000000 to FFFFFFFF.
Editorial.pp20: PowerPacker data file (PP20)
Editorial.pp20: failed to decrunch

(Editorial.pp20 was the name of the file I was using a test to unencrypt).


So does anyone have any ideas on how to proceed ?

[h0ffman]

Ohhh.. interesting.. when I get a moment ill check out that issue.

[Exodous]

After the article titles in the file, GV14 to 21 store the content of the articles in the files in blocks with the following format:

8 bytes: "GARTTEXT"
4 bytes: long for length of compressed article
length bytes: PowerPacker compressed article with header "PPAK" - just change this to "PP20" and it's a standard  PowerPacker file.

The basic pseudo code is:

load the file
repeat until end of file
    search for the next instance of GARTTEXT
    get the next long for the length
    change "PPAK" to "PP20"
    save from the start of "PP20" for length bytes

[h0ffman]

Attached is the C# code for extracting articles from issues 14 and above.

[Geordie-Jedi]

That worked perfectly.
Now we're cooking with gas !!!

@Exodous - Thanks for taking a look and for your insight.

@H0ffman - Thank you very much for providing V2 of your app/script.

I was able to re-compile it for Linux and it has worked flawlessly.
I've managed to extract all of the individual articles, and then convert them to ASCII text.

Now's all I need to do, is strip out the latent formatting codes from the ASCII text, and I should be done.


I know that I probably sound like a stuck record but............

A MASSIVE Thank you to you all !

[Exodous]

sed -e "s/{[0-9A-Z]//g" file >file-cleaned.txt

Does a pretty good job

[Geordie-Jedi]

@Exodus

Thanks - That bash script worked brilliantly on a single .txt file I used for testing.

Then I decided to try and get clever, and tried a wildcard to apply it to all of the text files
in the test dir..........BIG mistake

Here's what I tried

sed -e "s/{[0-9A-Z]//g" *.txt >file-cleaned.txt

I went from -
150 files = 1.9 Mb
150 files = 13. Gb !!

Looks like I've accidentally tar-bombed myself - DOH !

Is there an easy way to amend your bash script, to scan a number of text files in a dir,
to clean them all, without cooking my CPU and filling my SSD entirely ?

I have done a canny bit of googling and found some possibility's
but I don't want to make another pigs ear of things.

[Geordie-Jedi]

OK, I've managed to find a workable bash script

Based upon Exodus's version, but only seems to affect the dir I'm using for testing.

Here it is:

sed -i 's/{[0-9A-Z]//g' *.txt

Exodus's version worked brilliantly -

1. For x1 one file at a time, (to begin with).

2. I'm sure there's an easy way to make it work across an entire Dir, and even sub-dirs.

3. Kept the original .txt file, and created a separate (cleaned version) in case of any issues.


However I'll just use my bodged bash script on a dir by dir basis -
Easier than Nuking my test dir !
Ha ha ha.

[Geordie-Jedi]

Reet, that's done !

(With the hard work of, and very talented people here on this forum)
I've been able to -

1. Extract the individual files into .pp20 (crunched) format. From each issue of
Grapevine issues 10-21

2. Convert these .pp20 files into plain ASCII text files.

3. Strip away the latent formatting codes from each text file.


So at the risk of sounding cheeky (and I'm genuinely NOT trying to be a nuisance !)

The final step is to name & number these articles according to DH (The OP) of this thread.
To a specific format, namely:

Gv_10-Article###.txt
(the '###' represents the number allocated to the actual article (3 digit number).

Now one of the original guys in this thread - (StoneOakvalley) has created a list of almost all issues
of Grapevine (#05-21)

I have a text file of this complete listing

Sample - GV issue 10:

005: Grapevine subscriptions 005
006: Important 006
007: Notice to Public Domain libraries 007
008: The future of Grapevine Pazza of LSD 008
009: The Grapevine Charts Torch of LSD 009
010: The Grapevine Guru Not The Momma 010

Is there any way to tie the names of the actual articles / text files to this list
and produce a final renamed text file ?

E.G. GV Issue #10

The_Grapevine_Guru.txt = (The actual name of the ASCII text file)

010: The Grapevine Guru Not The Momma 010 = (The name of the article in Grapevine)

So this would become:

Gv_10-Article010.txt

As "Not the Momma" is the author of the article

However I would suggest an amendment (as follows):

GV-10_Article_010_The_Grapevine_Guru.txt

As there are approximate 3,790 individual text files that need to be renamed according to DH's
and Grapevine's schema.

Edit - Added the following files:
1. The text files of the Grapevine article names for each issue.
2. The complete set of GV issue 10's cleaned text file articles.


Questions:

Is there any way to:

1. Scan the dirs of the text files and the master list file for each issue.
2. Compare the names of the text files of the actual articles with the master list
3. Produce a final renamed text file for each article

4. Where should I post or upload the converted / cleaned text files
once this is done ?


Any thoughts or advice would be gratefully appreciated.

[h0ffman]

I had a quick look and I couldn't find where the master list of articles was. I guess it must be in the executable.

Best I could come up with was adding an article number to the filename. This isn't taken from any data, it's a sequential count inside the ripping code. Best bet would be to shove the resulting file names into a spread sheet and carve them up there.

I modified the code slightly and it now supports both versions of the file layout.

[Geordie-Jedi]

@H0ffman

Thanks again for your new script and graft
Much appreciated

[Geordie-Jedi]

Reet, looks like I'm done !

With thanks (again) to Holger from the Ubuntu forums who has created a fantastic
Linux bash script to -

1. Scan the dir against the text file with the names of each article from the
master list (that I obtained from StoneOakvalley's website)

2. Rename the files

I reckon that it produced about 90%+ accurate renaming rate.
This still left me with a few hundred files to rename manually, Which was a few
hours work.

The bash script is here:

#!/usr/bin/env bash
# script to rename Grapevine text files according to an index of the articles
# two parameters
# - the name of the index file
# - issue number
#
# GV-ren gv_issue_10_article_list.txt 10

for i in *txt ; do
    # remove extension
    name=${i%%.txt}
    # get the article number by searching the index file for the file name and split off the number at the beginning of the line
    anr=$(grep -i "$(echo $name|tr "_" " ")" $1)
    anr=${anr%%:*}
    # if we have a number, rename the file
    if [[ -n "$anr" ]] ; then
        mv "$i" "GV-${2}_Article_${anr}_${name}.txt"
    fi
    echo

I would say that this is about 95% perfect, there are a few remnants still left
However fixing that would take months to complete manually.

3. E.G. There were still (a very small amount of formatting codes left in)
However this was no fault of Exodus (his script was brilliant)
it was for something we didn't account for there were some codes like - {{j}
which may have been for fading text, or the like.

4. Also there are some slight variations in the naming of the articles
from Holger's script -

E.G.
Original filename = "Editorial"
New filename = "grapevine_issue_21_editorial.txt"

And some other's like that.

Being a bit of a perfectionist this annoys me, however Its near as can I get
without literally manually renaming every single file.
I think that's its close enough.

I've also tried to upload the extracted and renamed text files.
However the final .zip file size is 11.8 Mb, so won't let me.

I cannot currently upload this to the EAB fileserver (not enough posts yet)
Can one of the mods let me do this, or suggest something else please ?

Lastly I took the liberty to rename the files according to my suggestion =

GV-[issue No]_Article_###_Name_of_article.txt

So hopefully another Amiga project completed.
As I'm sure DH will really appreciate this, I know that I do

Thank you to everyone involved in this especially -
H0ffman
Exodus
andy2004
Holger

Cheers lads, you's are Legends !

[hitm4n]

Fantastic work from all. Nice.