SMBFS and UTF-8 Characters

[tygre]

Hi all!

On my Amiga, I use SMBFS to mount volumes that points to shared drives on my RPI, where I put all my MP3 and modules. It works well and, using the TRANSLATE option, I can read/write files back-and-forth, even with "typical" accented characters, like é or Ô, with: SMBFS USER=<username> PASSWORD=<password> DOMAIN=GIB SERVICE=//CHAMSAE/Music TRANSLATE=L:FileSystem_Trans/INTL.crossdos

Recently, I added MP3 files with names in Cyrillic or Hangul characters. These files are no problem for either the RPI itself or my Windows laptop because they both use UTF-8. But, these files to do not appear through SMBFS. I was expecting mangled file names, maybe like ????? ????.mp3, so I could have played them anyways , but they do not appear at all

Is there a way to "see" files with UTF-8 names using SMBFS?

Cheers!

[nogginthenog]

Confirmed! I'm running a self-built SMBFS from Olaf's github dated 20th August.
Montréal works OK

[tygre]

Hi Nogginthenog!

Yes, Montréal also works for me... But try ????? ? - ???????.mp3 (Cyrillic), it doesn't show at all for me

And, it just happens that EAB also doesn't support UTF-8! What appears above as ????? ? - ???????.mp3 should be the title of [ Show youtube player ]!

Cheers!

[Olaf Barthel]

Originally Posted by tygre

Hi all!

On my Amiga, I use SMBFS to mount volumes that points to shared drives on my RPI, where I put all my MP3 and modules. It works well and, using the TRANSLATE option, I can read/write files back-and-forth, even with "typical" accented characters, like é or Ô, with: SMBFS USER=<username> PASSWORD=<password> DOMAIN=GIB SERVICE=//CHAMSAE/Music TRANSLATE=L:FileSystem_Trans/INTL.crossdos

Recently, I added MP3 files with names in Cyrillic or Hangul characters. These files are no problem for either the RPI itself or my Windows laptop because they both use UTF-8. But, these files to do not appear through SMBFS. I was expecting mangled file names, maybe like ????? ????.mp3, so I could have played them anyways , but they do not appear at all

Is there a way to "see" files with UTF-8 names using SMBFS?

Cheers!

No, not yet. Unless you are forced to use an old Windows system or old Samba version ("old" meaning 20 years or older) the names of files and folders will be represented using 16 bit Unicode characters.

This is an option which smbfs (version 1.176 and beyond) can make good use of for the small portion of Unicode which maps exactly to the Amiga default character set (this being ISO 8859-1).

If this is all you need, then you don't even have to resort to the file name translation tables which use the original MS-DOS codepage-based scheme.

So, what about the remaining (roughly) 65,280 Unicode characters?

Because the Amiga cannot display these characters, smbfs will not attempt to return them during directory scanning and will not allow you to access them. The problem here is that these characters have no sound representation in the Amiga domain. Mapping them to UTF-8 sequences is tricky (what if you want to rename a file or directory?). Also, if you switch to UTF-8 then you would have to encode all characters except for those present in the US-ASCII 7 bit character set.

I have been pondering how to work around that problem since January this year and didn't make much progress beyond that, I'm afraid

The problem is solvable to some degree, but you'd still see the characters which don't fit the Amiga domain in an encoded form.

This will have its own drawbacks since you'd easily have to double or triple the length of the respective file or directory names. The upper limit for these names is 107 characters, and smbfs won't let you use names longer than that (they will not show up in directory lists and remain inaccessible).

[tygre]

Hi Olaf and thank you very much for your answer!

I see that that's a tricky problem... What about something like the MS-DOS/Windows "scheme": converting long file names into 8+3 chars? Let me explain

If I understand correctly, SMBFS knows that a file name contains chars beyond ISO-8859-1 and, wisely! , decides to hide it. Instead, could it convert the file name into a unique encoding of its own? For our beloved Amigas, SMBFS could keep the any ISO-8859-1 chars, up to 100 chars, and then append a unique number?

The unique number would distinguish two files that could have the same ISO-8859-1 chars by chance. (It could also be used internally by SMBFS to keep a correspondance between files "shown" on the Amiga side and the original files on the RPI although something more robust could be necessary.)

As with MS-DOS/Windows, this "scheme" would prevent renaming from the Amiga side but would allow accessing (yeah! ) and even moving.

Would that be possible? Could that create other problems?
Cheers!

[Olaf Barthel]

Originally Posted by tygre

Hi Olaf and thank you very much for your answer!

I see that that's a tricky problem... What about something like the MS-DOS/Windows "scheme": converting long file names into 8+3 chars? Let me explain

This is a well-known technique, but if I remember correctly, once you request that file names should be returned in Unicode form, you lose access to the 8.3 format which the SMB server helpfully provides along with the legacy directory records.

If I understand correctly, SMBFS knows that a file name contains chars beyond ISO-8859-1 and, wisely! , decides to hide it. Instead, could it convert the file name into a unique encoding of its own? For our beloved Amigas, SMBFS could keep the any ISO-8859-1 chars, up to 100 chars, and then append a unique number?

No, this is not a workable approach Files and folders are accessed through their complete path names, and if smbfs were to translate the path name components through a checksum scheme of its own making, then it would have to cache that information almost indefinitely. Also, it would have to cache how each checksummed path component relates to its parent.

The alternative is to rescan every directory that may contain encoded file or folder names upon access: the Samba server does that in order to allow for the 8.3 encoding to work but a client such as smbfs does not have this luxury. smbfs might have to ask the server over and over again for every directory that is part of a path.

The unique number would distinguish two files that could have the same ISO-8859-1 chars by chance. (It could also be used internally by SMBFS to keep a correspondance between files "shown" on the Amiga side and the original files on the RPI although something more robust could be necessary.)

As with MS-DOS/Windows, this "scheme" would prevent renaming from the Amiga side but would allow accessing (yeah! ) and even moving.

Would that be possible? Could that create other problems?
Cheers!

The only scheme which would work for smbfs is one which requires no caching of the replacement name (even listing the contents of a directory would require adding a replacement for a Unicode file name to the cache, in case it's needed later).

So this would have to be a 1:1 mapping, I'm afraid

This could work, but it would have to use an "escape character" (or a sequence of characters) which indicate that what follows it is Unicode data. I think this could work if the encoded Unicode data could be stored in a compact form. UTF-8 showed how this could be done In order to keep the ISO 8859-1 Amiga file/drawer names I could use an "escape sequence" of two characters, for example.

The drawback still is that the file/drawer name length is limited to 107 characters, and with each Unicode character becoming "escape sequence"+2 or 3 encoded characters this may quickly exhaust the available space. And that's not even considering how full path names will work out. Many applications don't allow path names longer than 100-300 characters (and there are, of course, those which don't even check if the full path name fits into the buffer).

This remains a thorny problem Question is which trade-offs are acceptable. For example, how much memory may smbfs commit to lookup tables, or how often it may rescan directories.

[tygre]

Hi Olaf!

The only scheme which would work for smbfs is one which requires no caching

I haven't read SMBFS source code so I'm certainly missing something here, sorry , but could you explain why caching would be a problem for SMBFS? (Except for memory consumption )

but a client such as smbfs does not have this luxury. smbfs might have to ask the server over and over again for every directory that is part of a path.

Would that be so bad? (Except for speed and network traffic )

once you request that file names should be returned in Unicode form, you lose access to the 8.3 format

Could you please maybe explain me (or point me in the source code) why the 8+3 format couldn't be made "compatible" with Unicode. My reasoning is: if SMB/SMBFS can match, for examples:

TextFile.Mine.txt <-> TEXTFI~1.TXT

ver +1.2.text     <-> VER_12~1.TEX

.bashrc.swp       <-> BASHRC~1.SWP

Then, couldn't it also match:

VilleDeMontréal.txt     <-> VilleDeMontréal.txt

XXXX - XXXXXX XXXXX.mp3 <-> ???? - ?????? ?????~1.mp3

XXX - XX.mp3            <-> ??? - ??~2.mp3

Where X is some Unicode code point and ? is just some ISO-8859-1 character chosen to replace any Unicode "code point" outside of the ISO-8859-1 chars. The ~ also could be different, maybe using \ to show that these files are for Amigas "consumption" only... Ironically, this is similar to what EAB does: when I copied/pasted file names with Cyrillic and Hangeul characters and saved my post, EAB replaced every "code point" (Cyrillic chars, Hangeul syllables) with "?" Wouldn't that work?

Cheers!

[tygre]

PS. I saw in proc.c the function

static int copy_utf16le_to_latin1(byte * to,int to_size,const byte * from,int len)

which is used in few places, into ifs like:

if(server->unicode_enabled)

{

	copy_utf16le_to_latin1(finfo->complete_path, finfo->complete_path_size, name, name_len);

}

Could this function help?

I had tried setting dos charset = UTF-8 but maybe I made a mistake and should try with a different combination of other parameters?

[tygre]

Hi all!

I'm happy to write that I have a proof-of-concept version of SMBFS that can handle files with non-Latin1 characters in their name (like the MP3 of this video: [ Show youtube player ])

It's rather "simplistic" right now: it replaces non-Latin1 names with a (unique) numerical name
but it could become smarter... Maybe using Jens' codesets.library?

I wonder if there is an interest (besides mine!) in improving this PoC?
Olaf, could I share with you my PoC, maybe via a pull request?

Cheers!

[tygre]

PS. Just sent a pull request to Olaf

[Olaf Barthel]

Originally Posted by tygre

PS. Just sent a pull request to Olaf

Thank you very much! I'm looking at integrating the changes right now. There's still some left-over stuff in my working copy of post 2.22 changes that need some scrutiny before I can proceed.

Hang on...

And the changes are committed Version 2.23 is now tagged and ready for tinkering.

[Olaf Barthel]

Originally Posted by tygre

Hi all!

I'm happy to write that I have a proof-of-concept version of SMBFS that can handle files with non-Latin1 characters in their name (like the MP3 of this video: [ Show youtube player ])

It's rather "simplistic" right now: it replaces non-Latin1 names with a (unique) numerical name
but it could become smarter... Maybe using Jens' codesets.library?

I wonder if there is an interest (besides mine!) in improving this PoC?
Olaf, could I share with you my PoC, maybe via a pull request?

Cheers!

A simple (not necessarily simplistic) solution is better than a clever solution which takes a long time to arrive

The best idea I had on how to achieve something similar would have involved encoding the Unicode characters in the drawer/file name. This would have required only small changes to smbfs, but it would have bumped against the name length limitations. With only 107 characters to work with and any encoding scheme taking up more than two characters to represent a 16 bit value, some names would never have fit. How do you show that directory entries have been omitted because of that? No idea The same problem already exists for file/drawer names longer than 107 characters.

Your solution does not have this problem by keeping the original name and its "alias" in memory. The extra memory spent will remain spent until smbfs shuts down, though. There's some room for improvement here, I'd say Also, your solution could be extended to file/drawer names longer than 107 characters which smbfs cannot currently represent. This looks like the way forward to me.

[tygre]

Originally Posted by Olaf Barthel

Thank you very much! I'm looking at integrating the changes right now. There's still some left-over stuff in my working copy of post 2.22 changes that need some scrutiny before I can proceed.

Hang on...

And the changes are committed Version 2.23 is now tagged and ready for tinkering.

Nice, thank you!

Cheers!

[tygre]

Hi Olaf!

Originally Posted by Olaf Barthel

A simple (not necessarily simplistic) solution is better than a clever solution which takes a long time to arrive

The best idea I had on how to achieve something similar would have involved encoding the Unicode characters in the drawer/file name. This would have required only small changes to smbfs, but it would have bumped against the name length limitations. With only 107 characters to work with and any encoding scheme taking up more than two characters to represent a 16 bit value, some names would never have fit. How do you show that directory entries have been omitted because of that? No idea The same problem already exists for file/drawer names longer than 107 characters.

Your solution does not have this problem by keeping the original name and its "alias" in memory. The extra memory spent will remain spent until smbfs shuts down, though. There's some room for improvement here, I'd say Also, your solution could be extended to file/drawer names longer than 107 characters which smbfs cannot currently represent. This looks like the way forward to me.

For the memory taken by the store, do you think that it could be freed between "uses" of SMBFS, like between a call to smb_proc_readdir_short() and one to smb_proc_open()?

Indeed the max. length of file names (and directory names) is really a hard constraint (in all sense of the term ). I actually limit the names to 31 characters because I met problems with the Ram Disk and some other programs... But these problems maybe came from my install?

Agreed on extending this solution for directory names!

Another thing I'd like to add is a real "transliteration" from Unicode to Latin1 but this seems complicated!

Cheers!

[n9yty]

I haven't seen any binaries for releases of smbfs for quite a while, and I don't have a build environment set up. Is there somewhere to download them from?

[tygre]

Hi n9yty!

No problem, I can share it with you... Where would be more convenient? The Zone maybe?

Let me know!