29 September 2023, 06:55 | #1 |
Registered User
Join Date: Sep 2019
Location: Finland
Posts: 378
|
Use for PIPE: HippoPlayer and aget
Use for PIPE: HippoPlayer and aget
Text by me & @patrik, early 2023. I thought I'd write something about this since it's the first time I've used the queue-handler provided PIPE: for anything and it was kind of an interesting AmigaOS learning experience. In fact I didn't know what the whole PIPE business was, even though I remember noticing the file "queue-handler" in the directory "l" back in the days. So, in the shell you can do: echo "hello" >PIPE: ...and then get it back with: type PIPE: Whee! Or even better, you can name your pipe: echo "hi again" >PIPE:SuperPipeline If the pipe is empty, trying to read it will block until data is available. Googling about pipe usage results in some disconcerting information, such as: "Up to OS 3.1, the queue-handler has some bugs which don't allow a really foolproof transmission of data.". There are also some alternative implementations which purportedly fix bugs and generally work much better. During this experiment kickstart 3.0 and 3.2 were used for testing, no 3rd party queue-handlers were used. HippoPlayer uses DOS file handles for reading data (obviously), so replacing the handle with something that doesn't point to the local filesystem would probably work just fine, right? Let's grab the fine UHCTools package and the aget tool, which is able to do HTTP-transfers wonderfully. Then, redirect aget output into a named pipe and on the other end, make Hippo read it. We get an interprocess communication network data streamer thingy without having to implement a HTTP stack or internet connectivity, since the UHCTeam already did all of that. What could possibly go wrong? After a not insignificant amount of programming, testing and fine tuning, eventually it started to work. There were some gotchas on the way. Here HippoPlayer is the pipe consumer, and aget the pipe producer. Hippo passes a url to an mpeg audio stream to aget and asks aget to write the data into a named pipe. Hippo will then start consuming the data from the pipe and feed it to mpega.library for decoding, and after that to Paula or AHI. Radio streams work fine, as well as MP3 files without needing to download them first. Nice! When the pipe runs out of data, ie. all data has been transferred, the consumer end will see an end-of-file, similarly as with local files. If the consumer wants to stop reading the pipe, that is, the user presses the stop button, the process is more complicated. First, CTRL+C needs to be sent to the producer, aget in this case. Then the pipe must be flushed by the consumer. Aget will be able to complete the current write because the pipe is now empty and then notice the incoming break signal, and exit cleanly. Stream playback completed. It’s possible to configure the pipe buffers. We tried a lot of buffer combinations, different sized pipe buffers and aget buffers. Some radio stations were difficult as they provided data in throttled manner, the bytes came barely fast enough to sustain real time playback. In these cases it also helped to just wait a few seconds before starting to consume the pipe data to allow some data to be buffered. Hippo also had a small rounding error in the MP3 downsampling part which contributed to data being consumed ever so slightly too fast. Another thing to take into account is that pipe file handles can only be read sequentially, so the seek operation won’t work. We did not notice any problems with the pipe usage in terms of data loss or reliability. It’s notable that there are differences in how the pipe device works in different kickstart versions, details below. CTRL-C Example under 3.2.1 where the CTRL-C behaviour can be observed: Code:
aget shell - starting aget: 7.Ram Disk:> aget se.aminet.net/INDEX PIPE:agetout Buffer size: 128kB Connected to se.aminet.net:80 Sent GET request! Server response: 200 OK Receiving file: 0% - [ ] - 100% (6.80MB) Code:
8.Ram Disk:> uhc:c/typeline PIPE:agetout 1 | Aminet index, created on 27-Sep-2023 8.Ram Disk:> Code:
7.Ram Disk:> aget se.aminet.net/INDEX PIPE:agetout Buffer size: 128kB Connected to se.aminet.net:80 Sent GET request! Server response: 200 OK Receiving file: 0% - [ ] - 100% (6.80MB) Ctrl-C signal caught, ending transfer! 160.79kB received in 13.46s - 11.94kB/s 7.Ram Disk:> Aget shell - starting aget: Code:
11.Ram Disk:> aget se.aminet.net/INDEX PIPE:agetout Buffer size: 128kB Connected to se.aminet.net:80 Sent GET request! Server response: 200 OK Receiving file: 0% - [ ] - 100% (6.80MB) Code:
10.Ram Disk:> uhc:c/typeline PIPE:agetout 1 | Aminet index, created on 27-Sep-2023 10.Ram Disk:> break `status COMMAND=aget` 10.Ram Disk:> type PIPE:agetout >NIL: 10.Ram Disk:> Code:
11.Ram Disk:> aget se.aminet.net/INDEX PIPE:agetout Buffer size: 128kB Connected to se.aminet.net:80 Sent GET request! Server response: 200 OK Receiving file: 0% - [ ] - 100% (6.80MB) Ctrl-C signal caught, ending transfer! 160.79kB received in 294.07s - 559B/s 11.Ram Disk:> |
29 September 2023, 07:14 | #2 |
Registered User
Join Date: Jan 2019
Location: Germany
Posts: 3,492
|
The Ctrl-C behaivor was added in 3.2 to address the case of the reading end of the pipe terminating, thus leaving the writing end with a "broken pipe" left. Unixoid systems have their own signal for this case, terminating the writer. AmigaOs does not, so it instead sends a ^C to the writer and ignores all further incoming write requests.
This is particularly useful if you use pipes along with "more" as a pager: list all | more will abort the "list" if "more" is terminated to list the entire file system. Note that you can also access reading and writing end of the pipe directly by name in a single line: copy foo to PIPE: | type PIPE: hex This "PIPE:" identifies (left) the writing and (right) the reading end of the pipe that is implicitly constructed by the shell for you. |
29 September 2023, 19:07 | #3 |
Returning fan!
Join Date: Jan 2011
Location: Montréal, QC, Canada
Posts: 1,456
|
Hi Koobo and Thomas!
Very interesting, thanks for sharing! |
29 September 2023, 21:13 | #4 |
Registered User
Join Date: Oct 2021
Location: England
Posts: 1,330
|
great functions added to both aget and HippoPlayer, being able to stream audio direct from the net is 1 of the best most recent features, previously i used AmigaAmp to listen to the retro goodness on Slay Radio, now Hippo can do the same by using your "Radio Stations" option and entering the information contained in 1 of Slay Radio's .M3U files, i use http://relay1.slayradio.org:8000 and it plays just fine.
Perhaps that could be automated by getting Hippo to just outright read those address(s) from an M3U file when we add a NEW mod/file. Last edited by DisasterIncarna; 29 September 2023 at 21:21. |
30 September 2023, 10:50 | #5 |
Registered User
Join Date: Jan 2005
Location: Umeå
Age: 44
Posts: 962
|
@DisasterIncarna:
Why not just search for slay radio in the radio stations search? Or is the question two questions: - Can hippo be made to to add urls to play to the playlist manually, without searching for them? - Can hippo be made to play .m3u files? Last edited by patrik; 30 September 2023 at 10:59. |
30 September 2023, 23:12 | #6 |
Registered User
Join Date: Oct 2021
Location: England
Posts: 1,330
|
yeah probably thinking the load/play m3u option as i typically have a saved list of fave stations including slay/digitally imported fm/etc which is why i rarely use or overlooked the radio station search, which yeah also works fine as it defaults to the 128k stream, no idea how you pick the 56k or 24k streams without doing it manually tho. Just accustomed to using my saved file list is all either way its 1 of the best new additions i use all the time.
Now that i think about it, maybe i could just slap a load of m3u links into a custom playlist and load that instead? Last edited by DisasterIncarna; 30 September 2023 at 23:20. |
01 October 2023, 12:03 | #7 |
Registered User
Join Date: Jan 2005
Location: Umeå
Age: 44
Posts: 962
|
@DisasterIncarna:
Continuing here asking things that are not obvious to me - Where/how do you download your .m3u file collection? - What is the actual use for the 56k or 24k streams? |
02 October 2023, 00:27 | #8 |
Registered User
Join Date: Oct 2021
Location: England
Posts: 1,330
|
the only use for different bitrates afaik is just if your using something that cant manage large internet speeds, emulated amiga and im all in on biggest bitrates, using real hardware or some fpga with a slow net capability then im using much, much lower bitrates.
Ive also defaulted to the lower bitrate M3U streams when the better/biggest ones go offline/die/arent available. Saving an M3U was/is as easy as just rightmousing a stream on a website and saving the m3u, for example: on https://www.slayradio.org/home.php#news looking top/left in the "Listen to slay radio" box, you can just rightmouse the High/Medium/Low links, and each of those M3U's has a list of relays/mirrors for each of the 128k/56k/24k bitstreams. ive done the same on other websites to get M3U's i need as i mainly used them on a radio station docker app on my NAS and since i had them i just defaulted to using them wherever i am able to. |
14 October 2023, 15:37 | #9 |
Registered User
Join Date: Sep 2019
Location: Finland
Posts: 378
|
The plumber brothers are back with another pipely adventure!
I got an idea and decided to push CD-quality wav-sample through PIPE:. It was really slow. I mentioned this to @patrik and this caused him to lose two nights of sleep, after which he decided to do some scientific measuring: PIPE performance Needed to know queue-handler isolated performance, measured on a super serious A3000 68030@25Mhz workstation. Code:
8.Ram Disk:> source ? DESTFILE/A,NUMWRITES/N,WRITESIZE/N: erfordrat argument saknas 8.Ram Disk:> sink ? SOURCEFILE/A,NUMREADS/N,READSIZE/N: erfordrat argument saknas Code:
3.2.1: 8.Ram Disk:> version full Kickstart 47.7, Workbench 47.3 (2021-07-13) 8.Ram Disk:> version full PIPE: queue-handler 45.2 (2017-10-15) 8.Ram Disk:> run sink PIPE:test [CLI 7] 8.Ram Disk:> uhc:c/time source PIPE:test 14.431280s Code:
3.1: 11.Ram Disk:> version full Kickstart 40.68, Workbench 40.42 (1994-02-18) 11.Ram Disk:> version full PIPE: Could not find version information for 'PIPE:' 11.Ram Disk:> version full L:queue-handler Version 37.12 (1991-03-01) 11.Ram Disk:> run sink PIPE:test [CLI 10] 11.Ram Disk:> uhc:c/time source PIPE:test 99.937655s so old queue-handler is approx 1/7 the speed of the new one End of scientific section A CD quality sample bandwidth is about 0.17MBytes/sec, so the old queue-handler can't handle it on the super serious workstation. Luckily the new one can also be used on older OS versions and kickstarts! Observations anomalous I was testing a pipeline where a WAV sample was produced into PIPE: and HippoPlayer was consuming it on the other end. I noticed that on my A1200/060 (OS 3.1) the WAV sample was intermittently destroyed and sounded like noise. The conditions for this happening were:
65536/2the problem went away, or altenatively removed Mungwall+Enforcer from the equation. These conditions seemed to unearth some of the bugs apparently hiding in the old queue-handler. With queue-handler 45.2 this effect could not be reproduced. |
14 October 2023, 15:51 | #10 |
Registered User
Join Date: Oct 2021
Location: England
Posts: 1,330
|
|
14 October 2023, 18:49 | #11 |
Registered User
Join Date: Jan 2019
Location: Germany
Posts: 3,492
|
The V37 and V45 Queue handlers work by entirely different principles, the v45 is completely new. The V37 creates a new process for every PIPE: file you open, and then uses inter-process communication with PutMsg() to send the data from the writing PIPE:-process to the reading PIPE:Process, so inter-process communication goes over four processes: The writer, the writing pipe, the reading pipe, and the reader.
The V45 queue handler only creates a single instance of the PIPE that keeps lists of waiting and ready packets, and directly copies from the writing end to the reading target buffer, avoiding one stage of inter-process communication. That certainly helps to speed things up, but it primarily simplified the design. The drawback is that you have a sequential bottleneck in the middle, i.e. if multiple processes want to talk to each other over the PIPE:, then a single PIPE process has to manage copying all data around. However, we only have a single CPU core on Amiga anyhow, so it is probably not quite that bad. |
14 October 2023, 21:03 | #12 |
Registered User
Join Date: Jan 2005
Location: Umeå
Age: 44
Posts: 962
|
@Thomas: Very interesting with the details about how the original queue-handler was implemented vs how the current is!
If anyone is interested in reproducing this PIPE: benchmark themselves, these are the components used: http://megaburken.net/~patrik/SourceSink.lha http://uhc.a1k.org/time |
17 October 2023, 19:49 | #13 |
Registered User
Join Date: Sep 2019
Location: Finland
Posts: 378
|
Thanks @Thomas, interesting
I performed another measurement on the miggy (A1200/060, OS 3.1) with the following HippoPlayer-pipeline:
For a three minute run the average CPU usage:
Here about 170 kB/s of data goes through the pipe. |
18 October 2023, 04:51 | #14 |
Registered User
Join Date: Oct 2021
Location: England
Posts: 1,330
|
wonder if someone can abuse the same setup to make an online animation streaming program, even if it was just for tiny iff animations or something.
|
Currently Active Users Viewing This Thread: 1 (0 members and 1 guests) | |
Thread Tools | |
Similar Threads | ||||
Thread | Thread Starter | Forum | Replies | Last Post |
PIPE: already mounted | spawnterror | support.AmigaOS | 21 | 15 June 2024 14:07 |
Pipe Mania (Empire) | Avanze | MarketPlace | 0 | 11 April 2016 21:39 |
\\.\pipe\WinUAE | boir | support.WinUAE | 17 | 23 October 2015 08:57 |
Amiga, vfork(), and pipe() | tygre | Coders. General | 5 | 03 December 2011 01:35 |
Pipe Dream (original) | MethodGit | request.Old Rare Games | 3 | 16 December 2010 22:17 |
|
|