Live RTP audio playback

Ajouté : 2026-05-25

[00:00:00]Yo yo yo. What's up? What's up? What's up everybody? Welcome back. Today we are back on our phone client, our soft phone client, our VOIPE client, however you want to call it. Um, and the context here is, you know, I ate [ __ ] outside on a jog. My phone shatters, my teeth shatter. Not actually that just it sounded cool to say. Um, you know, my phone screen shatters. I can't use it anymore. I buy a new phone. Uh, it works for a week or two. Um, I let it die all the way. Pixel 7s, apparently some of them just like don't charge again after.

[00:00:35]So, my phone, second phone in a month, just like totally screwed. You know, that's like fine except except when somebody calls me or someone comes to my door and tries to like deliver a package or something, the the doorbell rings my cell phone. I'm like, that's freaking annoying. I want the doorbell to ring my door. And so we need to make a phone that can connect to the doorbell over the phone network because there's no wire that goes to my apartment from the front. Um, and so that's why, you know, there there are Linux SIP clients, they call them, VOIPE clients, whatever. Uh, but I don't like any of them. And of course, in classic Seraphoria fashion, we go and we say we'll make our own. And we've been working on this like kind of like on and off for a month. We've kind of been like allowing ourselves to indulge in rabbit holes. Um but what we have so far is we have a client that can initiate a phone call. So this is an existing an existing Linux client that runs in the terminal. I with my client can say hey I would like to phone this guy please. And when I do that he goes ring ring ring ring ring ring answer the phone and he says answer 200. The ringing is done. My client sees the invite complete and we see like the negotiated call parameters. Um but we are not doing any actual audio transmission or receiving yet. Um and so we are kind of working towards that right now. We've kind of done some of the pieces for this thing. Let's let's kind of draw this out a little bit. We have uh there is like SIP. This is the protocol used to initiate the trans like the call. This is like done enough so that we can initiate one but not enough so that we can do anything else. Right?

[00:02:17]This guy then is going to in the SIP pack packet we are going to send an SDP session description protocol I think is what it's called and this is going to tell us what audio streams are where you know what we've decided we agree on for like what formats we support etc etc. So this is kind of like a definition of the parameters of the call. And in here you we we can parse this right now. Well, we can parse like enough of this to play a stream from ffmpeg or something like that.

[00:02:50]So this is like we're going to say like squiggle done. Squiggle squiggle check mark for both of these.

[00:02:57]In here he defines some RTP streams.

[00:03:02]RTP.

[00:03:04]This is like essentially like UDP packets come on the wire with like a small header header and some encoded data. Now through both the SDP and through the RTP header, they tell us what type of data that is. We only support one type of data right now or we've like only ever looked at one type of data. This is G7 11 PCMU which is basically like uh how do we say this? They're like floats but 8 bits. 8 bit floats essentially. Um so we can parse these and we have written these out. We have like taken a sine wave from ffmpeg that's streaming over RTP and we've written it out to like test.csv.

[00:03:53]And in test.csv CSV we see something that like resembles a sine wave. So we're like good, right? Um so but but what we what we are doing right now is one this is just like in a little test application that's running like in main um everything in line but two we are completely ignoring like some important pieces of this. Oh I forgot the last piece of this which is audio. So audio output on my machine is held by lib pipewire. So we have also gotten enough of this to do sound output. Um so if we want to like attach all of this together, we have to somehow pipe the data that comes in here into this pipe wire output. Um and we need to make sure that we're doing it sly, right? So there since the stuff comes over UDP, we have no guarantee that like it will come in order. We have no guarantee that we're going to get it all. we have no guarantee on how long it's going to take for like the next packet to get to us. Um, so we kind of have to figure out like the goal for today is mostly to figure out like how do we schedule insertion of the stuff that we kept from RTP into the audio output in a way that like doesn't suck.

[00:05:15]Right? So, so, so the goals goals, let's say goals because I we have I haven't thought this through yet. Um I guess like uh modules for RTP RTP G711.

[00:05:31]We have one for audio already. Um and then we also want like pipe like respect timestamps I guess and like feed audio correctly.

[00:05:45]That's kind of kind of the goal.

[00:05:48]So I guess we have all these pieces. How do we want them to play together? I guess like somewhere I guess. Okay.

[00:05:57]Yeah. So the plan the plan for today is that we are going to figure out how we are going to design this thing and we're going to implement it, right? So I guess we are on to figuring out how this thing is going to look.

[00:06:07]Um in spheroria projects we have settled on an event loop. Event loop.

[00:06:17]We'll draw a little loopy. Wow. Um, this thing is kind of based on e-le usually and then we typically will like hook up different anything that like triggers a thing to happen will kind of like go go to the root of the event loop, right? So the I guess like packets coming in. I I guess like at some point we are going to get a SIP packet, right?

[00:06:40]A SIP packet message is going to say we have initially initiated a call, right?

[00:06:46]So, this guy is probably going to spawn some form of like RTP stream, I would guess.

[00:06:55]I would guess, right? And he is going to have his own like UDP listener dispatching thing. He's going to he's going to bind to some port probably where he is expecting the data to come in. Um, and so he is kind of like attached directly to this thing and then he probably talks to him or he like creates, right? he like creates one of these.

[00:07:18]Um, that seems reasonable. Then we also have like our audio service loop who lives over here.

[00:07:27]And I guess when he creates this guy, he is going to have a handle to like an audio stream, right?

[00:07:36]Audio stream who's like kind of like linked to this guy, right? In some way.

[00:07:43]this audio stream, the way it's set up right now is we kind of have like a circular buffer where we just kind of like inject floating point samples into this thing and the audio service when it goes, "Oh [ __ ] I need to send more data over to the audio server, he'll like look at what data is populated in this buffer and pull it off."

[00:08:03]So I guess the like the flow of data here is like this RTP stream gets some data. He like decodes with D7 G711 decodes and he somehow like pushes into the audio stream. Right now the problem the problem right now is that the audio stream interface that we have is like it only like assumes that you have like sequential data which I think is not true. Right? This stuff comes in with like timestamp like each each RTP packet has a time stamp and they might come out of order and you might miss some, right? So what do we do to like handle that? I guess like ideally ideally we use this buffer that we have for audio and maybe we have like our circular buffer, right? So we'll draw it like the audio service has read up until this point and we have told it it can read up till this point and I guess I guess we probably want to like as data comes in like fill in the appropriate parts of this buffer and when we have pieces of data that are adjacent to this bump it forward.

[00:09:28]But what if we drop this packet, right?

[00:09:30]We might drop this packet that has the data right here. So, so using that as the event to move things forward seems kind of wrong.

[00:09:38]What if we use time?

[00:09:40]Time is who advances this cursor maybe.

[00:09:44]So we kind of like fill in some amount of future data just with the like we know this cursor is at some time step probably T1 right this is at T1 and we know where all of the data that we are getting comes in should be relative to T1.

[00:10:05]So we just feed every piece of data in here. this data, this buffer is going to be like pretty [ __ ] big probably like you know maybe like 1 second of audio data can live in here, right?

[00:10:19]And so maybe what we do is whenever we get data, we drop it in wherever it belongs and on some timer we advance this part.

[00:10:32]We advance the cursor that tells the audio service that he's allowed to listen to this stuff.

[00:10:42]And we just kind of have to like set that time. There's probably some sort of like like movement of this time step like where the time in the RTP stream is relative to our input time like rel relative to our schedule where maybe like if if we're dropping like over 5% of packets because they came too late because like you know if if we got a time we might we might end up getting packets for time back there, but like we've already maybe like back here or something, right? But we've already scheduled that. So like that stuff's useless. So we might want to like adjust the time step such that or like the the time the reference time I guess we'll call it such that we get most of the audio data in time.

[00:11:39]Okay, that seems reasonable. Okay. Okay. So, what is the plan? It looks like it's something along the lines of uh audio stream needs to support arbitrary right positions, which it doesn't right now, but we can make it so that it does.

[00:12:04]Then RTP stream holds um a UDP socket who who gets he gets RTP data and injects into and yeah so I guess we'll say gets RTP data uh feeds feeds into audio buffer. He also has an audio stream handle who like owns audio buffer, I guess. And he also is going to have like a timer handle. And this guy's job is to uh like update reference time and like advance audio buffer cursor.

[00:13:11]I think that works. I think that all and I think that like in terms of implementation, all of these things kind of just like they're all independent and reusable in a way that's nice, right? If we ever decide to do something else with these pieces, like there's there's nothing that is like too tied together.

[00:13:34]Yeah, let's try that and see how it feels.

[00:13:39]Okay, so we have this RTP exploration thing who right now is just doing everything in line. I guess step one, I think step one is probably to uh event loop uh RTP test bed test area. Then we're probably going to implement like I guess like stub out initial like overall shape and then probably at that point let's kind of do like uh ignore time and feed audio right so like just feed audio packets as they come in then do proper time scheduling I think that kind of makes sense. I think that's like a flow that will work. So, let's do that. Let's do that.

[00:14:36]Um, okay. So, let's make ourselves a little loop.

[00:14:45]And he needs a way to be able to like chain uh events together. So, I think we'll just say if you know each thing that happens can chain a 100 things together. If we have a a Q depth of more than 100 things, then [ __ ] off.

[00:15:00]Hello. Is this a tutorial? [ __ ] no.

[00:15:04]This is me coding for fun, and whatever comes of that is whatever comes of it.

[00:15:10]Um, okay. So, I guess what we have to do is we have this like existing UDP socket thing that we've got here. So this thing is going to live as part of the like RTP stream abstraction that we were just talking about, right?

[00:15:28]UDP socket get. Thank you.

[00:15:33]Uh so this is a this guy.

[00:15:38]Um and this is going to be analogous to something that we're already doing here, right? We already say somewhere in main try to receive data from the socket. So I guess We're going to have some service function and I guess ideally this RTP stream concept like is he ideally he's not tied to the idea of G711 encoded data, right? We kind of want this guy to just return back like an RTP frame probably, right, to the caller and the caller gets to choose how they want to manage that, right? Because like every application who has an RTP stream or every consumer of an RTP stream may not implement all of the same things, right? Like we can imagine if we were to make some video conferencing software, we would like have H.264 264 support for video decoding and that would maybe want to reuse this thing.

[00:16:42]But uh does does 7-Eleven have good slurpies?

[00:16:46]So true, dude. We love slurpie day around here.

[00:16:50]Right. So ideally ideally the codec part is like completely decoupled from here.

[00:16:56]Uh which means no. I've changed my mind. We're going to say we're going to say that the RTP stream is a application specific concept, right? So we'll call this like maybe this is even like call, right? Or like yeah, we'll call this appsp specific RTP stream because I I think that makes more sense, right? So if somebody else wants to like like all of the pieces that are needed to build up the overall flow of data, we'll put somewhere.

[00:17:30]Um but the uh the actual piping of the data is like application specific. I think that's fine.

[00:17:43]We'll just call this RGB stream. I'll remember.

[00:17:49]Okay. So, we also said this guy's going to have some sort of audio handle. Do I have audio in here yet? I think I do.

[00:17:58]Add import of audio here. And I guess we'll kind of make one of those somewhere, too. Oopsies.

[00:18:13]So, we'll say we have our audio library that we've been working on.

[00:18:20]And he is going to have a handle like audio handle is going to be maybe one of these audio stream.

[00:18:37]Yeah, it looks like this is where this comes from.

[00:18:43]Okay.

[00:18:45]So he is going to go let me receive some data sir. Basically everything that's in this loop in main is moving into this service loop. So he's going to give me be like let me pull some [ __ ] from this socket. And what was buff here? We just had like a megabyte of data. I guess I guess we will figure out where that's going to live later. For now we will just say buff has to be enough for like one UDP message, right? So like surely like probably this is enough. I think MTU is usually 1600 or something around that.

[00:19:20]So this should be plenty big enough.

[00:19:23]Uh we have this frame that we parse from RTP. And then here this needs this is like the G711 decoding stuff. So this needs to live somewhere. So maybe let's say G711 and we'll say PCM decode PCMU is going to be a bite and he is going to return a I16 sample maybe um maybe he wants maybe we want this to be a uh floating point because our audio our audio world is floating point right beware I saw 12 kilob UDP datagramgrams in the wild Not fun. That's kind of sick. I didn't know you could do that. I kind of assumed that like my router would like not forward that, but that's kind of sick.

[00:20:13]I wonder maybe it makes sense to do everything as I and then convert at the end.

[00:20:22]Or maybe this guy just works in I16 and we externally are uh why do all the G's hang on 7-Eleven?

[00:20:34]So true.

[00:20:37]Like maybe we as the caller responsible for converting that to floating point or maybe our audio library should support uh I16 data.

[00:20:46]That's probably the most correct thing to do, right? like pipe wires floating point thing is really just like one of the options. So yeah, let's do that. Let's do that. We we will work with I16s because that's what this thing is supposed to be. And why do extra conversion if we don't have to? Okay, so this guy's like G711.

[00:21:12]Maybe this should be organized somewhere, but for now we're just going to be like G711 decode PCMU for this bite. This is our sample and we have to figure out what to do with this sample which um you know I guess this is being a loop probably uh this sample we want to feed to the audio buffer right so we have like self audio handle which I guess is an audio stream Uh so we should we we should be able to be like stream buffer push uh sample I guess we we'll do sample F32 for now is going to be probably wants to be like somewhere between zero and one probably it's probably like float from int sample and then we probably want to do like a little uh you know max I32 divide or something or I16 sorry and then we push this for however many number of channels there are I guess right so if we have two channels we push this twice this will probably work for now probably and then we'll put like a fix me on here that's like Surely we should just have an S16 stream, right? Are you fat shaming IP packets? You guys talking about like 64k fat boys and they're like, "Whoa, whoa, whoa. No fat shaming.

[00:23:00]No fat shaming."

[00:23:02]Uh, yeah, that seems kind of reasonable.

[00:23:07]Here we're saying anything that is not a G711 packet is no good. We should probably uh like payload type uh should be an enum but from the RFC for RTP video profiles I think it is in chapter six here they define you know PCMU as type zero right so this is just like a thing that we know but we could maybe source this thing RFC 355 51 line 1802 RFC was it 3391 3551 uh chapter 6 uh zero equals PC.

[00:24:00]Okay.

[00:24:02]Uh which is the only thing we support.

[00:24:06]Perfect.

[00:24:08]Perfect. Okay.

[00:24:12]Uh so we this timer stuff is going to come later once we start actually trying to respect the time stamps. For now we're just feeding this stuff in directly. So we'll do a little fix me here too which is like a uh actually use frame time stamp right that's not a thing that we're doing right now but again we're just trying to stub out the overall shape of the thing.

[00:24:34]Um, okay. So, I guess that means that we need an audio service.

[00:24:44]Okay.

[00:24:47]And I guess we're going to be like loop like how does this guy register himself with the audio service? We have a get we have a file descriptor for this thing and a service function. So I guess it's like loop I guess this thing is maybe try.

[00:25:11]So we are like we have some audio of the pole FD for this thing. We only care about read events. Write events we I don't think matter. And we're going to say, "Hey, when this thing happens, could you do um a like audio audio tell me about the audio event, which means we need to allocate some ids, right? So, we're going to go um we have we're going to have like audio probably we're going to have RTP. These are like the two things we care about. And we actually can just hard hard allocate these, right? We can go like const audio um is zero audio or RTP is one.

[00:26:04]And in the future, we might want to do something more advanced than this, but for now it seems like, you know, we're going to have these two things plus timer, right? These are going to be like the three IDs that we have to service for now. So it it might be that this app gets advanced enough that we have to like do some dynamic allocation of these ids. But for now I think just hard coding these three is fine.

[00:26:29]Okay. So we are still going to parse the STP the like the the audio stream parameters up front.

[00:26:38]But once we have done that, I think we kind of want to do like RTP stream is like RTP stream in it with the IP that we got, maybe maybe that seems kind of reasonable. So this guy, he wants like an init function who gets the IP that he is working with.

[00:27:09]who actually also has a port. This is a bad type in my opinion, but whatever.

[00:27:17]And I guess maybe the audio stream is someone that is it a pointer? No, we'll we'll own it.

[00:27:27]We'll own it. So, he probably also wants the pipe bar service to come in here so that he can like allocate the audio stream as well. So he's going to like open the UDP socket, I guess, with this IP and probably if something goes wrong, we close that socket, I guess.

[00:27:54]Okay.

[00:27:56]Uh then we try to create some audio stream for this thing, which I thought we had. We do. Oh, I see.

[00:28:09]I see. We go. So, the audio stream has some sort of uh requirement that it has a stable memory address. So, we call this guy init pin. This is our way to flag to ourselves that like don't [ __ ] move this thing after you initialize it. So, we're kind of going to do it like this instead. Um which means that we can go like self audio stream init pinned with pipe wire and we have some sort of like parameters here.

[00:28:37]So what are our parameters? We have numbum channels sample rate buffer.

[00:28:46]Uh okay. So maybe our audio buffer lives here because we're already allowing ourselves to uh have self- reference and like stable stable structure address requirements.

[00:29:02]So that doesn't hurt. Um let's do one channel.

[00:29:08]Well, this is probably going to come from like the the def the the stream definition probably.

[00:29:23]which it maybe is not.

[00:29:27]Yeah, I'm not sure.

[00:29:29]I'm not sure. So, what happens when we look at this session description?

[00:29:34]Hold on. Everything's broken.

[00:29:37]Uh let's let's see what what does this boy have for us?

[00:29:41]We've got media descriptions.

[00:29:43]Who has formats protocol?

[00:29:52]Who defines the sample rate?

[00:29:56]Who defines the sample rate?

[00:30:06]It must be part of the audio format.

[00:30:10]It must be part of the audio format.

[00:30:13]Right? So if we go back to chapter six of this thing. Yeah. Yeah. Yeah. Here the sample rate is is defined here. So here we are assuming that we are type zero. So we know that it's num channels one. We know that the sample rate is 8,000.

[00:30:33]And uh what was the other thing we need here? The buffer.

[00:30:37]And we are saying we can use our own audio buffer for this thing. [ __ ] it.

[00:30:44]Uh, this feels too large. Let's do 512 kilobytes.

[00:30:51]Let's do that. Or 512 kilobytes times 4.

[00:30:54]So let's actually uh do that 512 kilobytes because the four bytes per float. Okay.

[00:31:05]And so this is now self.socket here I guess.

[00:31:11]And that seems fully initialized to me.

[00:31:16]So now we should be able to be like RTP stream.

[00:31:25]I guess we have like var RTP stream is an RTP stream.

[00:31:34]And we're like, hit me with it.

[00:31:40]Which means that this has to come after we have parsed the session description.

[00:31:46]Cool.

[00:31:49]Cool.

[00:31:51]And then this guy needs to like register himself with the event loop, I think.

[00:32:01]So he is going to be like loop register.

[00:32:09]Uh he's like, "When my socket is readable, I would like to let you know that RTP stuff happened."

[00:32:24]Seems reasonable. Seems reasonable, right?

[00:32:29]Sure. [ __ ] it.

[00:32:34]Okay, so now all of our [ __ ] needs to be driven by the event loop. So whatever the [ __ ] we were doing in main before, [ __ ] that [ __ ] And we're going to go while true. Let's try to like get an event from the event loop.

[00:32:50]We can wait as long as you want me to wait.

[00:32:55]Um if EV is I guess we have like uh probably if this thing is empty we just try again probably and if we get audio we say pipe wire service you know if we get what are the other options timer which we haven't done yet and RTP if we see RTP It's ids.audio, I guess, or ids.rtp.

[00:33:30]We're doing RTP stream service.

[00:33:36]And that's kind of it maybe.

[00:33:43]Okay. He's like, "Dude, I need the event loop, too." So true.

[00:33:48]here. He's like, "Yo, you got to hit you got to hit try on that shit." And then probably we have to dnit this thing if this fails. And then we probably also want some sort of DN on this guy, right?

[00:34:04]Which is like dnit this bad boy and uh clear the socket.

[00:34:13]Not that this really matters in our context, but it might matter in future contexts. So, we'll do this.

[00:34:22]Okay, that feeling kind of nice. Else crash.

[00:34:31]Uh, this thing needs to be variable for that to work.

[00:34:40]and sample F-32.

[00:34:47]Oh, uh, push no clober.

[00:34:53]If we run out of room, we'll freak the [ __ ] out. How about that? All right, let's see if that like does what I want.

[00:34:58]So I believe our test here is that we ask RT we ask FFmpeg to generate a sine wave and send it to localhost port 504 and write down where he is sending that stuff in this like STP definition here.

[00:35:16]So if we run this and then we run our RTP exploration, we look at that STP, he gets mad that we are out of memory.

[00:35:28]So that's interesting here. This our circular buffer is running out. We've got too much [ __ ] data.

[00:35:39]So that means that the audio stream is not getting serviced fast enough or we have not a too small of a buffer.

[00:35:49]So can we just try cranking this a little bit?

[00:35:57]Yeah, he's immediately running out of memory. I don't like that.

[00:36:01]I do not like that in the slightest.

[00:36:03]Right? Because like that there's no way we should be getting a meg of audio data immediately, right? That feels like a bug, right? It feels like a bug.

[00:36:16]So, can we look at what's happening here?

[00:36:23]Audio buff land. Let's split this.

[00:36:30]Um, self audio stream buffer count. How many things are in here?

[00:36:41]So, we are just immediately ripping that [ __ ] Okay, let's uh change the rules. We'll just allow overwriting.

[00:36:55]So uh right so in this scenario if we r if we get too much data we just start clobering stuff from the past and now can we check if the audio stream is getting serviced at all because I don't hear anything happening we did see servicing audio a single time which which seems bad.

[00:37:26]Is someone hanging the event loop?

[00:37:33]Let's try this.

[00:37:37]Yeah. What event did he get?

[00:37:41]Cuz someone is hanging.

[00:37:43]So event one RTP stream service this is hanging why because we forgot to open our UDP socket is nonblock probably no he should be nonblock but then this should be failing Right.

[00:38:13]Loop time. What is happening here? What is happening here? He is spamming loop time, which means that like maybe we're forgetting to check if the length is zero.

[00:38:27]Maybe we need to like if this is zero, we need to break.

[00:38:34]He's constantly getting aotted. He's constantly getting stuff.

[00:38:44]Are we just like receiving data faster than we can process it?

[00:38:55]There's no way, right?

[00:39:09]That seems like a bug.

[00:39:16]That seems like a bug.

[00:39:22]Something is wrong here.

[00:39:28]Something is wrong here.

[00:39:32]Um, so this guy's supposed to say, "I want a U size back.

[00:39:38]It's a fast cable."

[00:39:44]I wonder if we are like getting old data. No, I mean, okay.

[00:39:54]What is What is going on here?

[00:40:01]Sorry.

[00:40:04]So this guy man to receive from this guy returns an S-size T.

[00:40:16]What is the return value?

[00:40:19]Um oh I mean this these these uh how do you say this?

[00:40:28]These docs are for lib C, which is like not what we're calling. We're calling the SIS call directly, but like typically we're probably we're going into here. So this guy's returning us a sys call.

[00:40:46]We're trying to convert that into an arno which is how like how is this function work?

[00:41:01]Oh, this is a bug.

[00:41:04]I see.

[00:41:12]That's better.

[00:41:14]That's better. Now we're getting like a proper wood block. So we were using um we're linking against lib C which means stood pixo is the lib C arrow and the lib C arrow is like we returned negative1 and we stored ANO in the global variable but we are not calling the lib C version of receive from we're like manually calling the Linux system call. So we should just need to make sure that we're using like the correct uh thing there.

[00:41:44]So, we should probably make sure that we're doing that like this. This is a bug basically everywhere where we're doing this. We should just fix that really quick.

[00:41:56]Oops.

[00:41:58]Oops.

[00:41:59]Quick fix. Quick fix. Quick fix.

[00:42:04]Uh, we should be consistent here.

[00:42:16]We should be consistent.

[00:42:25]Uh-huh. Uh-huh. Uh-huh.

[00:42:28]Boom. God damn. We're making this mistake a lot.

[00:42:35]Okay.

[00:42:39]Okay. But now the question is is if we go back to this thing. So we do see that we eventually hit a wood block in there, right? But if we try to actually process the data, like are we literally just too slow?

[00:42:59]Ah, [ __ ] This one.

[00:43:12]I think we are.

[00:43:16]That's funny. We're We're just too [ __ ] slow.

[00:43:29]That feels like a bug. Who is taking that? Means that like We can't do audio in debug mode.

[00:43:41]Like that doesn't sound right.

[00:43:45]That doesn't sound right at all.

[00:43:51]Lov Can we Can we turn on Lovm and see if it's like Ziggs debug back end?

[00:44:05]It might be something that we're doing is just like really [ __ ] slow.

[00:44:10]Servicing RTP.

[00:44:14]Yeah. I mean, all right. I guess what we need to do is profile this then, right?

[00:44:20]Cuz like if we're too slow, we're too [ __ ] slow. Is FMP feeding the sign in real time or as fast as it can?

[00:44:30]Well, RT like FFmpeg shouldn't know how fast I'm consuming the data.

[00:44:41]Yeah, I mean there is something interesting there, right?

[00:44:44]Can let's um can we log like the time range that we're getting from this thing?

[00:44:56]Because if we are getting just more data than we should be, then that could be a mistake.

[00:45:10]Okay, let's um let's let's let's log here uh sample num samples and then we'll also log the start time.

[00:45:29]Uh what is it? It's now clock. C clock get time. Yeah.

[00:45:36]And then we like boot time I think is the one we like, right?

[00:45:40]Could have just said everything at once with time stamps to play later. But like I mean this is pretty pretty damning, right?

[00:45:54]That's pretty damning.

[00:46:00]But like, how what the [ __ ] right?

[00:46:09]Like how who why shouldn't someone be like, "Yo, dog." Like, shouldn't shouldn't feg be rate limiting its output?

[00:46:19]Shouldn't it? Cuz like cuz like surely an RTP stream is meant to be consumed live. But yeah, that does look pretty bad, right?

[00:46:38]I wonder if we can ask it to slow slow itself the [ __ ] down. And I his speed 1.63E to the power 10 * 10 the^ 4. Yeah, that seems bad.

[00:46:55]Okay. Can we say um uh uh the RTP stream this is sending is running at a thousand like 1600x speed. Can I ask FFmpeg to play it in real time?

[00:47:20]Uh, okay.

[00:47:27]Dash re is what he's claiming, man. FFM native frame rate. All right. I mean, that's like that's not an insane thing to suggest.

[00:47:49]Uh, okay. Well, it was close.

[00:47:58]Uh, he's he did say put it before, which I don't think should matter here. I think he's a [ __ ] Oh.

[00:48:14]That's a much ser speed.

[00:48:20]I mean, it is still faster than native, but maybe it's starting to normalize.

[00:48:30]Maybe they're just trying to get a little ahead.

[00:48:36]Yeah. Uh, okay.

[00:48:43]Uh, sure. [ __ ] it.

[00:48:46]That re does seem to do the job.

[00:48:51]Okay. Well, then that means that we can go back to using Zig's debug back in.

[00:48:54]I'm like, it shouldn't be that slow, man. It shouldn't be that slow.

[00:48:59]Perfect. Okay. Now, we're getting a wood block, which is makes sense because it would block. So I think here we need to like catch an error and we need to say like if E is equal to error would block like I don't give a [ __ ] otherwise it's a real error.

[00:49:29]There we go baby.

[00:49:32]Okay.

[00:49:34]So, I'm not liking that the audio service is not doing anything.

[00:49:42]Um, do we see the audio stream show up in like this audio thing?

[00:49:49]It does think it's here, but it does not seem like it's like pulling data correctly. Well, oh no, wait. I see servicing audio in between here, but I'm not hearing anything relevant. So, I guess the question would be why can we go back to push?

[00:50:17]No clobber here by the way.

[00:50:23]Yeah. So, that's working now. And can we check maybe just every time this happens, every time we get a packet, can we check the audio stream like head and tail of the buffer that we're trying to push into.

[00:50:49]I mean that thing is moving around.

[00:50:55]Right? Like that that there is data flowing through the system. Is it possible that this was wrong? What are like the min and max like maybe we have [ __ ] this up in some way?

[00:51:13]Um yeah. So let's just do a little We'll just let it get big.

[00:51:28]I still hear nothing.

[00:51:39]Yeah. Nothing relevant is happening.

[00:51:43]Nothing relevant is happening.

[00:51:47]Um Okay.

[00:51:51]So, what the [ __ ] Can we go back to look? Let's look at our audio example.

[00:52:06]So, this guy, what does he do?

[00:52:10]He says, Hold on.

[00:52:25]Oh, sure. So, he's got a service up here that he has defined and he gets the stream buffer for this thing and he generates a sine wave at some sample rate. Okay.

[00:52:44]And he just pushes the data in there.

[00:52:52]Okay, then what the [ __ ] is the problem?

[00:52:56]Like that feels like what we're doing.

[00:53:02]Oh, hold on. Can I go back to uh It looks like he's just pushing in the range 0 to one, right?

[00:53:15]Like that he's just running math sign.

[00:53:19]And when we were here, we were hearing that. So 0 to one seems correct.

[00:53:23]So that makes me now suspect of uh the actual data that we're pushing in. I would like to check that in some way. So let's um let's do this.

[00:53:40]Let's write the [ __ ] out.

[00:53:46]Um, so every time we push a sample, okay, I guess we're going to like open this thing. Selfwave file is our like uh outf. We'll call this we're going to open uh you know the wave.csv.

[00:54:10]We're going to say I want this to be write only. I want truncate to truncate the file.

[00:54:19]And I think that's probably fine.

[00:54:23]And I'm not going to bother closing it because this is like debug code.

[00:54:27]And let's do a little write.

[00:54:33]I guess let's uh maybe we have a writer for this thing.

[00:54:39]Uh, and let's do a writer buff.

[00:54:48]Okay, so we're going to go self writer is initialize with the out file and the writer buff and we will say writer interface print a number.

[00:55:13]Uh, this guy's mad because open needs some perms or something.

[00:55:19]We need to try this. And then let's say every frame we get or every time. Yeah.

[00:55:26]Every frame we flush the writer.

[00:55:34]All right.

[00:55:39]Okay.

[00:55:46]Cool.

[00:55:48]So, we can do something with this.

[00:55:56]Let's try to figure out what is the period of this thing. We're hoping for a 440 Hz wave sampled at 8,000 htz. Okay.

[00:56:06]Well, first of all, let's also uh let's look at like maximum of this thing and the minimum.

[00:56:19]Oh my god. Min, please. Please. There we go.

[00:56:24]So, those are like not full range, but I mean it's like fine.

[00:56:32]Um, can we look at so at 8,000 samples per second?

[00:56:42]It means that if we highlight 8,000 samples, we should see 440 hertz wave, which means we see 440 revolutions of this thing right up.

[00:57:01]This thing's just not G711.

[00:57:03]I'm just a [ __ ] idiot. I am just a [ __ ] idiot.

[00:57:14]Oh, no. It is sample rate 8,000.

[00:57:18]Oh, frequency is one.

[00:57:21]That is a really long sample rate.

[00:57:26]Let's try uh an actual 440 Hz wave. What if we did that?

[00:57:40]Sorry, I'm hearing something, but I can't tell if it's like the [ __ ] song and you guys aren't hearing it. But just kidding. I'm not hearing [ __ ] Well, I feel like I am. No, I feel like I'm hearing something. It's not good.

[00:58:01]Hold on. Let's uh Oopsies. This goes to right. This goes break this connection.

[00:58:11]This goes to left. Like, are you guys hearing that? It's so quiet. It's so unbelievably quiet.

[00:58:26]I mean, I mean that might it might just be that G711 is dog [ __ ] or we're like receiving the packets out of order. Those are all options. Can we um maybe run this a little bit and then can we let's get like a second of data or so and then let's look at the thing that came out.

[00:58:59]Okay. So, we are looking for over 8,000 samples, 440 peaks of a wave.

[00:59:09]So, can we like just quickly graph this?

[00:59:19]Uh, can we get like a line graph maybe?

[00:59:25]I mean it might be it might be right just like dog [ __ ] encoding right. Can we maybe is there a way to like check in Libre Office like number of peaks in this Libre like Libre XL maybe XL counts uh wave wave peaks some Jesus Nice.

[01:00:07]Um, do we just want to like do it in Python instead? I'm sure it's like doable. I'm just like too [ __ ] stupid. I guess we could like look at the period manually here. So we're going like here is a peak hereish, right? So 17. Where is the next peak here?

[01:00:44]Right. So, we're looking at a period of around uh 29 - 17 is 12.

[01:00:57]So, 12 over 8,000 or 8,000 over 12.

[01:01:05]Seems too fast.

[01:01:07]Seems too fast. Can we look at maybe let's look at like a small amount of this data.

[01:01:14]and see if it looks continuous.

[01:01:20]Oh, I mean that's [ __ ] right?

[01:01:23]That's just like obviously [ __ ] data, which is interesting because I I feel like we saw something like this before, but I thought we had fixed it.

[01:01:41]I thought we had fixed it.

[01:01:44]Okay, let's do a quick little Is this a law versus elaw? Maybe.

[01:01:53]Maybe. But I thought we had something that looked right before, which is what's bothering me.

[01:02:01]Um cuz this does say give me mu law which does turn into format zero.

[01:02:20]So I don't really see why that should be a problem. We could just double check.

[01:02:23]Uh G711 Ulaw.

[01:02:31]This is a law.

[01:02:34]Mu law inverts all the sign bits after the sign bit if the value is negative. Adds 33 binary and converts it. So this is the formula to decode the decoded linear value is given by this. Let's just double check that we're doing that -1 * s. So s multiplier is if s is zero then one negative 1. That seems good. 33 + 2 * n * the power 2 e - 33.

[01:03:19]That sounds correct.

[01:03:26]That looks correct to me according to this.

[01:03:31]This looks like where you where you forgot to reverse the bits before. Oh yeah, I did forget about that. Huh?

[01:03:43]Did we lose that piece?

[01:03:53]We might have lost that. I forgot that existed.

[01:04:01]Uh yeah, good call. Good call.

[01:04:09]Good call.

[01:04:18]Uh B in B is B in. All right. Good [ __ ] call. Thank you.

[01:04:28]Okay. So, let's uh Dude, your memory is good, by the way. That's crazy.

[01:04:36]Send.

[01:04:40]Oh, [ __ ] yeah. Do you guys hear that [ __ ] You [ __ ] hear that?

[01:04:49]That's a [ __ ] audio wave, baby.

[01:04:56]Uh, all right. We did this at some point. We did this.

[01:05:02]Um, and at some point I guess we did this as well. We've kind of done all of these three steps, which means that we're kind of at respect time.

[01:05:15]Okay. So, this is something that we are not doing in the slightest right now. We are just queuing audio packets as we receive them. So the question is is what the [ __ ] do we do?

[01:05:28]I think that we said that we wanted to just dump the data into the audio buffer as it comes or sorry. Yeah. So as it comes we dump it in the correct spot in the audio buffer.

[01:05:43]So instead of just pushing here, um, we want to say where the [ __ ] should it go, right?

[01:05:56]So, it sounds kind of like this circular buffer interface needs a way to be like we almost want like audio stream right relative to the tail or something, right?

[01:06:16]And we want to say Yeah.

[01:06:25]Uh and what what what is the interface here?

[01:06:33]So I guess we are going to save like time step or like time tail time step.

[01:06:48]Okay.

[01:06:51]And this is going to be a like what does the time step field of the RTP header look like? Time stamp is a U32. So we can say like tail time step stamp is a U32 who is in some unit, right?

[01:07:10]Can we look at what that unit is?

[01:07:20]Time step time time time uh fix header fields 5.1. Okay.

[01:07:30]Version extension pack time stamp. Here we go. The time stamp reflects the sampling instance of the first octed in the RTB data packet. The sampling instant must be derived from a clock that increments monotonically and linearly in time to allow synchronization and jitter calculations.

[01:07:45]The resolution of the clock must be sufficient for the desired synchronization accuracy and for measuring packet arrival jitter. The clock frequency is dependent on the format of data carried by payload blah blah blah blah blah.

[01:07:59]So it looks like it's defined by the profile. So that's in the other RFC.

[01:08:07]So we're looking for like PCMU, right? 4 514.

[01:08:19]Does he talk about the time stamps in here at all?

[01:08:27]Okay. Encoding independent rules. Since the ability to express silence is one of the primary motivations for using packets to transmit voice, the RTP header carries both a sequence number and a timestamp to allow a receiver to distinguish between lost packets and periods of time when no data was transmitted.

[01:08:48]Okay. Discontiguous transmission silent suppression may be used with any audio payload format. Receivers must assume that senders may suppress silence unless it's restricted by the signaling elsewhere. Okay.

[01:09:02]So some payload formats define a silence insertion descriptor or comfort noise frame to specify parameters for artificial noise that may be generated during a period of silence. Okay.

[01:09:11]Uh for applications which send either no packets or wait blah blah blah blah.

[01:09:14]Okay. For applications which send either no packets or occasional comfort packets during silence. The first packet of a talkert that is the first packet after sounds blah blah blah. I'm more interested in like the clock stuff first. We'll deal with sounds later. The RTB clock rate is used for generating RTB substamp used for generating this is independent of the number of channels and the encoding. It usually equals the number of sampling periods per second.

[01:09:39]For N channel encodings, each sampling period, say 1 8,000th of a second, generates n samples. Okay, what the [ __ ] did they just say?

[01:09:48]Independent of the number of channels encoding. Okay.

[01:09:52]So, if we are running at 8,000 hertz, the clock also advances at 8,000 htz is kind of what this is saying.

[01:10:07]I think it seems like also the RTP header has a channel function in there, but I think that our our encoding is single channel anyways.

[01:10:22]No. Okay. Where does channel come from then?

[01:10:30]I guess it's just embedded in the data in some way. So, we just won't worry about that. Okay. I think that this was saying that the clock should advance at 8,000 samples, right? So, can we just double check that actually?

[01:10:49]Uh so we're looking for this is the like time step and then we should also say uh like s num samples in this thing which is going to be the payload size I guess right something like Okay, we're spamming something else too much.

[01:11:31]Okay, so does this look like it's going up by around 10:24?

[01:11:38]Yes. Right. So here we see we got 10 24 samples and then this this time stamp increased by that much right like 4201 nice okay cool so that means that a reasonable interface here is like audio stream right past tail maybe the like distance and the sample right that seems like maybe something that we want to do right so time stamp is just a sample number but I think with at some arbitrary starting point I don't think it starts at zero but it does seem like that okay this seems reasonable so the distance is going to be we like write down tail fail time stamp here.

[01:12:40]And we must also write down like last tail update maybe like this instant I think where where is duration and instant come duration. Oh, time. It's time stamp, right? So, on a timer, we just bump the tail.

[01:13:16]And that seems reasonable.

[01:13:23]That seems reasonable. Okay.

[01:13:26]So, here's how to push over. We're saying right past tail, I think.

[01:13:35]our distance from the tail is like related to the number of sample indexes here.

[01:13:44]So we say like uh we'll say like like frame audio offset.

[01:14:00]Okay, so this is going to be like the tail time stamp but like the uh the frame time stamp minus this, right?

[01:14:24]But if this is too far, if this if frame time stamp is greater than less than the tail time stamp, we have to like increase amount we buffer which means means that we uh advance the like we we do like self like increase amount is the other way and how what do we do with this information? This is going to be in 8,000 of a second, right? So, it's probably like the last tail update needs to move, right? We need to make it seem like this happened later to give us more time, right?

[01:15:46]And we want this in sec in in like milliseconds or something maybe like nanconds.

[01:15:58]Okay, that's looking for like something.

[01:16:02]So this is in samples. So we have like increase amount nanconds is going to be the increase amount in samples times something over 8,000.

[01:16:16]I think let's do some grade 12 chemistry stoometry. This is the or grade 11. We spent like six months on this and it's just unit conversions. So you have 8,000 samples per second. We are trying to get to nanconds.

[01:16:38]So we flip this to get seconds per sample.

[01:16:50]And then we need to times by samples to get this into seconds. So this is the sample. So this is times n samples right? This turn this cancels out the units here which gets us in seconds. But we want this in nanconds.

[01:17:13]So we turn this into uh 1* nonds per second. And this gives us nse here.

[01:17:24]Yes. So we have uh nancond time ns per s over 8,000 times increase amount samples. There we go.

[01:17:45]So this is increase amount ns.

[01:17:49]That sounds right.

[01:17:52]Right. Right. Something like that.

[01:17:57]I hope. And then like what unit is this?

[01:18:00]This we want this to get like into uh we want this to like expand into an I64 as early as possible. So I think if we do this, the whole thing gets upgraded to an I64.

[01:18:14]But I think that I64 maybe we do U63.

[01:18:24]ns per s time samples over samples per second equals ns.

[01:18:30]I think that we got that's what I wrote, right? I think that's what I wrote. NS per second time samples over samples per second, which is 8,000 samples per second. Nice. Nice. Same answer. Let's [ __ ] go, baby. Let's [ __ ] go. Took me a minute.

[01:18:47]of what we got there. Okay.

[01:18:51]And then is ad the right way.

[01:18:56]I think so.

[01:18:58]I think so. And then we'll put like a fix me here. Like um if it's rare that this happens, we might not want to increase the buffer amount, right?

[01:19:13]Uh but that's fine.

[01:19:16]Okay. Okay, so we need a right past tail thing on the audio stream, right?

[01:19:22]So stream buffer is a circular buffer who right now has basically push and pop.

[01:19:38]So this guy might want to like right past tail. No, right past tail.

[01:19:48]distance is going to be a U size and a vow is a T and he is going to say uh self tail plus distance I I guess it's going to be like if items len minus the size of the thing is greater than or equal to the distance.

[01:20:27]So this is like the remaining space return error out of memory. Otherwise we're good to go. So we say like write pause is the tail plus distance and then we just mod this by the length of the buffer.

[01:20:46]And then we say self items at right pause is equal to val. And then we also need like like increment tail like um maybe like mark written is a name for this.

[01:21:09]And we're going to say like again we're going to check that this is correct. This is probably an assertion.

[01:21:18]Uh well no cuz here the mark written is like we're calling this on a timer and that timer might be pushing past uh the the amount that's been consumed as well. Right? So if we have like a we have like 10 bytes of stuff of room but we mark like a hundred bytes written maybe that should just collapse the head like empty the buffer.

[01:21:47]No overwrite the buffer.

[01:22:02]So, how how is our head and tail stored right now? Ink tail.

[01:22:11]Oh, perfect. We'll just do this.

[01:22:19]And we'll say uh fix me. LOL.

[01:22:25]Massively inefficient.

[01:22:28]Should that be less than equal distance?

[01:22:33]Yeah. So if the distance is greater than the amount of room remaining.

[01:22:41]Yeah, you're right. You're right. I agree. Maybe less than not less than or equal to.

[01:22:48]So if I have like a buffer size of four and three things left, then I can write one thing.

[01:22:57]So it should be equal, I think.

[01:23:02]Because yeah, I can write one thing.

[01:23:04]Distance would be one.

[01:23:06]Oh yeah. Yeah. So less than right. So because so here we have four space for four things.

[01:23:13]Three things in buffer. Now distance of one should work.

[01:23:20]Distance of two should fail.

[01:23:23]So here we have uh one on the left.

[01:23:32]Uh, distance is two.

[01:23:40]No, no, no. It's this way. Right. I'm back to thinking it's this way.

[01:23:49]Why?

[01:23:50]Space for four things. Three things in the B. This is a bunch of work.

[01:23:58]It feels like it should be less, but why is my brain not figuring that out?

[01:24:04]So, we say one on the left. If distance is one, we want that to work. If distance is two, we want this to fail.

[01:24:17]Yes. Okay. So, if distance is one and this is one, this condition does not happen. If distance is two and this is one, this condition does happen. Jesus Christ. my [ __ ] stupid ass brain.

[01:24:26]Dude, I sometimes sometimes I'm [ __ ] stupid. What can I say? What can I say?

[01:24:34]Okay, but this should be enough now for us to be like we write past this by some distance. So our distance is the frame audio offset plus sample index.

[01:24:59]Okay.

[01:25:02]Then this push no clobber goes away and we need to on a timer update this thing, right?

[01:25:14]So I guess here somewhere we need to have a timer like something that can handle time. So we're going to say var timer service here is a stood io timer service who needs allocations I guess. So do we have an allocator right now? We do. So we have Alec dot allocator.

[01:25:38]I guess this is here. And then we need like an expansion allocator for this thing. So this is like this guy. And we're going to say he's going to expect to be holding like one time like less than four timers at a time. And if we registered like 16 timers, then we've made a mistake somewhere. We should really only have one. So these are like plenty plenty big enough.

[01:26:00]Uh, just kidding. Those are not parameters for this psych.

[01:26:07]It's actually this.

[01:26:09]Uh, it seems like this guy registers himself with the event loop. So, we don't have to do that. But, we do have to pass the timer service over to our RTP service, right? So, he wants timer service here.

[01:26:33]Okay.

[01:26:35]And what's he going to do?

[01:26:39]I guess he is going to create a timer on this service and do something with it.

[01:26:56]I think that this is going to be uh we need an RTP timeout ID here to be registered with this thing. So, we're going to be like add something that triggers every 200 milliseconds.

[01:27:16]So, we'll buffer 200 milliseconds of audio data and we are going to add in we're going to say like, hey, let the [ __ ] system know if there's any that there's an RTP up.

[01:27:35]Okay, then this thing returns a handle which we must need we must write down.

[01:27:44]Surely, surely we need that somewhere. I don't know where, but it feels like we need it, right? So, timer handle is a stood IO timer service handle.

[01:27:55]Used to watch a while back, did you start using AI in your develop workflow?

[01:27:59]Um, every time I've tried, I've been like, this is just a waste of money. So, I've tried a few times. I tried um using it to like remove a bunch of Windows code from a like standard library function that I copy pasted. It couldn't even do that. I tried to get it to um what was the thing I tried before that I tried to get it to write a PNG decoder for me uh using like a specific interface that I was trying to target.

[01:28:28]That didn't work. I tried to get it to I tried something else recently as well.

[01:28:33]Basically, every time I've used it, it doesn't work. So I don't have to wrestle with the idea of it being uh immoral until it is functional.

[01:28:45]I And you might say you're an idiot and I'd be like maybe. But I just installed cloud code and asked it to do something that I thought was easy. And every time I do that, it spends like $15 and fails.

[01:28:54]So until that's not the case, I will not be using it. But I I will I will check back in every few months, you know.

[01:29:06]every few months I'll give it a shot.

[01:29:08]Um, okay. So, every 200 milliseconds we probably want some like on timeout thing to happen.

[01:29:19]And we probably want we need to initialize our like tail time stamp uh probably to null right probably we we don't know what this is until we have written something and I guess last tail update should be now.

[01:29:54]Sure.

[01:30:00]And then we'll say we're trying to push 200 milliseconds of data later. This initial parameter seems like maybe wrong, you know, like this feels like it might not be correct.

[01:30:16]Uh th this feels like we're missing some initial state, but we'll see.

[01:30:30]Okay. So I guess the first packet we get.

[01:30:44]We probably do something special which I guess is just is set this thing to the time stamp of the frame probably.

[01:30:58]And then here we can just kind of dreference it everywhere because we know it will be set. So on the first frame that we get we say that the tail is here. And so our audio offset will be zero essentially. That seems reasonable.

[01:31:14]That seems reasonable.

[01:31:17]And then on timeout what do we do? Do we like check now?

[01:31:26]I guess um and missing some initial state wrong.

[01:31:37]Yeah, I think I I think I clarified that. But yeah, I think that there's I think my initial state's [ __ ] but we'll see.

[01:31:43]Um I guess we should be able to be like We want to say how long has elapsed. So, uh, elapsed since update is going to be now like the last tail update duration to now.

[01:32:07]Now, is this going to freak the [ __ ] out if this is negative?

[01:32:13]No, that'll be fine.

[01:32:16]So, Can we convert this to nanconds? I guess um okay so if elapse since update nconds is negative uh we just don't do anything right that's usable. However, we do always have to rearm the timer.

[01:32:47]So, I think there's like we probably need to save like a reference to the timer service as well so that we can say um rearm this thing at 200 millconds again, I guess. Sure. [ __ ] it.

[01:33:13]Only in stud can we see times like I96.

[01:33:15]That's actually standard library.

[01:33:17]Believe it or not, that's not my library. That's the Zig standard library.

[01:33:21]Believe it or not, I would love I would love to take credit, but I did not. I That was not me. I wish.

[01:33:31]Um, okay.

[01:33:33]So, we say now we just have to advance it by some amount. So, um, advance like elapsed since update samples. And this is just the other way from whatever the [ __ ] we did here, right?

[01:33:51]So, we're going uh we do we just do some like shuffling.

[01:33:57]So, times 8,000 gets rid of the divide by 8,000 divided by increase amount samples, but that's what we're looking for. So, it's divide by nanconds per second.

[01:34:15]Looks like it's like this.

[01:34:20]Um, so this is elapsed since update 9 seconds and then we should just be able to be like self audio stream buffer like mark written elaps since update samples.

[01:34:39]Yeah, we're missing something where we need to zero out missing pieces, right?

[01:34:44]like some someone needs to write silence to the parts where we didn't get g data, right? I think that's the responsibility of the audio stream. As he consumes the data, he just zeros it out behind him, but we'll deal with that later. And then we just say RTB timeout. We say RTP stream on timeout. Okay, this thing I think is failable but not in any way that we want to try to handle. And then we also have timer which means that we need to service the timer here.

[01:35:29]Okay, that seems kind of nice.

[01:35:36]Uh, I do kind of feel like maybe this guy should take the timer service in here as an argument so that he doesn't have to like remember him, you know?

[01:35:51]That feels kind of nice.

[01:35:53]Feels kind of nice.

[01:36:02]And then I think that it's all kind of just kind of starting to work.

[01:36:07]Um, he does want a div trunk on here, huh?

[01:36:16]Yeah. What happens here if it doesn't divide nicely?

[01:36:22]A div trunk I think is make sense. I think we just say take the smaller of whatever.

[01:36:29]And this should be a U size. Okay, we can do a int cast here probably.

[01:36:39]There might we might want to do a fix me like safer cast maybe, but I don't think that this will ever come up as a problem.

[01:36:53]Okay, that might work. It would be interesting if it did. It does not. So, what did we [ __ ] up? Oopsies.

[01:37:04]Um, in circular buffer 48, the head minus tail has been broken.

[01:37:17]So, where was this? This was in integer overflow. So, oh [ __ ] Is this supposed to be ink head is t Did I have head and tails completely backwards?

[01:37:41]Yeah. Push updates the head. Oops.

[01:37:46]So this should be a right past head.

[01:37:50]This should be head.

[01:37:56]And then this should not be ink tail.

[01:37:58]This should just be uh self head plus equals amount I think. And then I think we just have to uh assert that the amount is less than uh the size of the buffer. I think we let the head get a little far.

[01:38:21]We let it get ahead. Um and then we fix it later on read if I remember correctly. It's been a while, but does that work?

[01:38:40]Right past head now. Oopsies. Okay. Does that work to some extent? To some extent that was working, but it sound like we were only getting like small chunks, right? So like were we forgetting to update the time?

[01:39:08]Yeah, we do.

[01:39:15]So let's update this and the time. Do we also have to update when we update it as well?

[01:39:35]Uh, so I guess if tail Yeah, we're missing something here about like what if the tails time stamp hasn't happened yet?

[01:39:51]If t self tail time stamp is null, we have nothing to write I guess.

[01:40:05]So I guess we kind of start by always rearm the thing no matter what. Then we start worrying about this stuff.

[01:40:19]I think when we get the first sample, we set the last tail update to now.

[01:40:31]That seems kind of reasonable.

[01:40:38]So that when the timeout happens, we increment the tail time stamp as well as the last tail update.

[01:40:48]Okay, that's getting closer.

[01:40:57]Okay, does that work?

[01:41:00]That's not bad. That's not bad. So, we could hear it like stuttering a little bit at the front as it like starts trying to figure out how much it needs to buffer, but then eventually it like stabilizes.

[01:41:15]I don't know if you guys like can tell that.

[01:41:21]That's kind of [ __ ] sick. That's kind of sick. Okay, so now we should be able to start like scrambling the packet order a little bit.

[01:41:31]Um, how do we do that? Can we scramble the packets a little bit?

[01:41:40]Um, what made you become a good programmer? All this knowledge now new language so much not to new now language so much well to not think about syntax I solve problems so much easily but make it feel natural okay I assume there's like some broken English in here but like what are you asking you're asking how do you get comfortable and fast I guess and I think it's just it's the answer's always for everything unfortunately get the reps in so like you are watching us kind of do a thing that doesn't need to be done right and I like don't have a job because I like doing things that need to be done that don't need to be done. Sorry. So much that like this is just like my [ __ ] You know what I mean? Like this is my hobby. This is what I do for fun. I just like I there's an infinite number of things to program and I want to program them all. You know what I mean? Well, I guess I shouldn't say them all. There's a lot of stupid things to program, but like there's a lot of things that I want to do more than I have the time for, right? So for me it's just like I have a lot of ideas that I want to play with and so I I I do them right and I also don't let the world tell me that things are bad ideas right so like to me trying a thing that shouldn't work is a good way to understand why it doesn't work you know what I mean and if it does work then like either the world is wrong or you're missing a piece and you have to look a little bit more carefully but like the the the the general answer is just get the reps in, you know. Got to program them all, Sparroman. Got to program them all.

[01:43:20]Yeah. Yeah. Um, okay. This strategy seems to kind of work. So, what do we do? What are we doing right now? We have I guess this service function should be called renamed to like on data or uh service UDP maybe is a better name for this. So we have like we have four guys working in parallel.

[01:43:48]We have the audio service who just pulls data from the audio stream as it's available. we have on UDP we we have like a little bit of initialization stuff but generally what we're doing is we are looking at where the data needs to go by looking at where the frame says it starts versus where the uh this should be called head time stamp by the way we had our head and tail backwards so basically how far are we from like the last thing that's been committed for read on the audio stream side, right? So like there's kind of like a buffer here where there's some amount of stuff that's like we can call this like committed committed which is like the audio system is allowed to read anything between these two arrows but then we like arbitrarily write past these arrows, right? And on a timer, we move this arrow forward, right? Every 200 milliseconds, we bump this arrow. And then, you know, every some amount of time, the audio service is also bumping this forward. We hope that that we anything that we got anything that is being played is between these two. We're not handling anything for that right now. And then we say any time any time that we get UDP data that would come in behind the stuff that we've committed, we go, "Ah, [ __ ] We wait. we need to wait a little longer and so we just kind of like pause a little bit and like like move the next update a little bit later.

[01:45:28]But that's not bad. I'm sure that we need a better strategy for a lot of this stuff, but this is like kind of the general shape of this thing. So we when we hook this up to like the SIP service, we're going to get like an STP.

[01:45:44]We're going to construct an RTP stream like a application specific RTP stream based off of the SDP that we get from the SIP service.

[01:45:57]We are going to have to we're going to have to end up having some sort of like well either either we're going to have like one active call. If there's only one active call allowed, then this type is enough. But if we want to support like multiple RGB streams, which we might need to just for like maybe even like send and receive. I guess send and receive are going to be different, I would guess.

[01:46:22]Um, but like there's a world where you could imagine like RTP stream manager or something who like holds a pool of of like many, you know, 100 RTP streams or something, right? and like his job is to like dispatch accordingly and service all of these guys as they need to be serviced and stuff like that. But like for now, this single thing is good enough. A single UDP socket that we read from um it feels like I'm I'm happy with like the split of all of these uh pieces. Like obviously like the code like all of this is uh ugly and like inline and stuff but like the audio codec living in its own audio codec spot that we just pull from from this application for this context. That seems reasonable, right? We have the RTP packet parser that lives in its own place. Maybe this scheduling stuff needs to live somewhere.

[01:47:23]RTP stream manager factory builder.

[01:47:24]Thank you. Like you could argue that that there's going like we probably want some sort of form of like like how do you call this uh like RTP time sync algo or something right whatever however we've decided to manage the time which is right now just saying just use the longest latency piece of information and use that as our buffer size but like later we're going to do something that says uh look at the statistics of how many times we got a frame that was like out of range.

[01:48:04]And so probably that's going to live in somewhere common because probably anybody who wants to use RTP is going to want to do scheduling but everybody who uses RTP is not necessarily going to want to write to the audio buffer. So like this stuff being specific super reasonable there. Yeah, there's definitely some like commonality that we could find in this stuff that we're going to have to look for.

[01:48:30]But yeah, I'm happy with the overall shape. For the most part, the responsibilities are like separated and dduplicated enough, I think.

[01:48:41]Cool. Cool. I think we call that a win.

[01:48:44]Let's hit play one more time.

[01:48:49]We love that [ __ ] dude. We love that.

[01:48:53]Uh, that's success. Success. So, next stream we probably work on hooking this up to like the actual SIP client. We're pretty [ __ ] close, I think.

[01:49:04]But yeah, we call it there. Thanks for watching, guys. If you like what you saw, we stream maybe it feels like it's kind of every other day at this point.

[01:49:12]We aim for, you know, at least every other day, sometimes more. We aim for 12:30 Pacific time for, you know, somewhere between 2 and 4 hours. So, if you're watching live, it is 2:30 now.

[01:49:22]You should expect to see us anytime between two hours ago and an hour and a half from now. Um, this VOD goes onto the YouTube. Some VODs do not, right? It really depends on if I think there is like a nice story and feature development thing to tell, right? So, here the story was um we got RTP audio playback working or something like that.

[01:49:42]But sometimes it's a lot more like I'm doing unit testing or last stream we kind of just implemented this like STP parser and it was like kind of [ __ ] boring and no new content considering we've like done all the parser stuff already. But if you want all of that access the access to the extra VODs um you either have to catch those ones live or they go up on Patreon or under YouTube member or get archived on Twitch. If you pay in the appropriate spot, you help the stream continue for longer because one day I'm going to run out of money and I'm going to have to get a job again. But every dollar I make, you know, pushes that day out. So, if you want to get access to those VODs and help the stream live for a longer period of time, uh, please consider supporting. But obviously, no pressure.

[01:50:21]U, that is our influencer spiel.

[01:50:25]Um, YouTube, peace out. Tuition finds someone to raid.