Boosting Software Efficiency : A Case Study of 100% Performance Improvement in ... - Gili Kamma

[00:00:10]How do you fix a legacy product that everyone from customer to developer wants to walk away from? In 2017, I inherited an embedded system struggling with unexplained resets and missing data. I worked on that product for two years and during this time I drastically improved its performance and capabilities and in this talk I'm going to speak about it. I hope you will get some useful ideas and inspiration and that you will find it interesting.

[00:00:42]So first about me my name is Gillik Kama. I'm really passionate about solving problem and making things work.

[00:00:50]I have more than 20 years of experience in software. I started as electronic engineer then I moved to BSP and drivers. I did embedded high level C++ thread over operating systems. I did some back end C and Python. And for the last several years I'm working in the cloud with microervices and go and currently today I'm working in priority as a manager. So I switch sides in the last few years and my story begins in 2017.

[00:01:27]I I moved to Denmark with my family and I started to work in Camrop. Camtop is a very Danish company and the business of Camrop is meter and when I am saying meters I mean water, heat and electricity and if to be more specific smart metering. So smart metering, it's exactly the opposite of what you see in this image. So instead of a person that need to come to your house, read the numbers from the meters and then transmit it to someone else, everything is automatic.

[00:02:04]I worked on the water domain and this is my system, the system I work on end to end. So on the right we have the water meter. The water meter is installed in the house and it measures the water consumption of the house and every 16 seconds several minutes depends on configuration it it transmit it wirelessly. And now imagine you have a big neighborhood with hundreds or thousands of houses and in every house we have such a water meter that transmits its measurements. On the other side we have the ready manager. This is the backend system the billing. So they're the one that in the end of the month supposed to send you the bill and in the middle this is the product I worked on and it was installed on a very high place next to a big antenna and it was responsible to collect a meter measurement from the water meters wirelessly on one side and then it does some logic and then send them to the cloud by cellular modem or Ethernet.

[00:03:16]And what is important to understand here that the business is data. So loss of data means loss of money and there are fines if we are missing some of the data.

[00:03:28]So let's speak about the problems.

[00:03:32]Originally it will work with less than 1,000 water meters and it worked very fine with it. But after several year there was a demand from the sales department to sell it to bigger projects, bigger neighborhoods. And then they came to a smart engineer. He did some calculation and he found out that the hardware of the unit can support up to 7,500 m. So everything is per perfect. That's what they sold to the customers. And then in reality after the site was installed we found out it's not working as expected. The product was unstable.

[00:04:15]We had some crashes. We didn't know to explain them. And of course customers were unsatisfied, right? They didn't get what they paid for and developers were very unhappy because they got a lot of complaints.

[00:04:34]And if to be more specific about the problem, we had memory issues. We didn't have enough memory even though that theoretically we supposed to have we had all these unexplained crashes and resets we didn't know to explain and also occasional data loss during network issues. So sometimes we had problems with the network and everything was fine. we didn't lost the data and sometimes we had network issues and we lost the data and of course none of the errors were seen in the office. So we could we couldn't reproduce anything we couldn't solve anything very common right you are familiar with that okay let's speak about technology to understand the environment so Linux kernel leop distribution it was all the image no sources it worked very fine but I couldn't touch 32 megabyte RAM okay it's important because later I'm going to speak about numbers it was C++ 17 and this is nice but not really. It was more in a style of C++ 11 and little bit earlier it means no smart pointers for you and I don't know if this is the best of all it was written in QT if you are not familiar with it's an application framework it's a rapid to low-level programming and the good thing about cut that it feels like C. So if you are not an embedded developer, it's very nice to write it. You you can write an embedded software without understanding embedded.

[00:06:13]The bad thing about cutie that it feels like C sharp but you don't have a garbage collector and it really encourages you all the time to allocate objects but it doesn't encourage you to to release the objects later. So it's a problem. If you have enough resource, it's fine. But if you don't have enough resource, it can become a problem. Okay.

[00:06:35]So let's speak let's speak about my goal. So I was new in the country, new in the company, new in the project team.

[00:06:42]Everything was new and and that was the goal. I was receiving reduced to zero loss of data and create a stable.

[00:06:52]Okay. So how am I going to do that?

[00:06:54]Right? Very simple. So that was my plan.

[00:06:58]I was it was in a time I was very naive about it. I was say okay I will analyze I will go over the code understand how to put a break point where are the logs how to run the software look at the main modules in in the software understand the main flaws.

[00:07:16]Okay. And after that the plan was to reproduce the errors on the table and then when I can reproduce them of course I can solve them. Okay. Okay. first impression. So I went to the code and start looking at and I saw a lot of news and the leads and I was like it was the first time I saw something like this.

[00:07:38]I was really shocked about it. But then it clicked me and I said okay I I I understand it. It explains everything right. It probably means I have memory leaks. It mean when I'm allocating but I'm forgetting to to release the memory and probably I have memory fragmentation and this is like 20 seconds about memory fragmentation. So let's assume I have 100 free bytes and now I'm allocating 40 bytes and 20 bytes and 40 bytes and now I'm releasing the 40 and the 40 and now I want to allocate 50 bytes. Ah, I can't right. It's pretty frustrating. Even though I have the memory, I can't. Now imagine all your memory looks like this.

[00:08:24]This is memory fragmentation. So you have the memory, but it's organized in a way you cannot use it.

[00:08:32]Okay. So it explain why I don't have enough memory.

[00:08:37]So my allocation fails.

[00:08:40]So I'm accessing null. Of course I am not checking before if you wonder and I'm getting this uncontrolled reset QED right wrong it's not wrong wrong but it's wrong in this case so let's go back to my previous slide so I had some memory leaks but not so many and I didn't have memory fragmentation at all but in this point of time I was sure this is my problem and all what I need to to do is just clean some of the news and everything will be fine.

[00:09:20]Anyway, so I I had my theory and assumption and I needed to test it. So I developed two tools for that and the first one was memory profiler. The memory profiler gave me a good understanding of the sizes and the quantities of the allocation.

[00:09:38]Unfortunately, I couldn't use standard memory profiler as the Linux kernel was old. So, I couldn't touch it and I needed a way to understand what is happening inside the software as I am not the one who wrote it.

[00:09:52]And in general, when we are looking when we have a dynamic allocation and we we are using memory profiler and everything is good, I expect to see something like this. So, I'm allocating and releasing allocating releasing. But I have some upper limit some constant uh constant value I am not crossing it.

[00:10:14]But when I have memory leaks it's going to look like this. So I am allocating releasing not everything allocating again not releasing everything and so on. So over time we see that my peak memory is just increasing increasing increasing in some point I don't have enough memory and I am crashing.

[00:10:36]So let's look at uh let's speak about interesting measurements I did. So the first one was number of allocations per second. Let's assume I have only two dynamic dynamic allocation per second.

[00:10:50]Maybe I don't like it but the system can work with it. Right? Not a problem.

[00:10:54]Let's take the other side. Let's assume I have 2,000 dynamic allocation per second. This is something else. Nothing is going to work.

[00:11:04]Current and maximum number of allocation.

[00:11:08]If we don't have memory leaks, we expect the maximum in some point to be stable constant. But if if it's keep changing over time, we know we have memory leaks.

[00:11:20]Also very interesting to look at the current allocation because we see how it changes over time.

[00:11:28]Also current and maximum bytes allocated similar note the same. And here also the same thing we expect the maximum to be constant in some in some point and if it's not it means we have memory leaks.

[00:11:43]We are not releasing everything we are allocating.

[00:11:47]This is another very interesting current and maximum allocation per size value.

[00:11:53]It means how many allocations of 100 bytes I have. How many allocations of 1,000 bytes I have. Why this is so interesting. Two reason one if I have a specific memory leak it's much easier to catch it because I see a specific maximum value of specific size that is keep changing. So to catch it now I know what is the size of the object that I have licks in it. The second thing it's about memory fragmentation. If all my allocation in the system will be 100 bytes I'm not going to have memory fragmentation at all. When I'm going to have memory fragmentation it's when I have a lot of allocation and in different sizes.

[00:12:39]Okay let's speak about implementation.

[00:12:42]So it was C++ of course and what I did I overloaded the new and the delete operators. It means that I replaced the default implementation with my implementation. And what I did in this new implementation two things the first one I updated my profiler with each allocation and release to do this all accounting I showed you before and I added extra four bytes to each allocation. And in these four bytes I wrote the size of the allocation. Why I did it? Not because of the allocation because when I'm allocating I know what is the size of the allocation. But when I'm releasing I have only the pointer.

[00:13:25]Okay. So I want to know when I'm releasing something I want to know what I'm releasing now.

[00:13:30]And let's look the implementation itself. So this is my new new function.

[00:13:36]So here I'm allocating side plus four bytes. I'm writing the size in the beginning updating my profiler and returning to the user pointer plus four bytes so it could use it delete exactly the opposite so I'm getting a pointer I'm taking four steps updating my profiler and release the memory so how it helped me for two weeks I just clean all these small memory leaks. I was very happy. It's very nice.

[00:14:16]But after two weeks, I understood all of them were minor. I understood this is not my main problem. I really tried hard to find the big memory leaks that explained everything, but I didn't find it. So, it took me two weeks to understand I am in the wrong direction.

[00:14:35]And I also didn't find any memory fragmentation.

[00:14:39]Okay, the second tool I was developing was a lot simulator and the lot simulator helped me to reproduce the real life scenario of thousands of water meters around me in the office. I had around 300 water meter and this is a really nice quantity, right? But you can imagine yourself my unit worked very well with this quantity and the pro and we saw a lot of problems in big sites when we had thousands of water me. So that was the motivation and uh the idea I had was to base my simulator on what I have in in reality.

[00:15:20]So for every real message I received from a real water meter, I was creating up to 100 fake messages from fake water meters. And in this way, I tricked my system to think that it had thousands of water meters around it and not just the real 300.

[00:15:43]So let's look at the code.

[00:15:46]So this is the strct of the message. So there is op code and ID. This is a water meter ID, a unique ID and value. This is the measurement.

[00:15:57]Okay. So we are getting in and the idea here that this is my real software. So this is the software of the unit. So this is internal simulator not external tool but internal but it's internal internal in my software. So I am in the received thread. I'm getting in. This is a new line here. I'm checking it was a Linux system. So here I'm checking if I'm working with simulator or not and it's based if there is a file there or not. So it's very easy to to put a file in the unit and remove it from the unit.

[00:16:34]Okay. While true we are getting in here we are receiving a real message and so we receiving the message is received true. Moving on and then I'm taking the real message and push it to a queue.

[00:16:47]Till now it's even not the simulator and then I have another thread that taking it from the queue the message and process it and does everything that it needs to do. Okay. So this is the simulator part. If simulator enabled true we are getting in and now what I am doing I'm duplicating the original message modifying the water meter ID to be a unique ID and take this fake message and put and put it exactly in the same queue as the original one and now I'm doing it up to 100 time depends on configuration and in this way I trick my system to think that it has thousands of water meters instead of this real 300.

[00:17:36]So look at this. This is the simulator.

[00:17:38]That's all.

[00:17:41]10 lines of code. And it worked. The unit crashed on my table when I simulated 6,000 water meters and put aggressive configuration. Okay, what does it mean aggressive configuration?

[00:17:55]So if I have message A that I need to send once an hour and message B I need to send every 10 minutes and message C I need to send every five minute instead of this message A I'm sending every five minutes message B I'm sending every four minutes and message C every two minutes. Okay I wanted to put a lot of pressure and load on my unit. I wanted to see it I wanted to break and I wanted to fight the battle mix. So that the way I did it.

[00:18:27]Okay. What is special here? No special hardware require required. So no need to fight over the signal generator with the hardware guide.

[00:18:37]A simple implementation. You saw it yourself. 10 lines of code. Easy access to simulator mode without additional building. It's nice because we want it to be easy to use. And then I and I used it all the time. So you don't need to build a a special version so to be able to to test your software and the best of all it consistently crash the system.

[00:19:04]Okay. So I have come across many problem during this two-year development.

[00:19:10]Some of them were big some of them were small. I'm going to share four four of them with you.

[00:19:16]Okay. So the first problem I want to speak about unexplained crashes and that's what I unexplained crash after crash continuously with gaps of several minutes in big sites and when I'm saying big site I'm speaking about the amounts of water meters so more than 5,000 water meters and we just crashed every several minutes we just crashed and of course the first time they told me about this problem I was sure this something about the hardware, right? We are really good to blame the hardware as embedded developer and so they brought me the unit and it's pretty expensive and everything was fine. So in this point I understood two things. One this is not something about the hardware, it's probably something about software and the second thing that this is something about the amounts of water meters. And that was my main motivation to to develop the simulator load I just showed you.

[00:20:16]But now I have in one end I have the simulator load. So I know to reproduce the crash and on the other end I have this my new implementation. So here in this line I put a conditional break point and this condition and my condition was if pointer equal null please stop. And this is very nice because I just ran the simulator and then I was stopping here and this is like it's before I am crashing. So it's very very nice. So you can just look in all the the call stacks and the variables and all the thread everything is open. It's before the damage is done. So it's very very nice and it's reproducible.

[00:21:02]So we can do I I did it again and again and again till I understood what is happening there and that what I found out I found out that two thread tried to create large messages about five megabytes each at the same time. The second one always failed. I didn't have enough memory in my system.

[00:21:24]Okay. Now imagine this issue on your system. How would you solve it? So take 20 seconds to think about it. In in a minute I will share with you how I did it.

[00:21:43]Okay. So the first thing I did was to change it from asynchronic work to synchronic.

[00:21:50]So instead of two thread that are running in parallel, I deleted one of the thread and now only one thread was responsible to prepare and send message A and then if it needs to prepare and send message B always in a row and by doing that I was reducing the memory I need from 10 megabyte to 5 megabytes.

[00:22:14]Okay, so I'm not crashing anymore.

[00:22:17]That's nice. Can I start here?

[00:22:19]Of course no. One, I'm greedy. And second, I wanted to support 7,500 water meter and not just 5,000. So, let's continue. So, we are here and continue.

[00:22:32]And then I start to think about it and I ask myself, why doesn't it happen in all sides? I supposed to see this collisions everywhere. I have thousands of sites.

[00:22:44]Why I didn't see it? Why? why I'm not seeing it.

[00:22:50]So I dived into the code to understand what is happening there and that what I found out that the message size was proportional to the number of the water meters the unit listened to. So when I was listening to 5,000 water meters, I took all their data and created a big epic JSON message and over that I needed also to encrypt it and after that I was compressing it. But in my peak I needed this 5 megabyte. And when I had only 1,000 water meters, I took all their data. But in my peak, I needed only one megabyte. That's all. And I had two megabytes in my system. So if I had two messages of 1,000 water meters, it worked fine. But if I try to do that in big sites, it didn't work.

[00:23:43]Okay. So my solution after understanding that so my solution was to split the large messages into smaller ones. So instead of doing everything together just do it in batches. So instead of again taking everything so I need the five mega instead of this I just every time I took a batches of one up to 1,000 water meters took all their data in my peak I need only one megabyte then I compress it and then I'm taking the next batch of the next 1,000 waterme I need only one megabyte that's Nice.

[00:24:24]The next step I succeeded to do it regarding the cutie. QT as I told you it's a high level framework. So you can imagine yourself that when we are speaking about character it will be two bytes and in C++ it's one bytes. So everything that was written in QT so I didn't touch it but I touched only the message the the function that prepared the message and the message itself. And by doing that I will reducing the memory I need from one megabyte to half megabyte.

[00:24:57]And then I I start to ask myself, okay, if I I have I need only half megabyte. I I don't need to do a dynamic allocation.

[00:25:07]I can do just a static array, that's it.

[00:25:09]I have enough memory, right? And of course, if I'm not doing a dynamic allocation, I cannot fail in dynamic allocation.

[00:25:19]And what I just showed you now was the bottleneck of my system. And after improve doing all these improvements, I could more than double the amount the amount of water meter I could work with.

[00:25:34]Okay, next problem. Not enough information after creation.

[00:25:40]Okay, so I had standard application logs. It was Linux system nonvolatile logs. It means that it survives reset.

[00:25:49]It's it is very nice. But unfortunately as the Linux kernel was old I didn't have a remote access. So in my office I have logs but uh but in reality in the real sites I didn't have but we had something else instead. We have event mechanism that was based on the RAM and every several minute it was collecting interesting events from from the unit and send them to the back end. So it's partial logs and when everything worked fine h it worked pretty well again it was implemented only in the run the problem was when I had this crashes and reset because it just then I couldn't see anything everything just disappear when I had this crashes because I don't have enough memory of course I didn't have enough memory now to allocate the event prepare the message and send them to the back end I didn't have anything I just I was crashing. That's it. And then after the reset, the event mechanism looked in its queue. It was empty, right after reset and I didn't get anything. I just saw the reset and and that's it. And I was very frustrated about that. But then I woke up in the morning one day and I got an idea how to get this information. And that was the idea. When the unit wakes up, I will send the last 100 lines from the application log to the back end using the event service.

[00:27:24]So I call t- n 100 with my log and put it to a string short log and then I just send it as event in when I'm just was waking up reset reason and that's it. So every time I had the reset I I also got in my back end the last 100 lines from the log and it was very very effective because it gives me eyes to understand what what was happening there before the reset and one of the thing I choose to do it's not to put any if there or any condition even though though there are cases that we don't need this 100 lines right we can think about it but I didn't care I want it to be there always and this is a good example to good enough solution. Okay, it's not perfect.

[00:28:18]There are age cases you can think about it but for me it was good enough. 80% results 20% effort.

[00:28:28]Okay, next problem. Occasional data loss during network network issues. H very popular if you work with network. Don't think there is one that is not familiar with this data for transmissions remains in RAM awaiting more processing.

[00:28:45]Okay. So what is the problem with that that in case of unstable communication it start to aggregate in the memory and it takes a lot of space and in some point we don't have enough memory and we are crashing and the second thing that sometime we have reset like a bad weather or some short maintenance event and then we are just losing everything we have in the run. We don't have any control of this situation.

[00:29:17]And my solution was to disconnect the logic from the network. Instead of doing everything in the same place, we have this tree decision if we have a network or we don't have a network what we should do. So instead of this, I just put it aside instead of this. That's what I did instead. So I created two thread. One responsible for the logic.

[00:29:38]So he's the one that knows when we're supposed to send the data. So it takes the data create the message and put it to nonvatile queue. Nonvolatile it means it survive resets.

[00:29:52]And the second thread responsible for the sending. So is all the time goes to the queue see if there is a message there and if there is a message there it takes it. It takes it and try to send it. And if it succeeds, great. You just go back to the same queue, remove the la the last message you just succeeded to send and take the next message.

[00:30:14]And if it fails because there is a timeout or a bad acknowledge, it's okay.

[00:30:19]It will just wait another minute and then go back to the same queue and take the message again.

[00:30:27]And the idea here that we always execute exactly the same loop and we store the results in nonvolatile memory regardless of the current network status. Network status is very tricky because there are two reason why one you can check the the network and everything is fine fine but now in two milliseconds from now it's in a different status. So it didn't help you that didn't help you that you check the the network before.

[00:30:58]And the second thing that maybe around you you have a really good network but not around your back end. So you are sending the network everything is fine but you still you are not going to succeed to send this message.

[00:31:13]Okay. Implementation achievement one.

[00:31:16]Maximum data loss is now limited. Think about it. If once an hour I am taking the data, put it to nonvolatile memory and I am deleting it only when I'm sure that the other side received it. I am never going to miss more than one hour of data. It's super strong. And the second thing, I'm not being sensitive anymore to network error. If for 23 hour I don't have a network and then I have one hour that I have a network everything will be sent this through one hour and I can do it a day after day after day. Or let's take another scenario. Let's assume that now for two weeks I don't have a network. It sounds silly but but sometimes someone forget to pay for the SIM and then after two weeks he will pay everything will be sent to the back end.

[00:32:08]you are not going to lose anything and indeed after implementing this change we no longer experience any significant data loss and this is a good example of always using the same scenario to avoid any age case.

[00:32:25]Okay, last problem and this is more embedded problem. Suddenly no meters are being detected anymore.

[00:32:34]So this is my system and next to me I had another unit I didn't mention before the RF receiver and it it was the unit that was connected to the antenna itself. So it received the RF frame and translated to message and it sent me to the 3D concentrator using UR. This is a very popular or common serial port and uh we had site and everything worked fine for weeks and then suddenly just we stopped receiving frames and what was interesting that the system doesn't recover on its own. So nothing was changing. We just have the unit with hundreds of meter everything is working and suddenly we are not hearing any any meters anymore and we are still connected to the back end. So the backend knows we are alive but we are not sending any data and customer of course complain because they didn't get their data and every time it was a different unit though I couldn't blame the hardware right I told you I I love to blame the hardware but I couldn't we didn't understand why it happened and the good news were that a controlled reset solved the problem so when we just reset ourself everything goes fine again and that's the reason we understood it's something about us and not the about the the other unit because control reset solved it and that was the first solution and my team did it before I was joining them and the idea was to limit the damage. So if we were detecting this one hour of silence, okay, we just reset we just reset the software and it solved the problem. After several months, I was working I was working in the project. I started asking myself why we are waiting one hour, right? If it not recovered itself, why why to wait so long? So I found a new sweet spot of four minutes. So I kept all the logic. I just changed the number there the threshold and my customer became very happy because now if before once in a while they saw this one hour missing data now they didn't see it. They have only four minutes and they didn't didn't notice it.

[00:34:58]I was less happy because I knew I didn't solve anything right. It's like a nicer band that what I did here but I needed to move on. So I moved on and then months later I was looking in logs of a unit that had this silence issue and I realized that the RF receiver the unit next to me reset itself not me the other unit. So what was happening that I was receiving frame one frame two frame three and then the other unit reset itself and I got some junk. So my your driver wasn't able to align again on the next frame even though that the other unit sent me frames all the time. And after I was understanding that I could choose a real solution. So I initialized the UART when I detected silence instead of a control reset. And the reason that control reset work because that after the control reset we were just waking up and then we were initializing all our driver. So instead of initializing all the driver I initialize only the URL and it worked and sometimes it takes time to identify the real problem. So we use temporary solution in the meantime.

[00:36:20]Okay let's speak about monitoring.

[00:36:23]When I started working every several days, we got complaint that some of the unit is not working and then we went to the event service and to the back end and we try to figure out what is happening there and most of the time we didn't understand what is happening there and most of the time it was like okay just reset the unit and 99% of the time it worked and after several years I was exactly in the same position and I was too super frustrated about it because I all the time got complaints. No one likes to get complain on its work. I didn't change anything yet and I understood I'm not moving anywhere because I still I'm exactly in the same spot. I don't understand what is happening. So I decided to change what I'm doing. So maybe the results will be different. So I started to look actively at the event service and the idea was that maybe if I could look there actively I could find some patterns and detect some pro problems and maybe I could reduce the number of the complaints I'm getting. So that was my motivation. So I started I started reading events for specific units but it was very uncomfortable. Okay, it was a simple UI with I don't know 20 lines in the screen. not very comfortable to understand the system. So I was complaining as a good developer and a two nice student that sat next to me suggested to write me a script to retrieve all the events and save them to a local text file. I told them this is a great idea and they did it. But then I had a really long local text file. It's not very nice to read it. H so I understood I need to do some improvement. So I wrote a Python script to analyze the text and export it to a CSV file. And the reason to do it for export it to a CSV file that I could put it in Excel and use the sort filter graph abilities of the exel when I'm processing the data.

[00:38:32]Okay. So this is an example. So I was sending data once an hour. So now when I'm putting it in Excel, it look like this.

[00:38:42]And in Excel it's easy to calculate gap and then maybe to find that I have error that when we are doing it manually it's very easy to to skip and what is nice here that I don't need to do it only for several hours or for one day I can do it for days and weeks and months. This is super strong. So you can just take your data go over months and make sure that your system behave exactly the way you want it to behave.

[00:39:16]Another example memory usage. I was very sensitive to memory. So here every 10 seconds every 10 minutes I was sending the memory usage to the back end. So it looks like this.

[00:39:29]And now I can graph it. And this is very nice. But this is not effective because this is like a very short time. But now again think about it days and weeks and months how powerful it is to to see your memory usage over time in your unit and to find problems and errors.

[00:39:49]So I ran the script every morning analyzing them for 10 minutes daily and then if I found some problematic unit I was connected connected the support guy to connect the customer and also all the time verified that what I'm doing is is right and I all the time use it to to improve my software and over time I added more units and more parameters and then I needed to take a step back because to look at graphs it's nice when you have one unit, two unit per unit. It's not working when you have tens of units or hundreds of unit or thousands of unit. I'm not speaking about four. So that's what I did instead. So I did a summary row for every unit in my in my system and I have thousands of units in production and and when we are looking this like this it's much easier to catch things like here my fware is not updated and if I just just looking in in its log I'm not sure I was seeing this right because it's very easy to mix but if you see that all the others are f_sub_2 and this is f_sub_1 say okay interesting why this is different errors so if all the others are zero and I have one unit with 87 error I know I have a problem there so it not explain me why I didn't I have a a a problem there and maybe I should maybe I will need to go to the lo to their specific logs of this unit but it direct me to this unit to look at this is also interesting thing last time. So in the end of the day my unit the ready concentrator all the time supposed to send data to the back end.

[00:41:41]So and this is the the last time stamp that the back end saw my unit. So it's supposed to see it all the time. So if it doesn't see it we have a problem. So here we see that all the other last seen around 10 and this is around one it means that there are nine hours of missing data.

[00:42:02]And in this case, I was connecting the support guy and 99% of the time it was or a pro or a power problem or a network problem.

[00:42:14]Over time, we automated the script to run them at night and receive result by email in the morning morning and we did some summariz summaries so we won't need to go over again thousands of of monitoring of results.

[00:42:30]Okay, let's speak about testing.

[00:42:33]After several months, I I felt I'm ready to release the software. So, I came to my team and asked them, "Okay, how did you release before it was here?" Right?

[00:42:43]You already released this once. And they gave me a really nice test paper, manual test paper, 100 pages.

[00:42:55]This is a real story. and uh and they told me we were three engineers and we work on this six months and I was just looking at them and it was Denmark so I really tried to be polite so I say okay and I just thought to myself okay I'm only one person and it's not going to happen but I need to test myself right so let's did what I did in the first thing I did was to rewrite a test paper from manual test to automatic test It mean to remove some of the test and modify the other test and because it was before the AI it took me a week and also in this time my team leader then had another team of students. So he gave me this team to help me to write the test and they are the one that did the implementation. I was guiding them and this is two decision we took regarding the implementation. one to have two pipelines, one for long test and one for short test. Some of my test took hours, some of my tests took took minutes. So I didn't want the long test to block the short test. And that's the reason why we did two pipelines and the second decision was about to run them 247.

[00:44:16]And the reason to do it, it's not because I think that the second time we are running exactly the same test with the same software is going to be different from the first time. I'm not there. It's because I wanted to promise that it always be stable and not and not flaky. So after we h got a new test, we just put it to the pipeline and if it didn't behave nice and behave nice, it's even one time it was flaky. I don't I didn't count to three. I stopped in the one. We just removed it from the pipeline. We gave it back to the student and ask him to prepare his own work again. And in the beginning it was little bit hard but in some point they understood the idea of of this stability that we are not go going to skip this and it became very very stable. So we always knew that if we see green this is green and if we see red this is red.

[00:45:16]Another thing I did was regarding system tests. So I had around 10 units that were running with load simulator and aggressive configuration 24/7 and another 10 units that were running regularly with standard configuration 24/7. And the idea here is not about running them because then we are just testing the power. It's not interesting.

[00:45:39]What is interesting is the monitoring.

[00:45:42]And I'm when I'm speaking about monitoring, I'm not speaking only about searching for errors. It's also about verifying the positive behavior. If we supposed to do something once an hour, let's let's verify we are doing it for long periods of time.

[00:46:01]And general about testing, the question isn't if there is a bug, it's who will find it first. That's it.

[00:46:10]So, how did it end?

[00:46:13]Eight months later, product without any resets and it supports up to 10,000 water meters instead of 5,000.

[00:46:24]Two years later, test automation, half automatic monitoring, and I'm saying because we still needed this person to go over the reports and connect the support guy if needed. I was able to release every two to three months instead three engineers six months before me and I was able to add features and improvement we couldn't put before because it was unstable.

[00:46:55]Take all messages.

[00:46:57]Measure everything. Avoid assumptions.

[00:47:01]Remember my first impression how wrong I was.

[00:47:06]Write robust software. We cannot control the inputs and the environment. But we can control our behavior. We shouldn't crash no matter what. What it's okay not to do what we are requested to do if we can, but it's not okay to crash.

[00:47:26]and examples from my talk. Split large messages into smaller, worked with fixed sizes, decoup logic and network operation.

[00:47:38]Reduce the number of thread. The most challenging bugs in the system arise from multiple thread running simultaneously. If it's if you don't must to put a thread a thread, don't put it there.

[00:47:54]monitoring. Be proactive. Don't wait for customer complaints. It's never getting better because someone is complaining about it.

[00:48:04]And last but not least, work with simulators. Don't wait until the end of the development. Start with it. And especially now with the AI time, no excuses why not to use simulator.

[00:48:19]H. Thank you.

[00:48:25]questions. I'm not sure we have time.

[00:48:28]Yes, we have plenty of time. Any questions or something?

[00:48:35]>> Yes.

[00:48:36]>> Yeah. Um, so I have a question regarding um regarding the uh thing that you shown for like me management. So uh what year was that and uh um I'm sure you must have tried the uh hardware upgradation as well. So um like at what extent could you uh solve the problem with hardware upgradation and uh where you have to go back to software to solve the problem of scaling?

[00:49:08]>> Again I'm not sure I understood the whole question. Um so um okay so first of all like what year was the um >> it was 2017 uh when I started to work on that project. I finished in 2019 but the product already was there 7 years old.

[00:49:29]So it was it was created in 2000 2010.

[00:49:36]>> Okay. So you mentioned there was like only 30 m of RAM available. Yes. Um >> it I think in general I will think about RAM. It really depends in which system you are coming from. There are some system that 32 megaby RAM. It's it's a lot. Okay. It's like it's imaginary.

[00:49:58]There are some system you have half megabyte and say thank you that you have this half megaby. So it really depends.

[00:50:04]If you're coming from a place that you think about jiggers, you just look at this and say, "Why what are you kidding me? Well, this is like a 32 megabyte RAM." But again, it really depends on on the unit. There are a lot of different unit with different RAM. Depends what you need to do. Depends if it's connected to a power or not.

[00:50:26]So um did you try you upgrade the RAM to something like 1 KB layer and how it affected affected the scaling issues >> the scale issues >> like uh with the meters number of meters you can support right >> again I'm not sure I understood the the question >> okay so I would say like uh you started with 30 MB RAM for the system right >> yes >> um and with 30 MB RAM it was able to support certain number of meters, right?

[00:50:57]>> Yes.

[00:50:57]>> So, um you must have tried to upgrade the lamp to something like >> I I couldn't upgrade. Again, this is a specific hardware. You cannot touch it.

[00:51:06]This is like a a live product that already was installed in thousands of sites. It's not something you can you can touch at all. So, and again, the the the problem wasn't the amounts of memory we had. it was that when it was first developed the wrote it to to support up to 1,000 water meter and it worked fine with this number. So everything was good. The problem that in some point h they just decided that we can support up to multip multiply it by almost eight and say okay it's just supposed to to work out of the box and it didn't it it's never tested to this number of meter so of course it didn't work. So that's the reason for this talk and all the optimizations I needed to do.

[00:51:56]>> Okay. Uh so you did load testing. Uh what was the most popular issue did you find and were there like patterns on there like what were the main issues did you find in the low testing >> in in the low >> um like low testing testing the system with more number of meters right >> again I'm not sure I I understand the the question again >> um by doing load testing what was the main error you And why is there are there >> ah okay in the in the in the lot simulator you are speaking okay again the the idea that I had problem that I could only reproduce when I had a lot of thousand meters and that's what I did in the in the lot simulator and I tried to do it as simple as possible you saw it yourself um again I was a one engineer uh So it's not of course you can develop a really sophisticated simulator. There are a lot of ways to write simulator but again I was only one engineer and I wanted to have results best and that's the reason I did it this way.

[00:53:17]So instead of inventing a some other unit that can send it or use a signal generator or doing anything that it's more common in the industry instead of this I did something that it's little bit dirty in one side but in the other side it was super effective and very fast to implement. You saw it yourself. It's like 10 lines of code. It's nothing it's nothing to write. just to figure out the idea. So what took me time is to figure out how to do it. The implementation itself and I think all the examples here of the code I showed you it's very very simple because again I was one developer and I think my expertise is not about writing a lot of code.

[00:54:08]>> Yes I have a question. So it looks from uh like your own uh overall message that you brought uh there are good practices in fact in this place best practices like why do you think most of your advices are related to best practices so they're very simple uh so why do you think hadn't happened before you >> I think that be I think several reason one when they first h wrote the software I think most of them weren't embedded developer most of the original team weren't embedded developer and that's the reason why they choose to write use Q for that they they just write it from a functional perspective they just want it to work and they test it again for this 300 around them and it worked fine and and that's it and and it was also I think good enough. Okay, this um ready concentrator supported up to 1 million water meters. It's a lot and the system now I think probably it already replaced with the new generator but for many years it worked. And the second thing I think about good practices there are some things today that we are looking at them and they are trivial. Okay. Uh writing a I don't know Python script to analyze logs. Today we are doing it with the AI but even using a Python Python to analyze things. This is like a very common practice today but it wasn't 10 years ago.

[00:55:52]It it was much less popular. I don't know if you remember most of us are old enough to remember this. So some of the things that today we are looking and say oh this is trivial it wasn't and I think in general in general in software there are a lot of things that's after it takes a lot of time to solve them and after we see the solution solution we say oh that's trivial so why didn't they solve it before me right I don't know how many times people told me that and I say okay so why didn't you just solve it >> betric is reaching has a lot of constraint here. I can tell he's wanting to ask this. Did you consider using fixed size pools for repeated allocations of the same size?

[00:56:39]No. Uh the reason I didn't want to do it is because this is a lot of uh programming. This is a lot of writing code. For example, take my um the the event monitor. Okay, I could write it in in a nonvolatile way, right?

[00:56:56]But I choose not to do that because I choose like a a faster a faster a faster solution to implement. So of course maybe if I couldn't if I didn't succeed to do it the way I did it, maybe this is another way I could do that. And maybe this is even a better way. I don't know.

[00:57:16]I never chose it. I chose I chose one way and it worked for me. And uh I'm not saying this is the best the best the best path but again for me it worked.

[00:57:28]And again the thing is that I really optimize my time and the amounts of code I wrote. Again it was a a working system. I didn't want to touch I wanted to touch as little as possible. I didn't want to refactor everything.

[00:57:45]And again I don't know if I was refactoring everything. it was better or not. But I I choose not to do that, not to to take everything and say, "Okay, this is garbage and put it away and and write it from scratch."

[00:58:01]Okay. So, so I prefer when I didn't need to, I prefer not to write a lot of code.

[00:58:08]I'm following up on the development practices question. Um you are painting a a picture of this organization um where if it works that's good enough.

[00:58:21]I'm wondering how did you convince management or whoever uh to give you the time and the people and budget whatever to to make all the like I'm going to invest a bunch of time also when writing our tests. How how did you sell that idea?

[00:58:39]>> The when I'm I'm I was joining the team, they were very frustrated.

[00:58:46]Most of the original team left the company. Some of them left to different projects. Most of them just left the company because they were so frustrated.

[00:58:55]They put me there myself to do that. and and again they were so frustrated and they didn't know what to do because again this is a project they sold to the customer and it didn't work and because they were so frustrated they just were happy and the first time I I they give they give me some feature to put there and I told them it's not stable I I cannot touch it okay I cannot put an extra feature of something that is not working and then we took the step back and I told them okay I think I can do that and and that what I was focusing and again I think they already were developing the next generation so all the other team members already work on the next generation I and I was the only one that worked on the legacy product and they just want me to all the till the new one will be ready so I think it really depends how much what they need what they can lose So it depends on on the pain. Okay. So if you are trying to solve something that is there is no pain for no one no one will give you the resource right but if it's painful they will give you the resource because again depends alternative. So in your case, I don't know if it's pain, it's it's your pain or also someone else pain to say, okay, we are developing a very slow because we don't have tests and we and I could double the velocity of my of of me creating a new feature. So I think maybe it will help to convince them.

[01:00:37]Anything else?

[01:00:40]>> Yeah. Okay. Thank you very much and have a nice day.