Rubik Pi 3 x PhotonVision

[00:00:00]All right. Hello everyone. Thank you for being here. Uh my name is Ramy. I gave a talk kind of similar to this but it will be way more similar to the next talk last year. Uh it was the Rubik pie and it was running like an experimental build. Um very custom software and we said this is exciting. We really want this part first. And a year from now later, it's really exciting to be here with Proton Vision, a couple teams that helped start this. Actually, just because I know some of you, can the people who helped beta test the Rivic raised their hands? Maybe just the only one.

[00:00:38]Okay. Um, but yeah, this has been really exciting. I'm really glad uh Vision decided to support 53. um as Qualcomm it's really exciting to see industrial tools in you know first hands and it's really nice to see that you've been at a good price they can use myself >> just another introduction my name is Larry I also work for quality right now I work for qualcom in my day job I do location technology we do Wi-Fi cell Bluetooth locations for all kinds of IoT devices cat trackers, pet trackers, the the body cameras, watches, things like that. We do location. But one of the cool things about the BCOM culture is it's a little bit open in culture. So we can go work on other things if we find it interesting. And so given my background at first, I've been doing first mentoring for probably 15, 18, 20 years.

[00:01:36]Um I was able to hook up with the STEM team to bring things like this into first and some other sort of exciting things that are coming up. So, it's pretty cool to be able to do that. As part of my my first experience, I worked with sabotage RC 1640 and we became one of the beta teams. We talked about that um what their experience was during the beta process and their advice. What we did was we we established the product.

[00:01:59]We're able to bring vision on board provide really of a worldass capability for for vision processing. And so then we established a major program. We asked some teams to be involved. We established socalled discussion with those conference calls with all photon vision engineers amongst the teams and really let them sort of digest what was going to happen. Gave them a little mini test plan. We asked them to run. They ran their own things.

[00:02:23]They ran on their own robots. We got their experiences back. The good, the bad, you know, the great, the ugly. We got all those back. We incorporated their feedback. They provided that feedback back to Qualcomm. And then we incorporated that before the before the launch. And so that's the process that we went through. and they're going to talk a little bit about what their experience was during that. So, >> all right. Hello everyone. I'm Sam Bruin. I'm a Photon vision developer and maintainer and I'm going to be an intern this summer. That's how I got connected with Rubric.

[00:02:54]So, first I'd just like to cover the scope of this presentation. Uh we'll be giving an overview of Photon Visions features and then during the presentation we'll be focusing on object detection. Funtime Vision gave a presentation a couple years ago on April tags and local mobilization and you can find that on our website. Then we'll be getting our advice from current students team uh 1640 sabotage and then finally we'll be talking about some of the future directions photon vision is exploring.

[00:03:22]So first vision basics uh going over some of the history of vision and FRC. We started with five colors and that was very simple detection. you just saw a color and you tried to go towards it. After that, we moved into retroreflective tape. This is where you would see those line lights, that blinding green light. I'm sure some of you in the room know what I'm talking about. Uh most recently, we've had April tags. Uh all of you, I'm sure, have seen them on the field, know what I'm talking about. Some of you may even have been around when they were introduced those.

[00:03:57]And then most recently, we had object detection. This is fairly new. Uh this is what I'll be talking about most in this presentation.

[00:04:06]Uh through vision, there are two main goals. The first is localization. This uses able tags to figure out where your robot is on the field. Uh and it'll help you uh perform various actions. Teams have used it to implement their shooting on the move algorithms this year. Uh and place it helps you align and place your objects. The second is object acquisition. This is where that color shape protection and object detection comes in. This is going to be most of what we're talking about today. So now we're going to segue into some object detection examples. The first is very similar to some of those uh very early naive implementations. Uh you're simply finding the object, turning towards it, and then driving forward to try and fire it. Uh this would be implemented very simply. uh and we have an example photon vision repo that shows us how to do this. You can find that in our documentation as well. Uh secondly, you can use uh the object detection to localize an object. If you can figure out where it is on the field, then you know that it's at ground level. You can perform some trigonometry to determine how far away it is from you and from that plot it point on a field 2D or a similar object and then place it um so that you can uh use that in your code.

[00:05:32]This has been used by teams in crescendo when we had those races to the center line uh teams trying to fight for those notes. Often teams would use this to try and figure out if there was a note there or not, whether they should try and get that one or proceed to a different note on the center line. Uh the next piece that I wanted to cover was training your own model. Uh this is I think one of the coolest things a team can do with regards to vision, especially given how new it is and how few teams have actually explored this ground yet.

[00:06:06]The first step is data acquisition and then labeling. Uh so you start by taking a lot of pictures and I mean a lot in the thousands. Uh you want to make sure these pictures are from varied environments. Uh especially when you're on the field. I'm sure most of you have seen that lighting conditions will vary.

[00:06:25]And if you don't get pictures in a wide variety of lighting conditions, your model won't work in a wide variety of lighting conditions. Uh this is especially important uh when we're at worlds. Uh because as you all know, lighting will be changing uh throughout the course of the event and going into events. And then when you're labeling uh you do that on a variety of different platforms, you can upload all the images to Edge Impulse. There's a link here kind of first robotics team to use. You can use chat. This is a self-hosted application or you can use robo flow.

[00:07:02]And then uh while labeling if you have multiple people labeling it's important to create a guidelines document so that all of them are consistent. If you have inconsistent labels that can affect the accuracy of your model. Once you've created all of your uh images and labelled them uh you need to proceed to train your model. Uh the first focus is the type of model you want to use on group five supports go V8 and go 11. Uh it should be noted that we do intend to add support for other models in the future but that have not yet come to pass. Uh there are a couple of different options for training your model. You can do it locally uh just using ultritics. This is typically aligned with a chat labeling approach that's all hosted. uh you can use edge impulse to go from finish to uh from start to finish train your model there as well. Uh it should be noted that if you do so it'll a different format than what photon vision currently supports though we have plans to support it in the future. Uh that format can be converted so that it can run on photon vision and that will be in our documentation very soon. And then finally, if you're using Rooflow, you can also train your model on there, but it should be noted that you have to be on a certain page tier to be able to download the weights for that train model.

[00:08:22]Uh once this is done, you'll need to convert your model. After training, you typically get uh pietorrch weights. And uh while these weights are very handy for uh doing math with the model or trying to train it and convert it, you can't really use them for inference on the edge or on smaller co-processors like the Rub Pi. So what we do for the Rubik Pi specifically is we convert it to TensorFlow light uh and then we can go ahead and upload it to Photon vision.

[00:08:51]for other platforms like the orange i5.

[00:08:54]You need to convert to rkn which is the rapchip format. Uh and there are notebooks for uh their jupyter notebooks for doing this located in the photon vision repository. Uh you can run these notebooks either on uh Google collab or loc. And then finally uh you need to upload your model and once that's done you can pack objects whatever you may have trained.

[00:09:19]Next, uh, we're going to have team 1640, Sabotage, help tell us about some of their experiences.

[00:09:30]So, we start off with benchmarks and performance data. Um, so we benchmarked against the uh Orange Pi 5 and 85 Max. Uh, I think the results are pretty good. Um since then we've made some changes to power delivery so it might not be accurate from then um so what we did with power delivery is we tested around with different voltages so initially we were running 5 volts that's what we met with um but we found that 12 volts the the rated power um is actually a lot better um and uh the peripherals like it will probably drop when you try to power it with 5 volts or anything lower than try as well.

[00:10:32]So then we tried to flash corrupted flash because I was restarting it and was updated at the same time. Um, so that was a little bit bumpy because, uh, we were using T Flash at the time, which was a pretty clunky flashing software. Since then, Qualcomm has released Qualcomm launcher, which is much more streaming.

[00:11:09]Then we were tasked with object detection performance testing that so we noticed that bounding boxes weren't getting performing properly. So we gave some feedback to Sam on that fixed. Um after that we had all the um things ironed out and moved on to um showing the golf like showcasing it at a competition like I think we're at the Bay Area competition.

[00:11:47]>> Yeah.

[00:11:51]So, um this is the USB that you have access to which you can plug in cameras to. There is two USB 3.0 ports. Uh they share a bus. Uh and then there's also a USB 2.0 port and a USBC uh 3.1 gen one port. Um for personally, we recommend uh plugging one camera into the USB 3.0 O port and uh one camera into the USB 3.1 gen one USB 64 port as that um was the best way to that was that maximized our resol we were able to maximize the resolution and frame rate with those two and uh we settled on this power supply to power two root cause actually on a robot and uh so this is a drop buck boost power supply buck boost converters are nice because uh Even though the battery may drop um it keep it at a nice stable voltage so that the pies are happy and um this is not a USBC specification the one we're currently running because um it's just giving straight 12 volts. If you do want that uh extra power security uh you should you could uh power a USB PD board so that you can do the USB negotiation or you could also uh just directly power the pie with 12 volts for the base.

[00:13:14]>> That is a picture directly from the robot powering as he said two two USB 2 Rubic Pi. So that's 12 volts at just about an amp. So we're running two Rubic pies at just about 1 amp steady state.

[00:13:26]So that's that's power you should expect.

[00:13:30]Yeah. So one of the most important parts of our project was creating documentation as we started off as a beta team. So we didn't have a lot of documentation.

[00:13:39]So we essentially made a documentation guide. So help teams out. It includes a setup video as you can see.

[00:13:49]Also, it's really important because you also have to correct install because if you mess that up, it can damage your request maybe.

[00:14:01]Make sure you're following exactly as it stands.

[00:14:04]>> And I do see some hands in the audience.

[00:14:06]So, I want to let everyone know that everyone will be taking questions at the end.

[00:14:16]>> All right. And then uh to uh sort of go over what cams we used, we use um cam for our um object detection. I use the NV 9281 specifications. Uh that's a color camera. It's 1 megapixel 100 frames per second. Um on photon vision, it will show but uh I think it goes down to around FPS. So there but uh we used actually three different uh tech processing cameras. So we used the Articam Ruby 981 which is 100 frames per second and that's only 1 megapixel. Um, we use the OB 2311 which is 50 frames per second at 2 megapixel which is actually a little bit better than the OB 9281.

[00:15:21]>> Regardless of the dropped FPS, the 2 megap difference is pretty spectacular. Um, and then the 30 cam is actually what we use on our robot. It's just uh really robust.

[00:15:39]It has an aluminum case with locking USBC connectors. Um which is really nice on like a rough field like this like this.

[00:15:50]>> There is one tabbo on that though. Um the $50 RGAM processor the the 100 meg one megapixel is a 9782.

[00:16:00]That was a copy paste from the previous slide. So that second camera there should be a 9782.

[00:16:07]The thrifty cam just the important thing there is to us if your teams are entirely different but the RG cam doesn't come in a case and you have to print a case which is easy to put it in but sometimes you have students probably not these three. They walk around with the thing not in a case and then that's not good for the likes of the camera.

[00:16:27]The thrifty cam comes in aluminum case and they don't have an opportunity to walk around without the uh the camera in a case. So the cam is a little extra money comes in aluminum case and has a lock USBC cable. It uh it pay for itself and we just had a little bit of sort of miscellaneous like tips and tricks um just so you guys have a smooth experience. Um, we recommend that you use static IDs on your because we had some uh bumpiness on certain networks.

[00:17:04]Uh, they might not have NDNS, but if you're connected to a Wi-Fi network, it might not have NDNS. So, your host name might not show up. So, if you have a static IP, you'll be able to really easily identify.

[00:17:20]Um yeah, that's especially um significant if you have multiple co-processors uh because then you can differentiate between the two. Um yeah, super easy to set up multiple processes as model tables. So there's definitely configuration there if you're having multiple processes. Um and what we do uh for our camera calibration actually is we import and export the camera calibration JSON and we can store them in our GitHub so we don't have to calibrate the camera every single time.

[00:18:07]um change some area here that that's definitely spent we definitely spent a lot of time in the bit calibrating cameras.

[00:18:23]So that's just to save some time. Um, and then just a little anecdote, protonation seems to run a little bit slower on Firefox like a little slow on Fire. So, we recommend you use like a Chrome based browser.

[00:18:48]So, um, the Room Pi 3 is $109, I think. Um, and it comes with a fan, so it's pretty great. It competes very well with the Orange 5 and a lot of the other processors in the price range.

[00:19:07]And I'll hand it off back to Sam.

[00:19:11]Thanks. Uh, let's give a warm round of applause.

[00:19:24]So, next I'm going to talk about some of the upcoming features for Photon Vision.

[00:19:28]These are things we've been playing with this season.

[00:19:31]We know that not all of them are expected to uh land in the 2027.

[00:19:37]The first is what we're calling YOLO tag. Uh this was created by a mentor from Koga Orange team 3476.

[00:19:46]Uh the way yolo tag works is we have a machine learning model. Uh this is also a yolo model that will look at the image and try and find any April tags. If it sees an April tag, uh it gives us that bounding box and then we pat it and that creates the light blue boxes that you're currently seeing on the screen. We then run the uh University of Michigan April tag algorithm on the regions inside of those blue boxes called ROIs or regions of interest. Uh after that we can detect those April tags. And the reason this is so much faster almost twice as fast is because we're able to skip processing that vast majority of the image using the University of Michigan April tag algorithm. And uh the thing about this algorithm that's important to keep in mind is that it does scale uh somewhat exponentially uh as the size of the image increases.

[00:20:43]So if you can shrink that region it's searching, it makes a huge difference.

[00:20:50]The next thing that we've been exploring is oriented bounding boxes. If you take a look at that picture there, the red box is what you would typically see if you had trained an object detection model on a cage from Reduscape 2025. The blue box, which is admittedly a little bit hard to see uh given that the cage itself is blue, is actually aligned with the cage. And you can see how it takes up much less area and gives you a much more accurate detection. Uh this helps particularly when there's something you want to align to or if there's something oblong that you're trying to determine the orientation of last year. Uh this would have been incredibly helpful with coral particularly for those robots who needed coral in a specific orientation in order to intake.

[00:21:39]And then another thing that we're exploring is gteam.

[00:21:44]Uh this uses backto graphs to improve post stability and uh we'll be passing in all of our April tag protections to it. Uh fair warning I'm not the one working on this. So if you have any questions about it, uh I can try my best to answer them, but you may be better off going to the Photon Vision Discord, which I'll be putting on at the end. Uh this may or may not be landing for 2027.

[00:22:10]The timeline has yet to be determined.

[00:22:13]Uh but we are hoping that with system core we'll be able to run it uh on the MRC and that shift speeds up quite a bit.

[00:22:22]Uh finally I'd like to thank all of our community developers all the teams who have worked with us uh and tested out some of our new features uh along with any of the contributors of vision and then I'm going to open the floor to questions.

[00:22:41]All right. This first question is more.

[00:22:43]You said the power supply 12 volts drive one amp. Is that with two cameras processing or is that just sitting?

[00:22:50]>> Yeah, that was with three cameras.

[00:22:52]>> Three cameras. Yeah, three cameras. I think it was like one to one.

[00:22:59]>> Three cameras total. Two on one and one on the other. Three cameras total. Not for each.

[00:23:04]>> Okay.

[00:23:04]>> So, it's three cameras total. Two and one. We originally were going to run all three on one, which is fine. that would have worked. We were having an unrelated issue that we thought may have been related to that. So we we switched to be two of one, but we could have run all three cameras on on one. Right.

[00:23:20]>> So then would that the lower current be due to the algorithm and processing for like a tag?

[00:23:26]>> No. So the there are a couple teams running tech this season. I believe is one of them, but to the best of my knowledge, sabotage is not running.

[00:23:35]>> That's correct. All right.

[00:23:39]Will the Rubic Pi be better supported than other co-processors like Orange Pi or something like that?

[00:23:45]>> Yes. So the question was whether the Rubic Pi is going to be better supported than other co-processors, excuse me, than other co-processors like the Orange Pi. Uh I can't necessarily speak to the level of support each individual co-processor will have. Um, I can say that the Ruby Pi, uh, we've been able to talk with Qualcomm if there have been issues and they have been very responsive with the Orange Pi, we haven't really had any information, any contact with Rack Ship. Uh, so we do have that going for us. Um, I can also say that the Orange Pi 5 and Chris, correct me if I'm wrong on this, is going to be exiting production soon.

[00:24:24]>> Um, so it they may become much harder to obtain. So for that reason alone I would expect uh developer to work on them. So >> one one thing uh on the power efficiency side because I work at Qualcomm I have to say this that is actually like what what we most mostly about power efficiency with our chips is like one of our biggest selling points and that's why we achieve that would you recommend but how many recommend Yeah. Rub.

[00:25:05]>> So the question is how many cameras I would recommend to run on the Rubic Pi?

[00:25:09]That is uh highly dependent question based on your situation. Uh so the first factor here is your USB bandwidth. Um there are aspointed out uh three USB buses that you can use. You have the USB 2.0 port, the two stacked USB 3.0 ports and then the USBC port. Uh if you have a camera on each one of those, you most likely won't experience issues with bandwidth. Uh if you have three cameras all running uh very high resolutions, all doing input tag processing, you might run into some CPU issues. Uh if you have three cameras, one of them is running object detection and you're running two input tag processing, that would likely be fine. And I can't speak to how YOLO tags will affect this as we haven't actually performed any benchmarks or collected metrics. Uh, and again this does depend on the resolutions you're running as well. I hope that answers your question.

[00:26:05]>> Yeah, our our our beta testing was exactly that. We did one camera with object detection, two cameras with April tags and we and when we suggest three cameras, we suggest the USBC, the vertical USB and one of the USB, you know, horizontal ones because those two, as they as the student said, share that bus. So having two on there can cause bandwidth issues. So you want one of the horizontal and one of the two vertical.

[00:26:29]USBC and USBA.

[00:26:31]>> And some of you may have seen that there have been issues with cameras in the vertical USB port uh switching. So they'll show up as a different camera in the Photon Vision interface. Uh this has been fixed in the latest uh release of Photon Vision in the latest uh image for the Rubic Pi.

[00:26:51]>> Yeah. What resolution were you running your cameras on?

[00:26:55]So we run um we did all the different resolutions for benchmarking on the robot right now.

[00:27:06]>> Well first >> uh yeah just adding up to that actually I know we talked about our two root pie setup uh and having cameras split between the two. Uh, one of our review pies is running and uh, the other piece for uh, the can reasons, but uh, we run it at a lower resolution whereas uh, 50 cams do generally cams like to put together pretty good setting over high with high bandwidth. um at high resolution um whereas the RD10 higher frame rates allow resolution. Thank you. And Jake noted that their object detection camera is running at 640x480.

[00:28:01]I'm not sure if this is common knowledge or not, but the object detection in the photon vision back end actually runs at 640x 480. If you have a resolution higher than that set, it'll actually get downscaled. So, you're losing performance by choosing a higher resolution for object detection. Uh that's something important to keep in mind.

[00:28:23]>> No, that is not the case for every >> I don't think it is comparable. Quest Nav is a very different solution and I think this is somewhat apples to oranges here. Um, I haven't been following QuestNav uh as closely as I perhaps ought to. Uh, but to the best of my knowledge, they haven't implemented April Tech support yet.

[00:28:48]>> That's what I've heard.

[00:28:50]>> Uh, and according to Chris, that's correct. Uh, so, uh, all we really do is April tags, object detection, and all of our other stuff. Of course, the way QuestNap works is they take all of the information the cameras are seeing around them and they compare it to where they were previously and they use that to somewhat build a virtual map or point cloud of the field and then they uh can compare their current position to that point cloud map to determine where they are. uh that is um as I said previously a very different class of solution uh and so I don't think I can truly compare >> um maybe a question to the volume supply is this something schematics about how people can do this we've been following the schematics about the looks video Yeah. So, the current zinc be supplies 5 volts. I believe they do have a model that can supply 12 volts that will be coming out soon or um is yeah, I believe it will be coming out soon. I don't have details on the release date, of course.

[00:30:08]Uh generally speaking, you can find some uh buck converters online that will work as well. We have some that are recommended in the community documentation. Um and I don't know if Larry has anything else on that. No.

[00:30:23]>> How would performance be running uh would would you be able to run object detection and april tag uh detection on the same co-processor?

[00:30:35]>> Yeah. So you definitely can uh I think there uh they're experiment with running two tag pipelines and one object detection pipeline and that work perfectly fine. Um something that should be noted is that you can't do it on the same camera stream. So if you have a camera and you want to do uh look for able tags and then run an object detection pipeline on that exact same screen as well that is not currently supported uh but it is something that we plan to explore at some point. We have no plans timeline for that future.

[00:31:07]>> So what will the interface be between the Rubik Pi and say system core? Is it can what are the >> Can you say that one more time? what what is the interface between the Rubik Pi and a system core?

[00:31:23]>> Sure. So the question is what the interface between the Rubic Pi and the system core is uh it will remain the same as the interface between the Rubik Pi and the Robo Rio for example. Uh everything connects to the radio via Ethernet and then all of our information is uh pushed onto network tables which you can then grab by the photon vision API and you get that simply by installing our metadata.

[00:31:48]Hopefully that if there are no further questions then I think that will be all. Thank you for coming.