ShellCheck Taught Me Something New About Bash

[00:00:00]So if you watch my videos and you think to yourself, "Oh wow, David is just the best programmer who's ever lived. He never makes mistakes." You'd be half right. Uh of course I am the world's best programmer. No, I'm just kidding.

[00:00:09]But I definitely do make mistakes. I want to talk about that today because ShellCheck, the tool for statically analyzing bash scripts, taught me something. Something that I maybe knew of in the past, but even that's a stretch. Let's just say I I didn't know it. I knew that there were remedies for it, but I didn't actually know how bad this issue could get. In my previous video, we wrote a bash JSON exporter. It takes a variable in bash and exports it to JSON. It was a ton of fun, go check it out. But during that video, this happened. Quote arguments to unset the so they are not glob expanded. Oh yes, you can do that.

[00:00:42]Um I don't think there's any chance of this being glob expanded. None of these are problems, they're just warnings. I don't think any of these are actual problems.

[00:00:49]I'm just going to leave it as is. Now I can make excuses here, and I think they're totally valid excuses. Uh it was getting towards the end of the video.

[00:00:53]This was a tough project. It was late at night, and I was just done. I even said that in the video. I was just done, I'll worry about it later. I can't anticipate this being a problem. The next day I decided to really look into it, and it could be a problem. You'd have to contrive a scenario, I'll show you that, but it could be a problem, and I should probably deal with it. In fact, this user even caught that issue and left a comment on my video, and I said I was going to make a follow-up video, so here it is because I think it's actually a really cool little gotcha with bash. All right, let's jump into the code, I'll show you this. So here's the code from last video. You don't need to have watched the last video to understand what's going on here. Just know that if I run ShellCheck on JSON var, there's going to be a lot of issues, a lot of them are warnings, but let's go up here and look at this one specifically. In JSON var line 185, we have this.

[00:01:36]Quote arguments to unset so they're not glob expanded. I kind of looked at this, and I was like, "We're just unsetting an array spot. How is that a big deal?"

[00:01:44]SC2184.

[00:01:46]All right, light mode jumpscare in 3 2 1. There we go, take a look at this.

[00:01:49]SC2184 on the ShellCheck wiki. Quote arguments to unset so they're not glob expanded. Here's the problematic code, and here's the correct quote code. Look, you put single quotes around it. Super interesting. I've done this in the past, but I never really fully understood the rationale.

[00:02:03]Arguments to unset are subject to regular glob expansion. This is especially relevant when unsetting indices in arrays, where this is considered a glob character group. So, this is really, really interesting. I will show you some examples of this, but take a look at this. If you know the pathname expansion is disabled, you can notice message set -o no glob. We can actually turn off globbing in our script. Now, in my JSON var program, I expect that to be running in someone else's shell, so I don't want to modify shell options like that. So, I think quoting it is the right thing to do. But yeah, that could be a little confusing.

[00:02:34]So, let's go ahead and just make a tempter where we can work out of here.

[00:02:37]There we go. SE2184.

[00:02:39]So, let's go ahead and do a simple example, where we have a simple bash script here.

[00:02:44]And what should we do in this? We should make an array, where it has three elements in it, foo, bar, and baz. Let's go ahead.

[00:02:53]Add the executable bit. If we run it, we get nothing, cuz all we did was actually just create the array. We didn't do anything with it. So, let's go ahead and unset ARR at spot one. Okay? And then if we declare -p ARR, it will print the array to us. So, let's go ahead and declare -p here, then unset it, then declare -p below. This will just print out what's in the variable and how it was created and all that stuff. So, let's pull that up here. Bat simple. And let's go ahead and run simple. And take a look at this. The first one we have array spot zero is foo, one is bar, and two is baz. Cool. Awesome. Now, we have array spot zero is foo, and two is baz.

[00:03:29]We have got rid of the array at indice one. That's awesome. We've gotten rid of the element there. That's great. We have a sparse array now. Bash is totally fine with that. So, when would this be a problem? What was it telling us? Well, the issue is unset is just like a command. It's not a command itself.

[00:03:45]Well, I guess it technically is. But if we were to do type on unset, we can see it's a shell built-in. So, it runs just like any other command. It's not a keyword. This is important because this undergoes all the same rules as glob expansion. So, you could imagine I would have something like this, a r r star. If you looked at that, you might be like, "Oh, yeah, that's a problem." because we're just going to look at files in the file system with the name star there. But, we have a r r open square bracket one close square bracket. Let me show you a different command instead of unset, we can just use echo. If we echo a r r at spot one, it actually just echoes it like a literal string. Echo the command runs and it gets this entire thing as an argument. What if we were to run, let's take a look what we have in our directory. If we were to run echo simple like this, do you see what the issue is?

[00:04:37]Look at this.

[00:04:38]Now, we just get simple back without the open and close square brackets. We can use quotes to avoid that happening, but without it, this is matching like a glob on the file system. This is very similar to running this. See how it matches a file on the file system? So, hopefully you're sticking with me here. If we were to look at our simple script here, this will become a problem if we have a file that matches that glob. And we can do that. We can create a new file called a r r one. So, let's go ahead and ls. We can see a r r one is there. Now, if we run our simple script again, look at this. Do you see the problem?

[00:05:16]Now, the arrays are the same. We haven't actually unset the first element or the element at index one here, index one.

[00:05:24]So, what's happening here is unset is actually doing glob expansion here and it's being called like unset a r r one.

[00:05:32]There's no variable name a r r one, but unset doesn't care. It'll happily unset a variable that doesn't exist. So, that's what's being run here. We're not running this like we think we should be.

[00:05:43]Now, there are a couple ways around this. The website said we could do this, we could do set minus o no glob. Now we will not be globbing at all. So this script, when it runs, will not be able to glob. So if we run it, look at that.

[00:05:58]It doesn't even look at the underlying file system. That's great, but this doesn't really work when we're using a function that's meant to be run in someone else's environment because we're just changing shell options. That's not good. We're kind of just messing with someone else's environment. We could, of course, do something like this where we set minus o null glob and then set plus o no glob here.

[00:06:16]And if we do that, it will also work. So we were technically able to unset it.

[00:06:20]That's great. The only problem is, we're just making the assumption here that the user had no glob disabled.

[00:06:28]We don't actually know what they had set. We could look at what they had set and try and reset it. We could get weird with it, but you know what? It's probably easier just to do this because unset will take this whole thing as a string and happily do whatever work is required on it. I'm I'm foreshadowing a little bit. So if we run that with the quotes around it, it works perfectly here. I'll show you.

[00:06:49]Simple. Run this. If you put quotes around it, this is what ShellCheck wanted me to do. This is the right way of doing it. This also works for other data types, which is super nice. So right now, we are just passing the number one here like as a string, but we could pass it in as a variable. We could say i equals 1.

[00:07:08]Unset array at spot i. And that might look kind of weird. Like why aren't we using a dollar sign? We're putting the whole thing in single quotes. Isn't that super odd? Well, don't worry. It still works. So that's super great. We could also get really weird with it. You ready for this? Let's make it an associative array. An associative array has keys and values, so we'll say the key foo is equal to a, the key bar is equal to b. You can see where I'm going with this.

[00:07:38]There we go. Now we have a a associative array with foo is a, bar is B, and baz is C. So, if we change this to bar, just as a string, no big deal, this will work. It's just because it's an associative array, it has to be a dollar sign. Now, if you look at this and you think this is crazy because the dollar sign is in single quotes, so it's not being expanded, it's actually just being passed unset, and unset knows how to parse this. I think it's a little crazy, too. It's a little weird. I understand why. It makes perfect sense, but it's a little uh I don't want to say crazy.

[00:08:08]Maybe it's a little unfortunate, but it works. See, we have our array with foo, bar, baz. Bar has been removed even though we've passed it in these single quotes. So, this literal string is being given to the unset built-in, and the unset built-in knows how to break that apart. It knows that when we're in these square brackets, we might just start uh doing some sort of math notation or whatever. I don't know if you guys know that. That might be something just to be aware of. Let's go back and change this, and let's get rid of this, so we just have a simple array of A, B, and C. And it's a simple indexed array. So, if we change this to Well, let's actually just do this. If we unset array spot one, it's exactly the same before. So, if we do this, and then we do this, look, we've gotten rid of the B because we've unset array at spot one. You can also do something like this cuz technically when you're in these uh square brackets, you're in what Bad Cop likes to call math time. So, Bash is now doing math here. So, we could do 1 + 2 - 1 - 1. Uh yes, that might look insane, but it's totally valid. That was just a really fancy way of saying one. But yeah, it's totally valid. You are in a math time when you're in this syntax notation, when you're dealing with indexed arrays. So, yeah, this is why you would quote here. I should be quoting it with the JSON var script. In fact, I did make a pull request where I went through and quoted it. So, it is fixed now. In fact, if you take a look at this, there's already been three fixes, which is super nice. Uh thank you guys. One of them was from me, but the other two were other contributors, so that is super cool. But if we take a look at this, we You see that I was dumb during the video. This is totally an issue, and then I actually just contrive a scenario where we actually break it.

[00:09:40]And the fix for this is so easy. We just put the whole things in quotes. So, I'll actually just show you that. I have the code checked out here. If you haven't seen the video, definitely go check it out. But if we do a shell check on JSON.var, we can see that the issue is on line 185.

[00:09:55]So, we don't quote it on 185. 185, there we go. It's right here. So, we check.

[00:10:00]This is where we filter out variables with the name under_jv.

[00:10:04]So, let me show you. If we were to run JSON.var -a to list all of our variables, and then we grep for under_jv, we can say that, "Hey, look. There really isn't much here." In fact, let's go ahead and tighten this up a little bit. Let's grep for any line that starts with 1 2 3 4 spaces and the opening JV for the name of the key. So, this is a JavaScript object we're parsing with grep. This is really bad. Don't do this, guys. But look, nothing shows up. That's super good. However, what if we were to touch a file with that same name?

[00:10:35]Remember this name right here? What if we touch a file that matches this glob?

[00:10:40]So, if we were to touch this file, and this should just be one of those characters. So, we could just give it, I don't know, a J was one of those characters. We run this. Now, we rerun this script.

[00:10:50]Look at that. See? The glob matches, so the unset fails. This is bad. This is not what we want. We should go here and put quotes around it. Like I said, on the main branch, the quotes are already there. So, I'm just showing you guys this as an example. We put quotes around it. That file still exists. We run it.

[00:11:05]We get nothing. That's awesome. That fixes that problem. If we remove that file and run it again, we get nothing. That's awesome. That fixes that problem. So, yeah. Super weird scenario, but I thought it'd be fun to show you guys. Like, here's a mistake from the previous video, and here's how to fix it, cuz it is totally a weird thing that you can run into.

[00:11:20]Um so, yeah. That is all I got for you guys today. It's a little bit of a short one. Let's go ahead and I can't end a video without thanking my patrons over at Patreon. Thank you guys for your continued support. If you want your name at the end of one of my videos, go ahead and sign up. And uh you can get your name here. And if you >> [laughter] >> Nice job, Prox DXD03.

[00:11:41]Wow, there's so much stuff happening right now.

[00:11:43]>> [gasps] >> Some people's names on Patreon have ANSI escape sequences, and and I don't do any filtering. So, as long as you guys aren't being ridiculous with it, I don't filter you out. So, Dark Space, you want to put a bunch of extra spaces in your name. Jordan, you want to use color output. Kieran, you want to make your name explode. Prox DX03, you want to put a hey there and have it show up up here.

[00:12:03]Incredible. That's incredible. It's beautiful. Nice job. That's I That's insane. I love it. I think it's so fun.

[00:12:11]Yeah, that's all I got. Thank you, guys.

[00:12:13]See you.