With my recent explorations into mouseless computing I have obviously begun using the terminal a bit more than I usually had. I've been comfortable for the most part in a terminal for quite some time, so it's not a big jump, but one thing has always bugged me about bash, and that is that I can't stand bash's syntax. Just taking a quick look at a for loop through a simple array can get the point across better than any number of words I can write:
array=( one two three ) for i in "${array[@]}" do echo $i done
(Lifted from Nixcraft) It can be slightly less confusing if you are using a static list, but not substantially, and who uses a static list anyway? The ${} syntax seems somewhat like how php or other languages do string formatting (which I consider abysmal, I definitely prefer the python way), but why are we trying to iterate over a string here? And what does the @ mean? I know that the quotes are a protector in case some of your data has spaces... Wait, we have to special case data with spaces? Shouldn't that already be handled by the fact that it is a different element in the array? Anyways, this isn't so much about why I hate bash. But bash isn't the only problem. Usually my solution when meeting with a problem where I don't feel like fighting with bash's syntax (recursive directory traversal, I'm looking at you) is to open up a Python shell and run it in there. But that has major problems too, namely calling shell commands from within Python is not really something you can do easily (you could do it in Perl, but then you'd be programming in Perl, so the syntax problems won't be fixed as much as shifted). The Python way of calling commands makes good sense for scripts that you store, but one off commands are a bit of a pain. Plus if you want to change directories, you have to import os and os.chdir(), and then you don't even get tab completion, which is terribly annoying. What would be great would be if there was a way to mix the two and get a kind of Python shell with the conveniences of bash. Enter IPython, an advanced Python shell. Right out of the gate we can see some major improvements.The first thing I noticed was that there is an alias for cd, so you no longer have to use bash to navigate to the directory you want and then open up a Python shell to write the script. Unfortunately it's a loud cd, such that it prints out the name of the directory you just changed to. It offers a -q option to silence that, but who wants to type -q every time they cd? There's a way around that, though, if we leverage the powers of Python. More on that later. The second most important thing is that it introduces the bang(!) operator which, when used as a prefix to a command, call the command as a bash command, so if you wanted to use find in a script, you could just type !find . and it would run find. Even better, the results of the commands can be passed back and used by Python so, again using find, if you wanted to recursively traverse through a directory and you didn't know about os.walk, you could call recdir = !find . and then use a normal Python for loop to iterate over it. Let's use as an example recursively converting all jpgs in a folder to png. The convert command line tool is very good at this, so we'll use that:
recdir = !find . for filename in recdir: if filename.endswith(".jpg"): !convert $filename {filename.replace(".jpg",".png")}
Compared to the original bash way of doing it, which might look something like this:
find . -iname "*.jpg" -exec convert \{\} {\}.png \;
Which isn't that bad, but it can be a mite confusing. And, of course, that's the cheap way of doing it, where you end up with a lot of .jpg.png files. If you wanted to have the file name replacement like in the IPython part it would take quite a bit more doing, involving sed or awk, and possibly creating a whole other script just to act as an intermediary. I don't have the time to work through that problem right now. So, in just those regards, we can see the advantages of IPython over bash. The problem is that I spend substantially more time just jumping around the filesystem, executing scripts, ssh, copying files around, etc., and it would get rather frustrating to have to type a bang every time I wanted to do a normal shell command. Well, they thought of this and added a magic "rehashx" command. If you type "rehashx" in an IPython prompt, the shell will go through your bash aliases and create the same aliases in your session, so you no longer have to use the bang if you just want to run a terminal command. Finally, the only thing that's left is the prompt. The original IPython prompt just looks like this:
In [#]:
Where # is the number of commands you have executed. This is not only ugly, it's totally unhelpful. Again, though, the IPython creators have your back. Provided with IPython is a profile called "pysh" that seeks to make IPython act as much like your shell as possible, just run "ipython --profile pysh". There is an editable config file under "/path/to/ipython/config/profile_pysh/ipython_config.py" that allows you to edit the some settings (and more, as we'll see later). For instance, on my Arch laptop, I changed the prompt formaters to:
c.PromptManager.in_template = r'{color.normal}[\u@\h \Y1]\$ ' c.PromptManager.in2_template = r'{color.normal}|\D> ' c.PromptManager.out_template = r''"
Which makes my lines look like this, respectively:
[huflungdu@HuFlungDu ~]$ |.>
without any extra coloring. The last one is empty, the output only prints the output values, not any kind of prompt before hand. There's still a couple of annoyances, like there being a banner, auto complete behavior, and the aforementioned "loud" cd. Some of these are easy to fix (getting rid of the banner is simply putting the code c.TerminalIPythonApp.display_banner=False in your configuration file) and some of them are a lot more fun. We're going to focus on the fun ones. The first thing that I notice is the loud cd. It makes sense in the original prompt since it doesn't give you your current directory, but with the pysh profile that is alleviated, and yet there is no configuration option to turn it off permanently. Only you may have noticed something, the config file that's given to us is just a plain Python script, and it turns out it's run without any protection (it's my script after all). So what does that mean? It means we can override IPython library functions from the cmd. Example, this is my fix for cd:
oldcd = IPython.core.magics.osm.OSMagics.cd def quiet_cd(self,parameter_s=''): parameter_s = " -q " + parameter_s return oldcd(self,parameter_s) IPython.core.magics.osm.OSMagics.cd = quiet_cd
This is pretty simple, we're just intercepting the cd command, prepending the quiet option, then passing it along back to the normal cd function to do it's job. The location of the cd command is different depending on which version of IPython you are using, this is for the latest version that comes on Arch, Ubuntu uses a different version but I don't have that one on me. The routine is the same, just oldcd has a different path. You were never meant to hack at this, so it's not surprising that they moved them. Another, and more substantial, problem was the autocomplete. The actual autocomplete algorithms are great, they are quite capable of replacing bash (in some ways better, like if you press up after you stat typing, it will only grab from the history that starts with what you have already typed. It takes some getting used to, but it's pretty cool) but it does some weird things. If there is more than one possibility, it immediately lists out all the possibilities, instead of waiting for you to press tab a second time as bash would do it. It also doesn't auto complete up to the highest common string, so if you type f<TAB> in a folder with foo and foobar in it, IPython will just list foo and foobar without completing to foo. Also, if there are multiple results found and it's part of the file system, IPython prints out the entire path to it, starting from /, which makes it rather difficult to read. Finally, it doesn't automatically add a space to the end of the auto complete if it is the only option, just leaves you right at the end of the string. Again, there's no real way to configure these things, but using the trick we used earlier for changing cd, we can hijack the rlcomplete function to make it do what we want. We can have a gander at my code for fixing these problems:
# I only want auto complete to happen if you press tab twice, by default it happens if you press it once. # I also want it to auto complete up to the largest common string oldcomplete = IPython.core.completer.IPCompleter.rlcomplete IPython.core.completer.IPCompleter.tabs = 0 IPython.core.completer.IPCompleter.linebuf = "" def newcomplete(self, text,state): if state == 0: self.line_buffer = line_buffer = self.readline.get_line_buffer() cursor_pos = self.readline.get_endidx() self.complete(text, line_buffer, cursor_pos) # If linebuf is the same as the text passed in, then we just pressed tab the first time, if it's not, we want to reset the tab counter if self.linebuf != text: self.tabs = 0 # Not consistent with bash, we insert a tab if the line is currently empty # Special case, if it's the first tab press and there are multiple matches, don't put anything, so you have to press twice to get all results if len(self.matches) > 1 and not self.tabs > 0 and self.readline.get_line_buffer().strip(): # Return the longest identical substring, e.g. cd fo with foobill and foobar as options will autocomplete to foob longeststring = "" if state == 0 and self.linebuf != text: self.linebuf = text for completion in self.matches: for i in xrange(len(longeststring),len(completion)): matchstring = completion[:i] if all(x.startswith(matchstring) for x in self.matches): longeststring = matchstring # This code may have been made redundant, investigate. # Possibly from when I stopped stripping the beginning off the matches for comparison if longeststring == text: self.tabs = 1 return None else: return longeststring # Return nothing so the string is empty, but set it so next time the options will be displayed else: self.linebuf = text self.tabs = 1 return None # Special case, if we find multiple matches, we don't want the whole path printed, just the path relative to where we've already navigated elif len(self.matches) > 1 and state < len(self.matches): return (os.path.basename( os.path.normpath( oldcomplete(self, text, state))) + ("/" if self.matches[state][-1] == "/" else "")) # Special case, don't print out the only option if it is the currently entered text, that's just silly elif len(self.matches) == 1 and self.matches[0] == text: return None # Special case, add a space if we found the only match and it's not a directory elif len(self.matches) == 1 and state == 0 and self.matches[0][-1] != "/": return oldcomplete(self, text, state) + " " # Normal case, just return the option else: return oldcomplete(self, text, state) IPython.core.completer.IPCompleter.rlcomplete = newcomplete
I know that's a lot to digest, it gets pretty complicated due to the fact that rlcomplete expects the function to return the values it found one at a time until it runs out (at which point it wants it to return None) instead of returning a list all at once. Anyways, with those changes to my IPython I've been successfully using it as a bash replacement for a couple of weeks now, and it's so good that when my coworkers use it, they don't even know they aren't using bash anymore. Pretty snazzy.