Setting up an OpenBSD Gitlab runner

Fri, 18 Dec 2020 00:00:00 +0000

I think one of the larger problems I've had when writing software has been doing proper testing CI on OpenBSD, which for me, obviously is a problem. Much of the time, I end up either just doing CI on Linux or not setting up CI at all.

There are a few popular CI options available:

Travis CI</a> (supports linux and macos)</li>
Appveyor</a> (supports Windows, linux and macos)</li>
GitHub Actions</a> (supports Windows, linux and macos)</li>

GitLab</a> (officially supports Windows, linux, macos and FreeBSD)</li> </ul>

Finally, I should mention that there is one platform that does absolutely solve my problem, definitely better than I will in this blog post. That's Sourcehut</a>! They officially have machines for a variety of Linux platforms, FreeBSD, OpenBSD and 9front</a>, and on a few different architectures as well!

I'm on the fence about sr.ht at the moment, and in general I'm pretty fond of GitLab, so I wanted to tackle the issue of having some degree of OpenBSD testing through GitLab.

NB: It's worth noting that the setup that I'm going to describe isn't particularly secure, and doesn't provide build isolation. CI is quite literally remote code execution as a service, so the setup I'm going to describe is not suitable for colocation with anything else.

Getting an OpenBSD VM</h2> This isn't particularly related to the guide, but if you're looking for a hosting platform that supports OpenBSD, I've enjoyed Vultr</a> quite a bit. I've had fairly good experiences with their support and had p good uptime with my VMs. They also offer cheap $2.50/mo, IPv6-only VMs (yes, GitLab and the GitLab runner have full IPv6 support, so this is an option).
I should also mention OpenBSD Amsterdam</a>, which is run by OpenBSD fans and makes monthly donations to the OpenBSD project with the proceeds.
Building the runner</h2>
Building is simple. We're gonna deviate from the official instructions a bit, because we don't need the helper images or anything.
$ doas pkg_add bash gmake $ git clone https://gitlab.com/gitlab-org/gitlab-runner $ cd gitlab-runner $ BUILD_PLATFORMS='-osarch openbsd/$(arch -s) gmake runner-bin </pre>
The runner binary will be in out/binaries/gitlab-runner-openbsd-$(arch -s)</code>
Machine setup</h2> I'm assuming you're starting with a completely fresh install. First, install the following packages:
$ doas pkg_add bash git git-lfs curl jq </pre>
Despite GitLab's docs saying they support sh</code>, they only support an sh</code> that supports long options (--login</code>), so we have to use bash</code> here.
We're going to run the runner (and our builds) as a separate user, not as root. There's no great way to do proper isolation of arbitrary binaries on OpenBSD outside of vmd</code>, which I don't believe supports nested virtualization. We could run this in a chroot</code>, but unfortunately it'd be a pretty loaded chroot</code>, since the deps for all of our builds would also live in the chroot</code>.
$ doas useradd -s /sbin/nologin -m gitlab </pre> If you have other users, I'd recommend making their home directories unreadable to others.
$ doas chmod -R o-rx /home/* </pre>
Resources</h3>
To reduce the impact of abuse, we're gonna try and restrict the usable resources a bit. We're going to modify /etc/login.conf</code>.
The following values are specific for my use-case, and won't necessarily be applicable for your system.
gitlab:\ :coredumpsize=0:\ # prevent coredumps :filesize=30M:\ # largest file can be 30M :maxproc=50:\ # max 50 processes :datasize=1G:\ # 1G heap per proc :stacksize=4M:\ # 4M stack per proc :openfiles=512:\ # 512 open files per process :tc=default: # inherit other values from `default` class </pre> Since I'm on Vultr, and they have a weird partition scheme where /home</code> isn't its own partition, I'm going to set up filesystem quotas for gitlab</code>.
# edit `fstab` to add the `quota` option to `/` $ doas mount -o update,quota / $ doas edquota $ doas quotaon $ doas -u gitlab quota </pre>Network</h3> We're also going to restrict network access for the gitlab</code> user. We'll do this through adding the following lines to our /etc/pf.conf</code> table <github> file "/etc/pf/github" table <allowed_hosts> const { gitlab.com crates.io static.crates.io } block out log proto {tcp udp} user gitlab # by default block outgoing traffic and log to pflog0 block in proto {tcp udp} user gitlab # block any incoming traffic pass out proto {tcp udp} to 127.0.0.1 port 53 user gitlab # pass dns traffic pass out proto tcp to <allowed_hosts> port 443 user gitlab pass out proto tcp to <github> port 443 user gitlab </pre> I'm primarily writing rust, so I just need the three hosts in the <allowed_hosts></code> table, as well as whatever IPs github is using that day. Part of the reason I log blocked outgoing calls, is that if a build fails due to a network issue, it makes it a lot easier to figure out what new rules you may have to add to your pf.conf</code> to get builds flowing again. To build the github table, I run the following script in an hourly cronjob: #!/bin/sh /usr/local/bin/curl -sSH "Accept: application/vnd.github.v3+json" https://api.github.com/meta | /usr/local/bin/jq --raw-output '.api+.git+.web|sort|join("\n")' > /etc/pf/github /sbin/pfctl -T replace -t github -f /etc/pf/github </pre> Finally, I also add the following to /etc/ssh/sshd_config</code> to prevent this user from logging in via ssh. DenyUsers gitlab </pre>Installing the runner</h2> To install, copy the binary into /usr/local/bin</code>. I'd recommend keeping it out of the home directory, so it's harder for a rogue job to overwrite. I drop the following config file in /etc/gitlab/config.toml</code> concurrent = 1 check_interval = 3 log_level = "info" [session_server] session_timeout = 1800 [[runners]] name = "openbsd-6.8" url = "https://gitlab.com/" token = "<secret>" executor = "shell" shell = "bash" [runners.custom_build_dir] [runners.cache] [runners.cache.s3] [runners.cache.gcs] </pre> I also have an rc script to start it #!/bin/ksh daemon="/usr/local/bin/gitlab-runner-openbsd-amd64" daemon_class="gitlab" daemon_user="gitlab" daemon_flags="run -c /etc/gitlab/config.toml" daemon_timeout=60 . /etc/rc.d/rc.subr pexp="${daemon} ${daemon_flags}" rc_bg="YES" rc_reload=NO rc_pre() { # There are some interesting network timeouts # if I don't refresh the pf rules /sbin/pfctl -f /etc/pf.conf } rc_start() { ${rcexec} "${daemon} ${daemon_flags}" >> /var/log/gitlab-runner.log 2>&1 & } rc_cmd $1 </pre> Following all of this, you can just follow the normal gitlab instructions</a>. Conclusion</h2> Does this work? Well, yeah. The big problem is that I wouldn't necessarily call this secure by any means. At the end of the day, users submitting PRs to my project that use this runner are still able to run arbitrary code on my gitlab runner. All CI/CD is remote code execution as a service. Even with the steps taken above, there are still ripe opportunities for abuse. I definitely don't feel comfortable integrating this machine with my personal infrastructure. I'm certainly not setting it up to forward mail to my mail servers, since that'd require a key on the box. For now, this is roughly the best I think I'll do. What would be better?</h3> Ephemeral VMs (also with similar settings) would be the ideal here. If vmd supported nested VMs, it'd be awesome to add vmd support to the gitlab runner and launch all of my jobs in individual VMs. Alas, as far as I'm aware, this would only be possible if I had a bare metal machine handy. VirtualBox on a non-OpenBSD platform could also be a fair option worth exploring. For right now, this works for my case, is cheap enough, and this VM is essentially ephemeral to me and easily trashed if I need to replace it for any reason. Feel free to reach out if you think that there are ways to improve on this setup, or if I got something super wrong.

Final Fantasy VI Tue, 06 Oct 2020 00:00:00 +0000 For the longest time, I always considered this game one of my favorites, despite never finishing. I started it for the very first time on emulator somewhere around 2002 or 2003? I started it a number of times since then, but never finished it nor usually ever made it to the World of Ruin. But the other day, I finally finished it! Now I feel like I can say with certainty that it's one of my favorite games, if not my favorite game of all time. There will be FULL SPOILERS for EVERYTHING. It ain't perfect</h3> So before I start heaping praise upon praise on this game, I really want to make sure I account for the problems with this game. The SNES version of this game launched with some horrible, game-breaking bugs. The most infamous, is the Sketch Bug, where using Relm's Sketch</code> ability could cause the game to crash at best or corrupt your save at worst. A smaller bug that still heavily impacts gameplay, is that the Evade%</code> stat just doesn't mean anything. Instead, MBlock%</code> (your chance of evading magic attacks) controls both magic evasion and normal evasion. It's also very easy to max out MBlock%</code>, which on an unpatched version makes you virtually invincible in many cases. The English translation is notoriously bad</a>. It's not just bad in the sense that there are outdated references, or that perhaps some characterization doesn't match the original dialog, but in some cases the dialog is outright incorrect. It does have some unique characterization though, a lot of which has been brought into other games (more on that later). One of the other big problems with this game is how unbalanced the characters are. Each character has different base stats, and one unique ability that they're able to use in addition to Attack</code> and Magic</code>. The problem here though, is that Magic</code> is far, far better than pretty much any ability late game, and many of the abilities suck. Some of the worst offenders: Relm - Sketch</code> uses an enemy's ability against it...although oftentimes enemies are resistant to their own abilities. It also tends to be pretty weak. Relm is honestly only saved by her monster magic stat. </li> Celes - Runic</code> prevents any magic from being cast, and converts it into MP for Celes. While it's good early game, late-game you should be casting too much magic for it to be useful anymore. Celes is saved by having some of the best equipment and being one of the best magic casters. </li> Strago - Lore</code> is essentially blue magic. I love blue magic in FF games, but unfortunately Strago's stats suck, and also Lore</code>s have wildly marked up MP costs compared to basic spells. It ends up not being worthwhile to use most Lore</code>s late-game, at least in my experience (the exception here being Mighty Guard). </li> Cyan - Bushido/SwdTech</code> is a skill where you wait while a bar slowly fills up, and then it performs an attack based on how long you let it fill up. The abilities are never anything too special, and it takes way too long to actually charge the bar up meaningfully. </li> Setzer - Slots</code> is a slot game. The attack depends on what the slots land on, and if you don't make a match, you get an incredibly weak MT heal on the party. </li> Mog - Dance</code> performs random attacks based on the dance you select. Most of these aren't great for much of the game? I honestly just found the most utility just equipping Mog with spears and the two Jump</code>-related relics. </li> </ul> The others are mostly passable, but ideally I'd love to see a theoretical version of this game with much better balanced versions of these special abilities, and perhaps slightly nerfed magic to make sure that specials keep being relevant late-game. So onto all the ways that this game is great! Characters</h3> I think one of the things that folks tend to remember about this game are all 14 (ok well you know the 12 important ones) characters. Despite them not having enough time for full arcs, they mostly have mini-arcs but also clear characterization, even in the English version. One of the more interesting decisions is that FFVI is not intended to have one main character? I know growing up, I always considered Locke the main character, since he was the first man that joined the party, and young me hadn't yet understood the idea that women can be protags too. Now after completing it, I definitely feel like Celes is closer to being something like a main protag, since she's one of the three required characters at the end, and she undergoes the most character growth. Celes</h4> Celes starts her story as a general of the evil empire that serves as the Big Bad for the World of Balance. Locke rescues her, but when they rejoin the party, there's this great scene where, as the party meets her, they all interrogate her as to her true motives. Once they begin to trust her more, they end up finding out that her dad is a researcher for the Empire, and the Big Bad lies to the group that Celes has been working for them the whole time. After the World of Ruin, Celes ends up on an island taking care of her father, Cid, and he most likely dies (more on that in a bit). Celes ends up trying to commit suicide, before going on an adventure to round up all of her friends. And it's wonderful to see them all reunite in a ruined world, and the slow reunion kind of highlights how they've grown to respect and care about Celes throughout the story. Celes was definitely the main character I was most invested in, especially throughout the World of Ruin. The whole of the island scenes really makes the despair of the World of Ruin feel truly palpable. You see someone who starts off as a fairly strong character get understandably broken by the end of the world and the death of her father, and that really moved me. That really made me feel like things in the World of Ruin were truly hopeless. Terra</h4> Terra is another one of the main-ish characters that's worth a mention. You start off as her, enslaved by Kefka and torching people in a neutral city to retrieve an Esper. Terra ends up being able to speak with the Esper, and her resonance with the Esper is what ends up kicking off the whole tale. She reluctantly joins the Returners, a group fighting against the Gestahlian Empire. As the game progresses, she discovers that she ends up being the child of an Esper and a human, stolen by the Empire to be turned into a weapon. She feels a kinship with the Espers who are being kidnapped and murdered by the Empire, and that kinship is what ends up driving her to fight against them. One of the things that I really thought the game did a good job of conveying is Terra's sense of empathy and her role as a caretaker. One of the first ways that we see this, is the first time she morphs into her Esper form. She flies away out of control, and ends up confronting a child, before flying away again. When your party encounters and talks to the child, he makes it clear that Terra had kind eyes, and it was clear to him that she wasn't going to hurt him, despite how his parents had feared for his life. Later, we see this in how she seems to feel so deeply for the Espers in captivity in the Magitek Research Facility. Seeing them trapped in tubes is clearly heartbreaking for her, moreso because she knows that she's essentially one of them. Finally, in the World of Ruin, Celes discovers that Terra saved all of the children of Mobliz before it was destroyed, and is now raising...all of them. The kids adore her, calling her Mama Terra, and are greatly saddened when you eventually take her away to help save the world. Kefka</h4> Kefka is the most memorable character though. He's this sadistically evil clown that's mad with power, and he has an absolutely dope theme song</a>. Kefka's rise to power and his worsening sadism is one of the stand-out points of this game. Kefka transforms from a character that wanted to ascend the ranks of the Gestahl Empire to a being whose thirst for power was unquenchable. At the start of the game, Kefka is a high-ranking general in Gestahl's army, leading a batallion of troops to Figaro Castle to interrogate the king about hiding Terra, who had escaped from the Empire's clutches earlier. However, the king is less than helpful, Kefka throws a bit of a tantrum and light's Figaro castle, ostensibly an ally of the empire until that point, on fire. When Figaro Castle submerges itself into the dessert, Kefka utters one of his most famous, incorrectly translated, lines: "Son of a submariner!"</a>. Some time later, Sabin and Shadow happen upon Kefka at a Gestahlian camp laying siege to Doma Castle. He shows up to the camp telling General Leo that the Emperor is calling him away on other business (an outright lie). Leo leaves, warning Kefka not to do anything evil, and Kefka promptly orders that the moat surrounding the castle be poisoned. Wildly, I guess that Doma's moat is their primary water supply, since shortly after, everyone but Cyan drops dead from poison, leading to a quick end to the siege of Doma. Towards the midpoint of the game, Gestahl tricks your party and General Leo in getting Espers to meet the Empire in a neutral town for a ceasefire. Kefka shows up, purportedly on Gestahl's orders, murders General Leo, murders all of the Espers and burns the town, killing most of the folks living there. In the transition from the WoB to the WoR, Kefka takes a turn in that he stops working for the Emperor, and instead betrays the Emperor in pursuit of becoming a God and destroying the world. Once a God, Kefka destroys towns seemingly at random, out of what appears to be hatred for humankind. These scenes, as well as a few others that I've left out, really had an effect of me. They cemented Kefka in my mind as unabashedly evil, and his lightheartedness in perpetrating these acts always made all of these things far more disquieting. His horrifying actions always felt like they gave the main quest urgency and real consequences. They also served as the most memorable moments in the game for me, stuff that stuck with me throughout childhood and well into adulthood. One of the weaker parts of his character is that there doesn't appear to be any motivation for his ascent to power. The best we get comes from some optional dialog in a random town, where someone in a tavern mentions that Kefka was the first Magitek knight and had suffered severe brain damage as a result. Honestly, I don't love that justification and wish that Kefka's backstory had been expanded on in a meaningful way. Characterization in general</h4> I think one of the more interesting things that this game does is the general case of how you learn about characters' backstories. The game almost never forces you to learn about most of the characters. There are a few cases, like with Celes and her dad, where through playing the main game, they do character development. In the character sections above, there are even clear examples. Without talking to a random NPC in a tavern, you'd never find out how Kefka became unhinged. Without talking to that random child, you'd never discover that Terra had that moment as an Esper with a child. Without discovering the hidden orphanage in the wreckage of Mobliz, you'd never realize that Terra had adopted a horde of small children. There are more extreme examples of this type of storytelling, one of them being Gau. Gau is a child that you discover on the Veldt, that joins your party when you toss him some meat. If you avoid talking to random NPCs, that's quite literally all you know about him. He undergoes no character development; at most you're treated to some fun scenes</a> where Gau calls Cyan "Mr. Thou" because of his speech patterns. However, if you talk to some NPCs in Mobliz, you find out that a neglectful old man had a mental break, confused his child for a monster and threw him out into the Veldt. Later in the game, when you get an airship, you can even find this old man, and lead Gau to him. If you do so, you get a cutscene where your party tries its best to dress Gau up in a fancy suit and reintroduce him to his father. It starts off touching, but when it becomes clear that his father is never going to remember him, Gau backs off saddened, but at least secure in the knowledge that his father is safe. That's a really cool scene! But it's trivially missable! I have very mixed feelings about this approach, but I think one of the strongest arguments for it is that these scenes feel more earned, in a way. When you think to bring Gau to see that random old man speaking nonsense in a hut, it feels like you're getting rewarded with backstory for making that connection. It feels much less like the game is forcing its narrative upon you, and instead that it's letting you discover its narrative. It's also one of the things that I think makes different people's playthroughs of this game unique. Not everyone is going to uncover the same parts of the characers' backstories, and that breeds discussion and commentary, which I think is pretty cool. At the same time, if any of my friends play this game, I wonder how much of these sidestories they'll actually discover. It's almost sad that they may not get the same experience with the characters of FFVI that I did. I love these characters, I enjoyed their stories and I want my friends to be able to enjoy them as well. Ambitious</h3> This game is one of the most ambitious games that I've played on the Super Nintendo, and honestly I think that that is both positive and negative. Really the only negative is that they crafted characters and a world that I wanted to see more of. There are 14 characters in this game, and 12 of them have some degree of backstory...and unfortunately there's not enough time or space on the cartridge to flesh all 12 of these characters out enough as well as handle the story. The story feels a lot more high stakes than a lot of Final Fantasy games that I've played. The world ends in this game. Whole towns are destroyed. Arguably the whole merged world situation from FFV seems like it should be more destructive, but it doesn't seem to have the impact that Kefka creating the world of ruin does. Compared to the merging of worlds in FFV, so many more people clearly die from Kefka creating the world of ruin. FFVII seems to have similarly high stakes, since the end of the world is also on the table in that game, but Sephiroth isn't successful with his plan. In FFVI you're cleaning up a world destroyed, whereas in many other FF's you're trying to prevent it. Once you're in the World of Ruin, everything seems incredibly bleak. Like I said, Celes tries to commit suicide. The overworld music has gone from traditional jrpg fare</a> to a sad, almost dirge-like tune</a>. When you get to towns, you see people living in absolute ruin. People talk about the deaths of their families and friends. Characters that you did quests for in the WoB are now dead, ultimately because you failed to save them and their world. You failed, and you are feeling the full effects of it. Now, instead of you and your friends working together to save the world, you're alone. You're alone and you feel it. And when you start off, you're just hoping your friends have survived and you're trying to find them. This tonal shift is also accompanied by a change in the gameplay. The first half of the game was a typical, linear jrpg. However, once we switch to the World of Ruin, the rails are gone. You can go anywhere as you try to find out if your friends even survived. Before, you had not just a goal, but you were told where to go and how to achieve it, by either the folks in your party or NPCs. Once the world ends, you're not just alone in the sense that your party members are gone, you're also alone in the sense that nobody is telling you what to do anymore. You just have to figure it out on your own. From here, the game turns into much more of an open-world affair. You can go anywhere, you can get your party members in any order and there's actually very little that the game forces you to do in terms of progression. One popular FFVI run that I learned about during this playthrough is called a CES run, where you only get Celes, Edgar and Setzer (the required characters) and complete the game. One of the wildest parts of all this is that the World of Ruin only happened because FFVI was ahead of schedule, and the devs decided on a whim to add it</a>. I cannot even imagine this game without the World of Ruin; it wouldn't have had nearly the emotional impact without it. Honestly, for the time, opening up a jrpg like this was quite a departure from the rest of the gaming landscape. Sidequests were of course pretty common, but making whole swathes of what's conceivably the story was markedly different from other popular jrpgs of the time. Even in other FF games of the time, when you got your airship, there'd still be a clear quest-line you would have to follow that was still very linear. Choice</h3> One thing that struck me as particularly unique about FFVI was the types of choice that the game presented you with, and how it can directly impact the gameplay experience you have. When we talk about choice in video games, it's oftentimes the bs Mass Effect Paragon/Renegade dialog-tree-based system: NPC says thing, you can reply in a certain way that adds or removes points on some kind of purity meter. FFVI goes about presenting options to the player in an incredibly different way. Honestly, it doesn't really present them at all. There are very few major choices that you can make that affect the story. I think there are only two? There are only two that I'm going to discuss here anyway. Killing Shadow</h4> Shadow is a merc that, in the WoB, will join your party for a fee and will leave when the mood suits him. We also see him work for the Empire in parts of the WoB, reinforcing that he's the type that'd slit his mama's throat for a nickel</a>. However, we see him change throughout the story, and develop affection for some of the characters in your party, particularly relating to both Terra and Relm. Once we get to the Floating Continent, we see that Shadow has been tossed aside by the Empire and that they even tried to have him killed. Shadow rejoins your party throughout the Floating Continent, and fights on your side for the remainder of the game, or his life. Right as the world begins to end, Shadow says that he's going to distract Kefka while the others escape, and to wait up before jumping on the airship and fleeing. However, when he says, "wait up," he means it. As you race against the clock towards the airship, if you don't wait until quite literally the last second before jumping on the airship, Shadow dies as the world ends. What I like about this, is that the signal to the player is subtle. It's a real thing that Shadow asks, and the game doesn't break the 4th wall in any way to let you know that not waiting will result in Shadow dying. It's even makes sense to think that if you get into the airship early, you'll just see Shadow run and jump aboard regardless of what you do. If you do end up killing Shadow, the game doesn't even really tell you that you could've prevented his death. Imagine killing Shadow, not realizing you could save him, and then chatting with a friend that did save him. I like that the subtlety of this choice very likely prompted these types of conversations, as well as wonder as to what else could've been a choice. Another big consequence of this choice is that you end up missing out on the majority of Shadow's backstory. You get some bits of it in the World of Balance, but a lot of it is discovered in the WoR. Killing him cuts out that whole storyline. Killing Cid</h4> When we start in the World of Ruin, we play as Celes sharing a cottage with a sick Cid. To stop him from dying, you need to go down to the beach, catch some fish and feed them to Cid. The catch though, is that not all fish help equally, and it's timed. There's no timer on-screen, his condition just quietly gets worse. If you don't have the running shoes equipped, and you don't already know which fish are better for Cid, it's very likely that he'll die. Any of the times I played without a guide, I never was able to save him. On this playthrough, I could've saved him, but I decided to let him die on purpose, because I think Celes' arc is so much more powerful with the death of her father. While this choice doesn't have gameplay implications like letting Shadow die, it does heavily change the tone of Celes' story. If you don't save Cid, Celes tries to commit suicide and only regains the will to live by finding Locke's headband - a sign that he might still be out there and alive. You find a note from Cid about a raft he's been working on, and you set out to find your friends. If you do save him, well, Cid just tells you about the raft he built and sends you on your way. I don't really like this route, since it's much more satisfying to see Celes pull herself together from the edge of utter despair. Both of these choices profoundly change the tone of the story, and the game never hits you over the head with them. There's no little ding, there's no meter tracking your morality and most importantly, it's very possible to play through the game not recognizing that either scenario could've played out any differently. Blue Magic</h3> One of the more unique types of magic that frequently recurs in FF games is blue magic. Blue magic allows you to learn some of the attacks of your enemies, although the mechanics vary significantly from game to game. However, where this game differs from previous (and as far as I'm aware, subsequent) entries, is that there are essentially 3 different types of blue magic: Lores, Sketch and Rages. Lores</h4> Lores are Strago's special skill, and function like blue magic does in previous entries in the series. To learn a new spell, Strago has to witness an enemy casting it while not being impaired by certain status elements. After he's seen an enemy use a magic attack, he can use it from then on. Like I mentioned earlier though, these kind of suck unfortunately. The big problem isn't that any of the actual skills suck, in fact many of the spells are traditional FF blue magic spells, but it's that they cost tons of MP compared to the traditional white, black and grey magic and tend to have worse damage output or effects as well. Since late-game FFVI is all about maximizing damage output through magic casting, blue magic just doesn't make sense in the magic economy. If Strago had a better magic stat and/or if the MP cost of blue magic was reduced, it could've had a much larger role in the game. Sketch</h4> Sketch is Relm's special skill, and it just executes one of the enemy's specific skills against it. As I mentioned before, this does unfortunately have the massive drawback of the enemy often resisting its Sketch move. Relm also has the opposite problem of Strago, in that her magic stat is super high, so using Sketch instead of casting high-powered magic almost never makes sense. Rages</h4> Rages are Gau's skill, and they're one of the best in the game. So far, of the FF games that I've played (FF4-8, FF10 and FF15), there hasn't been any other ability like it. I guess there are technically skills with the word "Rage" in them, but they have always functioned like traditional blue magic, rather than Gau's Rage. With Rage, you pick an enemy and cede control of Gau completely. Gau then adopts a limited version of that enemy's moveset, as well as their resistances and weaknesses for the rest of the battle. To actually acquire these Rages, you have to go to the Veldt, an area where every enemy you've ever encountered will spawn, and use another Gau skill called "Leap" on the enemy party. Gau will leave your party for a couple of battles, and will come back with the Rage of all of the enemies in that encounter. For background, blue magic in all FF games is generally limited to roughly 10-30 spells, depending on the game. But remember, I said that every enemy appears on the Veldt. That's right...pretty much every enemy, and even some bosses, are available to you as Rages. That ends up being roughly 250 different Rages that you can learn. There are (incredibly necessary) guides that cover this topic in great detail</a>. This provides an incredible amount of flexibility, and the payoffs of learning the system well actually turn out to be worth it, making Gau one of the better characters throughout the game. Really the only downside is that it's difficult to organically understand all of the Rages and their trade-offs, since there are so many, and their various trade-offs aren't always clear without a guide. Learning how to milk the Rage system ended up being one of the more fun parts of my most recent play-through. Limit breaks</h3> Notably, FFVI is the first game that featured the mechanic that turned into Limit Breaks! It's hinted at in the Beginner's house, which serves as kind of the in-game manual for new players. One of the people there says to "try using the Fight command when you're low on health." What that practically means, is that there's a chance that if you attack at low health, you'll execute a unique, powerful move that sometimes will also yield instant death to an enemy. Here's a quick video exhibition</a>, since it's kinda cool to see. I didn't encounter any of these in my playthrough, unfortunately. Setting</h3> In terms of setting, FFVI was the first game that really pushed the fusion of the industrial and more traditionally magical settings that are familiar in more modern entries of Final Fantasy. FFVI has a number of towns all in different stages of development. There's the highly-industrialized Vector, which serves as the location of the high tech Magitek Research Facility as well as the seat of the Empire. In this area, homes are made of brick and steel, and the brutalist fortress housing the Emperor towers over the populace. Vector seems very much like a scaled down version of Midgar in FFVII, in a way. It's the center of not just power, but also technology. Contrast this to a town like Mobliz or Thamasa. These towns are very similar to towns you'd encounter in FFV or earlier, where there are a few simple homes and shops, but no real industry. Conclusion</h3> FFVI slaps. I'm not sure that these posts will ever have a real conclusion. I also didn't really plan this out, so there's not really a point to this post, other than expressing some of the aspects of this game that really caught my attention. My hottest take on this game is that FFVI walked so that FFVII could run. Honestly, this game feels in many ways like a spiritual predecessor (and I guess it's also the literal predecessor) to FFVII. Many ideas, like the hybridization of magic and tech, the technology capitol being the seat of authoritarian power, the unhinged, sadistic general disobeying orders to become the Big Bad after spend their early life as a science experiment. It seems like FFVII fleshed out and refined a lot of the ideas that FFVI had developed. Despite this, my biggest dream would be for FFVI to get the FFVII:Remake treatment. I'd love an opportunity to revisit this world and to see how the developers would expand upon the environment and the characters with more time. I'd love to see Vector, Narshe and Figaro get the HD treatment. I'd love to see an update to the battle system that gave some of the worse characters more opportunity. The biggest problem that FFVI has in my eyes, is that I'm left wanting more, and I'm left wanting to see what developers today could do with the framework that it built on the SNES. Hopefully someday, I'll get my wish. What is this? Fri, 03 Jul 2020 00:00:00 +0000 I play a lot of jrpg's, and I wanted a place to just write about my thoughts. I figured rather than create a new tumblr or create a new site, I'd add them onto my personal blog. I mostly want to have this space as a place to ramble about jrpgs, so that when I'm trying to remember my favorite parts, or the story, or talk to someone about what I liked or disliked, I have a reference of my thoughts while playing it to refer to. I'm not writing formal reviews, I'm not trying be a games journalist or whatever, I'm writing first and foremost for myself. In light of that, everything I write will have full spoilers for whatever game I'm talking about. Honestly, I like sharing my thoughts on games' endings with my friends, so avoiding writing about them out of fear that I might spoil something for some rando reading this defeats the purpose. Extended Attributes Wed, 30 Dec 2015 00:00:00 +0000 While working on support for FreeBSD's extended attributes in python</a>, I tried to be conscious of other implementations of extended attributes in different operating systems, that way I wouldn't be inadvertently causing the sane problem that I was trying to fix: reliance on a particular API's semantics. What are extended attributes?</h2> To put it very simply, extended attributes are metadata that are attached to files. Typically, they're key/value pairs that the filesystem associates with a particular file on the filesystem, though that doesn't always have to be the case. How they're implemented depends on both the filesystem, as well as the operating system. This means that implementations on the same filesystem (UFS, for example) can be complete incompatible across operating systems (Solaris and FreeBSD). Extended attributes are not mandated by any standard. The tooling and APIs are quite different across operating systems and some operating systems (OpenBSD, HP-UX) don't implement them at all. Because support is non-standard and spotty, it's rare to see them used in cross-platform software. I'd be super-interested in seeing some counter-examples to this. Extended attributes are sometimes namespaced. That is to say, there exists some top-level grouping of attributes. Other than the top-level namespace, there usually isn't hierarchy to attributes, other than any arbitrary user-defined hierarchy. Namespaces are usually system</code> and user</code>, although this isn't necessarily consistent, as we'll see. Extended attributes under the system</code> namespace are only modifiable by root (and sometimes only queriable by root). Linux</h2> ssize_t getxattr(const char *path, const char *name, void *value, size_t size); ssize_t lgetxattr(const char *path, const char *name, void *value, size_t size); ssize_t fgetxattr(int fd, const char *name, void *value, size_t size); ssize_t listxattr(const char *path, char *list, size_t size); ssize_t llistxattr(const char *path, char *list, size_t size); ssize_t flistxattr(int fd, char *list, size_t size); int removexattr(const char *path, const char *name); int lremovexattr(const char *path, const char *name); int fremovexattr(int fd, const char *name); int setxattr(const char *path, const char *name, const void *value, size_t size, int flags); int lsetxattr(const char *path, const char *name, const void *value, size_t size, int flags); int fsetxattr(int fd, const char *name, const void *value, size_t size, int flags); </pre> The linux API is actually a fairly nice one, and for the rest of this post I'm going to use it as my point of comparison. The return values of the getxattr</code> and listxattr</code> functions are the total size of the attribute, not the size of the data returned. This lends itself to a nice idiom for checking whether or not truncation occurred: char buf[BUFSIZ]; ssize_t res; res = getxattr("/home/worr/foo", "user.foo", buf, sizeof(buf)) > sizeof(buf); if (res > sizeof(buf)) /* truncation occured */ else if (res == -1) /* error occurred */ </pre> If value</code> is NULL</code> in a call to getxattr</code> or listxattr</code>, the size of the buffer required to hold the contents of the EA will be returned. This allows you to query the amount of space required to hold the return value, allocate it, and then call the function again to populate that. That's unfortunately racey, so it's preferable to call and then realloc</code> if truncation. Linux extended attributes are namespaced, and the namespace is specified as part of the attribute name. Namespaces are separated from attribute names by a .</code>. Currently, they support the common system</code> and user</code> namespaces, as well as security</code> and trusted</code>. It's important to note that listxattr</code> will never retrun EPERM</code>. If there are EAs that the current user cannot access, they just won't be returned. The attribute list returned by listxattr</code> is NULL</code>-delimited, and all of the attribute names returned by listxattr</code> are fully-qualified. AIX</h2> It seems funny that I'm going to talk about AIX's interface right after linux's, but that's largely because it's...almost exactly the same. ssize_t getea(const char *path, const char *name, void *value, size_t size); ssize_t fgetea(int filedes, const char *name, void *value, size_t size); ssize_t lgetea(const char *path, const char *name, void *value, size_t size); ssize_t listea(const char *path, char *list, size_t size); ssize_t flistea (int filedes, char *list, size_t size); ssize_t llistea (const char *path, char *list, size_t size); int removeea(const char *path, const char *name); int fremoveea(int filedes, const char *name); int lremoveea(const char *path, const char *name); int setea(const char *path, const char *name, void *value, size_t size, int flags); int fsetea(int filedes, const char *name, void *value, size_t size, int flags); int lsetea(const char *path, const char *name, void *value, size_t size, int flags); </pre> Just like linux, getea</code> and listea</code> return the size of the actual attribute value, which makes checking for truncation super easy. They also support getting called with a zero size, which will just return the size of the list or attribute value without writing any data to value</code>. The only key difference is that is that they use the character 0xF8</code> to separate the namespace from the attribute name. So querying for system attributes involves querying the name 0xF8SYSTEM0xF8attr</code>. There's also the statea</code> family of functions, which will fill in a struct stat64x</code>, but that's of little consequence to us here. FreeBSD / NetBSD</h2> ssize_t extattr_get_fd(int fd, int attrnamespace, const char *attrname, void *data, size_t nbytes); ssize_t extattr_get_file(const char *path, int attrnamespace, const char *attrname, void *data, size_t nbytes); ssize_t extattr_get_link(const char *path, int attrnamespace, const char *attrname, void *data, size_t nbytes); int extattr_set_fd(int fd, int attrnamespace, const char *attrname, const void *data, size_t nbytes); int extattr_set_file(const char *path, int attrnamespace, const char *attrname, const void *data, size_t nbytes); int extattr_set_link(const char *path, int attrnamespace, const char *attrname, const void *data, size_t nbytes); int extattr_delete_fd(int fd, int attrnamespace, const char *attrname); int extattr_delete_file(const char *path, int attrnamespace, const char *attrname); int extattr_delete_link(const char *path, int attrnamespace, const char *attrname); ssize_t extattr_list_fd(int fd, int attrnamespace, void *data, size_t nbytes); ssize_t extattr_list_file(const char *path, int attrnamespace, void *data, size_t nbytes); ssize_t extattr_list_link(const char *path, int attrnamespace, void *data, size_t nbytes); </pre> FreeBSD and NetBSD both use the same functions for the extended attribute calls. The most obvious difference is that the attribute namespace is no longer part of the attribute name. Each namespace is defined as an constant, and must be passed separately. Almost seemingly as a result of this difference, extattr_list</code> can now error with EPERM</code>, rather than hiding the attribute names that the caller doesn't have access to. The other, more annoying difference, is the return value of the extattr_get</code> and extattr_list</code> functions. Rather than behaving like linux, AIX or OS X, they instead return the number of bytes written, making truncation detection harder. This basically requires that you make two calls if you want to ensure that no truncation will occur. OS X</h2> ssize_t getxattr(const char *path, const char *name, void *value, size_t size, u_int32_t position, int options); ssize_t fgetxattr(int fd, const char *name, void *value, size_t size, u_int32_t position, int options); ssize_t listxattr(const char *path, char *namebuf, size_t size, int options); ssize_t flistxattr(int fd, char *namebuf, size_t size, int options); int removexattr(const char *path, const char *name, int options); int fremovexattr(int fd, const char *name, int options); ssize_t listxattr(const char *path, char *namebuf, size_t size, int options); ssize_t flistxattr(int fd, char *namebuf, size_t size, int options); </pre> OS X differs in a few ways. Notably, their functions all take an options arg. Rather than calling an entirely different function to prevent following symlinks, you can pass the XATTR_NOFOLLOW</code> to prevent traversing symlinks. Another, fairly curious difference is the position</code> argument that's part of the prototype for getxattr</code>. To really get a handle on this, we're going to dive into the wonderful world of forks. Forks</h3> Forks are kind of like having multiple datastreams for the same file. The data that we typically think of being stored in a file is dumped into one fork (in the case of Mac OS, the data fork) and metadata, resources or any other type of data could exist in other forks, wholly independent. On Mac OS filesystems (MFS, HFS, HFS+), each file could have at least a resource fork for the purpose of storing resources about a given file. This was used for things like splitting up icons that Finder would use to represent a file, or for separating presentation and content of text documents. HFS+ (maybe HFS too? I'm not sure) allowed for any number of named forks. Back to OS X</h3> Extended attributes on OS X are actually just named forks. The extended attribute API wholly supplanted the old resource manager API. To ensure that applications could seek to arbitrary points in a fork, OS X's extended attribute API includes a position argument. getxattr</code> is similar to Linux, in that it returns the size of the attribute's data, not just the number of bytes read. This makes truncation detection pretty easy. It is worth noting that extended attribute names in OS X are not namespaced in any special way. Solaris</h2> Solaris gets weird. Solaris is probably closest to OS X in its implementation of extended attributes, in that extended attributes are just named forks. However, Solaris includes only one specialized function call to deal with extended attributes. int attropen (const char *path, const char *attrpath, int oflag, ...); /* the varargs can include a mode argument of type mode_t */ </pre> But even this isn't required, since you can get the same results from using a combination of open</code> and openat</code>: int fd = open(path, O_RDONLY); int attrfd = openat(fd, attrpath, oflag|O_XATTR, mode); close(fd); </pre> From there, all of the *at functions can be used to operate on extended attributes with some restrictions: no links between attribute space and non-attribute space</li> no renames between attribute space and non-attribute space</li> only regular files are allowed - no dirs, symlinks or devices</li> </ul> Otherwise, extended attributes are treated like regular files. This sucks</h2> This is awful when trying to expose a generic, cross-platform API for extended attributes; the only one that I've found is written for perl</a>. I had to add support for FreeBSD in Go, Python and Rust - and none of these deal with Solaris or AIX! Adding FreeBSD support was pretty rough, largely since implementors assume that every OS has a Linux-compatible API. No OS has a Linux-compatible extended attribute API Are attributes namespaced? Are namespaces strings? Are they int</code> constants?</li> Are they named forks? What happens if I need to seek?</li> How big can the data be? How do we check for truncation?</li> Error conditions differ radically</li> </ul> Honestly, I wonder if this contributes to the lack of cross-platform apps that use extended attributes. They're super useful in any case where it's necessary to track metadata about files without having to keep track of it in a separate database. That's honestly fraught with peril anyway, since you're dependent on the name of the file (or whatever identifier you use in your db) staying constant across renames, deletes, etc. Where to go from here?</h2> A C wrapper lib around all of these implementations would be nice, but there are some obvious trade-offs that need to be made. The way that I've done this in Python and Rust has been to: Assume linux-like namespaces, and translate accordingly. If there aren't namespaces in your OS's implementation, then just make the namespace part of the attribute name.</li> Make two calls to get the size of the extended attribute. This works across AIX, Linux and OS X. Solaris will have to use statat</code> to get the size. Unfortunately, race conditions abound.</li> When listing extended attributes, ignore EPERM</code> for system-level attributes</li> </ul> Maybe when I get some time, I'll start working on one. Finally: please, please stop assuming that the whole world is Linux. Sources</h2> https://en.wikipedia.org/wiki/Extended_file_attributes</a></li> http://man7.org/linux/man-pages/man2/listxattr.2.html</a></li> http://man7.org/linux/man-pages/man2/getxattr.2.html</a></li> http://man7.org/linux/man-pages/man2/setxattr.2.html</a></li> http://man7.org/linux/man-pages/man2/removexattr.2.html</a></li> http://man7.org/linux/man-pages/man7/xattr.7.html</a></li> https://www-01.ibm.com/support/knowledgecenter/api/content/nl/en-us/ssw_aix_71/com.ibm.aix.basetrf2/removeea.htm</a></li> https://www-01.ibm.com/support/knowledgecenter/api/content/nl/en-us/ssw_aix_71/com.ibm.aix.basetrf2/setea.htm#setea</a></li> https://www-01.ibm.com/support/knowledgecenter/api/content/nl/en-us/ssw_aix_71/com.ibm.aix.basetrf1/listea.htm</a></li> https://www-01.ibm.com/support/knowledgecenter/api/content/nl/en-us/ssw_aix_71/com.ibm.aix.basetrf1/getea.htm</a></li> http://man.netbsd.org/6.0/usr/share/man/html2/extattr_get_file.html</a></li> https://developer.apple.com/library/mac/documentation/Darwin/Reference/ManPages/man2/getxattr.2.html</a></li> https://en.wikipedia.org/wiki/Resource_fork</a></li> https://docs.oracle.com/cd/E18752_01/html/816-5175/fsattr-5.html#REFMAN5fsattr-5</a></li> </ul> Starting a dev blog Sun, 01 Nov 2015 00:00:00 +0000 I haven't really written anything here in, well, over a year. I've thought a lot about how I really need to start writing again, and what kind of stuff I'd like to write about. Over the last few weeks, I've realized that I'm much more enthusiastic about the stuff that I'm working on when I spend a little time writing or tweeting about it. That said, I think I'm going to move updates about the FOSS projects I'm working on from Twitter to my dev blog</a>. Every day that I work on something, I'll post a bite-sized update on my dev blog. If I come across something interesting to write about outside of just posting commits, then I'll circle back here and write a bit about it in detail. I'd like to keep this low-volume and long-form (except for this post, obviously), if possible. I'm going to try this out for at least the next few days, since I have a lot of FOSS work in my pipeline. An Obfuscated String Implementation for Go Tue, 25 Feb 2014 00:00:00 +0000 Recently, I purchased the domain rit.singles, and I was possessed with the urge of making an actual dating website. I thought it would be cool to write it in Go, since that's one of the hip new languages of the now, and also because it does a lot of things that I really like. I started by writing the account management and registration code. This is surprisingly tricky, because handling sensitive data with any semblance of security is a pain. Luckily, Go exposes a number of very useful syscalls for us to work with. Since I'm not offering enough click-bait on my block, I feel like it's only appropriate to turn the rest of this post into a listicle. 4 Weird Tricks One Florida Man Used to Protect Sensitive Data in Memory!!! (Hackers hate him!)</h2> 1. mlock</code> your pages</h3> Allocate as many pages of memory as you need to hold your string. Then mlock</code> them immediately. mlock</a> is a portable function that strongly encourages your kernel not to swap out a page of memory. Why is this useful?</h4> Swap doesn't get zeroed out, and there's no way to guarantee when it will get overwritten, so you don't know how long any persisted data will last. While encrypting your swap is a good idea, you can't expect everyone to encrypt their swap paritions. Any caveats?</h4> Hibernating will still persist passwords to swap. Sorry. Solaris and Solaris-based operating systems require that you grant the proc_lock_memory privilege before a user can run anything that calls mlock</code>. Memory that you mlock</code> must be page-aligned. I mmap</code> my sensitive memory to ensure this. 2. mprotect</code> your pages</h3> mprotect</a> assigns permissions to your pages. I typically write my sensitive data, and then immediately mprotect</code> the page such that it's read-only. If any process tries to write or execute that memory, that triggers a segfault, which is much better than tampering with sensitive memory. Why is this useful?</h4> Instead of a rogue buffer overrun wiping your sensitive data, you instead trigger a segfault. This helps prevent tampering, either malicious or accidental, of your sensitive data. Any caveats?</h4> Again, the memory that you pass into mprotect</code> must be page-aligned. 3. Encrypt your sensitive strings</h3> It's really just obfuscation rather than offering real security, but it's a worthwhile tactic. Passwords become less trivially identifiable if they're encrypted in memory. Why is this important?</h4> If an attacker gets a memory dump, or if a machine hibernated and an attacker has your swap device, it makes it far harder to identify a password. Any caveats?</h4> Hell yes. The key needs to be stored in memory. If the attacker knows where in memory your key is, then it's trivial to decrypt your sensitive string. It really just ends up being obfuscation more than anything. 4. memset_s</code> your pages before you munmap</code> them</h3> memset_s</a> is a new C11 function that will memset</code> with the guarantee that it won't be optimized out. If the value that you memset</code> isn't used after the memset</code>, smart compilers will optimize it away. If you're clearing sensitive memory after you're done using it, this is definitely not what you want. Why is this important?</h4> When you free</code> or munmap</code> memory, it isn't wiped before being given back to the OS. If an attacker gets a memory dump after you have freed a sensitive block of information, it's very possible that the value is still in memory. memset</code> isn't sufficient for the reason I outlined above. Any caveats?</h4> memset_s</code> isn't implemented in many libc's. If this is a case for your targets, then check out this implementation from CERT</a>. Now where the hell does Go come into all of this?</h2> Since I'm working with passwords for rit.singles in Go, I felt the need to write a secure password implementation in Go. In it, I did all of these things, save for clearing my data with memset_s</code>. As far as I'm aware, Go's compiler won't optimize away a normal memset</code>, so I'm safe there. The code for the secstring</code> package is on GitHub</a>. The documentation is on GoDoc</a>. I welcome any bugs/criticisms/questions/audits. Setting up RetroPie Thu, 17 Oct 2013 00:00:00 +0000 Today, after a few days of work, I finally got RetroPie</a> up and running at an acceptable speed. It was a little longer and a little more difficult than anticipated, so it makes sense to blog about it. SD Card Selection</h2> Surprisingly, your SD Card makes a significant difference in the time it takes to load games and write out game state. The other, non-obvious, impact of the SD card is contention for IO when other tasks on your pi are reading/writing from/to the SD card. I had initially chosen a slow, older class 4 card. Whenever some other process wrote out to disk, my games would noticeably lag a bit. I haven't had that problem since switching to a Class 10 though. Don't just rely on the class number to give you information about the performance characteristics of the SD card. Checkout this handy list</a> before buying. OS Selection</h2> Go with vanilla Raspbian. I've played with Darkbasic's Raspbian</a>, PiBang</a>, and vanilla Raspbian for the host image. Raspbian was the fastest by far (after you disable all of the crap, of course). It's also convenient to have raspi-config and raspi-update installed from the getgo. RetroPie expects that you're running Raspbian, and you can hit some interesting behaviour if you aren't. While I initially liked PiBang, systemd is a poor choice for the pi if you want to emulate. I noticed as I was playing games, systemd was eating a large portion of my CPU, and would occasionally cause lag when there was a sudden burst of activity. Moving to a non-systemd distro (Darkbasic's Raspbian) helped significantly. Darkbasic's Raspbian was slow enough by itself. I never worked out the reason, since it uses the same repos as vanilla raspbian, with all of the same optimizations, however performance just on the terminal was noticeably slower. I'll post benchmarks later (possibly never). Darkbasic's Raspbian image is also very minimal, and not configured the same way as vanilla Raspbian. This can lead to some interesting issues when trying to build and run RetroArch (more later). If you choose not to install Raspbian, and use some minimal RPi distro, I'll include some helpful hints throughout (I set this up several times on the above distros). Let's get started</h2> Install your favorite image normally. Don't stop any services just yet or do any configuring just yet. Stick to a vanilla install, since that's what RetroPie expects (and it doesn't like any deviation). Non-raspbian notes</h3> Add an input group. addgroup --system input </pre></li> Make sure that /opt/vc/lib</code> is early in your linker path Check out /etc/ld.so.conf.d</code>. Some distros will install versions of libEGL and related graphics libraries earlier in your library path. These generally do not work and will throw weird, un-Googleable errors. If you see libEGL warning: DRI2: xcb_connect failed libEGL warning: GLX: XOpenDisplay failed </pre> when starting emulationstation that's symptomatic of this problem. The libraries in /opt/vc/lib</code> are built to use the Raspberry Pi's VideoCore chip. </li> </ol> Pull down the source</h2> After you've logged in and gone through the initial setup for your pi, pull down the RetroPie-Setup source. git clone git://github.com/petrockblog/RetroPie-Setup.git cd RetroPie-Setup </pre> Before we compile, I'd recommend setting your CFLAGS. Thanks to a patch I submitted, RetroPie-Setup now has some sane defaults ( -O2 -pipe -mfpu=vfp -march=armv6j -mfloat-abi=hard</code>), but those aren't fast. I'd recommend -Ofast -fno-fast-math -pipe -mfpu=vfp -march=armv6j -mfloat-abi=hard</code>. -Ofast</code> turns on the fastest optimization level, while potentially building unsafe code. It works quite well for me, but YMMV. </li> -fno-fast-math</code> turns off a GCC optimization that -Ofast enables. fast-math uses some "faster" math that doesn't give you the guarantees of some of the standard math functions. In some cases, it's also slower</a>. </li> -mfpu=vfp</code>, -march=arm6j</code> both tell gcc</code> to generate code specific to the raspi architecture, as well as what kind of fpu-specific machien code to generate. These two flags are both used by the raspbian project to build their packages </li> -mfloat-abi=hard</code> tells the compiler to generate fpu-specific instructions. Without this flag, gcc</code> will generate floating point code that will not run on the fpu, but will link to library functions that emulate floating-point code. This is slow. All raspbian packages are built with this flag (that's what puts the 'hf' in 'armhf'). </li> </ul> If you're worried about the problems the new flags could introduce, then stick to the defaults. If you're just reading this and you already have RetroPie installed, you should probably rebuild to get the new compiler options ;). Set the CFLAGS with export CFLAGS=-Ofast -fno-fast-math -pipe -mfpu=vfp -march=armv6j -mfloat-abi=hard export CXXFLAGS="${CFLAGS}" export ASFLAGS="${CFLAGS}" </pre> After all of this, it's just a quick sudo ./retropie_setup.sh</code>. Pick and choose what you want; it's really none of my business, and it's quite easy. Make sure you pick the source-based builds (I guarantee that the binary ones weren't built with any good options) and that you tell the installer to start emulationstation</code> on boot. Post installation</h2> Disable everything. update-rc.d avahi-daemon remove update-rc.d bluetooth remove update-rc.d cron remove update-rc.d cups remove update-rc.d dbus remove update-rc.d ifplugd remove update-rc.d lightdm remove update-rc.d nfs-common remove update-rc.d rpcbind remove update-rc.d rsync remove update-rc.d rsyslog remove update-rc.d saned remove update-rc.d triggerhappy remove update-rc.d ntp remove ls /etc/rc3.d K06rpcbind README S02dphys-swapfile S02ssh S05plymouth S05rmnologin </pre> I use ssh to manage the system, so I leave that. Everything else must go. Don't get me wrong, usually I like logging, ntp</code> and cron</code>, but it's just not necessary on a pi that's dedicated to playing games. PiBang/systemd notes</h3> PiBang uses this half-assed systemd with sysvinit compatibility. It's a pain in the ass. Almost all of the packages that need to start at boot that you get on PiBang need to be disabled with update-rc.d</code>. Some others though need to be disabled with systemctl disable <service></code>. I'll leave it as an exercise to the reader to figure out how to tell. Minimal image notes</h3> You're probably sitting there laughing at how much bloat Raspbian comes with. While that's fair, keep in mind that the RetroPie script installs a ton of shit, cups and sane included. Make sure you audit your startup as well. Configuring a joystick</h2> Your joystick needs to be configured before you can start playing games. Emulation Station will configure it for you when you first start it up, however that configuration only applies to Emulation Station. You need to configure your emulators on their own. For RetroArch: cd ~/RetroPie/emulators/RetroArch/tools/ ./retroarch-joyconfig -o p<playernumber>.cfg -p <playernumber> -j <joysticknumber> # joysticks are 0 indexed (they map to /dev/input/jsn # players are 1 indexed cat p*.cfg >> ~/RetroPie/configs/all/retroarch.cfg </pre>Non-raspbian notes</h3> You'll need a udev</code> rule to automatically give the input</code> group permissions to access the joystick devices. Drop the following into /etc/udev/rules.d/99-input.rules</code> SUBSYSTEM=="input", GROUP="input", MODE="0660" </pre> This ensures that the input</code> group owns any new input devices, and that they are group-readable. But moooom, I want to use an XBox controller</h3> No. You don't. You can use xboxdrv</code> to hook up your XBox controller. It's a userspace driver that talks to the raw joystick device and interprets the XBox controllers's raw output into something meaningful. Since the driver lives in userspace, anytime you twiddle a joystick or press a button, you're hopping like mad between userspace and kernelspace. This is slow. I could actually emulate the slow-mo feature provided by RetroArch just by twiddle both joysticks. No. I'm a special snowflake and I demand to know how to use an XBox controller</h3> apt-get install xboxdrv sudo -s echo xboxdrv --silent > /etc/rc.local </pre> If you don't want to reboot, just execute xboxdrv --silent & </pre> as root. You've been warned. Starting emulationstation on boot</h2> Luckily, RetroPie did half of the work for you! If you selected the option (either in setup or install) to start emulationstation</code> on boot, it'll drop a handy snippet into /etc/profile which will do that on login. However, you still need to login. To alleviate that, edit your inittab. 1:2345:respawn:/sbin/getty --autologin pi --noclear 38400 tty1 #1:2345:respawn:/sbin/getty --noclear 38400 tty1 #2:23:respawn:/sbin/getty 38400 tty2 #3:23:respawn:/sbin/getty 38400 tty3 #4:23:respawn:/sbin/getty 38400 tty4 #5:23:respawn:/sbin/getty 38400 tty5 #6:23:respawn:/sbin/getty 38400 tty6 </pre> Go to line 45 (on raspbian, or where the gettys are started), and enable autologin for whatever user is going to run RetroArch. In the spirit of disabling everything, I also stopped all of the other gettys from spawning. Should I overclock?</h2> Well, it depends. Want to run SNES games that have Mode-7 graphics? Or any of the SNES games that packaged an extra processor on the cartridge? Yes. Overclock. In most other situations, you're fine if you follow this guide. The one thing I would recommend doing, however, is changing the memory/gpu mem split. I gave my GPU 384MB of memory. I'm barely breaking 20MB of memory usage running my pi, so I don't need very much system memory. Your game will start slowing down as the proc overheats, so make sure you have a heatsink and a case with some kind of airflow if you plan on overclocking. Notes for non-raspbian users</h3> Yeah, you probably do want to overclock. If your distro is compiled with dumb options, it'll help a bit. On both Darkbasic's raspbian and PiBang, I had to overclock to get decent performance. Conclusion</h2> That's about as perfect a setup as you can get for RetroPie. If you have any other suggestions, leave them in the comments. I didn't configure RetroArch anymore than shown above, because well, I didn't need to. For all of my ROMs built by the homebrew community that are totally free and legal to play, I ran into no slowdown with this setup on Raspbian. However, YMMV, as always. OpenSMTPD Sun, 18 Aug 2013 00:00:00 +0000 I love OpenSMTPD</a>. After years of running a postfix installation, OpenSMTPD is such a joy to configure and use. The configuration is dead simple, and it's pretty damn stable. I wrote a protip</a> about it over on Coderwall. I plan on following up with more protips about more advanced configurations, with lmtp, SSL/TLS and greylisting. Obviously, it's now the mail server running mail on worrbase.com</a>, as well as the mail servers for my spamd test cluster (remember to email ubuntu@spamd.worrbase.com</a>!). Oh, and my friend Michail Yasonik</a> redesigned my blog. It looks significantly better than what I had up before. Update: spamd on Linux Sun, 28 Jul 2013 00:00:00 +0000 I finally took some time this weekend to deploy spamd on some ec2 instances. I have 2 mail servers on ubuntu 13.04 running the latest copy of spamd on linux, with synchronized greylisting tables. I've run into a few bugs with deployment, which I've added to the github issues. Hopefully I can get them resolved soon. I did run spamd through a small gamut of tests with postal, a SMTP stress tester. They handled the excess load beautifully, keeping the synchronization tables up to date even at 1000 mails/min (well, it was less than that, due to spamd's resource draining feature) for 8 hours. If you want to help test spamd, it'd be super great if you can sign up ubuntu@spamd.worrbase.com</a> for all kinds of sketchy services. Maybe even forge the email address and reply to some spam messages (that's what I'm doing). I'd like to continue testing it for the next week or so with some real-world spam load. The only bit left to port is spamlogd. I haven't figured out the most logical way to translate it to iptables, since iptables lacks a logging device akin to pflogd. Again, please direct spam to ubuntu@spamd.worrbase.com</a> (or even hostmaster@spamd.worrbase.com</a> to test greytrapping). spamd on Linux Sun, 23 Jun 2013 00:00:00 +0000 Somehow, I became a mail administrator for CSH several years ago. I don't know why I did that to myself. Regardless, I started deploying a secondary MX earlier this week and I ran into a problem - greylisting. We use greylisting to greatly reduce the amount of spam amavis needs to process. We used gld on Debian, quite successfully might I add, for the last few years. Greylisting, in case you don't know, is a simple technique to reduce the amount of spam you receive. Your mail server responds to any connection from an unknown mail server with a temporary failure, and makes a note of the ip, mail from and rcpt to that the mail server sent. Now legitimate mail servers, upon seeing a temporary failure, will try again. Many spam mail servers, on the other hand, won't (why? I don't fucking know). This is surprisingly successful. There are graphs all over the internet showing the spam reduction, and I'll leave finding them as an exercise to the reader. Anyway, back to gld. gld is a simple greylisting daemon (hey, I wonder where the name gld came from...) that's backed by a postgresql db. That's fine, but the only solution for managing greylists across multiple MX's is postgresql replication. That feels a little hacky to me...not to mention that gld seems pretty unmaintained. So I looked at postgrey, which is a postfix milter that does greylisting. Well, that's not a great solution because we have even fewer options for greylist replication across MX's. Now, I run my own mail server (surprise! @worrbase.com) on OpenBSD using OpenSMTPD and spamd. spamd is the OpenBSD greylisting solution, and it's by far the best I've used. Features include: all the normal greylisting features (blacklists! whitelists! greylists!)</li> replication</li> it saps spammer's resources</li> there's basically no configuration</li> greytrapping</li> the SMTP conversations are hilarious</li> </ul> Well, armed with this knowledge, I ported spamd to Linux. I can't promise that it's terribly stable yet, but you're welcome to test it and file bug reports. Here's</a> the github page. Leaving sugar behind...or am I? Thu, 09 May 2013 00:00:00 +0000 Well, my H/FOSS class is coming to an end. In a few short hours, I'll finish my last H/FOSS class with a presentation on our work on Sugar (coincidentally it's also my last college class). At this point, I've realized that Sugar is a fairly disorganized project. It seems like it's constantly in flux, with multiple wikis, bugtrackers, git repos, etc. It's very confusing to get started deving on - probably the most difficult project to enter in my experience in FOSS. That being said, walking away will be hard...well, not that hard since I don't plan on walking away just yet. I was assigned some tasks by the team after my initial set of patches, so I'll make sure to finish those up. Then what? I'll probably run away and never look back. While I can appreciate the work the Sugar devs do, it's not a project that I find particularly compelling, so I plan on leaving it once I've submitted my last set of patches. Fixing Sugar bugs Fri, 03 May 2013 00:00:00 +0000 After a late start, Matt and I have been blazing ahead on development of Sugar. As of this writing, we have 6 (almost 7) patches merged into Sugar upstream. The biggest hurdle in getting Sugar installed was that I wasn't running it on a supported OS. I was building and testing on Fedora 19, where it only supports Fedora 18 and Ubuntu whateverthefuck. As a result, my first 5 patches fixed various problems with the build system, including adding support for Fedora 19. After some vigorous testing and discussion, all 5 patches were merged in. Afterwards, I found a crash in the journal caused by mousing over the Pippy activity icon. The crash was caused by a call to a deprecated function...a function prevalent across the entirety of Sugar. My 6th and 7th patches are fixing this sweeping problem. While we haven't quite fixed the kinds of usability bugs we first set out to fix, we are making a significant impact on Sugar and are helping fix the contribution process. Week 2 in Sugarland Thu, 11 Apr 2013 00:00:00 +0000 Hey all, It's Thursday during H/FOSS class, which means that it's time for an update on my progress with Sugar. We've made very little progress - we've uncovered some more network-related usability bugs, but haven't produced any working code yet. Our environments have been set up, however Matt's got blown away in an OS reinstall. Sugar Progress Thu, 04 Apr 2013 00:00:00 +0000 Sugar progress has been slow so far - Matt and I have mostly been working on administrative stuff. I specifically set up the project page, and communicated with the sugar community about the validity of some of our bugs. Not much more to report than that. Introduction to Perl Thu, 28 Mar 2013 00:00:00 +0000 Hey everyone! I made a Perl seminar for CSH</a>! Here it is!</a> And here is the code!</a> This isn't even close to conclusive, but it does give you a basic introduction. I cover certain topics that are important, and omit ones that are actually bad (default variable, 2 argument open). My thoughts on Open Source Mon, 11 Mar 2013 00:00:00 +0000 Surprisingly enough, this post is not for my FOSS class. Anyway, this</a> article has been circulating around my network lately, and it led me to think about my stance on open source. I love open source software, both writing it and using it. Why? First and foremost, I'm a student. Learning is quite literally my profession. And hopefully after I graduate in May, I'll continue to learn. I love technology because to be truly successful, you need to continue to learn throughout your career. Open source allows me to learn by working with other people's code. Whenever I jump into a large codebase, I tend to pick up techniques and design strategies employed by the original programmer. By doing this, I've also gotten progressively better at jumping into large projects quickly and figuring out exactly what I need to do to add whatever feature or fix whatever bug. I also tend to get frustrated by poor software. There are a lot of open source projects that I like that lack features, or are bug-ridden. Whenever I get pissed off by something that doesn't work, I can fix it. And then I can push my changes upstream and fix it for everyone. Not only is it empowering, but I don't have to change my workflow/program to account for other people's broken code. A benefit of pushing your code upstream is the feedback you get on it. Most open source developers will analyze your code before they merge it in, give you feedback and allow you to fix your change. It's a great way to get some constructive criticism on your code. But the best benefit to working on open source is how it enhances my résumé. As I mentioned before, I'm a student. There was a point where I was struggling to get my first post-college job in the field. Having a github filled with open source projects that I've pushed to as well as open source projects of my own was a huge advantage over the competition - it helped me really stand out. Never will I parrot the fucking notion that all software should be Free - it shouldn't be. I'll never support viral open source licensing - that's not freedom and that hurts businesses, both small and large. But I will say that open source is invaluable to students looking to get out into industry, and that it allows people already in industry to train old skills as well as learn new ones. FOSS Class Introduction Thu, 07 Mar 2013 00:00:00 +0000 So, first, I royally messed this up when I first committed this blog entry, and wrote about 3 things I enjoyed doing, rather than writing about 3 Sugar activities. That post has been removed - largely out of embarrassment That being said, here are my three favorite Sugar activities. Wine</h2> The first activity I find interesting is Wine. This isn't because I think the Wine project is particularly interesting by itself, but because I find it surprising that Wine has been "ported" to the OLPC as well as its reception. One reviewer basically says that it's how he does normal computing on his OLPC, which is curious to me, as there really ought to be Activities to handle his needs (in this case, playing games and listening to music). Etoys</h2> Etoys was pretty much guaranteed to make this list, as I'm (obviously) a programmer. It seems like a smart way to make programming more appealing to children, so that they can get hooked at a young age. GetBooks</h2> GetBooks looks great, as it makes book acquisition trivial for people who want to read. I'm not sure what kinds of books you can find using OPDS, but the idea by itself is laudable. I honestly can't wait to see what kinds of texts are available to me. Managing ports in FreeBSD with CFEngine3 Mon, 06 Aug 2012 00:00:00 +0000 CFEngine is a fantastic tool for managing configuration files and packages across heterogeneous environments. I figured I could use it to maintain a basic set of installed packages across my Debian, OpenBSD and FreeBSD systems. Except that the FreeBSD package_method promises in the cfengine standard library suck. They use FreeBSD packages, and unless you have a box building your own sets of FreeBSD packages...packages suck. They are built with all of the bells and whistles, ergo they pull in a ton of shit you probably don't need on your system. So I wrote a package_method promise that instead uses portmaster to install and remove packages. Here's the code: body package_method freebsd_portmaster { package_changes => "individual"; package_list_command => "/usr/sbin/pkg_info"; package_list_name_regex => "([^\s]+)-.*"; package_list_version_regex => "[^\s]+-([^\s]+).*"; package_installed_regex => ".*"; package_name_convention => "$(name)"; package_delete_convention => "$(name)-$(version)"; package_file_repositories => { "/usr/ports/accessibility/", "/usr/port/arabic/" "/usr/ports/archivers/", "/usr/ports/astro/", "/usr/ports/audio/", "/usr/ports/benchmarks/", "/usr/ports/biology/", "/usr/ports/cad/", "/usr/ports/chinese/", "/usr/ports/comms/", "/usr/ports/converters/", "/usr/ports/databases/", "/usr/ports/deskutils/", "/usr/ports/devel/", "/usr/ports/dns/", "/usr/ports/editors/", "/usr/ports/emulators/", "/usr/ports/finance/", "/usr/ports/french/", "/usr/ports/ftp/", "/usr/ports/games/", "/usr/ports/german/", "/usr/ports/graphics/", "/usr/ports/hebrew/", "/usr/ports/hungarian/", "/usr/ports/irc/", "/usr/ports/japanese/", "/usr/ports/java/", "/usr/ports/korean/", "/usr/ports/lang/", "/usr/ports/mail/", "/usr/ports/math/", "/usr/ports/mbone/", "/usr/ports/misc/", "/usr/ports/multimedia/", "/usr/ports/net/", "/usr/ports/net-im/", "/usr/ports/net-mgmt/", "/usr/ports/net-p2p/", "/usr/ports/news/", "/usr/ports/packages/", "/usr/ports/palm/", "/usr/ports/polish/", "/usr/ports/ports-mgmt/", "/usr/ports/portuguese/", "/usr/ports/print/", "/usr/ports/russian/", "/usr/ports/science/", "/usr/ports/security/", "/usr/ports/shells/", "/usr/ports/sysutils/", "/usr/ports/textproc/", "/usr/ports/ukrainian/", "/usr/ports/vietnamese/", "/usr/ports/www/", "/usr/ports/x11/", "/usr/ports/x11-clocks/", "/usr/ports/x11-drivers/", "/usr/ports/x11-fm/", "/usr/ports/x11-fonts/", "/usr/ports/x11-servers/", "/usr/ports/x11-themes/", "/usr/ports/x11-toolkits/", "/usr/ports/x11-wm/", }; package_add_command => "/usr/local/sbin/portmaster -D -G --no-confirm"; package_update_command => "/usr/local/sbin/portmaster -D -G --no-confirm"; package_delete_command => "/usr/local/sbin/portmaster --no-confirm -e"; expireafter => 240; } </pre> The only interesting thing about the body of the promise is that I'm using package_file_repositories in a way that it was never intended to be used. Since the port names needed to be provided to portmaster in a / format, I'm leveraging a cfengine trick to kind of fake that. If you provide a package_file_repositories clause, cfengine will prepend a variable called $(firstrepo) to the name of every package that it adds to the package_add_command line. Armed with this knowledge, we just add a package_file_repositories entry for each port category directory, and the problem's been solved. CFEngine SVN pre-commit hook Wed, 11 Jul 2012 00:00:00 +0000 I use CFEngine on my personal site to manage my configuration. I have an SVN repo on one box that I can push configuration changes to, and then from there, the changes get pushed out to all of my boxes. One problem that I encountered is that changes to my CFEngine configurations that had syntax errors would be pushed out to all of my machines, and cause them to fallback to the failsafe configuration, which is less than ideal. My solution was to write a pre-commit hook for my SVN repo that would prevent me from checking anything in that would break my configuration. #!/bin/bash REPOS=$1 TXN=$2 LOG="/home/svn/commit.log" CHANGES=0 if [[ "$1" = "" && "$2" = "" ]]; then echo "Most be called as $0 REPOS TXN" >> "$LOG" exit 1 fi AWK="/bin/awk" GREP="/bin/grep" RM="/bin/rm" ECHO="/bin/echo" SED="/bin/sed" SVN="/usr/bin/svn" SVNLOOK="/usr/bin/svnlook" SVN_URL="file:///home/svn/cfengine/trunk" CF_PROMISES="/usr/sbin/cf-promises" PROMISEDIR="/home/svn/.cfagent/inputs/" MODULESIDR="/home/svn/.cfagent/modules/" "${ECHO}" "REPOS ${REPOS}" >> "$LOG" "${ECHO}" "TXN ${TXN}" >> "$LOG" "${ECHO}" "Checking out working copy from $REPOS ...." >> "$LOG" "${SVN}" co --depth immediates "${SVN_URL}/cf-agent_policies" "${PROMISEDIR}" > /dev/null if [[ $? -ne 0 ]]; then "${ECHO}" "Failed to get fresh working copy" >> "$LOG" exit 1 fi "${SVN}" co --depth immediates "${SVN_URL}/cf-agent_modules" "${MODULESIDR}" > /dev/null if [[ $? -ne 0 ]]; then "${ECHO}" "Failed to get fresh working copy" >> "$LOG" exit 1 fi # Figure out what directories have changed using svnlook. FILES=`${SVNLOOK} changed ${REPOS} -t ${TXN} | ${AWK} '{ print $2 }'` > /dev/null for FILE in $FILES; do "${ECHO}" "file changed ${FILE}" >> "${LOG}" if [[ $("${ECHO}" "${FILE}" | ${GREP} cfengine/trunk) != "" ]]; then if [[ $("${ECHO}" "${FILE}" | ${GREP} generic_cf-agent_policies) != "" ]]; then if [[ $("${ECHO}" "${FILE}" | "${SED}" -e 's/home\/svn\/cfengine\/trunk\/cf-agent_policies\///g') != "" ]]; then CHANGES=1 "${SVNLOOK}" cat "${REPOS}" "${FILE}" -t "${TXN}" > ${PROMISEDIR}/$("${ECHO}" "${FILE}" | "${SED}" -e 's/home\/cfengine\/trunk\/cf-agent_policies\///g') fi fi fi done if [[ ${CHANGES} -eq 0 ]]; then exit 0 fi "${CF_PROMISES}" --dry-run > /dev/null if [[ $? -ne 0 ]]; then "${ECHO}" "Syntax error! Aborting commit!" >> "${LOG}" "${RM}" -rf "${PROMISEDIR}" "${RM}" -rf "${MODULESIDR}" exit 1 fi "${RM}" -rf "${PROMISEDIR}" "${RM}" -rf "${MODULESIDR}" </pre> This should be the pre-commit file in the hooks directory of your SVN repo. If that file exits with a non-zero code, the commit will fail. The file must be executable by the svn user. Book Review: Programming Perl Tue, 10 Jul 2012 00:00:00 +0000 NOTE: Hey! This is a book review I wrote some time ago that was published here</a>. This review is outdated, mostly because there IS a new edition about the last major release of perl, perl 5.14. Because of the release of the new edition, I thought it might be good to reproduce the review on my blog. The camel book is a book that needs no introduction. The Perl community considers it the ultimate Perl book, and rightfully so, as it is written by Larry Wall himself (also Christiansen and Jon Orwant)! The camel book is an in-depth look at the Perl programming language. The book does a fantastic job explaining how to use the language, the more powerful aspects of the language, as well as some of the implementation details of the compiler. As you progress through the book, it constantly builds upon itself. Different quirks/features of the language which are first introduced as black magic are explained in exhaustive detail later on. For example, you first learn about what the autoload pragma is, followed by advanced usage of goto, and then they tie that together by detailing how autoload is implemented with gotos. As a side note, I know this sounds gross, but gotos aren't what you think they are in Perl... at least not always. The writing is, for the most part, very clear and understandable, and contains a heavy dose of humor. The humor is definitely welcome, as the text would be horribly dry without it. The humor also tends to be representative of the Perl community at large, which is fabulous for the new Perl programmer. The only problem I had with the writing style was the chapter on regular expressions. In this chapter, the wording is murky at times, and more examples would be a welcome addition. Like I said though, this is only one chapter. The biggest problem with this book is that it's old. The latest edition (3rd at the time of this writing) only covers up to Perl 5.6. We're at Perl 5.12 now, with 5.14 on the horizon, and a significant amount has changed. However, that doesn't make this book irrelevant. I think it makes a wonderful companion to Modern Perl by chromatic. Once someone has a handle on how to write modern Perl (>=5.10), this book is definitely a great look at the rest of Perl as well as an interesting peek into the internals of the language. Regardless of recent developments in the language, this book still has a significant amount of relevant content. If you want to truly become a Perl master, you need a copy of this book, but keep in mind you'll also probably want a book that covers more modern Perl. CrashPlan on FreeBSD 9.0, A HOWTO Sat, 31 Mar 2012 00:00:00 +0000 This was an incredibly productive week for me. I got a lot done, including setting up Nagios for my servers, starting a CSH Photohunt Android app, working a bit on a game that I've been writing, and most importantly, rolling out network backups. And that's what this post is about. CrashPlan</a> is a cloud backup service that allows users to backup unlimited data for a small monthly fee. For people with large amounts of data, this is a significantly cheaper solution than Amazon S3 or Carbonite or similar. I quickly purchased the unlimited backup service, and decided to start backing up all of my computers using their service. While I really hate the GUI tool, I think the service itself is fantastic. Here's a fantastic review</a> of the backup service, as well as a coupon. Let me first start out by saying that I already do backups. I do Time Machine and rsync backups to my FreeBSD NAS. What I was looking for initially was a cloud backup service that allowed me to backup my entire NAS to the internet (if I say "cloud" one more time...). CrashPlan doesn't "support" FreeBSD. That is to say, they don't officially provide a client to allow you to backup from FreeBSD...but it's more than possible. So without further ado... How to backup your FreeBSD 9.0 machine with CrashPlan</h2> First, set up and install the linux compatibility environment. Open up your make.conf and add these two lines: OVERRIDE_LINUX_BASE_PORT=f10 OVERRIDE_LINUX_NONBASE_PORTS=f10 </pre> We want to install the Fedora 10-based environment, since it's the newest and best supported one at the time of this writing. Then, load the linux.ko kernel module, and add the option startup option to your rc.conf. # kldload linux.ko # echo "linux_enable=\"YES\"" >> /etc/rc.conf # echo "crashplan_enable=\"YES\"" >> /etc/rc.conf </pre> Now, install the compat environment, java and linproc # portmaster emulators/linux_base-f10 # portmaster sysutils/linux-procps # portmaster java/linux-sun-jre16 </pre> Add linproc to your /etc/fstab, as well as some other mounts while we're here # echo "linproc /compat/linux/proc linprocfs rw 0 0" >> /etc/fstab # echo "/usr /backup/usr nullfs rw 0 0" >> /etc/fstab # echo "/var /backup/var nullfs rw 0 0" >> /etc/fstab </pre> When you're in the linux compat environment, /usr and /var are resolved to /compat/linux/usr and /compat/linux/var transparently. We need those nullfs mounts so that we can actually back up /usr and /var. Those are a bit important. Now mount the shit out of everything. Download crashplan somewhere under /compat/linux, and then chroot into your linux environment. Run the install script, and back out. # chroot /compat/linux/ /bin/bash # cd CrashPlan-install && ./install.sh </pre> chrooting is not absolutely necessary, but whatever. Moving on! We have some stuff to configure with CrashPlan before it'll run well, and we're going to need to find/write ourselves an init script as well! First, in /compat/linux/usr/local/crashplan/bin/run.conf, modify the two lines to end with this: SRV_JAVA_OPTS="... -Djava.nio.channels.spi.SelectorProvider=sun.nio.ch.PollSelectorProvider" GUI_JAVA_OPTS="... -Djava.nio.channels.spi.SelectorProvider=sun.nio.ch.PollSelectorProvider" </pre> The important part is the last define on both lines. CrashPlan uses epoll, which isn't available to us on FreeBSD. That line essentially tells CrashPlan to behave itself on our platform. Another thing we ought to do is to tell CrashPlan where Java is. Add a line to /compat/linux/usr/local/crashplan/install.vars that does that JAVACOMMON=<YOUR PATH TO LINUX JAVA> </pre> Neat! Now we just need an init script...and we can get that from one of the two sources I got some of this information from!</a> All we care about is the init script at the bottom. We've already done all of the other steps listed (and more!), so don't worry about those. We've done some things differently, but IMO this is a better setup. Copy the contents of that initscript to /usr/local/etc/rc.d/crashplan, and you're good to go. You can start it with # service crashplan start </pre> Here</a> is the other source of my information regarding CrashPlan installation on FreeBSD. Here</a> is the official documentation for running CrashPlan headless. Happy backups! OSS and App::GitHub Tue, 27 Mar 2012 00:00:00 +0000 I have a love hate relationship with open source software. On the one had, I get incredibly frustrated by the lack of support, the lack of documentation and the spotty code quality. On the other, I love having the ability to fix other people's code. It's been an incredibly important tool in my education. I definitely would not have nearly the knowledge that I do now if it weren't for the ability to quickly and easily involve myself in open source projects. Not only that, it's also convenient to have the ability to fix broken software that I'd like to use. Honestly, while I find that closed-source software has significantly fewer bugs, better documentation and better support - I have to say that that last point is a very compelling case for the use of open source software...by developers. I don't understand why non-developers get so evangelical about open source software. I just can't understand what the appeal is for them. Sure, if the software is stable and fulfills a need, by all means use it - but why the hell are there so many non-developer Linux users that constantly and mindlessly praise open source? But I digress. Regardless, for me, as a part-time developer, open source software is tits, simply for the ability to fix bugs that piss me off. I ran into a great example of this this past weekend, actually. I found a fantastic Perl module, App::GitHub</a>. While the idea is fantastic, the version I tried to use was horribly broken. A beer and 4 patches later, I fixed it and was offered co-maintainership. So this is also half of an announcement of a new release of App::GitHub that doesn't break when you try and use it. AdiumQQ Tue, 17 Jan 2012 00:00:00 +0000 Sometimes, the best projects come out of the dumbest disputes. I was in my room, and my former roommate Clockfort</a> and Russ</a> got talking about old chat protocols that we used to use. Clockfort suddenly remembered his old ICQ number, and we went around adding each other on all of our old chat networks. I took this joke a little far, however. I started making all kinds of new accounts. I went through every protocol supported by pidgin, and signed up for accounts. At the time, pidgin listed QQ</a> as a supported protocol. I signed up for a QQ account, and tried to connect with pidgin. Unfortunately, it just didn't work. I found out that the builtin QQ plugin didn't support the latest QQ protocol. I fount a plugin called libqq-pidgin</a> that did exactly what I needed. So I installed it, and added my one Chinese friend to my buddy list. Later that night, I wanted to talk to her at work. I booted up Adium, and added my QQ account. Unfortunately, it didn't work, because like pidgin, Adium is built on top of libpurple, and libpurple dropped official support for QQ. My solution was to write my own QQ plugin for Adium. The end result is AdiumQQ</a>. It supports basic features right now, like chat and such. It does not support file transfers or captchas just yet. Those features are coming. Today, I'm posting about it in my blog because it just got accepted in Adium Xtras</a>. I'm really proud, since this is my first open source program I've written that people are finding useful. Thanks to the libqq-pidgin team for making a nice base for me to build my plugin off of! I will of course announce any new release information on my blog. rvm and perlbrew Tue, 27 Dec 2011 00:00:00 +0000 I was kind of an idiot in my last post. I wrote a ruby script to symlink my gems' executables to /usr/local/bin. Well that's dumb. There's a much better way to accomplish the same goal, with some healthier side effects! It's called rvm</a> (or Ruby Version Manager)! rvm allows you to manage multiple versions of ruby that are installed to a location in your home directory. You can select a version of ruby to use for current session with rvm use 1.9.2 That command will use ruby-1.9.2, provided you installed it with rvm. To install: rvm install 1.9.2 When you have a version of ruby selected, gems will install into the directory that houses that version of ruby, and won't pollute your system install of ruby. App::perlbrew (installable through cpan) does exactly the same thing, albeit with Perl! I heartily recommend both if you do development with either ruby or Perl. Ruby and Gems from Homebrew on Mac OSX Wed, 15 Jun 2011 00:00:00 +0000 Hey, so I bought a Macbook Air a month ago, and I've grown to absolutely adore it. It basically comes down to that I have an OS that I don't need to fix constantly, while also having access to all of the utilities I'm used to using on my Linux/UNIX environments. I use Homebrew</a> to manage my software, because it seems to be the best package management solution for OS X. I haven't really tried any others, but I have no inclination to after using Homebrew for the past few months. Oh and I also learned Ruby. And I kind of like it so far. Regardless, this post isn't about my Ruby experience thus far. It's just documenting a minor annoyance. When I install gems using ruby1.9 from Homebrew, the gems don't make a symlink in /usr/local, and the internet tells me the right thing to do is to make it myself. Well fuck that. Here's a brief ruby program that will insert these links for you! I'm not claiming to be a Ruby expert, or even competent, so run this at your own risk. It can get a little destructive, and makes some assumptions. #!/usr/bin/env ruby require 'RubyGems' installed_gems = [] prefix = '/usr/local/bin/' ruby = '/usr/local/Cellar/ruby' Gem::Specification::all_names().reverse.each do |gem| (name, version) = ['', ''] gem.match(/([\w\-]+)\-([\w\.]+)/) do |m| name = m.captures[0] version = m.captures[1] end Gem::Specification.new do |s| s.name = name s.version = version installed_gems.push(s) end installed_gems.delete_if do |i| name == i.name and Gem::Version.new(version) > Gem::Version.new(i.version) end end installed_gems.each do |gem| if File.directory? gem.bin_dir then Dir.entries(gem.bin_dir).each do |file| fullpath = gem.bin_dir + '/' + file newpath = prefix + file if not File.directory? fullpath then if File.symlink? newpath then if File.readlink(newpath).match %r{/usr/local/Cellar/ruby} then puts "Linking #{fullpath} to #{newpath}..." File.unlink newpath File.symlink fullpath, newpath end end end end end end </pre> Hope that helps. Hotel Internet Tue, 14 Jun 2011 00:00:00 +0000 Have you ever been to a hotel, or an airport, that served wireless? And the after you connected, you get redirected to a page where you pay money, or make an account? Those are called captive portals. Setting up your own is really easy! The old de facto FOSS software for captive portals was Chillispot</a>. However, this software is old, and broken, and doesn't really work anymore. So, I used its drop in replacement, CoovaChilli</a>. CoovaChilli is totally compatible with Chillispot, so if you've used Chillispot before, you can use your old configuration with CoovaChilli without any modification. CoovaChilli handles serving DHCP to your clients, as well as setting up routing for you. Your server acts as a router for traffic for auth'd clients. CoovaChilli creates a tun device that it tunnels traffic through, kind of like a VPN. Since it serves DHCP, you can configure it to give your clients DNS servers, proxy information, etc. CoovaChilli will also load up a whole bunch of firewall rules that will restrict what resources clients can access on your network. First step, set up a simple FreeRadius configuration. I left mine pretty with the Debian defaults, and just created some accounts for testing. This system isn't in production, so I don't care about tuning my radius setup. And FreeRadius isn't the point of this entry. Once, you have FreeRadius going, download, compile and install CoovaChilli. CoovaChilli doesn't have Debian or Ubuntu packages that work very well...and building packages from the source doesn't work perfectly either, so just ./configure, make and make install. Don't worry, there's a make deinstall in case you want to remove it cleanly. After building it, copy /etc/chilli/defaults to /etc/chilli/config and start editing. Some things you'll want to change: HS_DNS1 and HS_DNS2: the DNS servers given to your hosts</li> HS_RADSECRET: the secret required to talk to your Radius server</li> HS_UAMSECRET: the secret required to talk to your hotspot</li> HS_UAMSERVER: URL to the login page</li> HS_POSTAUTH_PROXY: we'll talk about this later</li> </ul> You'll also probably want to set your interfaces. Your WAN interface needs to be set through your OS, but your LAN interface requires absolutely no configuration through your OS - CoovaChilli will do it when it comes up. It's also important to run this: sysctl net.ipv4.ip_forward=yes That will enable IP forwarding on your system. The other values can be left with their default values. We're not going to set up a custom login page because we really don't care. The default is fine for us. You're going to need to install haserl</a> to use it. Well cool, we now have a captive portal! Run the init script, and bam. You now have a captive portal. Tomorrow, we'll talk about going beyond basic configuration, and content modification using Squid3 and GreasySpoon. Faking STDIN in Perl unit tests Wed, 11 May 2011 00:00:00 +0000 I had this question the other day when I was writing my Exherbo packager the other day. I never found a good, conclusive answer through Google or Identi.ca/Twitter, so I tried something fairly obvious, and it worked! When doing unit tests for an interactive program, oftentimes the need comes up to fake STDIN, and provide some test interactions. In Perl, this is actually rather trivial. A note on filehandles in Perl</h2> In Perl, we often see this syntax for opening files: open(HANDLE, 'path'); print while (<foo>); close(HANDLE); </pre> This is bad style for a number of reasons (not checking to see if open was successful, not using lexical filehandles, not using 3 param open()...), but regardless, there is a thing or two worth noting here that's relevant to the rest of the discussion. When you open a filehandle in this manner, you aren't getting a scalar, an array, or even a hash back. You're getting a global typeglob. What the hell does that even mean? Perl has three separate symbol tables for storing variables, one for each type. Typeglobs are a way to access all three, and are denoted by the asterisk sigil. Hopefully that'll clear up possible questions about the syntax below. Another note about Perl, but this time about scope</h2> Most newcomers to Perl understand lexical scope and global scope, but get confused about 'local' or dynamic scope. Dynamic scope is pretty hackish, but also very useful. Dynamic variables (declared with the 'local' keyword) are lexical, however their scope also includes the bodies of any functions that get called within the lexical scope. This simple example code should highlight what I mean. #!/usr/bin/env perl use strict; use warnings; use 5.010; our $foo = "foo!"; sub foofunc { say $foo; } foofunc(); { local $foo = "bar!"; foofunc(); } foofunc(); </pre> foofunc is expecting global $foo, however localizing (giving it dynamic scope) in that block allow us to change the value of the global the function sees. Dynamic scope ends in the same place lexical scope would end. Enough build-up</h2> Now that you understand these two key concepts, it should be fairly easy to see how we're going to fake STDIN. Here's the code: { open(FH, '<', 't/input.1') or die $!; local *STDIN; *STDIN = *FH; Exherbo::Packager::init_config('t/config.yml'); close(FH); } </pre> Here, we open a new filehandle (lexical filehandles will not work here), and then give STDIN dynamic scope and set its value to our filehandle. It's a simple enough hack, but it works satisfactorily. If this is the absolute wrong way to go about it, I'd love to know. This is just what I came up with when I couldn't find what I thought to be a reasonable solution with Google. PSGI Mon, 25 Apr 2011 00:00:00 +0000 Dancer</a> is the hottest new Perl PSGI-compatible web framework. You might be thinking, Perl web development? Doesn't that use CGI? Isn't CGI balls slow? Well...you're wrong. Most things you think you know about Perl are probably wrong. Let's start out with debunking the myth of CGI. CGI is slow, but there are a number of Perl-compatible alternatives. The most well-know, and best supported alternative is FastCGI. Don't hate on FastCGI because it has CGI in the name, it avoids the big pitfall of CGI (loading a new interpreter every time a new page is called) by having a FastCGI server load your interpreter, and then responding to requests sent to the FastCGI server from the web server. This is gives us a clear performance improvement (not having to load a new interpreter every time you load up a web page), as well as a security benefit (total isolation from the web server user/process). The other well-known alternative to CGI is mod_perl. mod_perl is Apache, specific, and it's also pretty old and janky. I'm not going to talk about it, mainly because I have never used it. It's poorly documented, and there exist better alternatives anyway. Let's move on. Ahh, PSGI. We love PSGI. PSGI is fast, and is under ridiculously rigorous development. PSGI applications get a hashref that provides all of the necessary information about the environment: request method, uri, variables, protocol info, etc. It then returns an arrayref of a status code, HTTP headers and the document body. Most of the time, you'll be using a framework, so you'll never have to care about interacting with the internals of PSGI directly. Astute readers are probably thinking, "Hey, if PSGI is perl, then what does that run on, and how does it interact with my web server?" Well, good reader, it runs on any of the above-mentioned frameworks and more! There's an Apache mod for it, it works with CGI, FastCGI, and mod_perl, and you have have to deal with any of that shit! Plack</a> is the reference implementation of the PSGI middleware. It sits between your webserver and Perl application serving up the PSGI environment for you, while translating the arrayref you return into meaningful data for the web server. Hopefully, this clears up some confusion about Perl and web dev, along with PSGI and its role in Perl web dev. This post also goes out to @seanmcgary</a>, the ignorant shit who inspired this post (and by ignorant shit I mean pretty cool guy who is very knowledgeable about PHP and Javascript, but knows nothing of the Perl world). Virtual Filesystems in Perl Fri, 15 Apr 2011 00:00:00 +0000 So I mentioned in a previous post that I was writing a scoring engine for CRTs, the hacking competition I'm holding for CSH. Well obviously I'm writing it in Perl, because I know Perl best, and it seems like the perfect sort of language for this task. Actually, what drew me to writing the scoring engine in Perl is the sheer power of POE</a>. But that's totally a topic for another post. When I started work on the scoring engine, for some reason I got the notion into my head of making it cross platform. I don't know why the hell I'd ever do that, since all I tend to use are UNIX-based machines, but whatever, fuckit, that's what I decided to do. The first (and so far only) real hurdle I hit when writing cross-platform code is dealing with configuration file paths appropriately across different OSes. However, I found an interesting article</a> on reddit that solved this addressed this exact problem...in Python. I figured there would be a Perl tool because why the hell not, and guess what? I was so blessed to find File::System</a>. File::System is a fabulous perl module that allows programmers to abstract the real filesystem away behind a virtual filesystem. You can arbitrarily set the root to whatever directory you want, or with File::System::Layered, merge bunches of directories into your root. Thanks to this perl module, it's trivial to set up OS-specific configuration locations without dealing with tons of bullshit in your code. Let me share some of my scoring engine code with you to show you how it's done. package Scoring::Engine::Backend; use strict; use warnings; use 5.012; require Exporter; our @ISA = qw/Exporter/; our @EXPORT_OK = qw/fs/; sub fs { state $fs; if ($Test::Scoring::Engine::test) { require Scoring::Engine::Backend::Test; $fs = $Scoring::Engine::Backend::Test::fs; return $fs; } given ($^O) { when ("MSWin32") { require Scoring::Engine::Backend::Windows; $fs = $Scoring::Engine::Backend::Windows::fs; } when ("linux") { require Scoring::Engine::Backend::Linux; $fs = $Scoring::Engine::Backend::Linux::fs; } when ("darwin") { require Scoring::Engine::Backend::Mac; $fs = $Scoring::Engine::Backend::Mac::fs; } } return $fs; } </pre> This right here is the bulk of the code. This uses some modern Perl constructs, so if any of this looks unfamiliar or even scary to you, I suggest you go pick up chromatic's excellent Modern Perl book. You have some catching up to do. :) Moving right along here, it's fairly obvious what this code does. We export a simple function called fs, which pulls your OS from the $^O variable and switches over it. Based on that, I load whatever OS specific module I need, and that gives me a File::System::Object to work with. Here's what the OS specific modules look like: package Scoring::Engine::Backend::Mac; use strict; use warnings; use 5.012; use File::System; our $fs = File::System->new('Real', root => "$ENV{'HOME'}/.scoringengine/"); 1; </pre> The first argument I pass into the File::System constructor denotes what type of File::System::Object I get back. In this case, I want the simple File::System::Real, since I only need to map one real directory to my virtual filesystem root. After doing that, use is simple: use Scoring::Engine::Backend qw/fs/; use YAML::Any qw/Load/; use constant DNS_CONFIG = '/dns.yml'; my $fs = fs; my $fh = $fs->lookup(DNS_CONFIG)->open("r"); local $/; my $teams = Load(<$fh>); close($fh); </pre> Calling File::System::Obect::lookup finds me the actual file that I want, and gives me a File::System::Object back that I can call the open function on. open works just the way it normally does, giving me a plain old filehandle to manipulate. After that, everything is peaches. Now, in all of my code that needs to access the filesystem, I have absolutely no need to worry about translating paths or anything like that. It's as easy as getting my File::System::Object and accessing files with that. Another benefit is unit testing. If you scroll back up to my first code post, you'll totally notice that there's a check to see if a test module is loaded. This is for ease of testing. In my unit test code (you are writing unit tests, right?), I can easily just load that module before I load Scoring::Engine, and I don't have to do any other trickery to have my code look for my cooked test configuration files in my unit test directory. File::System is pretty hot, and I really hope people have embraced it as the de-facto way to deal with filesystem access. New blog Mon, 11 Apr 2011 00:00:00 +0000 I've reworked my blog quite a bit. I transferred my some of my old Wordpress posts to a new blogging engine -- jekyll. Jekyll is pretty fantastic, in that it generates static sites from posts done in markdown, and templates made with the Liquid templating engine. It's gaining popularity thanks to being the technology that runs github pages. I heard about it from Ryan Clough</a> a few years ago (before it was cool), and then my friend Russel Harmon</a> encouraged me to start using it for my own blog. Since I accidentally took down my wordpress site sometime ago, I decided to give it a go. I also decided to roll out nginx instead of Apache since my servers are too old to really handle Apache effectively. So far, I definitely like it. I love writing my blog in vim (the best text editor) and doing version control with git. I still need to set up git hooks to auto-publish when I commit. I write all of the entries on my OpenBSD dev server, charmeleon, and then push the generated HTML to my web server, so I don't need to bother setting up the jekyll tools on my web server or anything like that. nginx is fast. It's really fast, and it's trivial to configure (for static sites, anyway). I'm going to keep using it, and possibly try and use it when I set up my personal Diaspora node. CRTS Sat, 09 Apr 2011 00:00:00 +0000 A week ago, I participated in a competition at RIT called ISTS, the Internet Security Talent Search. This competition challenges students in the area to form teams to defend and maintain a small network, while attacking other teams' network. And lemme tell you, this was a fucking blast. I did it last year, but now that I'm significantly more competent, I had so much more fun. The competition works something like this: there are 14 teams that all have identical setups of mixed Windows, Linux and FreeBSD machines. Three of the five machines host network services that need to have maximum uptime, and the other two machines are clients that can be used for attacking. Teams get points for every minute of uptime each service has, as well as for attacking machines of other teams. During the competition, teams are given challenges to complete called business injects, which can range from adding services, to adding features to existing services, to attacking other teams. It really runs the gamut. However, while we're busy doing our thing, the red team, made up of graduates, security consultants, etc. are indiscriminately attacking everyone. Since every team has identical machines with identical configurations, we all know the security vulns that other teams have as soon as we discover them ourselves, so really the red team is kind of at a disadvantage. For the CSH team (Team OPCOMM), I was mail bitch (because apparently no one wants to deal with postfix), and FreeBSD wizard. I maintained the postfix and dovecot installation on a teammates machine, while also maintaining FTP and DNS on my own. Unsurprisingly, my box was the only one box on the team that didn't get pwned during the competition. And I'm not saying that it's unsurprising because I'm pompous or anything, I'm saying it because I was maintaining the FreeBSD box. I mean really, next to all that Windows and Linux-y shit, FreeBSD is a fucking tank. Now, since I'm talking about ISTS, you're probably wondering why this post is titled CRTS. Well, through talking to people on CSH after the competition, I found that many people were interested in the idea of the competition, but they didn't think they had enough skill to compete. So I decided to hold an event just like ISTS for CSH. I'm calling it the Computer Science House Root Type Person Talent Search, or CSH RTP Talent Search, or CRTS. It's definitely focused at people who have little to no systems administration experience, but still challenging enough such that people who know what they're doingwill still have fun. Prior, there will be a fuckton of seminars given about topics people will need to be familiar with, starting with basic UNIX skills ranging all the way up to intermediate hacking techniques. It's going to be a great weekend, and I've already gotten a lot of interest from people on CSH. I've planned some fun vulns for teams to discover, and I'm writing a bitchin' Perl scoring engine using POE and Dancer. All of the code is on github, and I'll make sure to list the different setups in detail after the competition. Formspring API Issues Tue, 06 Jul 2010 00:00:00 +0000 I've been working on my Perl library for Formspring.me, and I've run into a pretty egregious issue. Apparently they aren't following the OAuth spec correctly, and aren't returning a oauth_callback_confirmed parameter. The problem for me lies in that the wonderful Perl OAuth library, Net::OAuth requires that the OAuth response return this parameter, and errors if it's not present. I've emailed api@formspring.me about it, but they have yet to answer my email. [1] http://tools.ietf.org/html/rfc5849#section-2.1 Formspring.me API beta Thu, 03 Jun 2010 00:00:00 +0000 formspring.me released a beta version of their API today. For those of you who don't know, formspring.me is the popular social networking site that allows people to ask and answer anonymous questions. It uses Facebook Connect for user auth, as well as letting users sign up on the site if they don't have a Facebook account. Apparently, as a joke some time ago, I signed up for the beta API. So I decided to develop a Perl module allowing easy use of the Formspring.me API. You can follow the development of it on my github</a>. I plan to release this code to CPAN when I'm done so that other people can use it. Since I'm new to the CPAN world, if one of you astute readers notices something blatantly wrong about my code, I'd appreciate it if you'd point it out. My first COBOL program on z/OS Sun, 25 Apr 2010 00:00:00 +0000 Today, I got Hello World working in COBOL on an IBM Z9. This is actually surprisingly more difficult than it would seem. But I'm gonna document it here so I remember how to do it, and also in case some random z/OS n00b stumbles upon this entry. Here is the Hello World file I wrote: IDENTIFICATION DIVISION. PROGRAM-ID. HELLO. PROCEDURE DIVISION. DISPLAY 'Hello, World'. STOP RUN. </pre> Seems pretty easy, right? Well, yeah, no shit. It's Hello World. I grabbed it from a Wikipedia article. It's not exactly innovative genius. However, compiling it and linking it is a different story. I tried at first using the ISPF foreground COBOL compiler. However, this was very non-obvious, and the documentation was lacking. All the documentation I found consisted mostly of how to do it through TSO, where it made mention that an ISPF interface did exist. So then I went the JCL route. IBM has some great documentation on how to do this. However, it is definitely not in the 900+ page Enterprise COBOL for z/OS Programming Guide. That was really just confusing. Here is the great documentation.</a> This is what got me off the ground. My JCL ended up looking like this: //CLGP JOB //CLG EXEC IGYWCLG //COBOL.SYSIN DD DSN=HLQ.EXPERIME.COBOL(HELLO),DISP=SHR </pre> It's really pretty simple. The first line just declares the name of the job (CLGP), the next line executes the terribly named COBOL compile, link-editor, and run JCL procedure. IBM wrote it so you don't have to. The last line just loads the dataset member that contains all of the COBOL code. See? Mainframes and COBOL are THAT easy! LTSP Thu, 22 Apr 2010 00:00:00 +0000 I finally finished rolling out the new changes to CSH's usercenter. I decided to move away from Fully Automatic Installation and move towards the thin client solution provided by LTSP</a>. FAI was really painful to test, as well as significantly more painful to manage. LTSP allows us to only update one machine, and install upgrades without having to reboot the machines to receive the softupdates (the FAI way). One machine, in this case a virtual machine on CSH's Xen cluster, runs all of the software for the clients. The terminal client connects to the terminal server's login prompt until the user logs in. Then the client automatically ssh's (with X forwarding of course) to the server and subsequently runs all of their software off of that. It is bandwidth heavy (on that segment of our network anyway), but it works very well. The clients are just as fast with the X forwarding as they would be with a locally installed OS (these machines are ultra slow). Really, there isn't much to say other than FAI is terrible. Related: I plan on setting up puppet</a> at some point for myself. COBOL Sat, 13 Mar 2010 00:00:00 +0000 So I haven't really posted in the last month, mainly because I haven't done anything interesting. I've been writing some Perl code using Catalyst and Mason. But that's not terribly exciting. I am however, taking a Mainframes course at RIT this quarter. It promises to be relatively exciting. In it, we get to learn the intricacies of z/OS, and how to manage one of those beasts. That'll be fun. Surprisingly, there's a programming project towards the end of it (it's surprising because it's a Networking and Sysadmin course, we don't program much). And since I'm learning how to administer and use mainframes, I thought to myself, "Hell, let's do this project in COBOL." So here I sit, trying to find a decent COBOL tutorial on the internet. If anyone finds one, please send it to me. It'd be nice. Google App Engine, Twitter4J and OAuth Sat, 13 Mar 2010 00:00:00 +0000 So I'm writing a Twitter app in Java on Google App Engine right now with my friend, Dave Bright. We're using Google Web Toolkit for our frontend, and App Engine for hosting and database and such. Twitter4J</a> is our Twitter library of choice. Since we ran into a spot of trouble using OAuth with the Twitter4J library, I decided to give a bit of a brief tutorial on how to get it working on App Engine. There is actually very different than the code in my as I do some more error checking, and need to do some other stuff with my datastore. Twitter4J OAuth Overview</h2> Set the consumer key and secret for your twitter object with setOAuthConsumer(key, secret)</li> Get a request token with getRequestToken(callback_uri)</li> Persist the request token somewhere, we'll need the original later when we need to get the access token.</li> Redirect the user to the URL returned by getAuthorizationURL()</li> Get the persisted request token, setOAuthConsumer again.</li> Call getOAuthAccessToken(request_token, oauth_verifier) to get an access token</li> Persist that token with the user data, you'll need this everytime you want to authenticate.</li> </ol> Some incomplete example code</h2> Please note, this is not code I actually have in production anywhere. It's original code just for the blog. public String authenticate() { TwitterFactory tf = new TwitterFactory(); Twitter twitter = tf.getInstance(); // This is actually really important, as Twitter WILL NOT send you back the oauth_verifier if // you don't provide the callback URI, even if you provide a callback on their site twitter.setOAuthConsumer(CONSUMER_KEY, CONSUMER_SECRET); RequestToken rtoken = new RequestToken(CALLBACK_URI); // This is a special class that just relates the token to the RequestToken object in the app engine // datastore. The token string itself is the key. OAuthTemp temp = new OAuthTemp(rtoken.getToken(), rtoken); // Our persistence manager is named pm pm.makePersistent(temp); return rtoken.getAuthorizationURL(); } </pre> The second bit of the code gets called when the user is returned to the page. You need to get the oauth_token and the oauth_verifier GET variables and pass them to this function. public void authenticatePartII(String oauth_token, String oauth_verifier) { TwitterFactory tf = new TwitterFactory(); Twitter twitter = tf.getInstance(); // Get the temp object out of the datastore, and get the old RequestToken // Yes, it MUST be the old RequestToken. Or at least, have all the same parameters // as the old RequestToken. Key k = KeyFactory.createKey(OAuthTemp.class.getSimpleName(), oauth_token); OAuthTemp temp = pm.getObjectById(OAuthTemp.class, k); RequestToken rtoken = temp.getrtoken(); // You MUST provide the original RequestToken and the oauth_verifier passed to you by Twitter twitter.setOAuthConsumer(CONSUMER_KEY, CONSUMER_SECRET); AccessToken ac = twitter.getOAuthAccessToken(rtoken, oauth_verifier); // You really don't need the RequestToken information after you got your AccessToken, so // delete it. twitter.setOAuthAccessToken(ac); pm.deletePersistent(temp); // However you store user data, you ought to associate the AcessToken object you // just got with your twitter user, and pop it in the datastore. I'll leave that bit up to you. } </pre> That's all there really is to it. The Twitter4J library isn't the best documented library, and just looking at the Javadocs and the Desktop-centric code examples isn't really that helpful. I hope this clears up some confusion on how to get Twitter4J working with OAuth. My major is too easy... Mon, 18 Jan 2010 00:00:00 +0000 This weekend, I had absolutely nothing to do. I didn't have any homework, studying, projects or the like because my major is far too easy. So I got bored and wrote a complete Brainfuck</a> interpreter in Perl. The interpreter was the easiest; it took me a morning to get that working albeit without support for nested parens. After that, I decided to write a text to Brainfuck converter, which (not surprisingly) turns text into executable Brainfuck code. Maybe if I get bored again, I could write a module to output the equivalent Perl or C code given some Brainfuck code. This was my first real project using object oriented Perl. In all honesty, I like that Perl's object system allows for many different types of objects (objects can be scalars, arrays, or hashes) and how flexible that is. Unfortunately, I didn't get to exploit that flexibility in my code. It's still cool though. What I wrote is 2 Perl modules that make up the linked list that store data generated by Brainfuck code, a Perl module that interprets and runs the code, and a Perl module that translates text to executable Brainfuck code. I know I could have used a pre-built linked list, but I thought it was more fun to write my own. The biggest challenge was definitely writing the translation code. I could've done it the easy way, and just output x amount of pluses for the character code of each letter, but I decided that was the lamest way to write Brainfuck. Instead, I looped through each character in the phrase, and added some nodes in my Brainfuck code that corresponded to the tens place of each character code. Then, when outputting a character, my Brainfuck code would change the data pointer to the node that was set to the tens place of the character code of the desired character, and increment or decrement it to the proper value. A more ideal way to do it (to generate cleaner Brainfuck, at least) is to set each of those prep nodes to the nearest multiple of 20 for each character code, and then use whichever node is closest to the desired value. Maybe if I feel like doing more with this, then I'll add that feature (probably not). In addition to writing all this, I also made a GitHub</a> account. My GitHub has all the above code, so you can clone the repo if you want to see what the code looks like (it's actually really easy to read Perl). This whole idea came from an idea on CSH. A guy here decided to publish a "Hello World of the Week" article in our newsletter, and to combat him I wrote a Brainfuck version of Hello World. This contest grew, and other people joined (albeit humorously) with such contributions as a boot loader that printed out Hello World, Hello World in White Space, and Hello World in Microsoft Word. My contribution (also in my git repo) generated valid Brainfuck code for the phrase Hello World, and then ran it through my interpreter (aka Hello World in 425ish lines). omirssi: An Omegle irssi plugin Fri, 25 Dec 2009 00:00:00 +0000 Sometimes I really hate going home. I'm on Christmas break, and I've spent the majority of my time just watching movies in my basement. I got really bored, so yesterday morning while I was watching "Juno" (love that movie), I wrote an Omegle</a> script for irssi. I just made up a google code</a> page for download, so have at it irssi users. Maybe I'll get around to adding it to the big list of irssi scripts sometime soon. Perl, Apache2::Request, and uploading files Mon, 07 Dec 2009 00:00:00 +0000 So I started writing a netboot server for floor. I decided to start writing the web interface first, and learn myself some mason, web perl, and database-y things. Also found some things out about the process that were a little undocumented/misdocumented (now a word). The biggest issue (so far) was file uploads. So here's a quick tutorial on how to do file uploads with perl, with mason. In index.html (or wherever your form HTML code lives): <form method="post" action="upload.mas" enctype="multipart/form-data"> <input type="file" name="file_name" /><br /> <input type="submit" value="Submit" /> </form> </pre> In upload.mas: <%init> use Apache2::Upload; # $r is an Apache2::RequestRec, not an Apache2::Request like some places say it is my $req = Apache2::Request->new($r); # file_name refers to the form name you had in your HTML my $upload = $req->upload('file_name'); my $fh = $upload->fh; open OUTFILE, ">$where_ever_you_wanna_save_the)file"; binmode $fh; binmode OUTFILE; print OUTFILE $line foreach my $line (<$fh>); close OUTFILE; close $fh; </%init> </pre> In your httpd.conf: APREQ2_READLIMIT 2G # Sets the max size of your user's uploads to 2GB </pre> I intentionally skipped over error handling code and all of the httpd.conf config for the sake of brevity (I did say brief tutorial). Since I need to allow for larger file sizes than Apache does by default (64MB), I spent quite some time looking for that Apache directive. Hopefully this saves someone some time. OpenBSD pf vs Linux iptables: A Comparison Sat, 14 Nov 2009 00:00:00 +0000 This weekend, I decided it would be a good idea to turn my Linux router/firewall into an OpenBSD router/firewall. Clockfort</a> recommended it, so I decided to grab backups of my iptables rules, install OpenBSD, and learn pf. I learned iptables about a year ago, when I first built this router/firewall. It acts like those little home gateways that you get; it does NAT, DHCP, DNS, etc. I've also used iptables to firewall my desktop and laptop (though those rule sets were significantly simpler than the firewall). Configuration</h3> The firstÂ notableÂ difference between pf and iptables: pf has a config file! It also has variables, lists and tables that you can manually populate which ease configuration. You can even include other config files in case you need to split your config for whatever reason. When you're done modifying the config file, just call pfctl -f /etc/pf.conf and it'll load that rule set and start filtering. iptables doesn't have any of that. iptables are primarily populated through the iptables command. You can use iptables-save and iptables-restore to save and load iptables rules from a file. The file is basically a bunch of iptables commands with the iptables bit omitted. Another alternative is to write a bash script that loads your iptables rules one by one. That option gives you the benefit of using variables so that it's trivial to change IPs or similar. I used iptables-save and restore to configure my iptables rules, and just wrote rules to that file in a similar syntax when I wanted to reconfigure parts of my firewall. Rule Processing</h3> The next most obvious way they differ is how they process rules. iptables has various tables, each with different chains that packets traverse, whereas pf just processes packets straight down the config file. With pf, packets traverse the flat pf.conf file. Even if a packet matches a rule, it continues to process the packet all the way down the configuration file. Only if a rule contains the "quick" option does pf stop processing and take action before hitting the end of the rule set. If a packet makes it all the way to the end of the config file, the last action specified from a rule that matched that packet is taken. With iptables, packets are processed by various chains in different order, depending on the source and destination in the packet. For example, normal outgoing packets are processed by the OUPUT chain on the filter table. Various rules within that chain may cause processing to hop over to a different set of rules on a user-defined chain, or might take action on a packet. When a packet matches a rule description, processing on that chain stops immediately, and the action is taken. Dynamic Modification</h3> pf really just owns iptables here. To dynamically update your rules with iptables, you just write new rules on the fly. If you want to do a bunch, you would write a bash script to delete some rules, and write new ones. pf, you can change tables, variables, lists and anchors on the fly. Anchors are a really cool feature of pf. They are basically sub-rulesets that have names. So if you define an anchor somewhere in your ruleset, you can call pfctl and totally redefine the rules within that anchor. You can even write anchors to files, and load them from different files. Packet Filtering</h3> pf is pretty simple concerning what it can filter for. It can filter based on protocol, TCP flags, source IP, destination IP, interface, and port. There is also some slightly more advanced filtering, like antispoof, unicast reversing, and passive operating system finger printing. For the majority of situations, this kind of firewall control is fine. iptables can do all of the same stuff. However with iptables, you can load all sorts of modules that do far more intensive filtering than pf. You can filter based on state (no, you can'tÂ filterÂ based on state in pf), where they are (geoip), time, statistics, ToS, and much more. There are so many target extensions for iptables, it is ridiculous. And if that isn't enough, you can pass the packet into userspace and write a script to filter it further there. Performance</h3> pf is fast. I said before that you can't filter packets in pf based on state. That doesn't mean that pf isn't a stateful firewall. It definitely recognizes state, as it passes packets that are part of an established connection without even processing them with pf. This means the majority of your packets skip your firewall rules entirely. This isn't nearly as insecure as it sounds, since most iptables rulesets pass packets that are part of an established state anyway. It definitely has the distinct advantage of making it faster though. With iptables, all of your packets pass through all of your rules. This can really slow things down, especially if you have complicated rulesets. If you use all sorts of crazy iptables modules, that will slow it down pretty heavily too. And if you pass the packet into userspace for further processing, it will slow it down even more. Conclusion</h3> pf and iptables are both great firewalling solutions, but cater to people of different needs. pf is ridiculously fast, but lacks some of the more avanced features of iptables. Since my router/firewall box doesn't really need those advanced iptables features, and since it needs to be fast, I'm gonna stick with pf for now. Google Wave Mon, 26 Oct 2009 00:00:00 +0000 Yeah, that's right. Just got an account. Well, by just, I mean I got one on Friday. Don't even ask for invites, because they were gone by Saturday morning. Also, if you don't know what Wave is, you should really go to this</a> site and watch the video. I know it's long, but it's well worth it. So Google Wave is pretty awesome. It's ridiculously buggy, but it works well enough if you aren't using it in a production environment. I have yet to try out any of the multimedia stuff beyond embedding YouTube videos in Wave, but I imagine it works just as well as everything else does. I'll probably talk more about that when I've played with it. What I've really been playing with is the Wave API, in Python of course. The jist of what I've been reading is that the Java API isn't as up to par as the Python API. Also, I hate Java, and I hate writing code in Java. The Wave API is built off of Google App Engine</a>, and has all the same limitations as App Engine. My roommate and I kinda half started working on an NNTP gateway to Wave. Unfortunately, the limitations App Engine sets makes this a little more difficult than we'd like (no sockets or anything of the sort). The one thing I've come across in the API that I don't like, is that you must include the entire Python Wave module in the app directory that you upload to App Engine. Not a huge issue, but definitely a bit of an annoyance. The only real problems I see are the possibility for people to modify the version of the Wave API they upload, and that the version of the Wave API they upload is outdated. Another issue with using App Engine as the platform for all of the Wave code and such, is that you must use Python 2.5. App Engine doesn't work with anything higher than that. So even though the Wave development site says you can use Python 2.5 or above, the above part is really a lie. The good news: Google Wave won't forever be reliant on App Engine. Sometime in the future, they will be allowing Apps from anywhere. This is will be awesome, and I can't wait. Oh, and I had other productive things I was gonna do this weekend, but they all got shot to hell when I got my Wave account. Now you can wave me at ay1244@googlewave.com Project Work Weekend Part II Mon, 19 Oct 2009 00:00:00 +0000 Yesterday ended up being extraordinarily productive for me. I learned GTK+ in perl, and wrote the GUI for my Omegle app. It's looking decent, although there is this annoying issue with the menu bar that's bothering me a bit. It's very simple, but I can't really think of any other features to put in the GUI. I also got a bunch of the abstraction for my CS project done today. But that wasn't that much code. Oh well. Also, in the middle of this post, got roped into helping fix our Xen management interface. And by help, I meant I installed Ubuntu server (ugh) on a machine and installed openssh. Woo. Project Work Weekend Part I Sat, 17 Oct 2009 00:00:00 +0000 So CSH is holding its first "Project Work Weekend" of the year, and so far, it's kinda been productive. Personally, I got a lot of work done. I've been kinda productive. I worked a little bit on my Omegle application that I've been playing with in my head, and I definitely got some of my CS project done. For my CS project, me and a partner have to write a game playing framework, and then a bunch of pluggable games for the framework. It's a lot easier than it sounds, as they're all little command line games like take away, kayles, or connect 3. It's still pretty cool though. It's weird to be working with objects again after mostly programming in C this summer. But yeah, so the project is coming along. It's nice, because right now I only really have to write the abstract framework while my partner writes the game parts that inherit from my framework. Also I have to fix the old game. But whatever, that should be easy. My Omegle app is coming along slowly. Trying to figure out perl GTK+ while figuring out perl OO while figuring out the ill-documented WWW::Omegle perl module. Fun times. Actually, typing this made me realize I should OO it up a little more. Gonna go back to work after this post. I definitely need to finish up the slides for the presentations I'm gonna give CSH. I'm gonna publish them on here too, under a new "Presentations" section. It should look pretty hot. Anyway, I should go back to being productive, cause I haven't done as much as I wanted. About Me Thu, 29 Mar 1990 00:00:00 +0000 Hey, I'm William Orr or worr. I'm currently a Devops Engineer at Adevinta. Currently, I work with Kubernetes and Golang in the cloud, like the rest of the dang ops world these days. I also have an interest in systems programming, and focus on OS/systems programming projects in my limited spare time. Currently, I'm located in Barcelona, Spain. I'm a staunch OpenBSD</a> user at home, with almost all of my systems running OpenBSD where possible (I have one system that has to run FreeBSD). Currently, I'm spending time getting more proficient in Go and Rust, and trying to make the experience of both on OpenBSD more palatable where I can. Contact Me</h2> PGP Key: 95D8 225C 2EFE 2DA8 954F 41A1 E588 5A8A 4D91 B964</a></dd> Gitlab:</dt> gitlab.com/u/worr</a></dd> Github:</dt> github.com/worr</a></dd> LinkedIn:</dt> linkedin.com/in/willorr</a></dd> Email:</dt> will@worrbase.com</a></dd> Fediverse:</dt> bsd.network/@worr</a></dd> Twitter:</dt> twitter.com/worr</a></dd> Openhub:</dt> openhub.net/accounts/worr</a></dd> </dl> onwednesdayswewear.pink</a> worrbase.com</a> </section>

Setting up an OpenBSD Gitlab runner

Resources</h3> To reduce the impact of abuse, we're gonna try and restrict the usable resources a bit. We're going to modify /etc/login.conf</code>.</p> The following values are specific for my use-case, and won't necessarily be applicable for your system.</p>

Final Fantasy VI

What is this?

Extended Attributes

Where to go from here?</h2> A C wrapper lib around all of these implementations would be nice, but there are some obvious trade-offs that need to be made.</p> The way that I've done this in Python and Rust has been to:</p>

Starting a dev blog

An Obfuscated String Implementation for Go

Any caveats?</h4> Again, the memory that you pass into mprotect</code> must be page-aligned.</p>

Any caveats?</h4> Hell yes.</p> The key needs to be stored in memory. If the attacker knows where in memory your key is, then it's trivial to decrypt your sensitive string. It really just ends up being obfuscation more than anything.</p>

Setting up RetroPie

Let's get started</h2> Install your favorite image normally. Don't stop any services just yet or do any configuring just yet. Stick to a vanilla install, since that's what RetroPie expects (and it doesn't like any deviation).</p> Non-raspbian notes</h3> Add an input group.</p>

Non-raspbian notes</h3> Add an input group.</p>

Notes for non-raspbian users</h3> Yeah, you probably do want to overclock. If your distro is compiled with dumb options, it'll help a bit.</p> On both Darkbasic's raspbian and PiBang, I had to overclock to get decent performance.</p>

OpenSMTPD

Update: spamd on Linux

spamd on Linux

Leaving sugar behind...or am I?

Fixing Sugar bugs

Week 2 in Sugarland

Sugar Progress

Introduction to Perl

My thoughts on Open Source

FOSS Class Introduction

Etoys</h2> Etoys was pretty much guaranteed to make this list, as I'm (obviously) a programmer. It seems like a smart way to make programming more appealing to children, so that they can get hooked at a young age.</p>

GetBooks</h2> GetBooks looks great, as it makes book acquisition trivial for people who want to read. I'm not sure what kinds of books you can find using OPDS, but the idea by itself is laudable. I honestly can't wait to see what kinds of texts are available to me.</p>

Managing ports in FreeBSD with CFEngine3

CFEngine SVN pre-commit hook

Book Review: Programming Perl

CrashPlan on FreeBSD 9.0, A HOWTO

OSS and App::GitHub

AdiumQQ

rvm and perlbrew

Ruby and Gems from Homebrew on Mac OSX

Hotel Internet

Faking STDIN in Perl unit tests

PSGI

Virtual Filesystems in Perl

New blog

CRTS

Formspring API Issues

Formspring.me API beta

My first COBOL program on z/OS

LTSP

COBOL

Google App Engine, Twitter4J and OAuth

My major is too easy...

omirssi: An Omegle irssi plugin

Perl, Apache2::Request, and uploading files

OpenBSD pf vs Linux iptables: A Comparison

Google Wave

Project Work Weekend Part II

Project Work Weekend Part I

About Me

Any caveats?</h4>
Again, the memory that you pass into `mprotect</code> must be page-aligned.</p>`

Any caveats?</h4>
Hell yes.</p>
The key needs to be stored in memory. If the attacker knows where in memory your key is, then it's trivial to decrypt your sensitive string. It really just ends up being obfuscation more than anything.</p>

Notes for non-raspbian users</h3>
Yeah, you probably do want to overclock. If your distro is compiled with dumb options, it'll help a bit.</p>
On both Darkbasic's raspbian and PiBang, I had to overclock to get decent performance.</p>

Etoys</h2>
Etoys was pretty much guaranteed to make this list, as I'm (obviously) a programmer. It seems like a smart way to make programming more appealing to children, so that they can get hooked at a young age.</p>

GetBooks</h2>
GetBooks looks great, as it makes book acquisition trivial for people who want to read. I'm not sure what kinds of books you can find using OPDS, but the idea by itself is laudable. I honestly can't wait to see what kinds of texts are available to me.</p>