Matt Hackmann

MattHackmann

The thoughts and goings-on of some programmer dad.

That'll End Your Fun Real Quickly

Deciding that my Taco Bell supper was too much to weigh on my conscience, I decided that a late night bike ride was the thing to do. This after dark guilt ride is something that happens entirely too frequently. But, that's beside the point.

Before I even get out the door, I was looking for my pump to top my tires off. How something that large can go completely missing is beyond me, but after a solid 10-15 minutes of looking, it was nowhere to be found. So, I decided to just live with the pressure at whatever it was and go. Get outside, start Strava and my music, hop on the saddle, clip in, and away I go... except I don't. As I'm rounding the corner towards the apartment exit, I realize I'm not wearing my helmet. Return home, grab helmet, rinse and repeat. I'm finally on the road.

So, I'm out biking, listening to some tunes and giving my new bike undies a test run. My outfit is said undies, a pair of gym shorts, and my Lockheed-Martin jersey (it has pockets which is handy for holding my hand pump). My ass was comfortable for sure, though there's not enough padding in the crotch. Probably too much information, but there it is.

Again, I'm out biking and not really taking any particular direction. Up over a bridge, under an overpass, around Fry's and to a side road that takes me past the rock climbing place. This drops me out on a road that goes by the Sunnyvale LinkedIn complex. Had I brought my badge, I could have stopped in and raided the pantry, but I didn't so I didn't. As I'm making a left out of that building and onto the road, I hear a loud "pop" behind me. I didn't need to inspect the bike to know what had happened; I'd been struck by the Curse of Jibjab.

One of my coworkers (well, technically two) have the same bike that I do. He blew a spoke on that bike not once, but twice, on the same wheel even. I pulled myself over to the sidewalk, flipped my bike over and began inspecting every spoke on the wheel. Lo and behold, a spoke has been sheared off.

At this point, I'm somewhere on the order of 2-3 miles away from home. I briefly consider my options and decide that, despite the fact that wheel won't even spin freely anymore, it's good enough under load to make it home. I wasn't really keen to the idea of walking that far back in my biking shoes.

So, I slow rode all the way back, worrying the entire time that my immense weight was going to crush the rest of the spokes. Luckily, it went by without incident. I suppose all in all this isn't such a bad ordeal as I was planning on replacing the wheels with something better anyways, but I don't really like being forced into these situations.

Fate, however, has its own plans.

Old Dogs Can Learn New Tricks, It Just Takes Longer

As I sit here whiling away the time trying to put the finishing touches on my Awwnime Bracket app, I realize that the infernal clock ticks away at the minutes, much as it always does. Of course, not that any of this matters since my server is still running in Central time and the dates will be messed up anyways.

As stated, I sit here coding in that which I have tried several times before: writing apps for iPhone using the abominations that are Objective-C and Xcode. Now, this isn't my first time in this rodeo. Indeed, in 2012 or 2013 (before LinkedIn was a thing I considered a thing), I purchased a Mac mini in hopes of carving myself a space in the ever expanding "app-o-sphere" and some of those tantalizing dollars that could come with. I did my damnedest to follow learn Obj-C and tried to follow the Stanford iTunes U course, but it was just too much all at once for my brain to process. I quickly got discouraged and gave up. Shortly after the move to California (or perhaps slightly before), I sold the mini and forgot all about that shit.

Then, mid-last year, one of my fellow webdevs at work organized a weekly session in which we would all meet for an afternoon and learn ourselves some iOS development. This arguably went much better, though my mind still had trouble wrapping itself around all those fucking square brackets and the hierarchy of views and view controllers and what have you. Despite being one of the handful that "graduated" that class, I barely pounded out a semi-functional app. Seriously, that thing was (and is) a piece of shit. It was supposed to be an "offline" client for the music page, but never quite made it entirely to that offline mode. I still would like to complete it so I can track my music listenings while biking or driving.

Fast forward to a few months ago. I don't recall how it came around, but as LinkedIn's membership continues its drive towards mobile devices, so too are we trying to scrounge up mobile support internally. One thing lead to another and my name hit a list of people that they wanted trained in iOS. I accepted (perhaps a little begrudgingly), and for one week in August, did nothing but iOS training. Perhaps it was all the repetition, but something about that class made everything stick. Perhaps it's because we were writing actual apps that did actual things and each of those exercises themselves had much repetition. But, when I came out of that class, it was as if the clouds had lifted and the chorus sang out from the heavens. I'd finally grasped these ever elusive concepts that had me so worked up in the months and years prior.

That's not to say I'm an amazing iOS developer. Far from it. However, I've been putzing away at this bracket app with relative ease, only looking up particular things for particular cases when needed. It's odd to see how even the way I've been writing javascript has changed in only the last few weeks. I find myself using Obj-C/Foundation style nomenclature when writing methods (initWithXYAndZ).

So, that's that. I was hoping to get this app submitted tonight, but I just profiled the thing and it's eating up a healthy 500MB+ (all in images), so I need to figure out how to mitigate that.

I do love doing this shit...

What's in a Nap?

The clock says 9:55pm and I have, more or less, just awoken from an evening nap... that lasted two hours. I make no secret of the amount of time I spend napping, but generally it's 20-30 minutes. Two hours is excessive and it's now eaten away at time that I was going to spend working on my bracket app.

I'm not entirely sure why I'm so prone to napping. The most obvious thing to me would be the fact that I keep such terrible hours. My alarm begins going off at 6:20am so I can snooze it a couple of times before rolling into work not too long after 7am. My final sleep generally begins around 12am-1am, which leaves five to six hours for actual sleep. The nap could exist as a little pick me up to make it just a bit farther.

I've also theorized that my job could be part of the reason. Dealing with office work and software development is more taxing than one might think for something that keeps you immobile for nearly 9 hours. Given that my brain has been running at high capacity for that time, once I return home, everything just grinds to a halt. I flop on the couch, flick on the TV, and basically do nothing for the remained of the evening. That is, unless I have some sort of deadline to meet in a project I'm working on.

My final thoughts concerning the naps, somewhat related to the above, is that I may be suffering from mild depression. When I'm at home, lying in my bed, it sometimes takes quite a feat to do anything. I don't feel like playing a video game, or reading a book, or watching TV, or any numerous things that require nearly zero physical or cognitive effort. Hell, sometimes it's so bad I'll even let my stomach go hungry because I don't want to move to make food. Instead I'll just lie in my bed drifting in and out of sleep, going through this same mental cycle as my mind breaches into consciousness. I'm not entirely settled on the depression idea as I've seen real depression play out. I'm a pretty content person most of the time, so I think this is more extreme laziness than anything else.

Alright, there's more words!

Return of the Daily Blog

As has been a semi-recurring thing for the last couple years, I will be attempting to jot my thoughts down once per day on this little corner of the internet I've carved for myself. Not sure what I'll say at this point, but it may be more serious and brain dumpy than the usual fluff affair. Or it may not be, I dunno.

I've barely even said anything over the last few months, so I guess a place to start is by catching up. I feel like my life has taken an interesting turn the last few months. Before moving to California, I was essentially the shining definition of a lone wolf. Wake up, go to work, go home. The only time I socialized with people was during those nine hours in the office. Now, however, I actively spend quite a lot of time (by my standards) away from home and doing things with other people. This evening, for example, I was at a friend's house rolling characters for an upcoming Dungeons and Dragons game, after which we played Munchkin. Pretty standard social fare on paper, but as stated, kind of a big deal for me. Add to this the biking excursions, evenings of drinking, and other random outings with people and I sometimes find myself in that spot of swimming unknown waters and having to figure things out as I go. I hate being in this situation and I know it's a major reason why I shy away from dating. When I don't know the rules, everybody winds up having a bad time, or perhaps a great time at the expense of my psyche.

But, whatever. There's more to say, but I'll leave this here to leave fodder for the future.

RedditBooru v2 - Postmortem

Back in January, I wrote a rather frustrated post about rewriting RedditBooru on a new tech stack and how everything sucked. Six months (almost to the day) after that post, I finally launched the second major revision of the site. That launch was not without a whole bunch of drama.

Codebase Direction

In the aforementioned post, I was originally developing on a stack consisting of nodejs and mongoDB. Shortly after, I abandoned nearly everything I had written, the only thing carried over being the few templates that were done. There were many lessons learned in that excursion, particularly "if it ain't broke, don't rewrite it". I decided that instead of again rewriting everything from the ground up, I would work off of the existing codebase and retool that. This proved immediately to be the right idea, as I was able to quickly port many of the new things I'd written in node to PHP but without any of the stability issues. The first of these was the cron job.

The nodejs version of the cron (which was more of a daemon than a cron, really) heavily used async operations which allowed it to index all sources very quickly, generally in a matter of seconds. PHP is by its nature not async friendly, so I employed some clever process management handled by node, something that I'd casually thrown out as an option to resolve my node woes. The node manager keeps a pool of 4 indexer processes running, allowing for multiple sources to be indexed simultaneously in individual hardware threads but with all the benefits of synchronous code for each individual process. This proved to be rock solid right from the start, though did result in high CPU usage as I'd noticed in the first few days after launch. It was enough of an issue that I dropped the maximum number of processes down to one. Turns out I was unnecessarily hammering MySQL for data, and by being smarter about my DB calls, CPU usage was dropped almost to 0 and I was able to ramp back up to 4 processes while still utilizing less CPU than in the previous single running process.

One major change I did make to the front-end was to use backbone as my JS framework. The primary reason I did this was to familiarize myself with the MV* frameworks that have been all the rage lately and I know the day is coming when I'll need to know this stuff for work. The other reason was that it provided logical structure to my JS instead of haphazardly throwing everything into a single JS file. The only issues I had with the framework were around it's handling of routes, a chunk of code I eventually just wrote myself to suit my particular use cases. All in all, I'm pretty happy with backbone, but I feel that I probably did not use it to its fullest advantage. I probably could've gotten away with no framework and using something like Fiber to give me a nice, classical style OOP scheme to work off of.

Release Day

So, development went pretty smoothly once I kept doing things I knew how to do. And everything was fine... right up until just before the hard date I'd set for release (June 30th, in this case). A couple days before release, I went over my hosting bandwidth cap on the existing server, something that had been threatening to happen for months. I quickly began scrambling to figure how to route traffic to places where I wouldn't be paying the enormous overage fees I would be getting from my hosting. During all that, I made the decision that I should just spin the new redditbooru up on a new machine, one with much greater bandwidth caps. Many hours of file copying later, I had all of my sites on the new machine with traffic slowly diverting over there... except for redditbooru. Because of how it names hosted files, this would have to be transferred over last. And everything probably would have gone okay... if I hadn't forgotten one thing. As the traffic was shifting from the old machine to the new machine, there were weird anomalies showing up with pictures people had posted to reddit, namelyt different people were seeing different things for the same file. At this point, it should be mentioned that a user's DNS could be in one of four states:

  • pointing to CloudFlare (which would be pointing to one of the following)
  • pointing to AWS
  • pointing to the old server
  • pointing to the new server

My initial thoughts were that one or two things slipped through before I migrated the database from the old machine to the new machine. I had disabled uploading on the old one, so there shouldn't be any collisions. Except for the fact that I'd forgotten to also turn off the indexer on the old box. This was causing the two separate machines to constantly overwrite each other's images on AWS and since their database IDs were very different, those images would be different as well. And, I was powerless to do anything while the DNS was propagating. Of course, the old indexer was shut down which resolved all issues going forward and I overwrote all the bad images on AWS using the new machine as the source of truth, but it was still a waiting game as that DNS finished up so I could turn off the old server.

It was a hellacious couple of days and I'm honestly surprised anybody still trusted my software after that.

One Month Later

It's now one month later and things have finally smoothed out. Of course, there are still some issues I've been experiencing with the new software and new machine. For instance, all sites on that machine stopped working a week or two ago. Turns out that MariaDB had enabled binary query logging by default and had exhausted all of the available disk space (some 77GB of logs). MongoDB, which I'd relegated to image caching and internal metrics logging, also just fucking died a couple times, bringing all of RB with it. I've since disabled mongo and migrated the image caching to a simple disk based system. This leaves me in a bit of a bind as many features I had planned were going to use mongo for a denormalized data store (stats type things, mostly). I'm currently looking into redis as a possible way replacing this.

All that said, RedditBooru is healthy now and people are taking advantage of the new features built in, particularly the inline image viewer. That feature is so popular that time-on-site has jumped a full minute and my bandwidth for last month approached 6TB, a full 2TB over what my previous bandwidth cap. This is again going to fast become a major issue, one that needs resolving pretty quickly.

I think overall what I can take away from this whole experience is that calm works makes for higher quality work. Panic only brings additional pain.