Matt Hackmann

RedditBooru v2 - Postmortem

Posted On
Tags
coding
general
redditbooru
webdev

Back in January, I wrote a rather frustrated post about rewriting RedditBooru on a new tech stack and how everything sucked. Six months (almost to the day) after that post, I finally launched the second major revision of the site. That launch was not without a whole bunch of drama.

Codebase Direction

In the aforementioned post, I was originally developing on a stack consisting of nodejs and mongoDB. Shortly after, I abandoned nearly everything I had written, the only thing carried over being the few templates that were done. There were many lessons learned in that excursion, particularly "if it ain't broke, don't rewrite it". I decided that instead of again rewriting everything from the ground up, I would work off of the existing codebase and retool that. This proved immediately to be the right idea, as I was able to quickly port many of the new things I'd written in node to PHP but without any of the stability issues. The first of these was the cron job.

The nodejs version of the cron (which was more of a daemon than a cron, really) heavily used async operations which allowed it to index all sources very quickly, generally in a matter of seconds. PHP is by its nature not async friendly, so I employed some clever process management handled by node, something that I'd casually thrown out as an option to resolve my node woes. The node manager keeps a pool of 4 indexer processes running, allowing for multiple sources to be indexed simultaneously in individual hardware threads but with all the benefits of synchronous code for each individual process. This proved to be rock solid right from the start, though did result in high CPU usage as I'd noticed in the first few days after launch. It was enough of an issue that I dropped the maximum number of processes down to one. Turns out I was unnecessarily hammering MySQL for data, and by being smarter about my DB calls, CPU usage was dropped almost to 0 and I was able to ramp back up to 4 processes while still utilizing less CPU than in the previous single running process.

One major change I did make to the front-end was to use backbone as my JS framework. The primary reason I did this was to familiarize myself with the MV* frameworks that have been all the rage lately and I know the day is coming when I'll need to know this stuff for work. The other reason was that it provided logical structure to my JS instead of haphazardly throwing everything into a single JS file. The only issues I had with the framework were around it's handling of routes, a chunk of code I eventually just wrote myself to suit my particular use cases. All in all, I'm pretty happy with backbone, but I feel that I probably did not use it to its fullest advantage. I probably could've gotten away with no framework and using something like Fiber to give me a nice, classical style OOP scheme to work off of.

Release Day

So, development went pretty smoothly once I kept doing things I knew how to do. And everything was fine... right up until just before the hard date I'd set for release (June 30th, in this case). A couple days before release, I went over my hosting bandwidth cap on the existing server, something that had been threatening to happen for months. I quickly began scrambling to figure how to route traffic to places where I wouldn't be paying the enormous overage fees I would be getting from my hosting. During all that, I made the decision that I should just spin the new redditbooru up on a new machine, one with much greater bandwidth caps. Many hours of file copying later, I had all of my sites on the new machine with traffic slowly diverting over there... except for redditbooru. Because of how it names hosted files, this would have to be transferred over last. And everything probably would have gone okay... if I hadn't forgotten one thing. As the traffic was shifting from the old machine to the new machine, there were weird anomalies showing up with pictures people had posted to reddit, namelyt different people were seeing different things for the same file. At this point, it should be mentioned that a user's DNS could be in one of four states:

- pointing to CloudFlare (which would be pointing to one of the following)

- pointing to AWS

- pointing to the old server

- pointing to the new server

My initial thoughts were that one or two things slipped through before I migrated the database from the old machine to the new machine. I had disabled uploading on the old one, so there shouldn't be any collisions. Except for the fact that I'd forgotten to also turn off the indexer on the old box. This was causing the two separate machines to constantly overwrite each other's images on AWS and since their database IDs were very different, those images would be different as well. And, I was powerless to do anything while the DNS was propagating. Of course, the old indexer was shut down which resolved all issues going forward and I overwrote all the bad images on AWS using the new machine as the source of truth, but it was still a waiting game as that DNS finished up so I could turn off the old server.

It was a hellacious couple of days and I'm honestly surprised anybody still trusted my software after that.

One Month Later

It's now one month later and things have finally smoothed out. Of course, there are still some issues I've been experiencing with the new software and new machine. For instance, all sites on that machine stopped working a week or two ago. Turns out that MariaDB had enabled binary query logging by default and had exhausted all of the available disk space (some 77GB of logs). MongoDB, which I'd relegated to image caching and internal metrics logging, also just fucking died a couple times, bringing all of RB with it. I've since disabled mongo and migrated the image caching to a simple disk based system. This leaves me in a bit of a bind as many features I had planned were going to use mongo for a denormalized data store (stats type things, mostly). I'm currently looking into redis as a possible way replacing this.

All that said, RedditBooru is healthy now and people are taking advantage of the new features built in, particularly the inline image viewer. That feature is so popular that time-on-site has jumped a full minute and my bandwidth for last month approached 6TB, a full 2TB over what my previous bandwidth cap. This is again going to fast become a major issue, one that needs resolving pretty quickly.

I think overall what I can take away from this whole experience is that calm works makes for higher quality work. Panic only brings additional pain.

Not lost, but certainly forgotten

Posted On
Tags
general

Somehow, I found myself poking around in the darker corners of my backup drive today. I don't even remember the original reason I hooked the thing up, but as is wont to happen, I headed in a downward spiral into my past. I like to use these things as a measuring stick to measure not only my progress as a professional, but also as a person. To that latter point, the jury's out on whether I've changed at all...

Somewhere in the early days of my Narutard phase

Somewhere in the early days of my Narutard phase

I'd like to say this was borne from a conversation with my mom. I am the babby.

I'd like to say this was borne from a conversation with my mom. I am the babby.

This picture of my little sister never fails to entertain

This picture of my little sister never fails to entertain

The only upside I see to this one is that I can tell I've lost weight since...

The only upside I see to this one is that I can tell I've lost weight since...

Getting tired of all this Impact...

Getting tired of all this Impact...

And a rage comic. So 2011

And a rage comic. So 2011

To be fair, there are some interesting things I ran across that I'd like to feature sometime. But, that'd be too much effort and I'm more about the self-deprecation.

Gravity, My Enemy

Posted On

If there's one thing I've learned since I came to California, it's that my limbs will find any way they can to test gravitational physics on any item close to them.

To Become the Most Boring Person I Can Be

Posted On
Tags
general
mundane

This blog has once again fallen into dusty shambles. So, I'm going to kick in another post-a-day type experiment... starting tomorrow.

What I'm going to do is keep a daily log for a week, but instead of my usual type rambling, it'll be just 100% facts, completely devoid of emotion, opinion, or adjectives/adverbs (using as few as possible). A little something like this:

I woke up at 10:15am this morning. I didn't actually want to leave my bed, so I fished my iPad out from under my bed and checked my email, Facebook, and reddit. I decided to make German potato salad today but lacked all the ingredients required. After deliberation, I decided I would bike. I geared up, but after getting out the door noticed that my back tire was again low. I put the bike back in my apartment and took the car instead. Since I was no longer worried about distance or the roads I would need to take, I went to Safeway because they have a better selection of food than Target. Once at the store, I purchased potatoes, scallions, bacon, and white wine vinegar. After returning home, I cleaned a large pot to boil the potatoes in. They are currently boiling now.

Seems a bit like those old journal entries of mine, no?. For contrast, here's how I'd rewrite the first couple of sentences.

Given that I'm only half way into this day, I would label it as almost entirely ordinary. Consciousness began to return to me somewhere around 10:15a, about an hour later than my usual 9a (largely thankful to my late night of playing Danganronpa, no doubt). Despite the lateness, I didn't feel like leave the warm comforts of my bed, so I grabbed my iPad (generally kept beneath my bed) to entertain myself with. I did a quick check of my email, Facebook, and skimmed reddit. Despite Alien Blue being a pretty nice app, I don't like to "deep dive" into my usual subreddits on a mobile device because I feel the experience would be diminished on a small screen (moe must be observed with as many pixels as possible).

So, yeah. That's a thing I'm going to do because I'm otherwise bored and have little to write about (or am not willing to expend the effort in doing so).

Head Sensations

Posted On
Tags
general

I'd been considering writing a blog post on this subject for some time, but balked because it seemed an odd thing to talk about. But, after the topic came up at lunch not long ago, I figured why not. Besides, I need to post something this month as I've not missed posting at least one thing per month since August 2010. Gotta keep the streak alive.

The aforementioned topic is a physical sensation that's been dubbed on the internet as "ASMR", or "autonomous sensory meridian response".It's honestly a really shitty name and I thought the original idea of "brain orgasm" explained it better, but the community doesn't take kindly to that since it's not a sexual thing. To actually describe the feeling is a bit difficult as there's nothing really to relate it to. The usual description is "head tingles" and, while that's not entirely untrue, it doesn't seem quite accurate either. I'd describe it more as a feeling of mild pressure sensation on the top and back of the scalp that will at times "bloom" in intensity across the scalp in response to stimuli. Yes, there are also tingles.

What triggers this seems to be hit and miss. Slow, steady, soft speech patterns or rustling noises can do it, so does close personal attention such as a visit to the eye doctor. The aforementioned community is a subreddit that deals in posting videos that cause this sensation for people. These generally range from creepy videos of people whispering things into a microphone to creepy videos of people making various noises with various things or sometimes role play videos that are a mix of the two. There's no getting around the creepy factor, because it is and doubly so if you don't know what the fuck is going on.

As for myself and my history with this... condition(?), I've had it for as long as I can remember. I recall fairly well the first time I actually tried to vocalize it to somebody. One of my cousins and I were coloring at my grandma's house and I said something about tickles in my brain. I believe I was six or seven at the time. My young self would try to figure out ways of inducing the "tingles", at one point settling on raising and lowering my eyebrows repeatedly in erratic patterns (the results were mixed).

As I rolled into my later teen years, I kind of forgot about whole thing except when I would go to the optometrist. It wasn't until mid-2012 when I saw an ad for the subreddit that it was brought back to my attention and a little bit of perspective put on the whole thing. Having been tuned back into the phenomenon and having a better idea of how to trigger it, I tested the waters with the creepy videos and, though it's hit and miss, they work. Now, late in the evening, I'll often put one of those videos on, lay down on my bed or the couch with my earbuds in, and take a train to Relaxation Town.

It's quite nice, really.