Matt Hackmann

Fun with Load Balancers and Running on Lean Disk Space

Posted On

A while back, I spun up a new server and threw a load balancer on top of it and my existing server. The primary reason at the time was to allow to continue to run while Linode performed some mandatory maintenance on the original server (we'll call it "Chitoge"). The size of that directory at the time was somewhere around 100GB; my new server (we'll call that one "Taiga") has something like 20GB of total space. Obviously, I couldn't clone from Chitoge to Taiga, so I got a little clever. Using some fun around the hosts file, the nginx site config, and a small PHP script, when a file can't be found locally, it'd be retrieved from the Amazon S3 store I use as long term backup. S3 is slow, though, so I later changed it to pull directly from Chitoge. This helped both with speed and didn't count against traffic since internal network traffic is free. In a bout of laziness, instead of opening a connection to that server via IP and hand massaging the 'Host' header of the request (or mounting some sort of file share), I just host overrode on Taiga to point to Chitoge. Everything worked great.

Last night, knowing that I was soon to exhaust space on Chitoge (a thing that has happened on more than one occasion), I decided to convert it over to the same setup as Taiga. This would allow me to kill dozens of gigabytes of locally stored images and extend the life of my server plan at least a couple of years. Knowing that I was now deleting Taiga's source of truth image store, I changed up the script to check from a pool of servers; first try a local server, then if that fails, go to AWS. That worked fine and, while I was setting this script up on Chitoge, I decided that it should check against Taiga just in case it had a copy before wasting time downloading from AWS.

I got everything set up and started deleting old image files. Everything seemed fine, so I let the deleting continue and went to bed.

When I woke up, I was greeted by many messages in my reddit inbox saying that images were experiencing 50x errors. My initial thought was that I was saturating the CPU/connection with that long running process, so I killed it. I was unable to repro any errors, so I went on my merry way... until somebody new complained. Well, fuck...

To make a long story short, because I had pointed each server to each other, I was creating an infinite network loop whenever an non-local image was requesting. Here it is in traffic logs:

[User Request -> Taiga] GET /the_image.jpg

Doesn't exist locally. Retrieving from first server in pool: (Forced pointing to Chitoge)

[Taiga -> Chitoge] GET /the_image.jpg

Doesn't exist locally. Retrieving from first server in pool: (Forced pointing to Taiga)

[Chitoge -> Taiga] GET /the_image.jpg

Doesn't exist locally. Retrieving from first server in pool:

So, basically that until the scripts start timing out. Now, in a normal infinite loop, you kill your own process but, generally, should be safe from affecting others. However, here, every time there's a new request, that's potentially a new instance of PHP being spun up because the other instances are deadlocked waiting for data. So, eventually, the entire chain collapses and you start getting orangereds in your reddit inbox.

In the end, I still have Taiga sourcing from Chitoge since all image uploads go directly there and, generally, Chitoge will have a copy of the file. Chitoge will always take the AWS hit when a file doesn't exist to avoid this loop scenario. There are many ways I could avoid this scenario entirely, but for now, this set up is getting the job done.

Stoned - My Experiences With Marijuana

Posted On

I also considered naming this “The Post My Mother Doesn’t Want to Read”, but that’d be about when I killed that kid in kindergarten. Maybe some other day…

Growing up in a highly conservative Catholic family, it was ingrained into me from an early age that drugs are bad, m’kay. Anybody using or trying to give you these substances were people you generally wanted to avoid. As an interesting aside, alcohol consumers were not subject to the same stigma. But, be that as it was, I carried on through most of my life with a contempt for such things. This feeling didn’t waver, even as I prepared to move to California and the joke running rampant was “huhhuhhuh, you gonna get your weed card?”.

I’m not sure exactly when that sentiment began to change. Through peripheral consumption of various research papers that were being published about the effects of THC on the human body, I knew that marijuana was, in general, less harmful than most recreational drugs and even alcohol. That probably softened the attitude towards the drug itself, but did little to change my view of the consumers and culture built around it. Adding to that, my only knowledge of consumption methods was via burning the leaf. “Less harmful” be damned if you’re still pumping your lungs full of carcinogens.

My perception of the drug and its effect on consumers probably actually changed at about the same time. I began to find out that a lot of the people I held in high regard were consumers, people I never would have suspected were it not pointed out to me. There was, of course, a lot of cognitive dissonance upon hearing this because, in my world view, if you smoke marijuana, you become a couch-bound bum who just smokes and says “duuuuude” a lot. In that same conversation, “edibles” were mentioned, and not the home baked weed brownies. Chocolate bars made by companies who had to adhere to the standards of a government body in order to sell their wares through legal venue.

That, I think, was the tipping point. THC could be made available to me in a form that was easily consumed and easily metered. And so, about a year ago, I tried a very small bit of marijuana laced chocolate just to test the waters. The outcome? Well, I was having trouble determining that through the more immediate feeling of being tipsy that evening. In retrospect, the profound tiredness I felt a couple hours later was the effect of the chocolate.

With the gates having been opened, and at the time thinking I had not felt the effects, I allowed myself a little more experimentation. Of course, this had some upper limits. Absolutely no inhalation of burned weed; being that I have asthma, this would not be good. Also, see above about carcinogens. Really, that left me with a couple of options: try a larger dose of edibles or vaping. And I tried both, with differing effects. Edibles took a very long time to set in and lingered for hours. Vaping was much faster to feel the effects, but shorter lived. I decided that if I wanted to continue using marijuana, these would be the two vehicles of delivery.

In May of last year, I decided to get my card. Or, in my case, I opted to not buy the card and just got the paper. This entailed a visit to a doctor (I want to put quotes, but she was a licensed M.D.) with complaints of some ailment. I said something about migraines, my mother used to have them, had tried herbal something or others, blah and etc. Doctor didn’t blink an eye (she knew I was lying) and, after signing the paper, we spent more time discussing her grandfather who used to live in Bartlesville and had died recently than any topic for why I was there. It was an odd experience for sure.

Thus, through the last year, I’ve been a fairly light user of the product. I eventually gave up on edibles because it basically destroys an entire day. Also, I was beginning to experience bad indigestion, the culmination of which involved me throwing up on a train about a mile from my stop. Of course, that could have also been the dozen-ish drinks I had that night (it was a good birthday). There have been times when I’ve gone without, both due to circumstances or intentional “quitting”. The “quitting” was because I believed that the vape pen I was using was causing the back of my throat to become and stay raw due to heat. I stopped and it went away, so there’s a high potential for correlation (to be fair, that particular pen was later removed from the market). Marijuana is a non-addictive drug, so I never feel like I need it. However, like anything, one can become psychologically addicted to the high itself. I’d be remiss if I said there wasn’t a little bit of that going on.

Today, I treat marijuana in basically the same way I treat alcohol. If I’m to use it, driving and most outside activities are banned. Unlike alcohol, I will use it by myself outside of social situations, but generally only before bed as it helps calm my mind for sleep.

Going forward, I’ve already decided not to renew my card this year. There are a few reasons, the first of which is that I don’t want to have to make another “doctor visit”. Secondly, it’s obviously cheaper to drop this. Granted, because I’m such a light user, the cost of a year’s worth of marijuana is still less than a handful of average nights of bar hopping. I will probably continue to partake in social settings, but I do not feel the need to have it on my person or in my home.

The takeaway from all this is that there are a lot misconceptions and old feelings about marijuana and its users. Becoming stoned will probably not turn a person into a stereotypical stoner unless that person was already susceptible to such things, as can be expected with almost anything. I still don’t like the culture, nor do I like the smell, but I think if one wants to argue about the negatives of using marijuana, one should also consider alcohol and cigarettes, both of which are (mostly) socially acceptable, legal, and much worse for a person’s physical well-being.

One good turn

Posted On

In my previous post, I mentioned the little "micro list" thing my mom is currently doing on her blog. Among these was a favorites list. Noticing that she had added a row for video games, I snarkily suggested she should have "favorite TNG character". To her credit, she devoted the entire next post to the topic of Next Generation. It's moments like these when I consciously consider the fact that my mom has a bit of a nerdy streak which is pretty cool.

But, enough of that, given that she went through the trouble of constructing that list, I feel it's only right that I do the same. So, following her parameters, here is mine:

1. Favorite Character - Gonna have to go with Picard on this one. How can you not respect the guy who is essentially a role model for humanity? Following closely behind would be Q, Data, and Riker (in no particular order).

2. Favorite Episode - Much to my mom's chagrin, I can't not give this award to Cause and Effect. I believe it's been mentioned before, but I love time travel stories and time loops in particular are a guilty pleasure of mine. On a funny aside, the first Google result for "enterprise keeps exploding" is the Wiki entry for this episode.

3. Least Favorite Character - You thought I was going to say Pulaski, didn't you mother? Well, despite the fact that I cannot stand her elitist attitude and the way she treats data, she's only number two in that regard. No, this coveted award goes to Tasha Yar, the overly emotional, too bull-headed security officer that preceded Worf. Her death is one of the shining moments of the series.

4. Favorite Doctor - I don't have much to say about Lwaxana Troi (though, I do enjoy the episodes she's in), but I do have words for the various doctors in the series, my favorite being (drumroll) the Doctor! Sure, the actual TV series only had Pulaski and Crusher (of two, I prefer the latter), but the good ol' EMH makes a wonderful yet brief appearance in First Contact making him eligible for this list. You can't not love his dickish, self-serving attitude and, as a main character on Voyager, was given much more time to be fleshed out as a character.

5. On the topic of Q - Who doesn't love Q? He's such an asshole, but a highly entertaining one (see above about the Doctor). Q episodes always tend to bring out the more interesting topics, such as "how would you relive your life given the chance"? He also introduced the Borg, which is kind of a big deal.

Okay, I have geeked out about Trek for long enough. I should run to the store to pick up ingredients for turkey soup. Should be a hoppin' good time.

I am thief, watch me steal

Posted On

Taking a page from the tiny little blog posts my mom is doing right now, I'm going to steal that idea.

February 5th, 2015

1. Went to work today

2. Larger breakfast because everything looked good.

3. Made coffee. Then drank it

4. Refactored touch code to be more modular. Such tedium, decent results.

5. What the fuck, Nick? I get that it's nutella day, but I'm trying to lose weight.

6. Doubled my morning water intake because I was thirsty

7. Salad with tuna for lunch. Such sadness

8. Leave work early to await my piano delivery

9. Oh, it's already here. And the stand is here too

10. "Hey, guy. My name is Mr. Headache. Let me throb for a while".

11. Fuck you, Mr. Headache.

12. Still waiting for the piano pedals, but it's good to be playing things again.

That almost felt therapeutic.

2014 Micro Retrospectives - Programming

Posted On

Getting back to that thing I was doing, today I'll cover last year's coding efforts. And what efforts those were. Actually, I think 2014 might be my most active year ever. Just going by GitHub, I totaled over 670 commits. Of course, that doesn't include anything I did at work (and there was certainly a lot of that). So what exactly happened last year?

RedditBooru v2

This is pretty well documented. January was spent rewriting basically everything as I attempted to port RedditBooru to nodejs. Then, when that didn't pan out how I wanted, went back to the original PHP stack with major modifications to the existing codebase. The new feature set in v2 wasn't the huge success I expected it to be (though, my bandwidth nearly doubled overnight). Indeed, views have been steadily dropping, though I think this is partially due to better RES integration that came along. Still, 875k page views by 214k unique users is pretty good and more than I would have anticipated anything I'd ever create would do. But, that doesn't even begin to compare to...

Anime Bracket

Some interesting stuff happened here (that is actually pending a full write up). In August, a user on the anime subreddit contacted me about using the awwnime bracket site for a bracket he wanted to do. I said yes, pulled some code strings, and let him on his merry way. Long story short, that bracket garnered over 1 million page views and 2.5 million votes by 24k registered users. Here's a fun little image showing the difference between that bracket and the one I usually run in the fall:

That tall peak is 45k users in one day

That kickstarted an effort by me to make the bracket accessible to anybody (because it has also proved to be financially sustainable). Indeed, as I write this, I'm taking a break from working on the new site. But, more details on that later.

Everything Else

As usual, those two projects continue to be my main concern while not at work. However, that's not all I've done. There was the aforementioned contributions to RES updating support for RedditBooru albums. There was also the work I did on the awwnime bracket app. I also did some work around my Hue lights and a couple of other trifling things that didn't gain traction. Of course, that's not including the stuff that I don't commit (and I do a lot of small scripts just to help out with day-to-day life).

Alright, so there's another post.