30×30

So I came across a blog recently that the writer started to document her successes at meeting goals for herself. The interesting conceit of the list of goals is “NxN”—that is N things done by the age of N. So, as the title of my post suggests, I would create a list of 30 things to do by the time I’m 30 years old. The idea is that you do a new list each year. If I were 23, I’d do 24×24. When I turn 30, I start a new list of 31×31. In chatting with her, I said that I thought it would be hard to come up with 30 things to do in the next year. I don’t know that I’ll actually make an effort to complete the items on the list. This is more about seeing if I can even come up with 30 things.

  1. Become comfortable leading Balboa
  2. Apply to the spring 2014 GW Master’s in International Science and Technology Policy program
  3. Get back to an average of 50 books read in a year
  4. Write at least 1 blog post a month.
  5. Begin biking to and from work at least twice a week
  6. Buy new bookshelves and fully organize my books
  7. Fully catalogue my books
  8. Compete in a newcomer’s jack and jill
  9. Travel to a lindy event other than Lindy Focus
  10. Go rock climbing semi regularly
  11. Take further Swedish or Polish classes

That’s what I came up with in odd moments of thinking while at work on 13 February.

  1. Visit California

That’s all I got on 14 February.

  1. Submit a short story to a science fiction/fantasy magazine

The end of an age

Last night, a rather significant era in my life ended. And it’s the end of an era that I am sharing with a lot of people all around the world, though when they experience it wasn’t necessarily last night. For some, it probably ended a week or two ago. For most, it’ll probably be in the next few months.

I suppose “significant” isn’t really that true in the grand scheme of things. But it is something that has definitely influenced one part of my life for at least the last 15 years. So, what happened last night? I finished reading A Memory of Light, the final book in the Wheel of Time series. For those of you who know the series, you understand already the significance, even if you’re not a fan of the series. For the rest of you, let’s just say that this is a moment that a lot of people didn’t expect would ever occur.

Robert Jordan began writing the Wheel of Time in the 1980s, and the first book, The Eye of the World, was published in 1990. Five more books followed, each one year after the previous. Book 7 slipped to a year and a half. Number 8 to two and a half. Book 9 took two morw. Book 10, just over two. With each book at least 650 pages, and most over 800, this is a prodigious amount of writing that Jordan accomplished. And the world he created was understandably complex. The initial cast of three expanded quickly to dozens and the plots expanded even more so. Corralling all of this made it understandable when book 11 took 3 years (though Jordan also publish a short prequel in 2004). By this time, a lot of people had already began to feel that the story was suffering because of its complexity, that a lot of nothing was happening in all of the pages being written. Two months later, in December 2005, Jordan was diagnosed with primary amyloidosis with cardiomyopathy, a disease with no cure. The twelfth book Jordan promised, would finish the series, even if it had be over 2,000 pages long. He died just less than 2 years after his diagnosis, book unfinished. But he left behind extensive notes, hoping that someone would be able to finish the story for him. Jordan’s wife and his publisher chose Brandon Sanderson to be that someone in December 2007. Sanderson is a prodigious and prestigious fantasy author in his own right. But even so, it was a year and a half before any real update on the final book, A Memory of Light, was given. In early 2009, it was announced that the finale to the series was going to be split into three books! The first of those came out six months later in October 2009 and part two just a year later in November 2010. But A Memory of Light itself was not released until just ten days ago, January 8, 2013. 22 years, 358 days after the publication of The Eye of the World.

My introduction to the series came around 1996 or 1997. My mother, wonderful woman that she is, got me the first 6 books in the series in a pair of box sets for Christmas. I tore through the books in just a few short weeks. I devoured the world that Jordan created. The adventure, the magic, the new ideas and depth of the history, cultures, mythology, and characters all combined in a way I absolutely loved. While the later books attenuated some of that love as the story became overly complex and slowed down, I still have those first six paperbacks, and the number of times I have read them has worn their bindings down and leaved their pages. When each of the new books came out, I usually re-read the whole series from start to finish. The characters became very well-known to me. When the first of the Sanderson finished books was coming out in 2009, I reread the whole series of 11 books in the span of a month. That was almost three and a half million words.

I picked up A Memory of Light last Sunday and I reviewed brief plot summaries of the last three or four books just to remind myself where things stood before I began reading it on Tuesday. I’m still not sure how late I stayed up that evening, but I was somewhere between a third and two-fifths through it. Yesterday after work, I did very little other than read and finished the book around midnight. And I experienced moments of joy and happiness as the characters achieved great things and moments of hurt and sadness as they suffered or died. 16 years of familiarity and wondering “What next?” and “Will they survive?” was finally brought to a resolution. But I know that A Memory of Light was not the ending to my relationship with these characters or novels. There are neither beginnings nor endings to the turning of the Wheel of Time. But it was an ending.

Learning to webscrape and playing with regex

When I posted about the trend in movies toward franchises, I briefly ruminated on the idea of doing a similar, more in-depth analysis for all of the wide-release movies in the US from 1980-2011. The biggest stumbling block was the size of the data set — 31 years with anywhere from 80 to 160 movies gaining wide-release status. Just putting together the spreadsheets by hand would be time consuming. But the other night as I was falling asleep, I realized that it should be possible to craft a webpage scraping script which would pull the lists off of BoxOfficeMojo’s site and which would store the lists as CSVs.

It took me a few hours to hack this together as I don’t do much programming regularly, though it’s something I enjoy challenging myself with on occasion. And this ended up being a nice learning experience. I’ll go ahead and drop my code here. It’s far from clean, and someone who programs regularly can probably find a dozen ways to clean it up with better ways to go about doing what I did.

<?php
for ($year=1980; $year<=2010; $year++) {
 $titles = array();

 for ($i=1; $i<=2; $i++) {
  $url = "http://boxofficemojo.com/yearly/chart/?page=$i&view=widedate&view2=domestic&yr=$year&p=.htm";
  $ch = curl_init($url);
  curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
  $curl_scraped_page = curl_exec($ch);
  curl_close($ch);
  //echo $curl_scraped_page;
  $content = preg_replace("#><#", ">\n<", $curl_scraped_page);
  //echo $content;
  $regex = '#<a href=\"/movies/.+\">.*</a>#';
  preg_match_all($regex,$content,$matches); // stores strings as $matches[][], size [1][102]
  //var_dump($matches);
  array_shift($matches[0]); // get rid of extra unrelated
  array_pop($matches[0]); // get rid of extra unrelated
  //var_dump($matches);
  $moviereg = '#>.*<#';

  foreach($matches[0] as $movies) {
   //echo $movies . "\n";
   preg_match($moviereg,$movies,$title);
   //echo $title[0] . "\n";
   $title[0] = substr($title[0],1,$title[0].length-1);
   //echo $title[0] . "\n";
   array_push($titles, $title[0]);
  }
 }
 //print_r($titles);
 $titlelist = "";

 foreach($titles as $title) {
  $title = str_replace(",","",$title);
  $titlelist .= "$title,";
 }

 $fileyear = "$year.csv";
 if (!$handle = fopen($fileyear,'a')) {
  echo "Cannot open file ($fileyear)";
  exit;
 }
 if (fwrite($handle,$titlelist) === FALSE) {
  echo "Cannot write to file ($fileyear)";
  exit;
 }

 echo "Success, wrote to file ($fileyear)";

 fclose($handle);
}
?>

The meat of the program opens the webpage at BoxOfficeMojo based on the year, and pulls the page’s HTML into a string. I had to do a bit of formatting by sticking new lines between any neighboring HTML tags because the regular expression (“regex”) that I cobbled together was just throwing out a massive block of text and seemed to be waiting to find a new line. Then it was just a simple case of creating an array of all the occurrences of a specific sequence — the a href=”/movies/…” which indicated the link and title of the movies in the list (and two other links that weren’t part of the list). Then, get rid of everything other than the actual movie title and any commas in the movies’ names and create another string which was just the list of the titles as separated by commas and save it as a CSV. Toss all of that into a pair of loops which incremented the year value and checked for the second page of the lists that had more than 100 movies on them and everything’s groovy.

Open all the CSV’s as sheets in the same spreadsheet, convert the row of movies to a column, and I now have a 31 tab file that’s ready to be evaluated for the nature of the scripts. When I get around to doing all of that evaluation is another question entirely. If only I had an intern.

Remakes and adaptations in the movies, part 2

On Reddit today, someone submitted a link to a page which attempted to take a look to see if Hollywood is giving up on original scripts. The approach the author took wasn’t very good, only evaluating the top 10 US grossing movies in 2011, 2001, 1991, and 1981. By taking such a limited sampling of points, without any evidence to show that there is any sort of continuity to the kinds of films in the top 10, you can’t actually make any significant conclusions. Any or all of those years could have been statistical flukes. And as I pointed out before, it seems just as likely that the originality of the scripts in the top 10 movies is just as likely to be more influenced by the movie-going public than by the studios that are making the movies.

I linked that old post in the Reddit discussion, but realized I couldn’t find the spreadsheet file I’d based it off of. It probably got lost among my files on my work computer when I got laid off. But, that just means that I had even more reason to recreate it and improve it. So I did. Wikipedia’s lists of the top grossing movies switches from worldwide gross to US gross somewhere in the early 90s or late 80s, so I ended up repeating myself after I realized the inconsistency. As a result, some of the tables show both the domestic top 10 (as given by BoxOfficeMojo)  and the worldwide top 10 (via Wikipedia), but on the domestic top 10 counts are charted. I also cleaned up my categorization of the movies. There are now 6 categories — original, adaptation, original franchise, adapted franchise, original remake, and adapted remake. I think they are all fairly self-explanatory.

For the record, GoogleDocs is not particularly useful if you want to do a scatterplot and then add a trendline. I had to use a function to find the y=m*x+b equations and then calculate yearly resulting values and then plot them. And that’s just for a linear regression. If I want to do polynomial regression, I can’t imagine how obnoxious it would be. But, given all of that, it is pretty clear that the number of original scripts is getting lower. Even if you include the number of original franchises and original remakes.

Franchise vs. Non-franchise

There is a similar growth in the number of top grossing movies which are parts of franchises, both original and adapted. Prior to 2001, franchise films were only about 1/4 of the top 10, since then, they have been closer to 1/2. And that doesn’t count the first film in the franchise.

Original vs Adapted

And, for all of the 1980s, there wasn’t a single top 10 movie that was a remake of an earlier movie. And even now they aren’t taking over the theaters. Just under half of the years since 1990 haven’t had a top 10 remake, though 2005 had 5 (King Kong, War of the Worlds, The Lion, the Witch and the Wardrobe, Charlie & The Chocolate Factory, and Batman Begins (which was really more of a reboot of a franchise than a remake of the original Batman movie, but actual reboots are so rare that there is no reason to create an entirely separate category)).

So, what can we conclude? More franchise movies in the US top 10 grossing films. More adapted scripts as well. And two of the last five years have had no original, non-franchise films in the top 10 in the US. Eyeballing the global numbers since the mid-90s shows pretty much the same trend. But we still can’t make any claims about whether the success of the movies is because the overall Hollywood trend mirrors the top 10 or if the American public just prefers to go see movies in franchises that it knows or based on other sources with which it is familiar.

I would still, at some point, like to take the time to take a look at all of the yearly US wide-release movies, but there were 146 such in 2011 alone. That’s just under half of the number of movies that were counted for this data as it is. But I happen to have some time on my hands, so we shall see.

On individualism

If you ask me about my political leanings, I’m likely to tell you that I’m a small-L libertarian. Or a philosophical minarchist. Small-L libertarianism — at it’s most basic, socially liberal and fiscally conservative — is the “new” big thing in American politics. One of the key characteristics of libertarians and of a lot of the conservatives who rail against social welfare programs is that they claim they favor individual responsibility and the rights of the individual. A lot of this argument goes back through Thomas Jefferson’s declaration, “We hold these truths to be self-evident, that all men are created equal, that they are endowed by their Creator with certain unalienable Rights, that among these are Life, Liberty and the pursuit of Happiness”, to Rousseau, Locke and Hobbes’ views of the state of nature from which man arose.

The three philosophers envisioned variations on a world in which man was a solitary creature and that encounters with other humans were violent by necessity. It was only through overcoming this inherent violence that humans were able to establish societies and rules for governing human interactions. But, without question, this is choosing the chicken when asking whether the chicken or the egg came first. (Evolution tells us it was the egg, because there were plenty of species which laid eggs before there were birds, let alone chickens in particular.)

The fact is that humans are inherently social creatures. All other currently existing primates are social creatures and we have evidence that early hominids were as well. The earliest humans, those residing in a “state of nature”, were not solitary creatures. They had families, often extended ones. Based on the behaviors of the other extant primates, when those extended families got large enough, they would split apart because of the stress they put on the food supply and other resources. Over time, after each group split enough times, the separate families would be unknown to each other. And when two distantly related families encountered each other, the result would often be a conflict over resources. In this, the philosophers were undoubtedly right. Violence was inherent in the life of early man. But it wasn’t a violence of individual against individual,  it was a violence of tribe against tribe.

And this is the root of something which I’m becoming more aware of amongst the various folks who claim to support individual rights, and that they want to protect those rights against encroachment from the state. In proclaiming individual rights uber alles, people often ignore the fact that individual rights have always been suborned to some extent to the health and security of the tribe. In their everyday lives, however, people tacitly acknowledge this truth in their behavior, acting beneficially for not just themselves, but for their family and their friends, and often for a local community that they don’t necessarily have close biological ties too.

It is a rare thing for a person who is defending individual rights to actually be a true individualist and not a voluntarily contributing member of a community. It is this seeming contradiction that has always kind of tickled at the back of my mind whenever political discussions develop. The truth is that almost everyone is a socialist to the extent that they want their social unit — their tribe — to be equitable to its members, and to ensure their health and safety.

What gets me curious is why some people are more able than others to expand the population of their “tribe” to include an entire nation, or even the global population as a whole. I seem to recall reading about some research study which found a potential connection between the presence of certain genetic markers and this ability, but my memory may be lying to me.

On a personal level, I’m continually getting better at recognizing that my tribe is as much of the world’s population as wants to be part of a single tribe. On the national level, I tend to disapprove of the methodology of many of the policies we’ve enacted in an attempt to most effectively benefit the nation because I don’t believe that the majority of them are effective at all.