(Click to make it bigger, which you’ll need to do!)

The graph represents a normalized-compression-distance comparison of 950 boxes described by the contents of their manuscripts. (Which to 22 hours of CPU time to accomplish.) Given the nature of the documents, it is a somewhat artificial comparison. But, you’ll notice that the clusters are almost always within series. That’s what we’d expect from institutional documents. But, I need to chart the years from the various series clusters and see if they come from roughly the same decades.

Earlier this week, Andrea Doucet published this piece in the Chronicle on her own experiences as an academic who blogs. I like Doucet’s distinction between the forms of writing and reading associated with, on the one hand, the academic’s usual labor and, on the other hand, the blogosphere. There are a number of academics who write prolifically for sites run in the mode of the blogger, as delineated by Doucet. One of my favorite such places is Lawyers, Guns, and Money. That’s obviously not the kind of publishing I do here on parezcoydigo, neither the frequency or range. Doucet notes that she didn’t start her own blog until she had reached mid-career status, a point at which she felt free to engage in less credit-minded forms of writing. It’s a shame that she, and many others like her, felt the need to wait.

I started this site in 2008, which isn’t that long ago and pretty late in the blogging game. I did so to have a place to write about the intersections between my formal work (I was revising my book at the time) and technology. There aren’t too many forums in which one can write about the process of research, including the nitty-gritty details of things like workflow. I also wanted to participate in what I saw as a new, and potentially powerful form of scholarly communication. As a new faculty, I decided to put stuff out on the web that I wish I’d had access to as a grad student back in the late 90s and early 00s, including that nitty gritty. I do write about my content work some as well, though most often in conjunction with process and workflow or, as yesterday, as part of conference reports. These types of posts tread somewhere between the tortoise and the hare, as divided by Doucet.

In my view, what’s spectacular about a blog as a platform is its flexibility. And I would emphasize in a way that Doucet did not that a blog is, above all else, simply a platform for writing. This is what I tell my students every semester when I require them to set up their own blogs. Platforms or frameworks in the digital realm are systems for building. Those that are most powerful are also often the most flexible (which can cause its own set of complexity problems). As platforms for writing, blogs are much more flexible than many seem to imagine. There is no limitation on the form or content what one puts out there, from one line quips to serialized long-form narrative. And that’s why, I think, academics should embrace the platform in its variability.

As the issue of credit was mentioned by Doucet, and seems to be a perennial concern, I’d suggest that there is a natural place for the blog to fit in one’s tenure dossier, and it isn’t with research. Given the flexibility of the platform, the varied forms of writing it inhabits, and the differing publics it reaches, I place the academic’s blog in the category of service (and occasionally teaching). Whether one writes about their specialty content and historiography/literature, their process, news and current events, or whatever, they are reaching out to new and different publics than they ever will through the lowly journal article or the staid monograph. They also are engaging in a form of community building through the web that is vital, I believe, for the academy’s social relevance. This is not to say that writing for the web should be, in historian’s parlance, primarily about popular narrative. Being public with what we do alone starts to break down the ivory tower.

In my own dossier, this is what I did– I included a bit of explanation about what parezcoydigo is and what I do here. I included stats, which while not impressive at web scale certainly are at academic journal article scale. And, using Anthologize, I put together “books” of everything I’ve written here, as well as a best-of bit. For my not-so-tech-inclined brethren, I also printed the Anthologize-produced pdfs to include in my physical dossier. So, I asked for credit and, in the narrative of my department, received credit for public service.

(I’m writing this on a plane, and as I was finishing that last sentence, two Delta jets going east and west and my plane, heading north, just passed within a few thousand feet of each other, split only by altitude! Crazy three-dimensional space. It’s the closest call I’ve seen with flights 35 weeks or so a year. Except for the time the collision avoidance system took over and put us into a pretty steep dive.)

Ultimately, though, I think credit is a secondary issue. I’ll go further than above and say that a blog isn’t just a platform for writing; it’s also a platform for engagement and smart community. Writing, engagement, and smart community are things that any academic should get behind.

So, here’s the paper:

Racial Silences in the Criminal Archive: Jail Censuses in Quito, 1750-1850
Chad Thomas Black, University of Tennessee

Introduction

In 1978, the city of Quito was named in UNESCO’s first class of World Heritage Sites. Those first twelve sites were nominated by their home governments, who established the claims upon which they were accepted. For the advocates of Quito’s case, those claims were built on the idea that, “the city has the best-preserved, least altered historic centre in Latin America.”¹ As Susan Webster has recently noted, the literature on Quito’s architecture (the basis of its World Heritage status) has long held that the city is unique in the Americas, and particularly for the Andes, as “the most European of any produced in Spanish America”² The perception of the European-ness of Quito’s architecture has served to reinforce a long-held sense that the city was a Spanish space, founded on the ruins of an indigenous city that was relegated to the hinterland, with a buffer population of mestizos in between.³

Webster goes on to unpack the European-ness of Quito’s architecture as a matter of vantage point, as the documentary record demonstrates that the construction of the city’s monumental buildings was led in design and execution by indigenous master builders and their own crews. In a (indigenous) world where authority was more potently expressed through the “control of the people and processes of architectural production,” the perception of European-ness is undermined.⁴ Hidden in plain site, in the walls of stone and adobe is an enduring indigenous presence. The metaphor is a fitting one for understanding the relationship between authority and race in the broader history of colonial, and early-republican Quito.⁵ Indeed, neighborhood census data from the late eighteenth century reveals many “Indians” hiding in plain site. Just take, for example, the house of Dr. Dn. Pedro Jijón, whose twelve family members shared the block-sized residence he owned in Santa Bárbara with a long list of individual and family tenants:

• María Nieto, single. Family 3
• Manuel Valeriano Escobar, married, with one son, laborer 3
• María José Villavicencio, widow. Family 3
• Martina de andújar, single 1
• Miguel sánchez, Indian, married, tailor. Family 3
• José Tituaña, Indian, carpenter, married. Family 4
• José Pillajo, Indian, weaver, married. Family 4
• Nicolás Pazmiño, married, tailor. Family 3
• Francisca Gualanlima, Indian, single. Family 3
• Catalina Carrera, widow 1
• José Issa, Indian, cobbler, married. Family 3
• María Manosalvas, widow. Family 2
• Total: 45⁶

So many “Indians” living in a nominally-Spanish space (a recurring equation in colonial and modern historiography: urbanity == Spanishness/whiteness; rurality == indigeneity). Quito was, and is, a space inhabited by and even defined by a mix of ethnic/racial identities, defined in many different ways during different points in time but pointing to a related set of issues as often as notable by their absence from the archival record as from their presence.

Archival silence does not itself equal absence. Today I’d like to talk a bit about one such silence–> in the weekly jail censuses of the late colonial and early republican periods for the city of Quito. I’ll provide a description of these censuses, the process of their making, the data within, and the context of that data. As I’ve worked with these sources over the last few years, I’ve become increasingly convinced of the importance of historicizing racial terms, as institutional records can very well be silent on race, even as we find in them categories that in the modern period come to inhabit racial meaning. That said, categories that were administrative, judicial, or fiscal were still sometimes doing racial work.

Defining some terms

But, before we can get into that, a word or two is needed on a few terms of bearing. Using jail records to look at the relationship between race and the state requires I be clear about what I mean about those terms, and about criminality or crime as well. And, when we put those terms in to the shifting context of the period under consideration, I need to be clear that the things those terms describe change in the course of that period.

Making sense of the jail censuses requires that we recognize the real difference of the “judicial state” of the Spanish Empire from the modern state that succeeds it. The term “state” has been seen as problematic for the historiography of the colonial period. Bureaucracy, yes. Institutional authority, sure. But, following Tamar Herzog’s definition of the state, as “entities that act in the name of public authority and political authority”, we have the problem of notions of public.⁸ Herzog argues that “public” nature of the state poses theoretical difficulties for a system in which there was no distinction between the public and the private, an argument that I share. I’ve argued before that the process of Independence was itself a process of constructing private and public spheres out of the conflated authority of the judicial state.⁹ This dynamic is key to understanding the racial silences of the criminal archive on both sides of the republican divide.

Notions of the state aren’t the only contested terrain in the colonial historiography that intersect for us today. The question of whether or not race, if not racism, existed during the colonial period have hinged on the definition of what race is. If looking for a scientific racism of biological inferiority, one will be disappointed. But, cultural prejudices abounded, including those tied directly to blood and lineage.¹⁰ One of the key problems in interpreting the distinctions between ethnic prejudices and race/racism for the colonial period has been the context in which terms we associate in the modern period with racial ascriptions were used. In the venue of the judicial state, ethnic or racial terms frequently carried with them very specific administrative and/or judicial meaning. In the context of actions by the judicial state, for example, a term/category like indio or negro would have very specific meanings. It’s my contention that is the case with these jail censuses.

The jail census represented one of these actions by the judicial state, and understanding the occurrences, or lack of occurrences of “racial” categories depends on understanding the administrative meaning of those terms in that context. The late-18th c. was a transitionary period on many fronts, including in the emergence of new forms of racial identification. It’s no accident that in Quito, as in New Spain at the same time, artists of a naturalist or taxonomic bent were making the now famous casta paintings. It’s also a transitionary period from the perspective of criminality, marking the initial emergence of more modernist forms of punishment in which crime, and criminal acts inhere in the individual. But, we weren’t there quit yet, and through the end of the colonial period (and even into the republican), individuals are not indictable as inherently criminal, regardless of the acts they committed. (Yes, individuals are indicted, but individuals are not seen as inherently criminal based on racial or ethnic categories.) Criminal acts represented a disruption in the flow of communal harmony that the judicial state existed to ensure, and flow was restored through punishment of acts.

Following Gotkowitz’s recent admonition, the best way to analyze the relationship between race (in its various, historically-specific forms), criminal acts, the state, and the Archive is to look at the work that race does in context. And, what work does race do in the jail census?

Padrones in Procedure and Description

As part of their jurisdictional responsibility, the oidores of the Audiencia of Quito were required to perform a weekly visita, or tour/account of the city’s jails (divided into three categories– the Royal Jail of the Audiencia Court, the city jail of the Municipal Council, and the women’s jail, the Recogimiento de Santa Marta), and keep an ongoing padrón, or census. In reality, the actual work was most likely done by a notary or one of his assistants⁷ — it’s not uncommon for hands to change a few times each year in the census book.

The weekly record can vary from year-to-year, but generally followed this pattern:

1. In the City of San Francisco del Quito, on Friday such and such day of such and such month such and such year. The Señores President and Oidores of this this Royal Audiencia, (in latter years this was sometimes followed by the names of individuals, usually included three names- the president and two Oidores who at the moment were serving as Alcaldes de Corthe), made a Jail Inspection (Visita de Carcel) in the following manner.

   • the Inspection was required on a weekly basis, though most years there are generally about 48 entries in the Padrons
   • the three main jails of Quito were all supposed to be inspected, and their records were kept in two separate books –> 1 for the Carcel del Corte, and another for the Carcel Pública + Santa Marta. Or at least, that’s the best I can reconstruct. It’s not exactly clear on a year-to-year basis.
   • In the 19th c., after Independence, the practice of making Visitas del Carcel continued. Even the boilerplate remained relatively consistent: “In Quito on such and such a day of such and such month of such and such year. Your Excellency the Superior Court of this District made an Inspection of the Jail and Santa Marta in the following manner:” –> and this is at least as late as 1855 (the last year I’ve found.)

2. Following this boilerplate preamble comes a list of all detainees present in any given week. (Note, this can also include individuals who are supposed to be there, but are absent because they escaped.) At a minimum, this census includes the following information: Name of the detainee and the ordering authority. For most of the 18th c., it also includes the specific reason they were detained, and occasional updates on the person’s status. And finally, on occasion entries will includes ethnic or occupational ascriptions.

   • Names also carry status information. Though the language of community citizenship (vecino/a) was never used, respected individuals were inscribed with the honorific Don or Doña.
   • In latter decades, the date a detainee entered the jail was included in each of their week entries as well, which does make it easier to track how long someone stayed in jail for different types of offenses.
   • After Independence it was increasingly likely that an individual would be in jail as a form of punishment for a criminal act, rather than awaiting the conclusion of one’s investigation/prosecution. Of course, during the colonial period people did spend time in jail for punishment– but almost always over debt. Other criminal acts were sentenced with different forms of service, humiliation, or detention (in, say, the Poor House or a Textile Factory).

3. Following the list of detainees, more boilerplate in the form of something as simple as: “Thus ended the Inspection, and it is initialed by said Señores.” Followed in turn by rubrics from two judges and the presiding notary.

   • Twice a year, this boilerplate ending would be replaced by a statement granting debtors or others jailed due to Civil cases (with the exception of those who owed the Royal Treasury) freedom in honor of the Santa Pascuas de Navidad y la Resurrección.

What was the purpose of these Visitas? What kinds of information can we glean from them on race and the state?

It should be noted that the padrones are incomplete sources for reconstructing the jail populations of Quito, but by 18th c. standards they’re pretty damn good. The padrones were not part of a modern drive to statistically know population (following Foucault), but rather were a function of the judicial state’s commitment to, well, the production of paper. Formulaic constructions are a hallmark of the judicial state’s paper production, which in the 18th c. reached epic proportions. Of course, the judges were keeping track of their workload in part, but the padrones could also be used to track the effectiveness of the alguacil mayor.

Terms found in the padrones that could be interpreted as ethnic/racial tags:

Afro-descendent people:

• esclavo/a
• negro/a
• mulato/a

The terms are used interchangeably, and almost always indicate an individual in jail for slave condition. The vast majority of said individuals were held under the reason, “busque amo” — awaiting the search for a new master, awaiting a pending sale, etc. Masters who used the jail for this purpose were required to provide food and clothing for the slave. Ecuador didn’t fully abolish slavery until 1852, and as you can see on your chart, in 1853 there are no afro-descendant entrants.

Indigenous people:

• before Independence –> indio/a
• after Independence –> indígena

Looking at the chart before you, it’s hard to imagine how I can claim there is a racial silence on indigenous people when indios/indígenas acocunt for a (sometimes extremely) significant portion of the individuals marked in the censuses. Except that, from 1789 onward in particular, super majorities of those entries were for unnamed indigenous tributaries delinquent in paying their head tax. (After Independence, tribute suffered a long drawn out death worthy of PeeWee at the end of Buffy the Vampire Slayer the movie. The tax went by various names, but was most often referred to as the contribución personal from the 1830s through its end in 1857.) Indigenous arrests were most likely NOT to be related to tribute in the period 1767-1769, which coincided with the period of the restoration of royal authority after the Rebellion of the Barrios.

Tributary status is a fiscal category, and not a racial one. Bourbon (and later Republican) attempts to capitalize on the head tax as a form of fiscal reform account for the dramatic increases of Indians in the city’s jails during periods of collection. They also spurred an explosion of late colonial petitions to be declared Mestizo. It’s here that fiscal categories intersect with something racial, as defined by lineage.

To be more specific, percentages of indigenous detainees held for tribute:

1789: 67%
1827: 3%
1829: 46%
1834: 68%
1845: 89%
1853: 95%

I don’t have any records after 1853, which is a shame because I think it would be interesting to see the decline in use of the term indígena after the final abolition of tribute. In fact, there is a lull in the mid-1820s, a time that coincided with uncertainty on the legal and fiscal status of indigenous people under the republic.

For indigenous people in jail by name, their most frequent offenses have a rural tinge– ie, they’re there for hurtos, abigeato, or occasionally for violence. This too belies the administrative prejudices of indio as a category. Most surviving Indian corporations were outside of the city, and the city became a penal destination for these communities. But again, the Indian community as category is a fiscal/administrative entity, and says little to the “identity” of residents (temporary and otherwise) of Quito.

The rest:

Beyond these markers, we find none of the terms associated with racial mixture from the late 18th century onward, those terms so exactly delineated by the casta paintings. The rest of the population is treated as an un-delineated mass, with the only distinctions manifesting in honorifics. We do find Dons and Doñas during the colonial period. Of course, those status markers are dispensed with immediately in the Gran Colombian period, and replaced by the terms Ciudadano/a. Other status markers occasionally creep in– casique, gobernador de indios, gendarme, alcalde, and other offices. But, other “racial” terms that do appear in litigation never do here– there are no mestizos or zambos, no europeos or blancos, españoles, etc. Dons and Doñas appear in the padrones invariably as debtors, and only occasionally for moral crimes such as adultery or concubinato. The rest are the great undifferentiated plebe, the mob. But, from a “racial” perspective, were these masses NOT indigenous or african?

What about people’s names? Can they give a clue as to ethnic or racial identities within the plebe? It’s difficult to say. There are certainly many indigenous surnames that occur in the mass of no-category people. Name like Toapanta, Quilago, Cañar, Tituaña, Tutungilla, Lema, Topan, Guamansara, and many others frequently appear without category. It’s just a guess though. In the 1740s, tributaries were still often listed by name, and most Indian tributaries by then already had fully hispanized names.

Speaking silences

What work, then, is race doing in the Padrones from Quito’s jails for the century 1750-1850? There is a silence here despite the prevalence of terms we associate with racial categories in the modern period, even well into LatAm’s modernity. From the perspective of the state, one’s presence in jail intersected with certain fiscal and administrative categories, but not as racial categories and not in the manner that race comes to indict individuals. These categories, even in the 1850s, are still corporate, and relate to legal rights and restrictions accorded to corporate membership. Interestingly, though, the administrative categories could reinforce stereotypes of indigeneity, such as its equation with rurality. There are other tantalizing clues as well, in that on occasion individual’s status will switch after weeks in jail. Evidence for litigation records would suggest that those individuals were declared Indian at the time they opted for representation by the Protector for the Indians.

In fact, if we compare the more ethnographic data of litigation records to the censuses themselves, we see the extent to which indio carried pejorative connotations in the form of a cultural and class-distinguishing insult. As Borchart-Moreno has found (and I have too), ethnic slurs were common insults on the streets of Quito. There was “culo verde”, cholo/a, zamba (which combined both African and Indigenous overtones), and runa.¹¹ Of course, other insults included slurs against someone’s credit worthiness, their sexual reputation, the intelligence, and a whole host of other creative turns of phrase. And, there are a whole host of structural racisms in operation as well. These are glimpsed at in the republican-era records, as indigenous individuals being to show up as huasipongeros, even under the phrase “busque amo” and at the request of a master. Those terms in the colonial period were restricted to afro-descendent slaves. Their presence in the padrones, associated with Indígenas demonstrates that underneath the state’s ambivalence over racial categories in the padrones lies some real insidiousness.

In comparing the decades and before Independence, we find an ambivalence over the fiscal and administrative category of Indian-cum-Indígena at a time when Ecuador struggled to abolish the tribute system, while also claiming the extinction of political identities outside of the “citizen.” This ambivalence was often expressed by individual litigants who found themselves involved in varying ways with the judicial state, an ambivalence that makes very hard the possibility of drawing conclusions on the experience or meaning of race at the individual level. For example, in June 1809, Ventura Guzman filed suit before one of the municipal council’s ordinary judges to compel Julian de Echeverria to resolve a disputed house sale worth 840 pesos.¹² Guzman was buying a property in the neighborhood of the la merced Church, in the parish of santa Bárbara, in an area known as de los Barberos (Neighborhood of the Barbers). In the instrument, she is described as “doña Ventura Guzman, resident of this City, of celibate status.” Echeverria did not have the right to make the sale. That sparked litigation to recover Guzman’s 840 pesos. This time, the calculus of her legal identity changed a fraction. In petitioning the municipal court, Guzman dropped reference to both her title (doña) and her marital status, and instead filed under the rubric, “Ventura Guzman, Indian resident of the city, in conformance with the law, appear before Your honor and say…”" The doña purchasing an expensive house in the parish of santa Bárbara, not far from the center of royal power in El Sagrario, became an indigenous woman in need of the special protections offered by the crown’s protector of the Indians. What work does race do in this archival moment, on the actual brink of the Independence moment?

Looking back on 2011, I can loosely organize the posts into four categories (something maybe I should do with my actual categories): research, teaching/professional development, conferences, and digital hacks.

research

I’ve been playing around this past year with a couple of computational techniques for analyzing the Archive. At this stage, that has largely meant organizing/analyzing/visualizing the Criminales Series of the Archivo Nacional del Ecuador (ANE). In February, I wrote on how word frequency plots can be useful in tracking shifts in policing priorities. Of course, policing is an anachronism for the 18th c. in Quito, but nonetheless, in plotting keywords from the Criminales Series guide across time, definite patterns emerge.

In that post I compared plots of the keywords concubinato (literally concubinage, but covers most types of consensual illicit heterosexual sex and especially illicit cohabitation) and muerte (means death, but almost always indicates homicide). I’m interested in those terms, and in the system of punishment that grew up around them in the late 18th c. as part of my current book project, described in this post from September in the form of a grant proposal. Later in September and again in October, I took a couple more stabs at how one would conceive of an algorithmic approach to studying legal culture, and then more specifically on how normalized compression distance works as an unsupervised method for clustering like texts (in this case, decades in the Criminales Series Guide).

Of course, the big news on the research front this year was the final release of my book, The Limits of Gender Domination: Women, the Law, and Political Crisis in Quito, 1765-1830. While it officially has a 2010 copyright, and was scheduled for release with UNM Press’s Winter 2010 list, it wasn’t officially available until April 2011. I’m happy to report that feedback I’ve received from readers has been overwhelmingly positive, including I’m told the external reviewers for my tenure file. Some of the other readers I’ve heard from are graduate students who’ve had it assigned for seminar. That prompted me to post a new prologue for the book on this site in August, in anticipation of just what a graduate student reading of any book looks like. Just in case The Limits of Gender Domination didn’t show up in your stocking, it’s still on sale from the publisher through January 15. Or, pick it up at the AHA and I’ll give you a heartfelt, personalized inscription.

teaching and professional development

I did a bit of writing on teaching this year, which is ironic since I was on research leave in the Fall. Maybe it’s being away from the classroom that sets my mind on my actual classes. As with other aspects of this blog of late, mention of teaching has frequently intersected with mentions of technology. In April I wrote about the experience of hosting one’s own course sites (using wordpress) and getting pwned. That experience put me on the trail of migrating my sites off of wordpress, and into a static site generator (hyde in my case, though jekyll’s a more popular one). Over the summer, I spent quite a bit of time doing just that, and migrated most all of my sites from wordpress instances to static sites. Links to my old courses are always available at my other home. Starting in a couple of weeks, I’ll be teaching a graduate seminar on teaching world history. Thinking through what that class will look like prompted two recent posts, on conceiving the class and soliciting your feedback on what you wished you’d known before stepping into the college classroom for the first time.

Finally, in a more professional development-oriented pitch, I published the teaching philosophy statement from my tenure file, and also a guide to writing a history position job letter.

conference reports

I attended some great conferences this past year, including a few THATCamps, the AHA/CLAH meeting in Boston, RMCLAS in Santa Fe, and SECOLAS in Wilmington, NC. For the latter I also served as a program chair, which gave me purchase to many new young scholars and others more mature who I’ve only recently gotten to know. I posted a conference report on SECOLAS in March. I published the text of a talk I gave on google’s ngram viewer for the Latin American history classroom. Finally, I posted session notes on CMSs, Intro to Omeka, and Project Management from this year’s THATCamp Prime, as well as a roundup of my experience this year.

digital hacks and workflow

Finally, I did a fair amount of documenting my attempts to learn python through scripts and modules I’m finding useful, as well as the requisite couple of posts on my current workflow. We have scripts for monitoring server memory, batch renaming photos, tweeting from the command line (here and here), bursting and OCRing pdfs, posting to wordpress.com using markdown, using easygui for pythonic historians, and on making a static-site digital history archive. I did a series of posts (here, here, and here) on integrating Google Tasks on the command line. And, in that never-ending quest for an efficient academic workflow, published pieces on the current state of my workflow, using bitbucket private repos to source control research materials, and on using markdown for all my writing tasks.

It’s been an enjoyable year on parezcoydigo, and I’m looking forward to a better 2012 with more Early Latin American history thrown back into the mix.

Cheers!

Terpstra is also responsible for another of my favorite little apps, Marked. Marked is, essentially, a universal previewer and exporter for md and mmd files. The simple idea is this– if you’re working on a file, you can simultaneously open it in Marked to see a live preview of what it will look like converted into HTML. The power, though, is this– you can take that file and save it directly into not just HTML, but also to RTF or PDF. Moreover, using custom CSS, you can control directly what the output looks like, essentially giving yourself typographical control over your documents. I set up a custom CSS sheet that uses Adobe Garamond Pro (my absolute favorite font for academic writing output). So, now I can write anything I’m working on– be it a blog post, note-taking for research, transcriptions of my documents, letters of recommendation, my c.v., and even formal papers or chapters in my favorite text editor (textmate) using mmd and have easy export to RTF (if I need to share them with a Microsoft Word user) or PDF (if I don’t). Moreover, the files are simple plain text files, which are small and easy to version control. I can write, if I want, custom CSS for any task, and have real control over the PDF appearance of my documents without the complexity of writing LaTeX. And if need be, I can easily import into Scrivener for serious writing.¹ And here’s a footnote on footnotes.²

Given the ability make footnotes, control headings, automatically produce a table of contents, and, with custom CSS, control typography, it’s hard to imagine what I can’t do with (multi)markdown as an academic with a whole variety of writing needs.

• Breaking some rule or another
• Design for this dummy
• Interactivity?
• What’s the point?

Breaking some rule or another

I’ve been on a static website kick lately. Last spring I got hacked on my old wordpress multisite install through a well-known third party forum plugin. That wasn’t fun. I decided then and there to go static, since php is a scripting language for security vulnerabilities.

So in May/June, I converted my personal site to a dynamically-generated static site using hyde. Hyde is a python site generator inspired by jekyll. It’s pretty damn cool, as I get more and more used to it. It uses jinja2 templates, and allows me to keep all my site content and what not under source control. To publish an update to the site, I simply push to my server with an update hook. I even figured out a usable workflow to manage multiple subfolder sites for my courses (for the Spring– here, here, here– ok, that last one’s not up yet). Each one of those has its own css, templates, etc.

Along the way, there were two domains still using php on my server, including an empty Omeka repository for my Bourbon Quito Project. I’ve been playing around for a couple of months with replacing that site with something else. I have tons of content for it, but I’ve had so many other things going on that I’ve never developed it. It’s quite ridiculous of me to put up an academic/DH site and leave it there empty for so long. I’m sure that’s breaking some rule or another about agile web development or something. But, the genesis of Bourbon Quito Project was to make an individual research archive for a book, if not something more. Turns out that when one is hand-keying transcriptions, building such an archive can take a long-ass time.

Design for this dummy

My static-site-fever was stoked by working on my Honors Historical Methods class for the Spring semester using a hyde port of twitter’s bootstrap toolkit. It’s based on a 940px grid, and comes with all sorts of ready-made css loveliness. It’s not hard to make beautiful, contemporary-looking sites with topbar navigation, nice typography, and a variety of layouts. There are also ready-made javascripts for nifty effects if you’s like to use them. Most of all, it helps someone like me with zero design chops.

While playing around with the toolkit, I thought it might be fun to reinvent the Bourbon Quito Project as a static site. The workflow would make quite a bit of sense. When I transcribe a case, I do so in textmate using markdown. It would be simple to simultaneously publish the transcription to Bourbon Quito Project, as it would be keep push it to a mercurial repo. There are some issues to work out, though. And, with the bootstrap toolkit, it wouldn’t be to hard to make the site look intentionally designed. I decided first and foremost that I needed to change the color scheme a bit as well as the font set. Twitter is, of course, full of sans-serif goodness. I prefer my fonts seriffed. (I know this isn’t necessarily preferred web design, but for me serif fonts communicate permanence in a way that sans-serif fonts do not. And, this is a research archive.) So, I resorted to my favorite serif font with Garamond, using google’s web fonts, for paragraph text. Headings use IM Fell English, which has a particularly attractive Q with a long tail. I minimized twitter’s black/grey/blue color scheme in favor of pale yellows and reds present in an iconic 1734 map I cropped for the header image of the landing page.

I think it looks nice– much nicer than the lame css hacks I’d done on the previous omeka incarnation of the site. But design was only one of the problems I’ve been thinking through, and posed by the use of a dynamically-generated static site.

Interactivity?

In putting together my course sites using hyde, I’ve had little problem in accomplishing the interactive features I used to rely on php for. Some of it is javascript, and some of it is outsourcing. On the course front, what kinds of interactivity do I really want? I want a comment system, a calendar, forms, and a feed aggregator to point to student posts on their own blogs. For the former, I followed many sites both dynamic and static, and outsourced to disqus. The calendar, forms, and feed aggregation are available through google apis, and easily embedded. I was especially happy to find a solution for aggregating feeds, as the feed-wordpress plugin was one of the things keeping me in wordpress. But, the issues for my research archive are a bit different. What do I need to accomplish?

1. Full-text search.
2. Photo Gallery for Manuscripts.
3. More granular comment capability for blog/chapter posts.

Search is the complicated one here, but there is probably a solution with Tapir (if not google site search). For granular comments, there is Highlighter. Both of these solutions lack permanency– they’re dependent on third party services. But, for now I’m OK with it. That’s because I’m also working on a django app behind the scenes that will replace this whole shebang on heroku some day. But, I digress.

Aside from outsourcing comments and search, I found a great solution for the photo gallery of Manuscripts that mirrors the development workflow of the main site. PhotoFloat is a very nifty set of python scripts that takes a folder (album) of photos, produces thumbnails of various sizes, and constructs a photo archive and viewer with javascript. It lives locally on my machine, and I sync the results with my server at for the manuscript archive.

The photos I have from my archive collection trips are quite large at more than 3MB each. That’s way overkill for the web. So, I wrote a small python script that takes a folder of .jpg files, reduces their quality to about 800KB, saves them in the appropriate place of for processing by PhotoFloat, and then runs its scanner to make the thumbnails. Then, it’s as simple as syncing with the web server.

What’s the point?

There are so many great CMSs out there to manage a site like this, why not use wordpress, drupal, or omeka (the top three for DH projects)? Static sites are fast and use very little RAM. In getting rid of the php from my VPS at dreamhost, I’m averaging 80-90MB of RAM for like 4 or 5 domains. It’s cheap to run such a server, and while now I’m using Apache2, it’d be simple to switch to nginx for even more performance improvement. It’s more secure. I write everything in a text editor, and manage it with source control. I deal with all my pics through the command line. Static site generators like jekyll and hyde score you some geek points. But, more importantly, I feel like I have a better grasp of the mechanics behind content management.

And, it’s fun.

I’d really like to hear, for those of you with experience in the college classroom, what you’d wished you’d learned before taking on your own courses. On any topic–> managing students, lecture pitfalls, course design, effective uses of technology, managing expectations, grading, assignments, etc. In the comments below, please share something that you wish you’d learned about teaching before you took to the classroom.

From the jacket:

Set against the backdrop of the tumultuous late colonial and early republican periods in Quito (1765–1830), this study examines women’s legal, economic, and social status in order to gauge the relationship between the increasingly centralized power of the Bourbon kingship and the local operation of social authority. A gendered reading of judicial documents, legal literatures, and institution discourses reveals that Bourbon attempts to restrict women’s access to legal resources were resisted by a traditional local legal culture based on practices of consultation, negotiation, judicial discretion, and contingency. This customary judicial practice, Black argues, played a fundamental role in limiting gender domination and prevented the full realization of a legal, economic, or social patriarchy in colonial Quito.

By documenting the progressive removal of limits to patriarchal power in the waning years of the Spanish Empire in Quito, this study traces the genealogy of legal patriarchy in Spanish America.

• Why Version Control
  • Choosing a VCS for research materials
  • Setting Up A Bitbucket Repo
    • Installing Mercurial
    • Setting Up a BitBucket Repo
    • You Need A Local Copy
  • The Workflow
  • Mercurial Resources
  • Other version-control-like options

Why Version Control

In the long, long ago days of March 2010, Julie Meloni posted a piece on the ProfHacker blog on version control systems (VCS) and what they might offer to the everyday academic trying to hack her workflow. If you don’t know what version control is, take a look at that post. VCSs originated with the specific needs of software programmers to save, access, restore, share, collaborate asynchronously on code. Version control has become a cornerstone of software development, and even more so of the social aspects that mark open source/free software communities. Sites such as Github have flourished because of the ease with which individuals can share, fork, and otherwise review and distribute code for open source projects super small and super large. (That’s right– the linux kernel.) There are a number of open source VCSs that are available free of charge, including Subversion (svn), Git, and Mercurial (hg).

That’s all well and good for code, but what does it have to do with an historian’s research repository? Well, the basics of version control boil down to this this –> documents placed under revision control are tracked by the VCS. Changes to those documents are committed, together with a message summarizing the changes, to a repository. Repositories can be replicated, or cloned, on any machine that has the VCS installed. In essence, every change to a document that you save and commit to the repository is eternally there, and can be returned to. Differences between versions of documents can be visualized. There are other useful aspects for academics as well– including branching, which allows you to create a parallel version of any document and develop it independently. Just as an example, say you have a single CV that you want to adapt for specific venues (grant applications, job applications to different types of universities/colleges, etc.)– with varying degrees of ease depending on the VCS, you could branch that CV into different forms tagged by their intended use. With a few simple commands, these changes can then be synced across your computers.

Suffice to say, a VCS offers an academic the ability to maintained versioned backups of his or her research and other personal files. And, for my own DIY predilections, I must admit that I like repurposing tools intended for wholly different audiences and putting them to work for historical research.

Choosing a VCS for research materials

Following Meloni’s advice back in March of 2010, I began to play with svn for maintaining a repo of my research and writing. (This post, for example, will be committed to my svn repo, which I haven’t yet migrated away from completely.) My webhost offers svn repos as part of its standard hosting package, and so I went that direction first. I was attracted by the idea in part because as a commuter, I have a variety of personal and work machines that I use on a regular basis — and I wanted all of my research files to sync on all those different computers. I also am attracted to the idea of redundancy in backup methods. I currently have offsite backups of my main laptop on backblaze. I have some of my stuff in Dropbox. I also clone the hard drive of my laptop once a week to an external drive. Paranoid, I know. But I’ve come by that paranoia honestly.

When I started in earnest to collect, transcribe, and analyze sources for my next project, I decided I wanted a project-specific, version-controlled repository for my sources and python scripts related to the project. I could have done this with my svn repo on my server. With svn, a single central repo tracks folders and files. You checkout either all or part of the repo, work locally, and then commit everything back to that central repo. Works great, though svn makes some things like branching overly difficult.

After putting a small bit of code up on Github, I discovered that I liked working with git, which operates on a different model than svn. Git clones repositories to wherever you want them in their entirety. Changes to documents are made to a local working copy of the document, committed locally, and then pushed/pulled to/from other repositories. Branching is much easier, as is merging changes back in. But, these are my research files, many of which aren’t ready for prime time yet. Github offers private repositories, but only for pay. I’m a cheapskate historian, so I went looking for a service that offered even a limited number of private repos for free. Enter Bitbucket, which offers free public and private repos for accounts with up to five users. In fact, BitBucket also offers an unlimited license to academics with a verifiable .edu address. I like companies that offer educator discounts. Given the availability of free private repos, I decided to go with Bitbucket for my research archive.

Setting Up A Bitbucket Repo

Bitbucket now supports git, but when I started this project, it didn’t. So, the rest of this post will be a tutorial on using Mercurial with Bitbucket. As long as you’re not scared of using the command line a little (and would you be reading this if you were?), basic usage is very straight forward.

Installing Mercurial

The most important first step is to install Mercurial on your machine. There are a couple of options for doing this. The easiest is to download the appropriate package installer for your operating system (OS X or Windows) just as you would any other piece of software. On Linux, install with your distribution’s package installer.

Alternatively on a Mac, if you use homebrew you can install with:
$ brew install mercurial

Finally, if you’re familiar with python and use either easy_install or pip, you can install that way. Mercurial is written in python, and is available as a python package. So, for a systemwide install, you simply need:
$ sudo easy_install mercurial
or
$ sudo pip install mercurial

Mercurial uses hg (the chemical symbol for mercury) as its alias on the command line. So, to verify that the installation worked, open a terminal prompt at enter:
$ hg --version

As of this writing, you should see something like:
Mercurial Distributed SCM (version 2.0)
(see http://mercurial.selenic.com for more information)
Copyright (C) 2005-2011 Matt Mackall and others

This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Setting Up a BitBucket Repo

Once hg is installed, the next step is to set up the private repository for your research materials on Bitbucket. It’s a painless process. Fill out the information on the sign up page. Please note that your username will define the urls for your repositories, so choose it carefully. If you may want an educator’s unlimited account in the future, provide your university email address.

Once your account is set up, log in to your account. From the Repositories tab, click on create repository. One the next form, create a new repository from scratch by providing a name (like, say, “Research”). Make sure that Private is selected, along with Mercurial as the repository type. You can also choose to use either an issue tracker or wiki for Project Management. This is actually a cool feature. For humanities projects, I like to use a wiki, which I then use for project planning, large project ideas, and the like. Click on Create repository, and voila, you have an empty repository. Navigate on over to your new repository page, which will be in this pattern: https://bitbucket.org/<username>/<repo>. Nothing to see there yet, but now we’ll fix that.

You Need A Local Copy

OK, so now we need to set up a repo on your local machine to work with your bitbucket repo. We need to make a folder for the repo, we need to clone it, and then we can start to populate it with research materials that we’ll push back to bitbucket. This is the same process you’ll follow on any other computers you’d like to have your materials on as well. So, here we go.

First up, make a folder to put your repo in. I like to use a projects folder in my home directory. YMMV. Time to get a little used to the terminal, so let’s do it there. From your home directory, which is the directory Terminal will probably open in (if not you can always get there with a simple $ cd), enter this command at the prompt:

1. $ mkdir projects

The road to version control forks here. When cloning your remote repo, you have two options. The first, using https requires you to enter your bitbucket password each time. The second uses ssh and requires that you enter an ssh key once in your bitbucket account settings. Let’s look at the latter. On a Mac, this is pretty easy. We need to change into a hidden folder. You should already be in your home directory, and if you are, enter this:

1. $ cd .ssh
2. $ ssh-keygen

This will create an ssh public/private key pair. Open the public key with vim or nano or your favorite text editor. It will most likely be named id_rsa.pub. You can check with $ ls. Highlight the entire key and copy it to your clipboard. Now, go back to your bitbucket account page and paste the key in the Add Key box. Now, back to your terminal. Change into your projects directory and clone the repo:

1. $ hg clone ssh://hg@bitbucket.org/<username>/<repo>

Of course, substitute your username and repo name there. This will create a new folder with the name of your repo, and where your files will reside. The https method is similar, but requires no keygen, so you would have gone straight to cloning the repo for step two:

1. hg clone https://<username>@bitbucket.org/<username>/<repo>

That said, I prefer the ssh method.

Either way, you’re ready to go!

The Workflow

Now that your repos are set up and linked, we can start putting files in them. Mercurial tracks files, and not folders. So empty folders won’t show up. Also, binary files (like, for example, .docx files) are more difficult than plain text files for VCSs to track. Mercurial will detect changes to a .doc file, but will store a whole copy of the new version rather than just the changes to the file like it does with txt. This can cause, over time, the repo to get large. But, as I’ve argued here before, plain text files are more durable, platform independent, and compact. I’ve switched to taking all my notes in .txt files, using markdown/multimarkdown syntax. I’ve also started switching excel spreadsheets to .csv files wherever I can. OK, back to the workflow.

On the most basic level, working with Mercurial will involve the following steps: 1. adding new files to version control; 2. committing changes in those files to the respository; 3. pushing changes from the local repo to bitbucket.

On a day-to-day basis, the workflow looks like this. If I’m transcribing a new document, I save that file to one of the folders in my local repo. The folder structure is normal– however you like to set up your research files. At some point, I add the new document to version control with the command: $ hg add. This will add any file in the repo to version control that isn’t already. As I accumulate substantial work on a version controlled file, I commit it to the local repo with $ hg commit -m "A message about the work." Note that hg requires a commit message with any commit. I find the log of messages a nice way to track my progress. At least once a day (depending on how much work I’ve gotten done), I push the local repo changes to bitbucket with $ hg push. I do this almost entirely using D-Term, which rules and provides a dropdown command line already in the directory you’re working from. And, that’s it for the most basic workflow. I now have an iterative back-up of all my research files both on my local machine at in the private repo at bitbucket. There are more advanced commands that are useful as well, but I’ll save those for another time.

A couple of final caveats — putting a file in the trash doesn’t remove it from your version control history. The proper way to erase a file is with the command $ hg remove <filename>. Likewise, the proper way to rename a file is with $ hg rename <filename>. Otherwise, a file will show up as missing, and will need to be removed at the next commit. You can also get into trouble with conflicting versions in different repos, which require merging.

If you’re not comfortable with the command line, there are a couple of nice, and free, GUIs for managing an hg repo. For the Mac, there’s the aptly named MacHg. For Windows, there’s TortoiseHg.

Mercurial Resources

There is, obviously, much more to Mercurial than I’ve listed above. In the future, I’ll cover branching and merging scenarios for typical humanities work. Luckily, in the mean time there is some excellent documentation available. Brian O’Sullivan has a free online version of his **Mercurial: The Definitive Guide. There’s a nice tutorial by Joel Spolsky. And Bitbucket’s support pages are also useful.

Other version-control-like options

If you’re not ready to make the plunge into a full-fledged VCS, there are other options likely already at your finger tips. That’s certainly the case if you use a Mac. With the release of OS X Lion, Apple has integrated version control more explicitly into its operating system. If you look at the save dialogue from the File tab on any program, you’ll see that Save now says Save Version. This form of save is actually tracking changes, and earlier versions can be browsed if the application supports the feature. Together with Apple’s TimeMachine, that takes you a pretty long way towards version control of your documents. Additionally, for Mac users, programs like ForeverSave offer version control for pretty much any program and their binary files. If you use DropBox, you can pay a little bit extra to turn on version control that can be accessed from their web interface for anything you keep in your DropBox.

Dropbox isn’t platform specific, but I must admit that I don’t know what the ForeverSave or OS-Specific options are on Windows or Linux. Feel free to share such in the comments.

Historians who use computers to manage their research, which at this point has to be approaching 98% at least (I won’t go higher than that, I know there are still legal pad-istas out there), are used to file-based workflows. It makes sense, of course, because for the archival historian a computer file represents an easy metaphor for the physical source world in which we work. In fact, I think that the folder/file/document metaphor may be more cogent for historians than for other everyday computer users. At the archive, we are likely to take folders from boxes, and in those folders we find texts that we call “my documents” and take notes on or transcribe into files on the computer that are likewise stored in folders. In fact, as opposed to cultural or literary scholars, it’s rare to hear archival historians speak of “texts” as opposed to “documents”. So, it’s a convenient replication that orders knowledge on the computer in a manner familiar to its organization in the archive. If I had to guess, I’d say historians are likely to be pretty tied to the folder/file/document metaphor for computer interaction, and likely resistant to calls for its demise, or for criticisms of the robustness of the metaphor’s graphical-user-interface.

Document-, or file-based workflows are mediated by a Finder-type window or a file/folder dialogue box. Programming such gui-elements can be a real pain in the ass. But, in the interest of the re-usability of code, they can also be pretty handy. In learning techniques to manipulate text with python, it’s nice to be able to pick files or folders of files to mess with, without having to put long paths and filenames through raw_input or in the code itself. But again, guis can be a real pain in the ass to program. Not if you use easygui. As the name of the module suggests, easygui provides several very-easy-to-use dialogue boxes that look native to the operating system and that provide a comforting means of choosing directories and files or entering bits of text into a program. They also help abstract one’s script from specific example texts or current obsession. For the rest of this post, I’ll provide a quick and easy tutorial on using easygui and some usage scenarios.

Installation

easygui is available through Python’s package managers, pip and easy_install. The current stable release is v.0.95, and you can also download the source files on the project’s page.

Install with the package manager of your choice in the normal way from a terminal prompt:

$ sudo easy_install easygui
$ sudo pip install easygui

To install from source, download and unzip the package from the site linked above. Change into the directory of the unzipped folder in the terminal and enter:

$ sudo python setup.py install

In both of those cases, I’m assuming you’re on a *nix computer, be it Linux or OSX. sudo is necessary if you’re using the system’s default python, and installs easygui to your python’s site-packages folder. Things are a little different on Windows or if you’re using homebrew or macports python on your Mac. For beginners, it’s probably easiest to just use the default installation of python on your Mac and not have to worry about editing you PATH or PYTHONPATH.

To test if easygui is installed, open a python interpreter in your terminal and try to import it–> >>> import easygui.

File and Directory Dialogues

Once installed, inserting file and directory dialogue boxes into your code is very simple.

The directory box simply allows you to chose a directory path, which can be used to open or write files. From the directory box, the only thing you can do is chose a directory. That said, in the case of OS X, for example, you can also use system built-in elements like the New Folder button, which is useful if you want to write files to, well, a new directory. In basic usage we have–

easygui.diropenbox(msg=None, title=None, default=None)

This line will display an open directory dialogue box. The optional arguments allow you to customize the dialogue box a bit. msg and title both put text at the top of the box, and do so in order with a dash in between. Seems redundant to me, so I usually just use one. default sets the directory that the dialogue box will open in. So, for example, on my laptop to open in my Documents folder as this would be default='/Users/chadblack/Documents'.

To get this Directory Open Box, the code would be easygui.diropenbox(msg='Choose a directory.', default='/Users/chadblack/Documents/sex_crime_empire/'). Notice that files are grayed-out. You can’t choose a file with this subclass, only a directory. This returns to the script a directory name, listed fully, such that if I had chosen the default above, it would return /Users/chadblack/Documents/sex_crime_empire.

How is this useful? What sort of usage scenario? If, for example, one wanted to produce word counts or keywords-in-context or, for that matter, NCD clusters from a set of documents, one could use this dialog to choose the directory of files to iterate over. It’s trivial to produce a list of files from the returned directory name with the os module. In the interpreter it would look like this:

>>> import easygui
>>> import os
>>> directory = easygui.diropenbox()
>>> fileList = os.listdir(directory)

That would give you a list of file names in that directory, which can then be iterated over. (Note, to open those individual files, you either need to concatenate them to the directory path, or change the working directory to where those files are located. For the latter, it’s as simple as os.chdir(directory).)

Basic usage for the file open dialog looks like this:

easygui.fileopenbox(msg=None, title=None, default='*', filetypes=None)

In this case, msg and title work the same way, but default points to a file name. More importantly, though, filetypes determines what type of files you can open, which needs to be defined as a list. So, to be able to open .txt files, you need filetypes=['*.txt']. Any file that is of a type not listed will be grayed-out, and you won’t be able to open it.

For this File Open box, the code would be easygui.fileopenbox(msg='Choose a file.', filetypes=['*.txt']). This returns a file name to the script, which in the case above would be /Users/chadblack/Documents/00-Writings/Posts/trunk/102011/thinkingWorld.txt. You still need to open that file, which can be done in one assignment step like this:

text = open(easygui.fileopenbox(filetypes=['*.txt']), 'r').read()

With that line, you chose, open, and read a file, assigning it to text as a string all in one fell swoop.

The save file dialog is just as easy. Basic usage is:

easygui.filesavebox(msg=None, title=None, default='*', filetypes=None)

The arguments for this function work the same as with opening files, and it also returns a filename.

For the example here, we have easygui.filesavebox(msg='Save file.', default='MyFile.txt', filetypes=['*.txt']), and this would return to the script the full path of MyFile.txt, which can be used in a write statement. So, let’s say we did something with text above and want to save it to MyFile.txt:

open(easygui.filesavebox(msg='Save file.', default='MyFile.txt', filetypes=['*.txt']), 'w').write(text)

There are many other types of dialog and text-display boxes that easygui provides. In the past I’ve used, for example, a text entry box to set the base name for batch renaming photos.

To conclude, easygui is, well, very easy and adds both a familiar interface and reproducibility to file and directory interactions for historians so used to working that way. I’m looking forward to v.2.0 of The Programming Historian, which I hear will be open source, to add easygui into the excellent example scripts.

• Introduction
• What is NCD?
• Limitations: Size Matters
  • bzip2
  • gzip
  • lzma
• The Code
• Comparing results
  • The Series Guide
  • The 18th Century
• Conclusion

Introduction

As I wrote here a couple of weeks ago, I’m playing around with a variety of clustering techniques to identify patterns in legal records from the early modern Spanish Empire. In this post, I will discuss the first of my training experiments using Normalized Compression Distance (NCD). I’ll look at what NCD is, some potential problems with the method, and then the results from using NCD to analyze the Criminales Series descriptions of the Archivo Nacional del Ecuador’s (ANE) Series Guide. For what it’s worth, this is a very easy and approachable method for measuring similarity between documents and requires almost no programming chops. So, it’s perfect for me!

What is NCD?

I was inspired to look at NCD for clustering by a duo of posts by Bill Turkel (here, here) from quite a few years ago. Bill and Stéfan Sinclair also used NCD to cluster cases for the Digging Into Data Old Bailey Project. Turkel’s posts provide a nice overview of the method, which was proposed in 2005 by Rudi Cilibrasi and Paul Vitányi¹. Essentially, Cilibrasi and Vitányi proposed measuring the distance between to strings of arbitrary length by comparing the sum of the individually compressed files to a compressed concatenation of the two files. So, adding the compressed length of x to the compressed length of y will be longer than the compressed length of xy. How much longer is what is important. The formula is this:

NCD(x,y) = [C(xy) - min{C(x),C(y)}] / max{C(x),C(y)

where C(xy) is the compression of the concatenated strings. Theoretically, if you concatenated and compressed two identical strings, you would get a distance of 0 because (Cxx) - C(x) / C(x) would equal 0/1, or 0. As we’ll see in a bit, though, this isn’t the case and the noise caused by the various compression algorithms at our disposal make a 0 impossible, and more so for long strings depending on the method. Cilibrasi and Vitányi note that in practice, where result is r, NCD will be 0 ≤ r ≤ 1+ ∊, where ∊ is usually around 0.1, and accounts for the imperfections of the compression algorithm. Suffice to say, though, that the closer to 0 the result is, the more similar the strings (or files in our case) are. Nonetheless, the distance between two strings, or files, or objects as measured with this formula can then be used to cluster those strings, files, or objects. One obvious advantage to the method is that it works for comparing strings of arbitrary length with one another.

Why does this work? Essentially, lossless compression suppresses redundancy in a string, while maintaining the ability to fully restore the file. Compression algorithms evolved to deal with constraints in the storage and transmission of data. It’s easy to forget in the age of the inexpensive terabyte hard drive what persistent storage once cost. In 1994, the year that the first edition of Witten, Moffat, and Bell’s Managing Gigabytes was published, hard disk storage still ran at close to $1/megabyte. That’s right, just 17 years ago that 500GB drive in your laptop would have cost $500,000. To put that into perspective, in 1980 IBM produced one of the first disk drives to break the GB barrier. The 2.52GB IBM 3380 was initially released in 5 different models, and ranged in price between $81,000 and and $142,000. For what it’s worth, the median housing price in Washington, DC in 1980 was the second highest in the country at $62,000. A hard disk that cost twice as much as the median house in DC. Obviously not a consumer product. At the per/GB rate that the 3380 sold for, your 500GB drive would have cost up to $28,174,603.17! In inflation-adjusted dollars for 2011 that would be $80.5M! An absurd comparison, to be sure. Given those constraints, efficiency in data compression made real dollars sense. Even still, despite the plunging costs of storage and growing bandwidth capacity, text and image compression remains an imperative in computer science.

As Witten, et.al. define it,

“Text compression… involves changing the representation of a file so that it takes less space to store or less time to transmit, yet the original file can be reconstructed exactly from the compressed representation.”²

This is lossless compression (as opposed to lossy compression, which you may know from messing with jpegs or other image formats). There are a variety of compression methods, each of which takes a different approach to compressing text data and which are either individually or in some kind of combination behind the compression formats you’re used to– .zip, .bz2, .rar, .gz, .tar, etc. Frequently, they also have their roots in the early days of electronic data. Huffman coding was developed by an eponymous MIT graduate student in the early 1950s.

In any case, the objective of a compression method is to locate, remove, store, and recover redundancies within a text. NCD works because within a particular algorithm, the compression method is consistently imposed on the data, thus making in comparable. What isn’t comparable, though, is mixing methods.

Limitations: Size Matters

Without getting too technical (mostly because I get lost once it goes too far), it’s worth noting some limitations based on which method of compression you chose when applying NCD. Shortly after Cilibrasi and Vitányi published their paper on clustering via compression, Cebrián (2005) published a piece that compared the integrity NCD between three compressors– bzip2, gzip, and PPMZ.³ The paper is interesting, in part, because the authors do an excellent job of explaining the mechanics of the various compressors in language that even I could understand.

I came across this paper through some google foo because I was confused by the initial results I was getting playing around with my Criminales Series Guide. Python has built-in support for compression and decompression using bzip2 and gzip, so that’s what I was using. I have the Criminales Series divided into decades from 1601 to 1830. So, my script was walking through and comparing every file in the directory to every other one, including itself. I assumed that the concatenation of two files that were identical would produce a distance measurement of 0, and was surprised to see that it wasn’t happening, and in some cases not even close. (I also hadn’t read much of anything about compression at that point!) But that wasn’t the most surprising thing. What was more surprising was that in the latter decades of my corpus, the distance measures when comparing individual decades to themselves were actually coming out very high. Or, at least they were using the gzip methodology. For example, the decade with the largest number of cases, and thus the longest text, is 1781-1790 at about 39,000 words. Gzip returned an NCD of 0.97458 when comparing this decade to itself. What? How is that possible?

Cebrián (2005) explain how. Different compression methods have upper limits to the size of a block of text that they operate on before needing to break that block into a new blocks. This makes little difference from the perspective of compressors doing their job, but it does have implications for clustering. The article goes into more detail, but here’s a quick and dirty overview.

bzip2

The bzip2 compressor works in three stages to compress a string: 1. a Burrows-Wheeler Transform; 2. a move-to-front transform; and, 3. a statistical compressor like Huffman coding.⁴ The bzip2 algorithm can perform this method on blocks of text up 900kb without needing to break the block of text into two blocks. So, for NCD purposes, this means that if a pair of files are concatenated, and the size of this pair is less than 900kb, what the bzip compressor will see is essentially a mirrored text. But, if the concatenated file is larger than 900kb, then bzip will break the concatenation into more than one block, each of which will be sent through the three stages of compression. But, these blocks will no longer be mirrors. As a result, the NCD will cease to be robust. Cebrián (2005) claim that the NCD for C(x,x) should fall in a range between 0.2 and 0.3, and anything beyond that indicates it’s not a good choice for comparing the set of documents under evaluation.

gzip

The gzip compressor uses a different method than bzip2′s block compression, one based on the Lempel-Ziv LZ77 algorithm, also known as sliding window compression. Gzip then takes the LZ77-processed string and subjects it to a statistical encoding like Huffman. It’s the first step that is important for us, though. Sliding window compression searches for redundancies by taking 32kb blocks of data, and looking ahead at the next 32kb of data. The method is much faster than bzip2′s block method. (In my experiments using python’s zlib module, code execution took about 1/2 the time as python’s bzip on default settings.) And, if the text is small, such that C(x,x) < 32kb, the NCD result is better. Cebrián (2005) find that gzip returns an NCD result in the range between 0 and 0.1. But, beyond 32kb they find that NCD rapidly grows beyond 0.9 — exactly what I saw with the large 1781-1790 file (which is 231kb).

lzma

Cebrián (2005) offer a third compressor, ppmz, as an alternative to bzip2 and gzip for file that outsize gzip and bzip2′s upper limits. Ppmz uses Prediction by Partial Match for compression, and has no upper limit on effective file size. PPM is a statistical model that uses arithmetic coding. This get’s us to things I don’t really understand, and certainly can’t explain here. Suffice to say that the authors found using ppmz that C(x,x) always returned an NCD value between 0 and 0.1043. I looked around for quite a while and couldn’t find a python implementation of ppmz, but I did find another method ported to python with lzma, the compressor behind 7zip. Lzma uses a different implementation of Lempel-Ziv, utilizing a dictionary instead of a sliding window to track redundancies. What is more, the compression-dictionary can be as large as 4GB. You’d need a really, really large document to brush up against that. Though Cebrián, et.al. didn’t test lzma, my experiments show the NCD of C(x,x) to be between 0.002 and 0.02! That’s awfully close to 0, and the smallest return actually came from the longest document –> 1781-1790.

The Code

In a way, that previous section is getting ahead of myself. I started with just zlib, and then added bzip2 and gzip, and eventually lzma for comparison sake. Let me clarify that just a bit. In python, there are two modules that use the gzip compressor: 1. gzip, which is for file compression/decompression; and, 2. zlib, which is for compressing/decompressing strings or objects. So, I was unsettled by my early zlib returns, and tried turning using gzip and file I/O, but got the same returns. Anyway, initially I was interested in speed, but reading Cebrián changed my mind on that. Nonetheless, I did time the functions to see which was fastest.

I based the script on Bill Turkel’s back from 2007. (Bill put all of the scripts from the days of Digital History Hacks on Github. Thanks to him for doing that!)

So, for each compressor we need a function to perform the NCD algorithm on a pair of files:

# Function to calculate the NCD of two files using lzma
def ncd_lzma(filex, filey):
    xbytes = open(filex, 'r').read()
    ybytes = open(filey, 'r').read()
    xybytes = xbytes + ybytes
    cx = lzma.compress(xbytes)
    cy = lzma.compress(ybytes)
    cxy = lzma.compress(xybytes)
    if len(cy) > len(cx):
        n = (len(cxy) - len(cx)) / float(len(cy))
    else:
        n = (len(cxy) - len (cy)) / float(len(cx))
    return n

There are small changes depending on the API of the compressor module, but this pretty much sums it up.

We need to be able to list all the files in our target directory, but ignore any dot-files like .DS_Store that creeps in on OS X or source control files if you’re managing your docs with git or svn or something:

# list directory ignoring dot files
def mylistdir(directory):
    filelist = os.listdir(directory)
    return [x for x in filelist
            if not (x.startswith('.'))]

Just as an aside here, let me encourage you to put your files under source control, especially as you can accidentally damage them while developing your scripts.

We need a function to walk that list of files, and perform NCD on every possible pairing, the results of which are written to a file. For this function, we pass as arguments the file list, the results file, and the compressor function of choice:

def walkFileList(filelist, outfile, compType):
    for i in range(0, len(filelist)-1):
        print i
        for j in filelist:
            fx = pathstring+str(filelist[i])
            fy = pathstring+str(j)
            outx = str(filelist[i])
            outy = str(j)
            outfile.write(str(outx[:-4]+"  "+outy[:-4]+"  ")+str(compType(fx, fy))+"\n")

That’s all you need. I mentioned also that I wanted to compare execution time for the different compressors. That’s easy to do with a module from the python standard library called profile, which can return a bunch of information gathered from the execution of your script at runtime. To call a function with profile you simply pass the function to profile.run as a string. So, to perform NCD via lzma as described above, you just need something like this:

outfile4 = open('_lzma-ncd.txt', 'w')
print "Starting lzma NCD."
profile.run('walkFileList(filelist, outfile4, ncd_lzma)')
print 'lzma finished.'
outfile4.close()

I put the print statements in just for shits and giggles. Because we ran this through profile, after doing the NCD analysis and writing it to a file named _lzma-ncd.txt, python reports on the total number of calls, the time per call, per function, and cumulative for the script. It’s useful for identifying bottlenecks or some such in your code if you get to the point of optimizing. At any rate, there is no question that lzma is much slower that the others, but if you have the cpu cycles available, it may be worth the rate from a quality of data perspective. Here’s what profile tells us for the various methods:

• zlib–> 7222 function calls in 16.564 CPU seconds (compressing string objects)
• gzip–> 69460 function calls in 18.377 CPU seconds (compressing file objects)
• bzip–> 7222 function calls in 21.129 CPU seconds
• lzma–> 7222 function calls in 115.678 CPU seconds

If you expected zlib/gzip to be substantially faster than bzip, it was until I set all of the algorithms to the highest available level of compression. I’m not sure that’s necessary or not, but it does affect the results as well as time. Note too that the gzip file method requires many more function calls, but with relatively little performance penalty.

Comparing results

The Series Guide

A little bit more about the documents I’m trying to cluster. Beginning around 2002, the Archivo Nacional del Ecuador began to produce pdfs of their ever-growing list of Series Finders guides. The Criminales Series Guide (big pdf) was a large endeavor. The staff went through every folder in every box in the series, reorganized them, and wrote descriptions for the Series Guide. Entries in the guide are divided by box and folder (caja/expediente). A typical folder description looks like this:

Expediente: 6
Lugar: Quito
Fecha: 30 de junio de 1636
No. de folios : 5
Contenido: Querella criminal iniciada por doña Joana Requejo, mujer legítima del escribano mayor Andrés de Sevilla contra Pedro Serrano, por haber entrado a su casa y por las amenazas que profirió contra ella con el pretexto de que escondía a una persona que él buscaba.

We have the place (Quito), the date (06/30/1636), the number of pages (5), and a description. The simple description includes the name of the plaintiff, in this case Joana Requejo, and the defendant, Pedro Serrano, along with the central accusation– that Serrano had entered her house and threatened her under the pretext that she was hiding a person he was looking for. There is a wealth of information that can be extracted from that text. The Series Guides as a whole is big, constituting close to 875 pages of text and some 1.1M words. I currently have text files for the following Series Guides–> Criminales, Diezmos, Encomiendas, Esclavos, Estancos, Gobierno, Haciendas, Indígenas, Matrimoniales, Minas, Obrajes, and Oficios totallying 4.8M words. I’ll do some comparisons between the guides in the near future, and see if we can identify patterns across Series. For now, though, it’s just the Criminals striking my fancy.

The 18th Century

So, what does the script give us for the 18th century? Below are the NCD results for three different compressors comparing my decade of interest, 1781-1790, with the otehr decades of the 18th century:

zlib:

cr1781_1790  cr1701_1710  0.982798401771
cr1781_1790  cr1711_1720  0.987881971149
cr1781_1790  cr1721_1730  0.977414695455
cr1781_1790  cr1731_1740  0.97668311167
cr1781_1790  cr1741_1750  0.975895252209
cr1781_1790  cr1751_1760  0.975088634189
cr1781_1790  cr1761_1770  0.975632632389
cr1781_1790  cr1771_1780  0.973381605357
cr1781_1790  cr1781_1790  0.974582153107
cr1781_1790  cr1791_1800  0.972256091842
cr1781_1790  cr1801_1810  0.973325329682

bzip:

cr1781_1790  cr1701_1710  0.954733848029
cr1781_1790  cr1711_1720  0.96900988758
cr1781_1790  cr1721_1730  0.929649194095
cr1781_1790  cr1731_1740  0.923066504131
cr1781_1790  cr1741_1750  0.906271163484
cr1781_1790  cr1751_1760  0.903237166463
cr1781_1790  cr1761_1770  0.902912095354
cr1781_1790  cr1771_1780  0.849356630096
cr1781_1790  cr1781_1790  0.287823378031
cr1781_1790  cr1791_1800  0.850331843424
cr1781_1790  cr1801_1810  0.850358932683

lzma:

cr1781_1790  cr1701_1710  0.965529663402
cr1781_1790  cr1711_1720  0.976516942474
cr1781_1790  cr1721_1730  0.947607790161
cr1781_1790  cr1731_1740  0.94510863447
cr1781_1790  cr1741_1750  0.931757289204
cr1781_1790  cr1751_1760  0.931757289204
cr1781_1790  cr1761_1770  0.92759202972
cr1781_1790  cr1771_1780  0.885106382979
cr1781_1790  cr1781_1790  0.0021839468648
cr1781_1790  cr1791_1800  0.880670944501
cr1781_1790  cr1801_1810  0.887110210514

First off, even just eyeballing it, you can see that the results from bzip and lzma are more reliable and follow exactly the patterns discussed by Cebrián et.al. The bzip run provides a C(x,x) of 0.288, which falls in the acceptable range. The lzma run returns a C(x,x) NCD of 0.0022, not much more needed to say there. And, as I noted above, with gzip we get 0.9745. Burther, by eyeballing the results on the good runs, two relative clusters appear in the decades surrounding 1781-1790. It appears that from 1771 to 1810 that we have more similarity than in the earlier decades of the century. This accords with my expectations based on other research, and in both cases the further back from 1781 that you go, the more different the decades are on a trendline.

If we change the comparison node to, say, 1741-1750 we get the following results:

bzip:

cr1741_1750  cr1701_1710  0.888048411498
cr1741_1750  cr1711_1720  0.919398218188
cr1741_1750  cr1721_1730  0.826189275508
cr1741_1750  cr1731_1740  0.80795091612
cr1741_1750  cr1741_1750  0.277693730039
cr1741_1750  cr1751_1760  0.785168132862
cr1741_1750  cr1761_1770  0.803655071796
cr1741_1750  cr1771_1780  0.879983993015
cr1741_1750  cr1781_1790  0.906271163484
cr1741_1750  cr1791_1800  0.883904391852
cr1741_1750  cr1801_1810  0.886378259718

lzma:

cr1741_1750  cr1701_1710  0.905551014342
cr1741_1750  cr1711_1720  0.932600133759
cr1741_1750  cr1721_1730  0.862079215278
cr1741_1750  cr1731_1740  0.848926209408
cr1741_1750  cr1741_1750  0.00587055064279
cr1741_1750  cr1751_1760  0.830746598014
cr1741_1750  cr1761_1770  0.844162055066
cr1741_1750  cr1771_1780  0.90796460177
cr1741_1750  cr1781_1790  0.929573342339
cr1741_1750  cr1791_1800  0.908149721264
cr1741_1750  cr1801_1810  0.913968518045

Again, the C(x,x) show reliable data. But, this time bzip’s similarities look a fair amount different that lzma when eyeballing it. I’m interested in the decade of the 1740s in part because I expect more similarity to the latter decades than for other decades in, really, either the 18th or the 17th century. I expect this for reasons that have to do with other types of hermeneutical screwing around, to use Stephen Ramsey’s [pdf] excellent phrase, that I’ve been doing the records lately. Chief among those (and an argument for close as well as distant readings) is that I’ve been transcribing weekly jail censuses from the 1740s the past week and some patterns of familiarity have been jumping out at me. I have weekly jail counts from 1732 to 1791 inclusive, and a bunch others too. I’ve transcribed so many of these things that I have pattern expectations. And, the 1740s has jumped out at me for three reasons this week. The first is that in 1741, after a decade of rarely noting it, the notaries started to record the reason for one’s detention. The second is that in 1742, and particularly under the aegis of one particular magistrate, lots more people started to get arrested that previous and subsequent decades. The third is that, like in the period between 1760 and 1790, those arrests were increasingly for moral offenses or for being picked up by during nightly rounds of the city (the ronda). The differences are this– in the latter period women and men were arrested in almost equal numbers. There are almost no women detainees in the 1740s. And, there doesn’t seem to be an equal growth in both detentions and prosecutions in the 1740s. This makes the decade more like the 1760s than the 1780s. The results above bear that out to some extent, as distance measures show to be more like the 1760s than the 1780s.

I also had this suspicions because a few months ago I plotted occurrences of the terms concubinato (illicit co-habitation) and muerte (used in murder descriptions) from the Guide:

If you click on that chart and look at the larger version you’ll see that right on decade of the 1740s there is a discernible, if much smaller than later, bump for concubinato. I was reminded of this when transcribing the records.

Conclusion

OK, at this point, this post is probably long enough. What’s missing above is obviously visualizations of the clusters. Those visualizations are pretty interesting. I’ll write another post this week just on the visualizations because they deserve to be higher up on the page for sure. We also have some issues to deal with on parameters for plotting the connections. For now, though, let me conclude by saying that I am impressed initially to see the clusters that emerged from this simple, if profound, technique for clustering. Given that the distinctions I’m trying to pick up are slight, I’m worried a bit about the level of precision I can expect. But, I am convinced that it’s worth sacrificing performance for either bzip or lzma implementations depending on the length of one’s documents. Unless your txt files are longer than 900kb, it’s probably worth just sticking with bzip.

There is a bit of irony in my teaching this particular class, as I haven’t taught world history in a really long time. And it wasn’t to college students when I did. (I was a history teacher at Enloe High School in Raleigh, NC in the mid-1990s.) But, I agreed to do the seminar for two reasons—

1. I’ve been rather critical of the notion that the World Survey is a useful course because it is so damn ungainly. The narrative choices that one must make in order to teach the history of The World, either before or after 1500, border on the absurd. So, it’s a challenge to conceive of this course in a way that will prepare our students to do something useful with, or be able to say something useful about it in a job interview.
2. As I said, it’s the one chance to teach about teaching. This is a challenge too. I know that for much of my career I have taught reflexively, out of the synapse habits formed from sitting in classrooms on one side or another of the lectern essentially since 1977. I’ve been working on reversing some of those synapses of late. I need to think more intentionally about the whole process of course design and implementation. It’s fun to do that with graduate students.

Still, that leaves me with the initial task of designing this class. What I’ve decided is that I want the class to be as much lab as traditional seminar. I want the students to do all of the practical tasks of building and implementing a class. Maybe I can recruit some undergrads to come complain during their office hours, or something. It is, of course, impossible to read for any kind of coverage. But, in addition to the lab aspect of the class, we will be reading monographs as well.

What I’ve decided on is to use a somewhat playful notion of metaphor to organize our approach to narrating ‘world’ history. Not only are there endless narratives that one could construct, there are endless methods to organize those narratives. I hope it will be useful to conceive of these methods as metaphors. We’re going to read Lakoff and Johnson’s Metaphors We Live By, and then in successive weeks treat world history through a series of metaphorical statements. I should note that I’m only covering modern world history (ie, from 1500 forward), so the myths are directed towards that period.

Metaphors I’ve settled on, for now:

• World History is Transition
• World History is State Thinking
• World History is Myth
• World History is Commodity
• World History is Representation
• World History is Space
• World History is Information

Using these seven metaphors, we’ll look at different organizing principles for constructing a World History survey. I’m excited by some of the books we’ll read to spark conversation around these themes— Mike Davis, Late Victorian Holocausts, Matthew Edney, Mapping an Empire, James Glecik, The Information, James McCann, Maize and Grace, Ken Pomeranz, The Great Divergence, Matthew Restall, Seven Myths of the Spanish Conquest, James Scott, Seeing Like a State, Topik, Frank, and Marichal, From Silver to Cocaine, Wilson and Stewart, Global Indigenous Media, Thongchakul Winichakul, Siam Mapped, and a few others.

I haven’t, though, settled on the practicum side. What do you wish you knew before teaching your first class from start to finish? What do you wish you’d unlearned from assistantships, or known to consider beforehand?

I don’t mind repeatedly writing for grants, but I do mind soliciting letters all the time.

I am applying for this fellowship to complete research and begin to write a book on the relationship between sexual behavior and Spanish imperial authority during the age of Charles III (1759–1788). Sex, Crime, and Empire: The Political Economy of Intimacy in Bourbon Quito will use prosecutions of illicit sexual activity (adultery, cohabitation, sodomy, bestiality) as a means to explore the shifting theory of state power and empire in the 18th century. Between 1760 and 1790, the Spanish imperial state began to take intense interest in the sexual lives of its subjects. (See plots of criminal sex prosecutions in the additional materials.) Other historians have analyzed the relationship between the colonial state (and church) and sexuality as one in which institutions arbitrated a Mediterranean honor code that hinged on the control of women’s sexuality and the purity of religious heritage. I argue instead that through judicial mechanisms, the late imperial Spanish state intervened in popular sexual norms in an attempt to make social roles and sexual activity conform to the Bourbon monarchy’s dream of patriarchal state authority. I will show that the Bourbon state used these methods in the pursuit of an economically rational absolutism. Further, criminal prosecutions of illicit sex during this period reveal not only the extent to which royal expectations deviated from customary norms. They also mark the emergence of new models of punishment emphasizing moral correction through convict labor in royal-owned tobacco and textile factories. I argue that during the imperial reign of Charles III this active criminalization of sexual behavior turned individual bodies into contested ground upon which imperial authorities established a new underlying political philosophy that combined moral and political authority with economic rationality. As such, the case files utilized provide compelling evidence for the late 18th century as a fulcrum point between early-modern judicial empire and modern disciplinary colonialism.

The study of sexuality in an imperial context integrates my recent research on women’s legal capacity in late colonial Spanish America with my longstanding interest in the organization and performance of governing authority. My first book examined the interaction between custom and law that enabled women in the early modern Spanish world to make autonomous legal acts despite formal strictures on their legal capacity. Contrary to much of the established literature, I found that in property contracts, marital disputes, and criminal activities, magistrates predominantly respected women’s customary right to make legal acts. In the course of researching that book, I also discovered an explosion in moral policing, prosecution of men and women for sexual offenses, and corrective punishment through economic service to the Crown. The intersection of these three phenomena, and the impressive documentary record left behind, allow me to investigate the tension between custom and prescription in a new way, and to draw out the political implications of late 18th-century family values.

More recently, I have also been researching the complicated notion of jurisdiction in Spanish imperialism that simultaneously perpetuated petty conflicts within the bureaucracy and provided social stability through the flexible application of royal dictates to local circumstances. From the classic work of John Leddy Phelan to Alejandro Cañeque’s recent The King’s Living Image, historians of the Spanish empire have recognized how jurisdictional imprecision facilitated the administration of early modern empire. I argue that jurisdictional imprecision is itself a metaphor for other forms of authority and place during the colonial period. Integrating jurisdiction-as-metaphor into the study of colonial sexuality helps to explain the failure of the Bourbon state to successfully transition to an effective modern disciplinary colonialism.

The state’s interest in illicit sex coincided with a crisis in royal authority caused by fallout from Spain’s participation in the Seven Years War. Saddled with debt and with its defensive shortcomings exposed by the British navy, the court of Charles III pursued a series of administrative, fiscal, and military reforms intended to rationalize, centralize, and magnify the Crown’s authority. Traditional scholarship on the period has centered on these Bourbon reforms, along with the often-violent reactions they elicited from subject populations in the Andean region. More recent scholarship has begun to turn attention to social reforms pursued by Bourbon officials, including the construction of poor houses and orphanages, the secularizing of marriage regulations, and controls on alcohol production and consumption. The two strains of reform, those aimed at economic rationalization, tax collection, and the centralization of power and those aimed at social control shared an underlying political theory of absolutist sovereignty that clashed with a centuries-old political culture of negotiation, consent, and decentralism. This tension was born of the “modernization” of colonial administration, and was nowhere more intimately felt than in state actions to monitor and control sexuality.

The Audiencia of Quito, roughly equivalent to modern Ecuador, provides an excellent venue to study this aspect of Bourbon reform. Situated between the Viceregal capitals of LIma and Bogota, Quito became the object of tax reformers in the 1760s, attention that led to a significant citywide revolt from 1765–1766. The impulse of civil authorities to monitor and control the city’s barrio residents and particularly their sexual proclivities emerged first with the Crown’s restoration of royal authority in the wake of the Rebellion of the Barrios. But the attention lasted far beyond the aftermath of the rebellion. Royal officials constructed an infrastructure of neighborhood policing together with a new network of royal factories staffed with prison laborers. This new system was fed with explosive growth in prosecutions of moral crimes, particularly illicit cohabitation. Luckily for me, it also produced an extensive trial record of prosecution, testimony, confession, and punishment that provides the raw data for documenting the connections between sexuality, colonial authority, and economic reform.

Spanish criminal prosecutions developed along regular and predictable steps, from allegation to appeal. Court appointed notaries, utilizing a combination of boilerplate and organic language, carefully documented each of these steps. Defendants were presumed guilty. Prosecuting magistrates directed both the investigation and adjudication of criminal acts, framing the interrogation of witnesses in terms established by the complaint. The initial witnesses responded to interrogation within a discursive space defined and controlled by the magistrate and the plaintiff. Confessions, on the other hand, temporarily diminished the magistrate’s control over the discursive terrain as defendants responded to charges and questions by often redefining the terminology used. Defendants then submitted their own witnesses and interrogatories, further seeking to change the language of the case. If found guilty, defendants invariably and immediately switched back to the language of the magistrate in an appeal for mercy. This predictable flow of cases reached to all corners of the Spanish empire, allowing for comparative evaluation of local popular sexual practices in disparate settings.

Ethnohistorians and social historians have long used court cases as rich sources for mining ancillary detail and anecdotal data in an effort to give voice to the popular sectors. I use these cases with an eye to their procedural context and to the “extra” information contained within their files. The regularity of criminal prosecutions open the possibility of a wide variety of readings. While I do closely read the documents in the tradition of ethno- and social history, I also submit my case files to distant reading techniques offered by texting mining and mapping. Using digital photography, I have collected all of the extant criminal prosecutions of illicit sex, murder, assault, and insult in the National Archive of Ecuador for most of the 18th century, and am in the process of transcribing these documents. Additionally, I have transcribed the weekly jail censuses of the city of Quito for the years 1732–1791 inclusive. While the individual cases and names represented in these documents provide compelling stories ripe for interpretation in their own right, taken together as a corpus and subjected to machine-reading techniques new analytical avenues open up. I am able to extract and compare procedural segments of cases and to map their occurrences in the city’s barrios. With the weekly jail censuses, I am able to demonstrate which magistrates took the lead in prosecuting moral offenses, and against which types of barrio residents. Using text-mining techniques to cluster similar documents such as Normalized Compression Distance (NCD) Analysis and Latent Semantic Analysis (LSA) topic modeling, I can compare prosecutions over time to demonstrate innovations in criminal prosecution during the 1780s. This algorithmic approach to legal practice is completely innovative in the colonial Latin American historiography. By controlling for the procedural development of a prosecution as well as for gender, age, ethnicity, and other social markers present in the cases, I am able to document allusive popular, customary sexual norms. I then put this analysis of criminal conduct into the broader context of imperial governance and the economic, penal, and medical literatures produced by the Spanish Enlightenment. Finally, I analyze the system of punishment constructed to discipline those caught in the city’s sexual dragnet, and specifically the role of royal tobacco factories in convict reformation. My conclusions demonstrate that the barrios of Quito operated with a level of tolerance for sexual behavior by both men and women that would never be predicted nor accepted by the strict Mediterranean honor code that has dominated the Latin American literature on gender and sexuality (Twinam, 1999; Lipsett-Rivera and Johnson, 1998, Chambers, 1999, Lavrin, 1989).

The evidentiary base of this project is built on approximately 1000 cases of sexual and sexual-related crimes prosecuted in the Audiencia of Quito between 1760 and 1790. I have digital photographs of the case set in its entirety. In addition to the prosecutions, I have developed a database based on weekly jail censuses for more than 10,000 arrests in Quito during the 18th century. The database contains information on arrestee’s gender, crime, supervising magistrate, ethnicity, and occupation. These records are further complemented by institutional records of the Presidency of the Audiencia of Quito and the jails, royal factories, and poor houses of the city. Together, they provide comprehensive data to combine close and distant readings of practices of sex, neighborhood surveillance, and correction for the city and its hinterland.

By the time this fellowship begins, I will have completed much of the research. I have collected the documentary base for the project with trips to the National Archive of Ecuador in 2009 and 2011. Archival visits are complete. I have been transcribing cases with some graduate student assistance for the past two years. I have also written the necessary code for text mining over the past year. I will use time afforded by an ALCS fellowship to finish my analysis and to write the first draft of this manuscript. I hope to reach not only Latin Americanists, but also the rapidly growing audience of the Digital Humanities. All materials produced for this project, including maps, the criminal database, transcripts, and code will be hosted as a project archive at http://bourbonquito.com.

At any rate, having written my teaching statement, I figure I might as well post it here. As you will see, it represents the shift in my thinking about historical education from a traditional and un-articulated sense of critical-historical thinking, into an every present interest in and concern with the collateral learning that I hope to encourage in my courses.

My Philosophy

My teaching philosophy hinges on a set of values that have progressively clarified over the course of the past five years. As an educator, what I value most are: 1. The importance of critical-historical thinking; 2. The importance of collateral learning; 3. The importance of openness in educational resources; 4. The importance of empathy to understanding the past; and, 5. The importance of a Do-It-Yourself attitude for the cultivation of life-long learning.

I began my career at the University of Tennessee committed to the traditional notion of the nature and importance of History and Humanities education. My philosophy of undergraduate teaching was rooted in the belief that a university education and particularly a humanities education are about participating, in the words of Robert Hutchins, in “the great continuing conversation.” Following on this principle, I saw the historian’s responsibility as cultivating critical thinking skills built on a sense of temporality. The addition of time is, after all, the defining characteristic of historical education. And it is with temporal causality that historians are uniquely capable of participating in the great continuing conversation. My sense of critical thinking, and its place in my educational philosophy, was tied to a set of skills common to all historians that define critical-historical thinking:

1. Reading comprehension.
2. The capacity to evaluate evidence based on the typical historian’s typology of primary vs. secondary sourcing.
3. The ability to identify and build causal explanations for historical events, structures, and movements (be they examples of continuity or change).
4. The ability to communicate such explanations in written form.

I structured my courses to provide practical experience in each of these areas, mixing primary and secondary source material on the various topics under consideration and emphasizing in lecture and discussion the connection between explanatory narratives of the past with that evidence. I increasingly sought to include a wide variety of media as part of the source-based building blocks of my courses (texts, film, photographs, artifacts, etc.). My hope for each of my courses has been that students will leave with a keen sense of the problematic relationship between the present and the past and of the problematic nature of the relationship between information and knowledge.

The onset of my university teaching career has also coincided with a renewed crisis in Humanities education sparked by the economic instabilities of the past four years. I would be remiss to exclude the impact that this predicament has had on my teaching philosophy. In particular, the Humanities have come under attack for a perception by some of their irrelevance in an age dominated by an economic instrumentalism. I reject the notion that university education should be above all else vocational, as mere preparation for the job market. In fact, the tendency to see a university degree as essentially a credential for participation in the 21st-century economy is a source of real frustration for me. But the conversation surrounding the future of the Humanities and its critics has caused me to rethink the collateral learning that occurs in my courses.

Writing in 1938 in Experience and Education, John Dewey argued,

Perhaps the greatest of all pedagogical fallacies is the notion that a person learns only the particular thing that he is studying at the time. Collateral learning in the way of the formation of enduring attitudes, of likes and dislikes, may be and often is much more important than the spelling lesson or lesson in geography or history that is learned. For these attitudes are fundamentally what count in the future.

I would define collateral learning a bit more broadly than Dewey, including skills as well as attitudes. What collateral learning occurs in my classroom? There is the basic exposure to the culture and lifeways of peoples in Latin America that can impact attitude formation. There is the process of critical-historical thinking. I strive to combine that exposure and process to encourage empathy for others in my undergraduates. Increasingly, I have also sought to include basic technical skills that intersect with my overriding values of openness and a do-it-yourself (DIY) attitude in the educational process. Historians have been slow to reckon with the information revolution and its impact on our disciplinary culture. But, we are also well positioned to teach the type of information literacy so important for students navigating the flood brought on by the World Wide Web. For my courses, this has meant working with my students on being savvy users of problematic information, but also as producers of their own content on the web. By having students produce their own content on their own sites, I hope to demystify the web a bit for a generation that has grown up with the internet as a given in their lives and to cultivate a capacity for DIY that is collateral to the specific content of my courses on Latin American History.

Philosophy in Action

What have these educational values meant for my teaching? Above all, they have required of me willingness for experimentation. I have taken seriously Departmental needs for a wide variety of Latin American offerings at the Graduate and Undergraduate levels. As the lone tenure-track Latin Americanist in the History Department, I have sought to offer a wide variety of courses covering time periods and regions outside my immediate area of expertise. I have taught course ranging from the Conquest of the Americas to the Mexican Revolution, from the Early Latin American Survey to Modern Latin American History through Film. I have taught on gender and sexuality in the early period, on the history of indigenous peoples in both colonial and modern Latin America, and also honors and graduate courses on research methods and historiography. I have lobbied hard to keep the survey of Latin American History active, despite staff pressures, in support of the Latin American Studies program and Ready for the World. It is within this context of broad teaching that I have experimented with the incorporation of technology in ways intended to make relevant the process of my courses to the world of information we now inhabit.

In course design and execution, I have experimented extensively with both modeling and teaching the values of the DIY approach and the implementation of critical-historical thinking in new technological venues. Some of these experiments have been successful, and some have not. In the column of successes I would include:

1. Requiring students to write regularly and critically online, with their peers as a significant part of the intended audience. My first attempts to do this in 2008 utilized discussion forums inside Blackboard for such writing and discussion. I decided, though, that I wanted my students to instead write (reacting to readings, discussion, films, news, etc.) in an online venue that they could take individual ownership over. Beginning in Fall 2009, I have required my students to sign-up for their own free website using a blogging service such as wordpress.com or blogger.com, and to write and comment on weekly essays. In addition to the specific Latin American History content they discuss, I have witnessed my students come to see this space as a space of their own on the Internet, and come to understand more clearly where content on the web comes from. This has been the case in both upper division courses and graduate seminars.

2. Encouraging digital literacy. Sometimes small steps can be important in developing digital literacy skills. Each semester I try to help students understand how search works, and to help them become better searchers. I have tried to combine the use of online and physical resources from the library for research and historiography across the courses I teach. Likewise, I’ve started to use simple strategies such as only providing citations for work that is available through the library catalogue, requiring students to find it rather than directly linking.

3. The design and implementation of the online presence of my courses using free and open source software, including on the WordPress platform. I have left Blackboard for hosting my courses, and their online interactions, in part to model DIY for my students (both graduate and undergraduate). Doing so forced me to look under the hood, as it were, of how the web and web publishing platforms work. As importantly, I am able to make all of my teaching materials freely and publicly available for curious individuals, other Latin American Historians, and graduate students.

4. Experimenting with the hybrid classroom. Many of our graduate students will be faced with the prospect of online teaching in one form or another in the coming decade. Very few faculty in our Department have experience with online teaching, podcasting, or the like. In Fall 2010, I decided to experiment with video-casting one lecture per week from my History 255 Survey of Colonial Latin America to gain in understanding of the mechanics of online and hybrid teaching. In addition to putting half of the lectures for the semester online, I produced response quizzes using Google Docs to track students’ participation in the online lecture. Students also had discussion sections for the more practical exercises in evaluating primary documents related to the course. By both mid-term and final surveys of the students, the use of online lectures for part of the class was very successful. Students complained that I packed more information into the video lectures, and they spent much more time watching and note taking than in the live lectures. From the experience of producing and delivering the online lectures, I also learned much about the technical and time demands required. From this experience, I know that I can better advise our graduate students on handling questions on online teaching on the job market.

In the column of less successful experiments, I would include my Spring 2011 course on Modern Latin American History through Film. Spring 2011 was the second time I have taught the course, only in this case I tried to adapt it to a 90-seat class to meet demand for upper-division courses in the Department of History. Having 90 students significantly challenged discussion, which I again tried to move online to account for the awkwardness of the hall in which we met. My blog infrastructure, which as worked very well in classes ranging from 10 to 35, was overwhelmed by the shear volume of students writing. Additionally, I assigned a semester-long collaborative project as the main task of the assignment that was hosted on a wiki. Student groups were charged with constructing a comprehensive wiki on an assigned film, including critical engagement with the historical issues presented by that film. The students and I struggled over our mutual expectations for what constituted a semester’s worth of work on such a project. Likewise, I offered tutorials for writing on the wiki, but few attended and many later struggled with the technical realization of their projects. I also assigned too many films. In the end, I will have to rethink the course’s implementation if I need to teach it to such a large group again.

A note on graduate education

Much of what I have written above relates to my philosophy of undergraduate teaching. I have enjoyed significant interaction with out graduate students as well, despite the fact that we offer no degree in Latin American History. I have forged this ground for myself because I value graduate education highly. I taught History 510 three time from 2008-2010. As our Foundations class, 510 is a joy to teach in part because of the practical skills of reading, analysis, and communication that form its core. Readings are often theoretical and outside of the scope of experience many of our graduate students bring with them. I have also taught seminars on the Spanish Conquest and on the authority and power in the early modern Spanish Empire. More importantly, outside of the seminar room I have sought opportunities to support and train our graduate students, most notably with the Dissertation Writing Group (DWG) I founded and have directed since 2007. With 510 and the DWG, I have had pleasure of being a sort of bookend for our PhD students progress, from the foundational skills of 510 through their realization in writing the dissertation.

(All of my course materials are publicly available through individual course sites linked on my website)

Note for this post

I recognize that this statement is pragmatic more than it is philosophical, at least in the way I would have imagined writing such a statement as a graduate student or at the beginning of my career. I wonder, in looking over it, to what extent I’m actually falling prey to the functionalism I detest. What I want to communicate is that I see the traditional sense of historical-critical thinking and writing that has formed the core of history education at the collegiate level as still salient. I don’t like notions of “relevance” — ie, that students will only learn or care about things that their 18 or 20 year-old brains perceive of as relevant to their lives. But, what I want is to port those traditional skills, if you will accept the metaphor, to a context that is relevant, ie to the context of the post-industrial world.

At any rate, what are your educational values? How do they impact your teaching choices?

• There’s a problem
• But wait, there’s more.
• Wither text analysis
• Cluster techniques

There’s a problem

I’ve been playing around for a while now with various algorithms designed to cluster unstructured text. The reason I want to do this is I’m working on a project to test for plural legal cultures in the Americas during the early modern Spanish empire. During the course of writing my first book, I was struck by the extent to which my findings on women’s participation in the legal system of late-colonial and early-republican Quito confirmed earlier findings for the 17th-century in Quito, and frequently clashed with books on the same theme in other areas of the Empire. There is an important caveat here. Much of the work on 17th-cent. Quito was done by my dissertation director. While I stand by my (and her) evidence for Quito, I’ve always wondered if this is simply a case of the influence of graduate apprenticeship, of if there is something more historically fundamental at work?

This is a difficult nut to crack. The Spanish legal system, as a scaffolding, was uniform on this side of the Atlantic. Cristobal Colon, Hernan Cortes, Francisco Pizarro, and the rest of Spain’s 16th-century marauders made there conquests under explicit authority of the Crown of Castile. And, unlike kingdoms on the Iberian peninsula that accrued over time to the Crowns of Castile and Aragon each with their own recognized fuero laws, the kingdoms in the Americas were unified under two specific overarching codes, the Leyes de Castilla and the Leyes de Indias. The former transferred the code of Castile to the Republica de españoles, while the latter dealt with legal conditions specific to the Americas, and particularly its indigenous inhabitants. These were not the only codes with currency. Not by a long shot. But, at a schematic level the authority of the Crown of Castile over the kingdoms of the Americas was enacted through these codes.

Furthermore, magistrates in the Americas were royal appointees, occasionally even at the city council level. While city councilors were chosen locally (and the Alcalde Mayor was a city councilman), in cities or regions with a resident corregidor, he would sit on the council. He was also a magistrate for his jurisdiction. What is more, magistrates above the alcalde level were invariably royal appointees (oidores of the Audiencia or the virrey himself). Litigants could also shop jurisdictions as needed, and one need not first seek a local magistrate to settle a dispute, but one could appeal directly to any of the levels above.

But wait, there’s more.

The journeymen of the Spanish legal system were notaries, and their handbooks and often there education were rooted in the peninsula. There is, in fact, a fair amount of uniformity across the Empire in the form of legal documents, prepared as they all were by notaries and their apprentices drawing on boilerplate examples from notarial manuals.

Where does this leave me? Well, with many reasons to see a unified legal system for the Empire. But there’s a catch. Central to the operation of the legal system, the administrative system, the economy, and everything else in the Spanish Empire was a notion of decentralization, of the devolution of power locally under the guise of customary practice. It had such currency, in fact, that its operation frequently went unnamed. In addition, magistrates were prevented from explaining their decisions with reference to specific codes or regulations (though the Crown’s attorney, or fiscal, frequently did express such opinions). So, the justification for legal decisions is almost never given. Even though the form and the function of the legal system is the same throughout the Americas, the opportunity for significant (and I do mean significant) local inflection was a built-in feature. There very well may be no singular legal culture. We don’t know if there was, though, because scholars (myself included) have tended to generalize from their particular archives

Wither text analysis

This seems like a job for a variety of text-mining techniques. I’ve had in my mind for a while now that a variety of clustering techniques may well help identify local legal culture in operation beneath the boilerplate of standard legal documents. In essence, I need to measure distance between documents, or rather groups of documents, and look for clusters by geography and time. It may well be that legal cultures in the Viceregal capitals were very similar, but not so much in other cities, towns, and villages. It could also be that local legal cultures are greatly impacted by local indigenous cultures or by the migration patterns of initial settlement in an area. Families frequently brought over extended kinship groups, drawn from specific regions in Spain. Did they bring with them the legal expectations of the area they left?

Don’t know. Yet. I’m hoping that text analysis of a corpus of civil and criminal trial transcripts from across the Americas, and across the centuries, will help me find out.

Cluster techniques

So, what techniques am I (planning) on playing around with? I’m only going to list them here. I’ll be writing follow-up posts on early experiments with each of the methods in the coming weeks.

1. Normalized Compression Distance (NCD), which uses compression algorithms to establish the similarity of two documents.

2. Latent Dirichlet Allocation (LDA)-based topic modeling, for which I’ll be using the killer gensim vector-space modeling module for python.

3. Term Frequency-Inverse Document Frequency (TF-IDF), also used in vector-space models.

4. KWIC and n-gram plots, which will be useful to chart both the occurrences and context of terms and phrases indicative of locality in legal outcomes.

I’ve also been playing with ways to include fuzzy orthography, so typical of early modern record making, into the analysis. Right now, I’m leaning towards incorporating some sort of Levenshtein Distance threshold to account for the heterogenous spelling.

At any rate, in order to test the scripts I’m writing to use these various techniques, I’m initially analyzing a relatively small corpus derived from the Criminales Series Finder’s Guide for the Archivo Nacional del Ecuador (ANE). The corpus consists of 21 documents, each of which comprises a decade of criminal prosecutions beginning in 1601 and ending in 1830. A rough word count (whitespace delimited) of the whole corpus amounts to 198,000 words, with the longest document coming in somewhere around 39,000.

Stay tuned for experiments with NCD.

One of the nice elements of optparse is that it automates producing a help option for a command line program. In the spirit of the CLI, I rewrote the command options so that they are more bash-like. I’ve also changed it a bit to better handle posts w/o due dates, blank posts, and subtasks.

Also, because of the awkwardness of sharing scripts through wordpress.com, I moved the source to github. I also put the script up there to post to wordpress.com using with markdown that I wrote about here. That one needs some work, though. Particularly handling ascii/unicode issues and with uploading images.

At any rate, the v0.2 of myTasks works thusly (assuming you have the script on your path and/or with an alias):


Usage: tasks [option] arg1 arg2 arg3

Options:
--version show program's version number and exit
-h, --help show this help message and exit
-l Lists all tasks. Takes no arguments
-n
Adds new task. Pass the name of the task list and the
new task as arguments in double quotes. For example:
tasks -n Main "Add this task to the Main list."
-c Clears completed tasks from your lists. Takes no
arguments.
-u
Updates a designated task as completed. Pass the name
of the list and the number of the task. The number is
available by first listing tasks with the -l command.
For example: tasks -u Main 1. This command would mark
the first message on the Main list as completed.
-d
Deletes a designated task. Pass the name of the list
and the number of the task. The number is available by
first listing tasks with the -l command. For example:
tasks -d Main 1. This command would delete the first
message from the Main list.


It does just about all I want to do with it. I’m going to add the option to output all lists, or a designated list, but that’s about it for now. Please feel free to fork on github and make a pull request to improve it.

Update 9-20-2011: I pushed v0.3 of myTasks to github this morning. The new version uses a different commandline parser, and adds functionality with task listing options, the ability to rename or delete Lists, and a bit more.

Note that the advice (for whatever it is worth) is directed towards new PhDs in History looking for traditional academic employment. In general with our grads, I usually spend most of our time working on the C.V. and the cover letter. Other elements of the dossier should be more comfortable for the potential job seeker- you need a very polished writing sample, particularly a chapter or two from your dissertation that exemplifies its argument and the power of your writing; you need a teaching portfolio with sample syllabi, student evaluations, and a statement of your teaching values. If, during your tenure as a grad student, you ever get laudatory emails from students or in comments from your TA evaluations, keep them. Print them out. They can be very important in proving your readiness to teach.

The ancillary materials requested by specific institutions will clue you in, in part, to what the school is looking for. Regardless of those materials, though, all institutions want a CV, a cover letter, and letters of recommendation. The latter you have little control over. I don’t think it matters whether a letter comes from a dossier service or directly from the individual. But, I would suggest that you cultivate your relationship with committee members and other letter writers such that if there is a job that you really want, they will be willing to write customized letters for it. A strong dossier letter is certainly sufficient. You do, though, have complete control over your CV and your cover letter.

What does the cover letter need to accomplish?

The cover letter must accomplish a variety of tasks –> express your enthusiasm for the job, communicate your degree readiness, your research chops, detail the importance of teaching to you, and your availability to interview. It also implicitly expresses your capacity to communicate in a clear and concise manner, your attention to detail (particularly the details listed in the job ad), and your understanding of the type of institution that you’re applying to.

You cover letter must be tailored to the specific type of institution to which you are applying. Generally, there are four types of institutions to which we apply- the R1 research university; the R2 research/teaching university and research-interested liberal arts college; the comprehensive regional university and fully-teaching centric liberal arts college; and the community college. The first important point for writing a cover letter is know the type of school you’re applying to, and customize the letter for that category. By customize, I don’t mean that every letter needs to be written from scratch. Rather, develop a template for each institutional category. Each template will rearrange the order and emphasis of the kind of information that your letter needs to convey.

So, a template for an R1 would look something like this:

Introductory paragraph
   |
Current research/dissertation/competitive funding received
   |
Future research
   |
Graduate-level teaching
   |
Undergraduate-level teaching
   |
Service
   |
Concluding paragraph

Whereas, for an R2/Research-oriented liberal arts school, you might have to omit graduate level teaching, downplay your future research agenda a bit, and beef up the undergraduate teaching for a template like this:

Introductory paragraph
   |
Dissertation/current research
   |
Teaching
   |
Future research plans
   |
Service
   |
Concluding paragraph

It is always a very good idea to connect one’s research interest to one’s teaching. And furthermore, to specifically mention your ability to teach whatever the ad mentions you need to be able to teach. Likewise, for teaching-centric schools, that aspect should be front and center in your template:

Introductory paragraph (with dissertation title/topic included)
   |
Teaching
   |
Research Interests current and future
   |
Service
   |
Concluding paragraph

In any case, the sections on research and teaching need to reflect the interests and needs of the institution to which you are applying. How to do that? Well, I would imagine you need to research the school and its department at least a little bit before applying to it! The ad should give first clues to what they’re looking for, but sometimes they do not.
What you are trying to convey is that you are able to walk in directly to the job on August 1st, 2012 and thrive.

The Introduction

We don’t write letters too often any more, and it’s important to remember that formal letter writing has its own boilerplate. If you can, put it on heavy-stock paper or letterhead.

Use the formal boilerplate:

Your Name
Address
City, State Zip
email

Date

Name of the Committee Chair, or the Search Committee
Address
City, State Zip

Dear Professor __:

In the opening sentences, it’s important to establish upfront your bona fides and what you are applying for:

I am pleased to submit my application for the position of Assistant Professor of Latin American History at the University of Whatever as advertised on H-Net. I am currently in my fifth year as Assistant Professor of Latin American History at the University of Tennessee, Knoxville. I completed my Ph.D. in Latin American History at the University of New Mexico in 2006 under the direction of Dr. Kimberly Gauderman. My primary areas of specialization are legal culture, gender and sexuality, and empire in late colonial and early republican Spanish America.

That’s it for the introduction. The whole paragraph. What comes next, of course, depends on which template you are following. It goes without saying, though, that you need to check, double check, and triple check to make sure that the Professor Name, the Address, and the University you name in the opening sentence all match!

I’ll give a few more examples of the kind of text I’d write now. Note, though, that this text differs from what an ABD or a new PhD would need to say.

Research

Remember, the cover letter is not the space for modesty. You need to, well, brag about your work and state why it is important. If it is at all possible, explicitly connect your work with the departmental priorities that you determined in your scouting. This section can be 1-2 paragraphs long, depending on how much really needs to be conveyed:

My research interests have focused on the relationship between governance, legal culture, gender, and the political upheavals caused by the Atlantic liberal revolution in the northern Andes. My book, The Limits of Gender Domination: Women, Law, and Political Crisis in Quito, 1765-1830 (University of New Mexico Press, 2010) provides a social, legal, and gender history of the corregimiento of Quito (the city Quito and its immediate hinterland in modern-day Ecuador) that tracks the interplay between these themes. Utilizing a variety of primary sources gathered in Ecuador with support from the Fulbright-Hays Dissertation Fellowship, my book argues that through the end of colonial rule, Spanish legal and social practices resisted the full consolidation of a legal patriarchy, a resistance that was counter-intuitively undone in the transition to republican rule. Women of all ethnicities and social ranks effectively used the legal system and their property rights to mitigate domination by the region’s men and an increasingly aggressive and invasive state. I analyze women’s activities within the context of shifting ideologies of governing authority and through criminal and civil litigation, notarial and city council records, and legal commentaries intended to guide judicial practice in the eighteenth and nineteenth centuries. My sources reveal that the women of Quito inherited, bequeathed, loaned, and otherwise controlled property, engaged in public market activities, pursued sexual relationships, and used the legal system to pursue or defend their interests against each other and against the men in their lives. Such actions were discouraged by Bourbon legal theory and judicial manuals, and then drastically curtailed as Quito made the transition from status-based to contract-based governance in the early republic. Across that period, the infrastructure of customary practice that afforded women autonomous legal and economic identities was dismantled by liberal political and judicial forms that were attached to citizenship.

I would probably add a second paragraph to this, but only if I was applying to an R1 or R2 school. Otherwise, I’d leave it at the above. A second paragraph might look like this:

My book provides a significant revision of late colonial gender relations, within the context of unstable legal and political cultures. I document significant legal autonomy among women of many different social estates. For example, despite legal strictures requiring married women to secure license from their husbands or the court to make legal acts, I have documented that this was routinely ignored. Overall, approximately 70 percent of female plaintiffs and defendants engaged in the legal system independently, without regard to their marital status. This was made possible through the structural currency of derecho vulgar (customary law) and the messy labyrinth of law and jurisdiction that afforded magistrates significant discretion. It was also rapidly dismantled by the emergent liberal state as Quito’s legal and political structures were transformed by the transatlantic liberal revolution. The new state of Ecuador constructed a private sphere of contract and property that had never before existed, walling off a man’s home as his castle, a space exempted from the police powers of the state. The important conclusion of the book, then, is that the full consolidation of patriarchal rights in Quito was itself a product of the liberal revolution that made nation states of kingdoms.

The structure of all this is to go from general description of one’s dissertation (themes, funders, sources, sub-disciplinary identities) to specific and (hopefully) important conclusions. If you have spun a publication out of your dissertation, work that into the first research paragraph. Substitute dissertation for book, and then mention your article(s) as a place that you’ve argued some salient point from your diss. “As I demonstrated in my article, “Between Prescription and Practice” (Colonial Latin American Review, 2007), ….” The exercise of writing this kind of paragraph is useful because it helps you articulate before sitting in an interview what it is your dissertation does. It shouldn’t be that difficult to write either, assuming you wrote grant proposals along the way in grad school.

Finally, it’s useful to recognize that your dissertation will need revision for publication. You don’t need to necessarily include this in your letter, but you should have prepared before the interview a mental list of revisions or new info that will be needed for publication as well as a list of two or three presses that you have in mind for submission.

Future Research

Even though you’ve just finished your dissertation (or are frantically working to finish it), you need to indicate especially to institutions that put a high value on research (which is an increasing percentage in the last 10-15 years) that you’re not a one-trick pony. Even if you don’t have a well developed idea post-dissertation, you need to have something to say in your letter and in interviews about where you’re heading.

It doesn’t need to be much, and you won’t be held to it in the future. If there is something in your research that will be unanswered in your dissertation but that you have a feeling for, go with that. In my original letters, I explained that we had no feeling for the extent to which women and convents were the originators of both formal and informal loan instruments. I suggested I’d be doing a project that used notary books and civil litigation to try to determine that. Haven’t done it (though I’d still like to!).

Teaching

It matters whether or not the institution you’re applying to has undergraduate-only instruction, a Masters program, or is a PhD granting institution. Regardless of the case, your teaching paragraph needs to connect to the needs of those programs:

I am an enthusiastic proponent of undergraduate education. As UT’s only tenure-track Latin Americanist, I have sought to continually develop a broad range of undergraduate offerings for our students. In my time at UT, I have taught ten different courses ranging from the “History of Indigenous Peoples in Latin America” to “Women in Colonial Latin America,” to the “Mexican Revolution” to the “Spanish Conquest.” I have also taught more survey-oriented courses in both Early and Modern Latin America. I am always looking for effective means to incorporate media and technology into my teaching, as well as intensive discussion of primary and secondary readings to engage students on a number of levels. Finally, I am developing a study-abroad course to take undergraduates to Quito, Ecuador to study the history of indigenous politics and modernization in the Andes. Copies of all of my course syllabi are available at http://chadblack.net/teaching/.

This paragraph in and of itself is too short to constitute the whole of a teaching section, particularly for more teaching-oriented programs. I would amplify it by researching the individual department and emphasizing the areas in which my teaching experience would compliment or expand the department’s current emphases. The same would go for graduate teaching. If you have any special training or experience for teaching, this would be the place to explain it. Also, you must mention any and every course listed in the ad, and how you’re prepared to teach it.

Service

By service, I mean service to the discipline. If you’ve worked for any conferences, or helped edit a volume, or managed a website related to you discipline, put that information here. If you’re committed to public history, or to public consumption of your writing and run a blog for that purpose, put it here.

Conclusion

The concluding paragraph needs to list the enclosures sent with the letter (writing sample, transcript, syllabi, etc.), the names of your recommenders, your plans to attend the AHA (and the reason why if its for more than interviewing), and an invitation to contact you if the committee needs any more information. It would look something like this:

Please find with this letter of application a copy of my C.V., Chapter 2 from my book, and whatever else you asked for. Letters of recommendation will be forthcoming from Drs. Smart Person-Who-Knows-Me, Another-Smart-PersonW-Who-Knows-Me, and Third-Impressive-Recommender. I will be at the AHA in Chicago in January as a participant on a really nifty panel, and would be happy to meet with the search committee at any time. Please contact me if I can provide any additional materials that might help the committee evaluate my qualifications.

Finish respectfully:

I appreciate your consideration of my application.

Sincerely,

Graduate students I’ve worked with have had success in getting interviews with such a letter. Note, though, that the letter is more a piece that can get you excluded from consideration than it is one that will get you included. By that, I mean to say you can torpedo your file with a poorly executed cover letter, but the merits of the rest of your dossier combined with the X_Factors of any give search committee are what will get you the interview.

Good luck this go round! I’d love to hear cover letter advise from others… what they like to see when serving on search committees. Or, what you don’t want to see.

Idle seminarian: I don’t have to swear any oaths to persuade you that I should like this book, since it is the son of my brain, to be the most beautiful, elegant and intelligent book imaginable. But I couldn’t go against order of nature, according to which like gives birth to like. And to what can my barren and ill-cultivated mind give birth except the history of a dry, shriveled child, whimsical, and full of extravagant fancies that nobody else has ever imagined– a child born, after all, in the prison of graduate school, where every discomfort has its seat and every dismal sound its habitation? Tranquility, peaceful surroundings, the pleasures of walking amongst stacks of mildewed tomes we once checked out viscerally, the morning cigarette at eleven after a long dark night of excursions through archival anxieties, the serene smells of brewing coffee — these are the things that encourage even the most barren muses to become fertile and bring forth a progeny to fill the world with wonder and delight.

It can happen that a man has an ugly, charmless son, and his love blindfolds him to prevent him from seeing the child’s defects: on the contrary, he regards them as gifts and graces, and describes them to his friends as examples of wit and cleverness. But although I seem like this treatise’s father, I am really its stepfather, and I don’t want to drift with the current of custom, or beg you almost with tears in my eyes, as others do, dearest seminarian, to forgive or excuse the defects that you will certainly see, discuss, and dismiss with the peculiar vitriol of graduate student contempt, in this my son; and you are neither his relative nor his friend, you have your own soul in your own body, and your own free will like anybody else, and you are sitting in your own department’s seminar room, where you are lord and master just as much as the king is of his taxes, and you know that common saying, ‘Under my cloak a fig for the king.’ All of which exempts and frees you from every respect and obligation, and so you can say whatever you like about this history, without fear of being attacked for a hostile judgement or rewarded for a favorable one. Even, it could be said, with hope and yearning for the reward of a hostile judgement and avoidance of the humiliation of a favorable one.

I’d have liked to give it to you plain and naked, undecorated by any prologue or the endless succession of historiographical and theoretical musing, epigrams, and eulogies that are usually put at the beginning of books. Because I can tell you that, although it was quite an effort to write the book, producing the preface that you’re not reading was far worse. Many times I opened my MacBook to stare at a blank screen, and as many times I didn’t know what to say; and once when I was in this quandary, with the computer in may lap and a short double latte on the table, an elbow beside it and my head in my hand, wondering what I could write to impress you, dear seminarian, a friend of mine wandered by the corner of the coffee shop I have sequestered as my own these last months, a lively and clever man who, seeing me thoughtful or despairing according the to observer, asked me the reason, and I didn’t keep anything from him but said that I was thinking about the prologue that I had to write for my tenure book, and that it had reduced me to such state that I didn’t want to write it at all, still less publish it for the knaves of the discussion table to parry and poke.

“Because how do you expect me not be worried about the opinion of that ancient legislator called the academic apprentice when he sees that after all this time sleeping in the silence of oblivion, and burdened by the years as I am, I’m coming out with a book as dry as esparto grass, devoid of inventiveness, feeble in style, poor in ideas and lacking erudition and instruction, a triumph of tenurability riddled with superfluous endnotes in need of exposition, unlike other books I see that, even though they must certainly be fictional, are so crammed with maxims of Foucault, Marx, Bourdieu, Homi Bhabha and the whole herd of theorists that amaze their readers, who consider the authors to be well-read, erudite, and eloquent men? And when they quote the Holy Writ of Deleuze and Gattari! Anyone would take them for no less than so many aspiring Derridas and other canons ministering to the enfeebled body of knowledge. And here they maintain such an ingenious decorum that having depicted a dissolute lover on one line they provide on the next a little deconstruction, a pleasure and a treat to hear or read. There won’t be any of this in my book, because I haven’t anything to put in the margins or anything worthwhile for the notes at the end. Following their burgeoning skills as readers, my dear seminarians, I’ll have to invite them to simply skip the notes. Still less do I know what authors I have followed in my text so as to list them at the beginning, as others do, in alphabetical order beginning with Adorno and on to Gramsci and Lacan and finishing with Williams and Zizek, even though one that last is a slanderer and impostor. My book will also lack historiography at the beginning, or at least extensive exposition of those deans of the colonial literature from Madison and Austin and New Haven to generations beyond.” I continued, “I have decided that my tenure book will remain buried in the stacks of the few libraries that still have the money to purchase codices, and in the judgement file of my department’s promotion process. And though no parent longs to outlive their child, this seems a dignified death, less heaven or the press provide someone to adorn him with all these attributes that he lacks– I’m no up to it, because of my inadequacy and my scanty learning, and because I’m naturally lazy and disinclined to go hunting for authors to say for me what I know how to say without them. this is why I was so perplexed and distraught when you arrived, my friend: there is justification enough for it in what I’ve just told you.”

When he heard this my friend slapped his forehead, burst out laughing, and said:

“Good God, my dear fellow, you’ve just corrected a misconception I’ve been laboring under all this time I’ve known you, considering you to be sensible and judicious in everything you do. But now I can see you’re as far from being that as the heavens are from the earth. How can matters that are so trivial and easy to remedy have the power to engross and perplex a mind as mature as yours, accustomed as it is to demolishing far greater difficulties? I assure you this isn’t caused by any lack of ability on your part, but by an excess of mental indolence. Do you want to find out whether I’m telling you the truth? Well, pay attention, and you’ll see how in the twinkling of an eye I destroy all your problems and remedy all those deficiencies that, you say, are perplexing you and discouraging you from publishing the history of women and the law in late colonial Quito.”

“Tell me,” I replied once I’d heard him out, “how do you intend to fill the vacuum of my anxiety and turn the chaos of my confusion into clarity?”

To which he replied:

“Your first problem, about the epigrams and eulogies written by important theorists that you lack for the beginning of your book, can be remedied if you take the trouble to write them yourself and christen them and give them whatever names you like or simply use a search engine to google up some quotes, saving even more labors. The applicability of such epigrams matter not, and you can always adapt them to your purposes whatever their original meaning with a simple bracketed interjection or two; and even supposing the quotes don’t really work and some pedants and academics (are they distinguishable?) start their backbiting and their nit-picking about whether this is relevant or not, you mustn’t care a hoot about that, because even if they do find out that you were telling lies or proof-texting they aren’t going to cut off the hand with which you wrote them down. As for references in the text and notes to the books and authors from whom you take the sayings and maxims and models and theories that you include in your history, all you have to do is to stick in a few relevant bits here and there, whatever you can look up without too much trouble, such as when you’re writing about power:

"There is no power relation without the correlative constitution of a field of knowledge, nor any knowledge that does not presuppose and constitute at the same time power relations"

And then in the margin, mention a little more Foucault or whoever it was that said it. You can even add some agency to your subject by universalizing its opposite:”

"Where there is power, there is resistance."

If you’re talking about the triumvirate of race, class, gender, go straight for the verbal kung fu of Butler, which you can do if you take just a little care, in a manner that will appear incontrovertible to the neophytes of the profession and allow you to ignore the subjects you wish not to write about: “Race and class are rendered distinct analytically only to produce the realization that the analysis of the one cannot proceed without the other. A different dynamic it seems to me is at work in the critique of new sexuality studies.” Or, simply take an aggressive stance towards all criticism and follow the Deleuzian path:

"The shame of being a man - is there any better reason to write?"

and,

"Every time someone puts an objection to me, I want to say: 'OK, OK, let's go on to something else.' Objections have never contributed anything."

And with these scraps, and other similar ones, you’ll be taken for a scholar, at least; and that brings no little honor nowadays.

As for beefing up your endnotes, you can easily do it like this: if you include a discussion of gender relations in the book, just include the term patriarchy, which will hardly be any trouble for you, and will give you a splendid endnote, because you will be able to say, “Exclusion from political power is indeed a necessary but not a sufficient condition for the particular form of patriarchal male domination, particularly in light of the poststructuralist insight into articulation of power throughout social and cultural forms,” in the chapter where you have written it. After that, to show that you’re erudite in the legal and cultural history, you could contrive to name a process as a thing, and there you have another fine endnote: “By judicial form, I mean to indicate the actual physical form used to submit petitions, document actions, make declarations, notate transactions, and so on, as well as the discursive form in which the documents were formed. There was a remarkable discursive continuity across the variety of acts that have entered the written record, encapsulated in the judicial form.” In short, all you have to do is contrive to mention these names or touch on these stories in your own story, and leave it to me to provide the endnotes and marginalia; I swear by al that’s scholarly to fill your margins, and use of reams and reams at the ned of your book.

Let us now consider the list of authors cited, which other books include and your lacks. The remedy for this is simplicity itself, because all you have to do is look for a book listing them all from A to Z, as you say, then copy this list into your own book; and if your deception is plain to see this won’t matter in the slightest, because you hardly need to use the authors anyway, and there could always be someone stupid enough to believe that you have used them all in shit simple, straightforward story of yours. Even if it serves no other purpose, you long list will at least lend your book an instant air of authority. Besides, people aren’t going to take the trouble to check whether you follow your authors or not, because they haven’t anything to gain from doing so.

What’s more, unless I’m much mistaken, this book of yours doesn’t need any of those features you say it lacks, because from beginning to end it is an invective against colonial histories. All that has to be done is to make the best use of imitation in what one writes; and the more perfect the imitation the better the writing. And since this work of yours is only concerned to destroy the authority and influence that certain books enjoy in the world and among the graduate-seminar reading public, there isn’t any need to go begging maxims from theorists, counsel from historiography, fables, clauses, or miracles from the virgin cults, but rather to attempt, using express, decorous and well-ordered words in a straightforward way, to write sentences that are both harmonious and witty, depicting what is in your mind to the ver best of your ability, setting out your ideas without complicating or obscuring them. You should also try to ensure that the melancholy woman is moved to laughter when she reads your history, the jovial woman laughs even more, the simpleton is not discouraged, the judicious marvel at its inventiveness, the serious-minded do not scorn it nor the wise fail to praise it. In short, always have as your aim the demolition of the ill-founded fabric of these books of history, despised by so many and praised by many more; and if this is what you achieve it will be no mean achievement.”

I listened in profound silence to what my friend said, and his words so stamped themselves on my mind that I accepted them without any argument and decided to use them for this prologue, in which, gentle seminarian, you will discover my friends intelligence, my good fortune in finding such a counsellor at a time of such need and your relief on finding that there will be no deviousness or circumlocution in this history of the women in late colonial Quito. I have no desire to extol the service I am rendering you in introducing you to such a noble and honorable subject as this; I won’t even solicit your thanks or appreciation for providing a book such as this for your dissection table. And so, may the grace of Clio grant you health, and maybe even employment someday my dear graduate student. Farewell.

*This is, obviously, Cervantes remixed. Thanks Penguin Classics!

• Australia (Melbourne, Canbera)
• Asia (Saigon)
• North America (Victoria, Montreal, University of Western Ontario, Kansas, New Englinad, New York, Philly, Michigan, Texas, New Jersey, Georgia, SoCal, Chicago, SanFran, etc.)
• Europe (Paris, Madrid, Florence, Cologne)
• Middle East (Cyprus)

One of my goals over the next year is to link up with people in Mexico or Ecuador and help host a THATCamp south of the US border.

The movement grew out of CHNM’s initial experiment with the unconference form just a couple of years ago. How to explain the astronomical growth? Is there a thirst for technological hacking of humanities questions? Do people yearn for conference experiences that offer time and experience value beyond the traditional model of sitting in rooms being read to? Are THATCamp attendees simply effective evangelists for DH and the unconference model? My guess is it’s a combination of the two.

This year, the main conference at CHNM was much larger than ever before, with some 150 attendees. It also offered for the first time a preliminary bootcamp day of set tracks that combined technology presentations and directed hands-on experience. It was clear from day one, though, that with the expanded participation in this years conference, deciding what to attend and what to forgo would be quite a bit more difficult.

I posted my notes from the three bootcamp sessions I decided to attend, on the most novice track: Intro to CMSes, Intro to Omeka, and Project Management. The hack track offered sessions on HTML5 and CSS3, hacking WordPress themes, and an introduction to jQuery. Finally, a special track by Google included instruction on data and mapping visualizations. I was torn between the three because I have experience enough with CMSes not to need basic instruction on installation or maintenance, but I’m also in the midst of building an omeka site, so I figured having a firmer grasp on the structural imperatives of omeka would make me more effective in utilizing its strengths and figuring ways around its weaknesses. That, and I’m not quite ready yet to start using the admittedly powerful new capabilities available in HTML5, CSS3, and jQuery for any sites I’m working on currently. Too much other stuff to do. (That will change, I reckon, when I migrate chadblack.net over to jekyll or hyde.)

I was happy with my choices larger because of the presentations given by Raf Alvarado in the Intro to CMSes and by Tom Scheinfeldt on project management. Raf explained the differences between wp, drupal, and omeka in the clearest terms I’ve seen yet, and with an eye towards the ontological implications of choosing one over the the other. He did this by exploring the content model employed by each of the systems, and connecting that model to a logic of presentation and form of hypertext. It’s the first time I’ve heard such an analysis, and it was particularly well suited to the audience. Tom’s 10 points on project management were nitty-gritty, and dealt with funding, managing relationships to funders, staff, and partners; with project leadership; with execution; and more. It also produced the most entertaining fake twitter stream of the weekend.

One of the day’s highlights included a live-studio-audience broadcast of Digital Campus, much of which was spent discussing the suit that has been filed by Oxford UP, Cambridge UP, and Sage Publishers have filed against Georgia State University over copyright and fair use. The suit has serious implications for universities and the way that we currently assign readings for classes. The plaintiffs are advocating an electronic reserve policy that would restrict fair use rights to something on the order of 6 pages per work. Anything longer than that would require paid licensing. The digital campus crew wondered if such a change would actually accelerate the movement towards open access and Creative Commons work in scholarly work. I’ve argued before that the nature of historical evidence and inquiry provide ample justification for making CC a disciplinary standard. More importantly, though, I think the OUP/CUP/Sage lawsuit is evidence for the growing disjuncture between the market-oriented mission of scholarly publishers, and the non-market needs of scholarly communication. Up to now, the scholarly publishing industry has existed in an artificial market place that represents a convergence of trends related to the tenure track, library purchasing, an explosion in new scholarship. The tenure track has demanded an ever growing standard of publication, and in history at least, that means new monographs (not just at R1s anymore, either). Libraries have traditionally been the main purchasing venue of these works through broad agreements with the leading academic presses who were more than happy to make overly expensive hardcover editions available to them. Finally, in connection with tenure expectations, and with a growth in graduate school attendance, there are simply many more people writing many more books that cover increasingly specialized subjects. The system isn’t sustainable. Departments have enjoyed for three decades the luxury of outsourcing tenure decisions to the press board and the free peer review labor of the academic press. Academic presses enjoyed the purchasing power of libraries that were willing to pay $100 or more per volume. That purchasing power has been shifted to the out-pacing-inflation costs of maintaining journal subscriptions in the sciences (and the humanities). And yet, new scholars need a venue for their work– and not just for 30 page journal articles. There is, I think, self-justifying merit in the long form argument codified by the bound codex that is a book.

That the model of the university press was developed for an age of information scarcity has become a truism of these discussions of late. We are, obviously, no longer in such an age. Recognizing this always leads back to a discussion of what it is that UPs offer scholars. The list isn’t long:

1. Peer review.
2. A distribution channel.*
3. Copy-editing.*
4. Advertising.*
5. List focus.
6. Gate-keeping.

If there’s something I’m missing, I’d love to know what it is. The three points above marked by an asterisk are relatively easily surmountable by the internet. List focus, gate-keeping, and peer review are, I think, linked to one another as disciplinary functions. They are more difficult, but I doubt they are insurmountable. In fact, I’d like to see the sub-disciplinary organizations (such as the Conference on Latin American History) to take over managing peer review, a clearinghouse model that would undo the need for list focus and certainly provide appropriate levels of gate-keeping. By removing those functions from the UP, we would also open the doors to more creative exploration of long-form narrative, research presentation, and the like. Once procedures are set up, we’d have a much better opportunity for the kind of buy-in necessary to marshall a transformation of scholarly communication that takes advantage of the possibilities of the post-scarcity age.

That said, without buy-in (which would include rethinking the connection between profit and scholarship) History as a discipline will continue its slip into the abyss of a UP/tenure/market model that is no longer tenable for the majority of work we produce. I don’t want to sacrifice serious scholarship to the altar of “readability” or “marketability” that some say is at the root of problems in scholarly publishing.

This is a big discussion, and one the DH world keeps turning to even as many of our colleagues that share departmental halls refuse to recognize or talk about. (I’ve had mind boggling discussions with people who haven’t published new work in 15 years who have no idea that the world of books and the world of scholarly publishing are hurling headlong into a crisis analogous to that of the newspaper.) It’s also a discussion that carried on over the next two days in sessions of the idea of an “unpress,” peer review, and journalism. (Allow me to add here that I think it’s time to put a fence around new uses of the “un” prefix.)

Day 2 returned to the traditional THATCamp model, and we started the day with a huge list of potential sessions that were pitched on the conference blog prior to convening. (All THATCamps, including regional camps, have a wp site hosted by CHNM that allow for pre-conference discussion of potential sessions and other ideas for the conference.) There was a bunch of yack and a bunch of hack that was proposed that I was interested in, and it was obvious that tough decisions would have to be made. I wanted to participate in discussions on digital literacy, advocacy and diversity in the humanities, avenues to establishing DH initiatives/centers, to get some hack on, design a wordpress theme from scratch, consider the future of Anthologize, and more. There were sessions on all this, and more. Alas, I didn’t make it to most of them. What I did do was this–

1. A session on intro to hacking led by Patrick Murray-John that put us into javascript and greasemonkey in order to manipulate web pages. It was useful for a couple of reasons. First, I’ve never written any javascript, but I need to. I could, at least, read the code and understand what it was doing. Getting used to a new syntax always takes some time, and coming from python I found myself repeatedly omitting ‘;’ and ‘{}’. Second, it’s useful to learn how to manipulate web pages locally on one’s machine in order to arrange or extract particular information for researching online resources. A number of browsers now support local scripts. Mostly, though, the session was fun. As with any coding (read, new coding experience or problem for me), it took a little bit to get things set up right, and to figure out the problems I was interested in. What was that problem, you ask? Of course it was a practical joke. I got so sucked into playing with greasemonkey, that I didn’t manage to slip out to attend one of the other sessions I wanted to during that slot. This seemed to happen again and again.

2. A session on advocacy that turned increasingly into a discussion of managing expectations in the relationship between a DH Initiative, faculty, and staff developers. There is a deep well collecting the frustration of people involved with DH who resent be treated as if they’re the digital equivalent of the copy shop. The theme cropped up on several occasions, as people reported some combination of naivety and arrogance propelling faculty to treat library and #alt-ac staff disrespectfully. I was a bit disappointed with the direction the conversation took, because what we started off wanting to talk about were strategies to evangelize/advocate to faculty and administrators who aren’t in to DH. Instead, we slid into a bit of the recurrent complaints on notions of credit, promotion, etc. At the same time as that session, in the big room at Research I there was a lively conversation on the notion of an “unpress” and what practical strategies are to encourage the development of alternative publishing models. I slipped in for the last bit of that session, which also had a lively twitter backchannel. In many ways, the unpress discussion was an extension of the discussion that dominated the taping of Digital Campus.

3. A session on practical steps towards encouraging diversity in THATCamp specifically and DH more generally. One of the results was to start a page as a forum for planning and discussion. It’s a bit of a tricky situation in part because of the DIY spirit of DH and THATCamp. As Alex Gil noted with respect to the circum-Caribbean, getting DH discussions going there would require a commitment of expertise and hardware. There are regions of the world where the digital divide is very stark. Increasing diversity will require a concerted effort to outreach. Something I failed to mention was that last fall THATCamp NM met at the National Hispanic Cultural Center in Albuquerque, and included a large number of hispanic participants. In part this has to do with the demographics of NM, and in part it has to do with the sponsoring organization and its ties to cultural preservation in NM. But, it was also the least academic-oriented THATCamp I’ve seen. The diversity question in DH seems to be slice of the diversity problem in academia.

4. Finally, the last session I attended on Saturday was on the future of Anthologize. Boone Gorges demonstrated some new functionality that’s on its way, including the ability to keep comments with exports. It was also an interesting moment of checking on the development cycle for a piece of software that was conceived, designed, and originally implemented in just one week.

Speaking of Anthologize…

Dan Cohen and Tom Scheinfeldt have been using Anthologize to construct the edited version of Hacking the Academy, a crowd-sourced book that was produced in a week last year. Well, the book wasn’t produced in a week, but the original site that aggregated the pieces did so in a week. Almost two hundred scholars submitted pieces – some short, some long – to that site. Tom and Dan selected and edited pieces for the final book, which will be published by the University of Michigan in both open access digital form, and in a bound volume. I’m happy to say that my piece made the final cut, and has been combined with one my Mark Sample for the Hacking Scholarship section of the book. Our chapter will be called “Voices: Sharing One’s Research.” I’m looking forward to seeing its final form. The table of contents of the new book is available here.

Edited to add: A whole bunch of collaborative google docs from the various sessions can be found here.

Brought to you by — Tom Scheinfeldt

There’s also a collaborative google doc here.

“Project Management is a low tech business. It’s almost entirely about people.”

10 things project managers are asked to do

[TOC]

1. Picking a project.

Interest is important, but it’s not enough.

Is it fundable?
* Projects don’t happen without funding. Or shouldn’t. Pick projects that are fundable from some identifiable source external or internal.

If it’s not fundable, is there a way to modify to make it fundable?
Audience, time table, technology, etc. –> can these be adjusted to make it more attractive?
Flexibility.

Do I have the capacity to do the project?
Money’s not enough.
Expertise, infrastructure, time, etc.

Do I know people who have the capacity? –> collaboration external to home department/institute/etc.

2. Build partnerships.

Content partnerships, expertise partnerships, etc.

“Strong partnerships build trust amongst audiences.”

Reputation ties.

Partners don’t have to be big, but don’t sell yourself short. Shoot big! Go for the person, place, etc. that you would want most.

Be careful in partnerships to manage expectations from the very beginning. Make it clear.

3. Fund projects.

90% of CHNM funding is grant money.

Read the guidelines + Follow the guidelines

No, really. Do those two things. The only secrets.

4. Setting budgets.

What goes into a digital humanities budget?

99% of the budget is about labor. Seriously. The whole budget should be staff time. That’s what they expect to see in budgets.

Ask for the maximum.

What can I promise based on that amount of money?

Under promise. Over deliver.

take something manageable. Make it sound bigger. Over deliver.

5. Staff the project.

Dependent on skills needed to pull of a particular project.

Figure out what you do well, find projects that fit your skills. Most projects aren’t full covered. So, you’ll need to find people.

Look for people who have a proven track record of being able to learn things quickly.
Look for people who have a proven track record of finishing tasks.
Look for people who have an ability to work on a team. Personality absolutely matters.

As a project manager, one of your main jobs is to shield the staff from the headache of project management, shield the staff from your job.

“A quick word about meetings….”

Individual meetings are much more important than whole project meetings.
What are you working on?
What are you working on next?
* Are there obstacles to getting these things done?

Communication with staff, especially when corrective info is need, should be done at a scheduled meeting. A scheduled meeting is more like constructive criticism.

6. Developing a workplan.

Meh.

OK, so you’ll write work plans for grant proposals. Know that it won’t likely survive.

More important:

Having a couple of key deliverables.

Outline 4 deliverables, then give them 5.

Quick word on PM software — you need a collaborative space. But, start lightweight and get more elaborate only if you need it.

Start, say, with a Google Doc. Go more complex as you need to– basecamp?

7. Manage/report to funders.

Meet your deadlines on reporting. Do it on time.

Reflect early and often on the work. Don’t wait till the end of the project, if that’s when you have to report.

Make notes along the way.

8. Publicize the project.

Need for publicity can’t be overstated.

Think about it. Do it. And not just at project launch, but at every stage of the project cycle. Have a project blog. Write substantive blog posts.

Give conference presentations.

Social media.

engage an audience.

Swag. Why not? T-shirts, stickers. Especially with cafe press. There is buy-in with swag.

9. Sustain the project.

Have it mean something to a community of people.

No easy answer here.

Funders want to see a detailed sustainability plan. You might not have an answer, but thinking about it and having a set of answers is very important. At least they want to see that you’ve been thinking about it.

And again, this is a reason why partnerships are important.

Scale? 5 year business, 10 year business….

Find a partner who is in the 50 year business? Whose job it is to think about the 50 year question.

Remember, you may not own it in 5 years. Build from the beginning that it might not be your own baby forever.

Also, continued funding won’t be to do what you’ve been doing. But, you could get funding to go in a new direction.

Revenue models in keeping with your values.

10. Lead!

That’s your job– you are the leader. Take the lead. Be the leader you need to be.

Cultivate relationships with stake holders. Know your institution’s procedures. Take an interest and leadership role in elements that will affect your ability to problem solve.

Know how to respond to user inquiries. Know when to say yes, when to say know.

Leadership –> that’s your job.

“People will forgive bad decisions, but not indecision.”

The instinct amongst academics is to take time to understand and get to decision. Not the way of project management.

Deliverables… Leadership… the MBA of DH Projects.

as @miriamkp tweeted it: “The best collaborations are about shared doing, not shared decision-making.”

For what it’s worth, this was a great session.

Notes on Intro to Omeka Session.

Omeka self-install vs. Omeka.net is essentially the same distinction as with wordpress.org and wordpress.com.

The Omeka Codex includes Site Planning Tips.

Omeka.net would be great for student projects.

If you have an facsimile of a letter and a transcription, is that one item or two?

Depends — if the letter is the item, then the facsimile and the transcription are parts of the item, rather than separate items.

Omeka has CSV importing, that allows you to map columns from a csv file to item types, elements, tags, or files.

Customizable output formats — atom, json, omeka-xml, rss-2 — for export. Issue of long-term sustainability.

Adding items.

Dublin-Core has 20 basic fields. And there are extended fields as well.

Also, 12 or 13 item types built-in. Best to use those, but it’s all customizable.

Item types are made of elements, and you can add elements to the basic types to customize them as well.

An item can belong to only 1 collection, but to endless numbers of exhibits.

Collections.

Omeka was modeled on museums. I think it’s interesting to see how this initial mindset impacted the ontology of the system– which goes beyond the notion of item, collection, exhibit I think. Might make for an interesting discussion on how stage-one decisions significantly shape downstream realization of a project for a methods/historiography class.

Sheila comments that though Omeka designed for museums, they’re finding that very few museums seem to be using it. Hmmm. Wonder why.

Geolocation plugin — literally mapping items.

Exhibits.

Exhibits are collections of items.

Exhibits have sections, which are made up of pages. The linked page just above has a nice diagram explaning this.

You don’t have to use collections. But, when using a CSV import the decision is all or nothing, just as making items public at that moment is all or nothing.

What about citation? Important question for use with students, but also for projects. It’s a sticky process because of the DCMI model for all items.

User rights — super, admin, contributer, researcher

Getting historians to think like curators, and blend that with narration makes audience and outreach/public orientation core concern for constructing publication. That’s an interesting element for rethinking academic communication.

Patrick Murray-John

Intro to CMS

-Principle of modularity. What CMSs do is connect ‘buckets’ of data (header, contents, main content, plugin content, etc.)

-Where they differ isn’t in how they “look”, but how they construct their modularity.

CMS designed for extension.

The job of theme is to filter the buckets onto the page in attractive ways.

umwgeography — a good example for utkhistory.org.

In fact, I like UMW Geogrpahy better than UMW History/American Studies

Raf Alvarado

Raf’s principle: “In general, if you can use WordPress, then you should use WordPress.”

Media
-Digital Media
–Hypermedia
—Websites
—-Database-driven Websties
—-Content Management Systems (CMSes)

Content Model:
data model — database
emergent information architecture
this allows site builders to represent and organize content — eg source material in teh form fo images, text, and metadata
Content is complex — media objects (ef image and detail, book and page), technical metadata, semantic metadata

How much complexity to model?

CMSes content modeling affordances:
Elementary units (pages, nodes)
Connective devices (tags, categories)
* Display methods (widgets, feeds)

Drupal calls pages nodes

Combined, the three produce patterns.

In the hypertext world, the elementary unit is the page and the connective device is the link. No display devices for HTML, just pages.

Emergent structure? happy chaos.

WordPress:

Pages –> Subpages –> [flyout menues, menu widgets]

Posts related to others through tags and categories, as opposed to subpages

Tags –> Tag widgets, list pages
Categories –> Category widget, list pages

Content accessed by browsing, menus, and widgets.

Clay Shirky, “Ontology is Overrated”

WP results in “Dendritic Hypertext” — a dual structure, both hierarchical and rhizomic

Contetn MOdeling in WP

Think of course modeling —

Pages = Syllaubs Items
Posts = Lessons resources
Categories = Course Schedule
Tags = Resource subject and format

Thinking “what are we mapping things to?”

categories have parents and children, they’re hierarchical. Tags don’t.

I need to restructure my categories on parezcoydigo

Omeka has ITEMS

ITEMS
–COLLECTIONS
–DC metadata
–Keywords
–Tags –> Browsable list pages

Content is accessed by Browsing and EXHIBITS

Results = “Axial Hypertext” (term borrowed from Landow, Hypertext)

Allows for sequential hypertext, or sequential content model

You want the reader to go through a sequence — allowing the scholar to construct a sequential argument.

This is done in part by constructing exhibits.

Drupal

NODES
–Menu paths –> Menus, blocks
–Arbitrary fields –> Views
–Node references –> LInked Views, backrefs
–Taxonomies –> List pages, Views
–Outlines –> Books

Content si accessed by Browsing, Views, Books, Panels, and other devices.

Drupal content models are like Relational Models — results in “Rhizomic Hypertext”

Nice slide demonstrating the content models of the three biggies.

Top level cuts:

Specific vs. general content modeling
* WordPress is general

Manual vs. Autamatic site organization
* Omeka is manual

Paradigmatic vs. Syntagmatic affordances
* Drupal is highly paradigmatic

WordPress
* Course sites, personal portfolios, blogs (of course)

Omeka
* Exhibits, simple thematic research collections

Drupal
* Collaborative projects, complex thematic research collections, data meshing

Some Drupal examples:

A House Divided –
implementation, using nodes to organize content.

Civil Rights Movement as Theological Drama — UVA site
Interviews as Actors
Documents
* Scenes, etc.

In New Mexico, though, the wind is an ever-present companion. It reveals not discontent, not unrest, but rather endurance. Persistence. It lays bare the desert sandstone, the tops of mesas, the dry backs of not-quit green enough tree leaves. On many days, to look out to the horizon is to see a low hanging cloud of dust. It creeps into the house through minute cracks in window sills. It’s a fine sand, almost invisible; it becomes invisible in ubiquity. But it’s there, in a blackened sneeze lest you forget. That low hanging stratum simply moves remnants from place to place, remixing the layers of a persistent past with those of a tentative present. The prevailing winds in Albuquerque are westerly, or southerly. Except when they roar from the east through the canyon that divides the Sandia and Manzano Mountains. When Costco moved to Albuquerque’s southeast, they built the entrance facing southwest, away from that canyon and towards Kirtland Air Force Base and a plot of land that eventually became the third (and final?) home of the National Museum of Nuclear Science and History. On one particularly windy visit, I asked the beleaguered card checker how she was handling the wind in the large, always open doorway. She said it was wearing, but that the company had studied the wind and chosen that orientation as the least likely to face it. And that’s true, except for every day when it’s not coming from the canyon, which is most.

To get here, the winds frequently blow across the mesa, through the volcanic boulders on the escarpment’s face that were pounded by ancient artists to frame fascinating glyphs. It roars across the tops of cottonwood stands that line the banks of the Rio Bravo del Norte, across the railroad tracks that gave Albuquerque its first taste of the particular form of future bearing optimism that’s characteristic of modernity, and back up the hill to my house. Or, if they come from the south, they blow over Tome Hill, a site of more petroglyphs and of Good Friday pilgrimages.

The Rio Bravo del Norte, or Rio Grande I should say, once flowed more to the east to the base of Tome Hill, and alongside was the old Camino Real. The Spanish marked passage by Tome as they fled the Pueblo Rebellion in 1682, and also their return at the end of the century. The middle Rio Grande valley is green, extending outward a bit from both banks of the river, with fields and stands of trees. In winter, giant flocks of birds leisure the fields each day before returning to their aquatic refuge at Bosque del Apache. Neither is hard to find, as green and water are minority landscapes here.

There are three places I like to run on a regular basis in Albuquerque– on single track trails along the rio in the bosque, around USS Bullhead Memorial Park the the VA hospital, and in the Sandias. Most days its to Bullhead, a series of soccer and softball fields and dirt trails tucked behind the VA Hospital and a second hospital building that was once the clinic of Dr. William Randolph Lovelace II. Lovelace was a decorated WWII veteran, former assistant to Dr. Charles Mayo, and important figure in aerospace medicine. His former house sits across Ridgecrest Drive from the maze of trails at the eastern end of Bullhead. Ridgecrest is a strange street for Albuquerque, a diagonal in a city of squares. And it ends, almost as if pointing, to this spot. Lovelace worked with NASA and in January and February in 1959, the Lovelace Foundation ran a series of physical and mental tests on 33 pilots hoping to make the final cut of 7 who were the core of the Mercury Space Program. The Lovelace Foundation, and the nearby VA Hospital were built in neo-pueblo style, an aesthetic yearning back.

To get to the VA and Bullhead park, I run south to “Cruising San Mateo I,” better known around here as Chevy-on-a-stick. The sculpture was erected in 1991, and its turquoise and blue tile work, its title, evoke mid-century Albuquerque in the heyday of Route 66.

Through the arch I run between the Hospitals to my favorite dirt trails. USS Bullhead Park is named for a WWII submarine, one of the last to be sunk. In a cruel twist of that highlights the extent to which modern NM’s recent past is patterned by its own desolation and isolation, the USS Bullhead was sunk on or shortly after August 6th, 1945, the day the atomic bomb was dropped on Hiroshima. Of course, much of the development of the bomb was done in Los Alamos and the first live test explosion in southern NM at Trinity site on “the day the sun rose twice.”[1] There is little beyond the name to make the connection to the submarine, or to the unsettling days of August 1945. (Were the winds in Ecuador howling that day?) My trails at the park mark a labyrinth of sun-drenched, unprotected, wind-blown maze. There are no trees here, and lying next to manicured fields of irrigated grass, the running trails speak to reversion of condition. Left alone, Albuquerque turns to desert again. Turns to its past. There are cacti, tumbleweeds, sage, road runners, rabbits, and snakes. When we do it to our yards here, we call it xeriscaping. Left alone, the dry and the wind will do it for you.

Ultimately, mine is just a normal neighborhood run. Every place has its story. But, when I’m out there and the wind is scouring my face at 25mph, and I look around at landmarks and landscapes laid bare by that wind, the sense of place and persistence that is New Mexico to me distracts just enough from the suffering of running to put me both in the moment and and back beyond it. And then, I grit my teeth on the grains that snuck between them and run some more.

[1]: Ferenc Szasz, The Day the Sun Rose Twice: The Story of the Trinity Site Explosion (Albuquerque: University of New Mexico Press, 1984). Frank Szasz passed away on June 20, 2010. He was a fantastic human, and taught the graduate foundations seminar when I arrived at UNM in 1997.

Both of these tasks — resizing images and switching formats — are easy to accomplish using the Python Imaging Library. Getting PIL up and running on Snow Leopard, not so much. I first installed using easy_install, and everything seemed to be working fine except for jpeg encoding/decoding. I apparently didn’t have libjpg installed. So, I used macports to install the library with $ sudo port install jpeg. After that, I needed to reinstall PIL. I uninstalled the library using pip (I don’t know why I did it that way instead of with easy_install.) Then, I reinstalled PIL with pip and, low and behold, jpeg encoding was available and fine, but PIL’s _imaging C libraries stopped working. Grrr. I tried tons of ways around this, but simply couldn’t figure out one that worked.

What finally worked was a two-step process in Emotu Balogun’s comment on the top rated answer to this stackoverflow question. Apparently the problem has to do with incompatibilities of the gcc version, libjpeg, and PIL. At any rate, this worked for me:

1. Reinstall libjpeg with Macports and adding the +universal argument to the command: $ sudo port install jpeg +universal.

2. Install PIL with Macports in the same manner: $ sudo port install py26-pil +universal.

3. I use Snow Leopard’s default 2.6.1 python installation, so I copied the PIL package from Macports’ site-packages folder in /opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages to Snow Leopard’s default site-packages library.

After doing that, PIL is working. Worth it? Not so sure. But, with PIL I can convert all of my images for this blog to .png files, and also resize the images to upload one that will fit in the dimensions of my posts.

To post this piece, I pull up dterm and enter post dterm_popup_cli.txt.

Pretty sweet.