Using devonthink for historical research: Constructing the Database
DevonThink Pro Office (DTPO) has become my research database of choice. In the Mac world, particularly with the advent of Spotlight which allows for full text searching of everything on your hard drive, one may ask why bother with a database program. I find, however, that Spotlight searches become quickly ponderous. With a certain amount of discipline, it is possible to refine Spotlight through the use of keyword tags. To do this, you would tag each file with a number of relevant keywords attached to a symbol (@ or * or &) so that later, when searching for related notes keyword searches are possible. The addition of the symbol (ie, @Quito) would limit the returns to files that have been appropriately tagged rather than any file that includes the word within the text. The comment field is accessed through the more-info option whenever a file is highlighted in the finder, as any Mac user would be familiar with. There are also other apps you can download to help with the tagging process.
I’m not that disciplined.
Before getting more specifically into DTPO, let me note that there are a number of other excellent note-taking and research database programs out there that might be better suited to the way you think, like to take notes, and organize your data. Circus Ponies Notebook is well regarded. Mind-mapping applications like NovaMind… MacJournaler or other journaling apps can be adapted for note taking… There’s Yojimbo, Voodoo Pad, Soho Notes, Omnioutliner, the more expensive Tinderbox for crazy hypertext note taking… The list goes on and on, and even something like Zotero which is primarily intended as a reference manager can be used for some types of note taking. I need something more powerful than a simple note taking tool, though. And this is where DTPO really excels.
What I like about DTPO is its ability to organize information in a hierarchical notetree while also maintaining the flexibility to organize and access the information in freer form. Some people have no need for hierarchy, the but information anarchism doesn’t work for me. I like to visualize my sources in a way that reproduces their organization in the archive, for the primary stuff, as well as the thematic and narrative points they will later support.
This is the database that supports my current research, though it is not complete. Much of the info that worked into the late colonial part of the project hasn’t made it into DTPO at this point, and may never!
As you can see, the tree contains root folders titled Inbox, Incoming (redundant/feeling my way around), Tasks, Notes (short notes not related to specific sources, but related to the project), Article Manuscripts, Book Project, Theory/Gender/Sexuality, Padrones (for extensive jail census), My Writings (dissertation chapters, conference papers, manuscript chapters, etc.) and a few other largely unused bits.
Book Project is divided chronologically into the Bourbon Period and the Independence/National Period, which are further subdivided by primary and secondary sources both printed and manuscript. Whenever possible, I will embed the actual source in the database as well. With relatively low-cost/high-volume storage available, it would really be possible to store the photos of each of my case files with the notes as well. Right now, however, this database is on my Macbook- so not enough storage there!
I further subdivide based on the physical location of documentary sources– AGI, by section or AGN by section, then by box, then by date or other organizing tag within the box. I now try to do all or most of my notetaking directly into devonthink, in the form of rtf files inside the note tree. In Screenshot 2 you can the transcription and notes taken for a series of letters titled Memoria de la Revolución de Quito en cinco cartas escritas a un amigo, octubre 1809. I like to have DTPO in 3-pane mode, so as to see the tree, the contents of an individual folder, and a note-taking space. I take most of my notes now directly in that third pane. When taking a note, I enter the title on the title pane, and then replicate that title at the top of the note, which becomes important for export down the road. (Yes, I export my notes still, both for redundancy purposes and because I haven’t yet made the full jump to paperless workflow). Notes seem to work best when limited to no more than about 1000 words. Shorter actually works better when later using DTPO’s search and classification functions.
It is at this point that Bookends comes into play. Though Bookends as a bibliographic manager plays pretty well with DTPO, Scrivener, Mellel, Pages, and Word- the apps I’m most likely to use– it is not the only option. Other popular options include Sente, EndNote, and Zotero (players big and small).
For any given source, I have an entry in Bookends, which produces a unique source number for the entry. I do two things in linking Bookends to DTPO — first, I will almost always add a hot link to the source root folder in DTPO to the entry in Bookends. More importantly, though, for rearranging information later, I insert in at the end of each note a short citation from Bookends that is generated by the program and indicates the author, year, and database #. I also add the relevant page #s for the note. It looks something like this: {Andrade, 1934, #5906@419-420}. When inserted into the text of a manuscript, Bookends can recognize this notation and replace it with a properly formatted citation. For me, though, the notation acts as an anchor for when the notes become disconnected from their position in the tree.
I don’t keyword tag each entry or folder. This is possible in the current version of devonthink through a work around that isn’t exactly a keywording function. You just use the comment field for an entry, and best practice would be to to reproduce the system I described above for spotlight. I don’t bother to do this because devonthink has the best, most powerful search functions of any application I’ve ever seen, allowing for fuzzy word searches, phrase searching, and much more sophisticated searches of blocks of text for linguistically similar other blocks of text within your database. More on that in another entry.
For now, back to constructing the database.
DTPO offers the option to link any note file to any other note file in the database, or to link any term to another note. Thus, you can grow, over time, a wiki or glossary of important terms, concepts, or people that links to central entries on those things. Because the data doesn’t have to be hierarchically anchored, you can set up a separate folder dedicated to these explanatory notes. Likewise, DTPO offers the option to replicate or duplicate files and store them in multiple positions within the database. This enhances the organizational flexibility of the program, enabling a combination of storage options. Once a wiki is set up on a name or term or concept, it seems DTPO automatically generates a link to that file, which is convenient.
For secondary sources, especially pdf files imported from sites like jstor or from your own scanner, DPO offers a couple of nice features. For files that don’t come with a pre-embeded text layer, DTPO will OCR (optical character recognition) pdfs to make them text searchable. These pdfs can then be read in preview or skim or adobe and marked up with notes, highlighters, underlines, and the like. Blocks of selected text can be dragged into DTPO. The next version of the program is supposed to integrate with Skim, which will make this type of notetaking even more powerful. Otherwise, I treat note files in secondary folders the same as I do in primary folders, linked to Bookends.
As far as it goes, I’m one of those people who prefers to quote in full a source in my notes rather than to summarize. Whenever I take a note that is wholly my own, or that is commentary on a source quote, I try to change the font color just to be clear. This paranoia goes back to a particularly adamant professor in undergrad at Appalachian State University who taught the Civil War and had a reputation for sitting in the library, looking up citations in term papers. I hear he would also do this with job candidate’s dissertations with any referenced source he could get his hands on.
So, there is the process I engage in for building the database. Next time I will look at using DTPO’s search, classification, and analysis tools to manipulate the data, as well as note export to Scrivener in the move towards writing.
One last note– Devon Technologies is currently beta testing a long overdue update to its devonthink application. DevonThink 2.0 will add, I hear, significant new integration with Apple’s quicklook to support more file types, the option to work on two databases simultaneously, new user-friendly GI, tagging, and more. So it may be that even as I write this series of posts, I’m already an anachronism.





November 25, 2008 at 8:49 am
Hi (maybe must I begin with the most traditionnal “My dear colleague”)
I’m glad to see I’m not the only one historian to use DT with great benefits.
It’s nice to share your daily experience with it. Thanks !
BGT
P.S. Please excuse my poor english, I’m french and as everybody know, french speaks… french.
December 1, 2008 at 5:38 am
Chad wrote: “DTPO offers the option to link any note file to any other note file in the database, or to link any term to another note. Thus, you can grow, over time, a wiki or glossary of important terms, concepts, or people that links to central entries on those things.”
This is good to know. I have been looking at DTPO with some cursory interest, but this might cement my inclination to organize my research. Scrivener does a great job within a project, but I think DTPO might be more useful across several interconnected projects.
December 17, 2008 at 4:23 am
Hi,
Much thanks for this. As I am in the near end of my dissertation (Andean archaeologist here), I am still trying to figure out how to organize my research and writing. I am a convert to scrivener, and am also enjoying Things as a to do list program. I am still with endnote, unfortunately. I am trying Devonthink at the moment — still in the very early stages. I wonder if you import all your pdfs into devonthink, or if you simply index them. I have a huge number on an external drive, but am not sure (yet) how DT can best deal with them, and I can benefit from DT system. Any ideas? But again, thanks for this overview…
December 17, 2008 at 10:16 am
Hi Andy-
I guess what to do with the pdfs depends on your internal storage capacity and whether or not they need to be OCR’d. If they already have a text layer, indexing would be fine. I have a different database for all my course materials that is totally an index, and which I sync on a regular basis. If the pdfs need to be OCR’d, you have to import them anyway into DTPO so you might as well store them there. If you use a different OCR engine, then again indexing would work. I like to have the pdf in the folder where I have the notes I took on it, and I like to put those bits in the note tree in places that make sense to the organization of my research.
Actually, this won’t be an issue in a few months when DT2.0 comes out of beta. As I understand it, all documents will be stored outside of the database, and you will be able to edit them in their original form. So, it will be like indexing plus whatever metadata is applied inside DT.
One last comment– I waited till after my dissertation to make any real changes in my workflow. So that meant using Scholar’s Aid as my notetaking and bibliographic program together with loads of transcriptions in Word. I printed out every note and case for a given section/chapter and put them in three-ring binders. I had an outline pointed me to tabbed sections of those notebooks. It was cumbersome, but effective and it got the job done.
December 17, 2008 at 2:52 pm
Hi again,
Yes, I have read about some of the possibilities of DT2.0. Perhaps I will wait to see what happens there. I have a certain anxiety about the growing pdf mess on my external drive, but will not really address it until the diss is finished. I am also not really thinking of changing my workflow at this point. Happily most of the research is done and I am either in Scrivener or the dreaded Word. Thanks again for the overview of your workflow.
December 17, 2008 at 2:59 pm
Andy– Have you thought about a program like Yep (www.ironicsoftware.com) for organizing your pdfs? With Yep you can keyword tag each one, and find things pretty easily. With the increased direction towards electronic publishing, what to do with pdfs becomes and increasing problem. Some people I know store all their pdfs in Bookends with the citations.
Hey, good luck in finishing and defending. Best piece of advice I ever got on the diss front– the best dissertation is a done dissertation! By the way, what area of Andean archaeology do you work in?
December 17, 2008 at 6:35 pm
Funny you should ask — I am giving yep a whirl at the moment. (There is something incredibly ironic about procrastinating by researching productivity tools). Of course in the process of playing around with it I have found a large number of duplicates on my machine, primarily due to imap gmail issues in mac mail. I may look into storing in endnote, as I can do that too. Thanks for the well wishes — I work on Late Formative (200BC-AD200) sites in the Lake Titicaca Basin of Bolivia/Peru. I do have interests in historic/colonial archaeology of the region as well.
December 18, 2008 at 6:43 pm
Having now looked at your other website, and learned more about your reserach, I believe we may know a few of the same people — Mary Weismantel, or Blanca Muratorio?
BTW, looks like Devon 2.0 is now available.
December 18, 2008 at 11:55 pm
[...] Devonthink for Historical Research, part II [...]
December 22, 2008 at 11:26 am
Circus Pony new Notebook version has many problems . Aquaminds Notetaker is more versatile with Applescript support, applet plugins, library tab. It has also better sharing options . CP’s is much more beautiful with less features. I struggle between both until I decided both are superceded by a more flexible DTPO . NT remains useful perhaps for other things, a travel planner or collecting a single subject info per book, but none have the flexibility and power of Devon .
February 2, 2009 at 4:10 pm
I use ‘Papers’ to store my PDFs; it’s got integrated searches on the web (and JSTOR), and indexes etc. But I’m currently wondering about an overall app to organise research and writing as well.
http://mekentosj.com/papers/
February 2, 2009 at 10:25 pm
I gave Papers a brief shot a while back and soon gave up as it seemed to be centered exclusively on scientific research. All my research is in literature (most journals I use are not yet available as PDF for download). My PDF collection is mostly scanned from current or older journals I get through inter-libary loan.
Oliver, I’m curious: how has your experience been? Are you in the humanities?
February 15, 2009 at 1:18 pm
Thank you for your helpful guide to organizing research with DevonThink. I’d like to ask for details about how you get the short citation note out of Bookends. If I drag a reference, it includes the author, year, number of pages, but not the database number. I assume you add the page number of the citation manually once the bracketed info comes into DT.
February 15, 2009 at 1:20 pm
Nevermind. I found it in the Bookends Preference pane.
February 15, 2009 at 1:56 pm
Just in case anyone else is interested– On the Bookends Preferences pane, under the Scan and Bib tab, I chose {} as the citation delimiters, and Author, Date, Unique ID from the Cite By drop-down menu.
Here’s a screen shot.
To insert the citation, I link Bookends to Devonthink, and when an entry is highlighted in Bookends, press ‘Shift-Cmd-Y’, which copies the short citation, takes me back to Devonthink (or Scrivener, or whatever app is linked to Bookends), and inserts the short citation. Then, yes, I add the page numbers, @2-3 for example.
March 1, 2009 at 12:28 pm
Another question for you. Do you store the pdf in Bookends, in Devonthink, or outside of them in the standard file structure (where they are then indexed by DT)?
Thanks again.
March 1, 2009 at 1:16 pm
Hi Chad- for my main research database, I store pdfs in the database. For my courses, I keep them in the file structure, and index the folders. I’m going to convert that to a standalone database before the fall that I can sync to dropbox and have available at any of my computers. I don’t store anything in Bookends, except citations.
I also use hazel to move pdfs out of my download folder and into a special pdf folder, which I periodically clean out, renaming the files and moving them to permanent home, etc.