Using devonthink for historical research: Constructing the Database
DevonThink Pro Office (DTPO) has become my research database of choice. In the Mac world, particularly with the advent of Spotlight which allows for full text searching of everything on your hard drive, one may ask why bother with a database program. I find, however, that Spotlight searches become quickly ponderous. With a certain amount of discipline, it is possible to refine Spotlight through the use of keyword tags. To do this, you would tag each file with a number of relevant keywords attached to a symbol (@ or * or &) so that later, when searching for related notes keyword searches are possible. The addition of the symbol (ie, @Quito) would limit the returns to files that have been appropriately tagged rather than any file that includes the word within the text. The comment field is accessed through the more-info option whenever a file is highlighted in the finder, as any Mac user would be familiar with. There are also other apps you can download to help with the tagging process.
I’m not that disciplined.
Before getting more specifically into DTPO, let me note that there are a number of other excellent note-taking and research database programs out there that might be better suited to the way you think, like to take notes, and organize your data. Circus Ponies Notebook is well regarded. Mind-mapping applications like NovaMind… MacJournaler or other journaling apps can be adapted for note taking… There’s Yojimbo, Voodoo Pad, Soho Notes, Omnioutliner, the more expensive Tinderbox for crazy hypertext note taking… The list goes on and on, and even something like Zotero which is primarily intended as a reference manager can be used for some types of note taking. I need something more powerful than a simple note taking tool, though. And this is where DTPO really excels.
What I like about DTPO is its ability to organize information in a hierarchical notetree while also maintaining the flexibility to organize and access the information in freer form. Some people have no need for hierarchy, the but information anarchism doesn’t work for me. I like to visualize my sources in a way that reproduces their organization in the archive, for the primary stuff, as well as the thematic and narrative points they will later support.
This is the database that supports my current research, though it is not complete. Much of the info that worked into the late colonial part of the project hasn’t made it into DTPO at this point, and may never!
As you can see, the tree contains root folders titled Inbox, Incoming (redundant/feeling my way around), Tasks, Notes (short notes not related to specific sources, but related to the project), Article Manuscripts, Book Project, Theory/Gender/Sexuality, Padrones (for extensive jail census), My Writings (dissertation chapters, conference papers, manuscript chapters, etc.) and a few other largely unused bits.
Book Project is divided chronologically into the Bourbon Period and the Independence/National Period, which are further subdivided by primary and secondary sources both printed and manuscript. Whenever possible, I will embed the actual source in the database as well. With relatively low-cost/high-volume storage available, it would really be possible to store the photos of each of my case files with the notes as well. Right now, however, this database is on my Macbook- so not enough storage there!
I further subdivide based on the physical location of documentary sources– AGI, by section or AGN by section, then by box, then by date or other organizing tag within the box. I now try to do all or most of my notetaking directly into devonthink, in the form of rtf files inside the note tree. In Screenshot 2 you can the transcription and notes taken for a series of letters titled Memoria de la Revolución de Quito en cinco cartas escritas a un amigo, octubre 1809. I like to have DTPO in 3-pane mode, so as to see the tree, the contents of an individual folder, and a note-taking space. I take most of my notes now directly in that third pane. When taking a note, I enter the title on the title pane, and then replicate that title at the top of the note, which becomes important for export down the road. (Yes, I export my notes still, both for redundancy purposes and because I haven’t yet made the full jump to paperless workflow). Notes seem to work best when limited to no more than about 1000 words. Shorter actually works better when later using DTPO’s search and classification functions.
It is at this point that Bookends comes into play. Though Bookends as a bibliographic manager plays pretty well with DTPO, Scrivener, Mellel, Pages, and Word- the apps I’m most likely to use– it is not the only option. Other popular options include Sente, EndNote, and Zotero (players big and small).
For any given source, I have an entry in Bookends, which produces a unique source number for the entry. I do two things in linking Bookends to DTPO — first, I will almost always add a hot link to the source root folder in DTPO to the entry in Bookends. More importantly, though, for rearranging information later, I insert in at the end of each note a short citation from Bookends that is generated by the program and indicates the author, year, and database #. I also add the relevant page #s for the note. It looks something like this: {Andrade, 1934, #5906@419-420}. When inserted into the text of a manuscript, Bookends can recognize this notation and replace it with a properly formatted citation. For me, though, the notation acts as an anchor for when the notes become disconnected from their position in the tree.
I don’t keyword tag each entry or folder. This is possible in the current version of devonthink through a work around that isn’t exactly a keywording function. You just use the comment field for an entry, and best practice would be to to reproduce the system I described above for spotlight. I don’t bother to do this because devonthink has the best, most powerful search functions of any application I’ve ever seen, allowing for fuzzy word searches, phrase searching, and much more sophisticated searches of blocks of text for linguistically similar other blocks of text within your database. More on that in another entry.
For now, back to constructing the database.
DTPO offers the option to link any note file to any other note file in the database, or to link any term to another note. Thus, you can grow, over time, a wiki or glossary of important terms, concepts, or people that links to central entries on those things. Because the data doesn’t have to be hierarchically anchored, you can set up a separate folder dedicated to these explanatory notes. Likewise, DTPO offers the option to replicate or duplicate files and store them in multiple positions within the database. This enhances the organizational flexibility of the program, enabling a combination of storage options. Once a wiki is set up on a name or term or concept, it seems DTPO automatically generates a link to that file, which is convenient.
For secondary sources, especially pdf files imported from sites like jstor or from your own scanner, DPO offers a couple of nice features. For files that don’t come with a pre-embeded text layer, DTPO will OCR (optical character recognition) pdfs to make them text searchable. These pdfs can then be read in preview or skim or adobe and marked up with notes, highlighters, underlines, and the like. Blocks of selected text can be dragged into DTPO. The next version of the program is supposed to integrate with Skim, which will make this type of notetaking even more powerful. Otherwise, I treat note files in secondary folders the same as I do in primary folders, linked to Bookends.
As far as it goes, I’m one of those people who prefers to quote in full a source in my notes rather than to summarize. Whenever I take a note that is wholly my own, or that is commentary on a source quote, I try to change the font color just to be clear. This paranoia goes back to a particularly adamant professor in undergrad at Appalachian State University who taught the Civil War and had a reputation for sitting in the library, looking up citations in term papers. I hear he would also do this with job candidate’s dissertations with any referenced source he could get his hands on.
So, there is the process I engage in for building the database. Next time I will look at using DTPO’s search, classification, and analysis tools to manipulate the data, as well as note export to Scrivener in the move towards writing.
One last note– Devon Technologies is currently beta testing a long overdue update to its devonthink application. DevonThink 2.0 will add, I hear, significant new integration with Apple’s quicklook to support more file types, the option to work on two databases simultaneously, new user-friendly GI, tagging, and more. So it may be that even as I write this series of posts, I’m already an anachronism.
Associate Professor of Early Latin America Department of History University of Tennessee-Knoxville
parezco y digo 
Hi (maybe must I begin with the most traditionnal “My dear colleague”)
I’m glad to see I’m not the only one historian to use DT with great benefits.
It’s nice to share your daily experience with it. Thanks !
BGT
P.S. Please excuse my poor english, I’m french and as everybody know, french speaks… french.
Chad wrote: “DTPO offers the option to link any note file to any other note file in the database, or to link any term to another note. Thus, you can grow, over time, a wiki or glossary of important terms, concepts, or people that links to central entries on those things.”
This is good to know. I have been looking at DTPO with some cursory interest, but this might cement my inclination to organize my research. Scrivener does a great job within a project, but I think DTPO might be more useful across several interconnected projects.
Hi,
Much thanks for this. As I am in the near end of my dissertation (Andean archaeologist here), I am still trying to figure out how to organize my research and writing. I am a convert to scrivener, and am also enjoying Things as a to do list program. I am still with endnote, unfortunately. I am trying Devonthink at the moment — still in the very early stages. I wonder if you import all your pdfs into devonthink, or if you simply index them. I have a huge number on an external drive, but am not sure (yet) how DT can best deal with them, and I can benefit from DT system. Any ideas? But again, thanks for this overview…
Hi Andy-
I guess what to do with the pdfs depends on your internal storage capacity and whether or not they need to be OCR’d. If they already have a text layer, indexing would be fine. I have a different database for all my course materials that is totally an index, and which I sync on a regular basis. If the pdfs need to be OCR’d, you have to import them anyway into DTPO so you might as well store them there. If you use a different OCR engine, then again indexing would work. I like to have the pdf in the folder where I have the notes I took on it, and I like to put those bits in the note tree in places that make sense to the organization of my research.
Actually, this won’t be an issue in a few months when DT2.0 comes out of beta. As I understand it, all documents will be stored outside of the database, and you will be able to edit them in their original form. So, it will be like indexing plus whatever metadata is applied inside DT.
One last comment– I waited till after my dissertation to make any real changes in my workflow. So that meant using Scholar’s Aid as my notetaking and bibliographic program together with loads of transcriptions in Word. I printed out every note and case for a given section/chapter and put them in three-ring binders. I had an outline pointed me to tabbed sections of those notebooks. It was cumbersome, but effective and it got the job done.
Hi again,
Yes, I have read about some of the possibilities of DT2.0. Perhaps I will wait to see what happens there. I have a certain anxiety about the growing pdf mess on my external drive, but will not really address it until the diss is finished. I am also not really thinking of changing my workflow at this point. Happily most of the research is done and I am either in Scrivener or the dreaded Word. Thanks again for the overview of your workflow.
Andy– Have you thought about a program like Yep (www.ironicsoftware.com) for organizing your pdfs? With Yep you can keyword tag each one, and find things pretty easily. With the increased direction towards electronic publishing, what to do with pdfs becomes and increasing problem. Some people I know store all their pdfs in Bookends with the citations.
Hey, good luck in finishing and defending. Best piece of advice I ever got on the diss front– the best dissertation is a done dissertation! By the way, what area of Andean archaeology do you work in?
Funny you should ask — I am giving yep a whirl at the moment. (There is something incredibly ironic about procrastinating by researching productivity tools). Of course in the process of playing around with it I have found a large number of duplicates on my machine, primarily due to imap gmail issues in mac mail. I may look into storing in endnote, as I can do that too. Thanks for the well wishes — I work on Late Formative (200BC-AD200) sites in the Lake Titicaca Basin of Bolivia/Peru. I do have interests in historic/colonial archaeology of the region as well.
Having now looked at your other website, and learned more about your reserach, I believe we may know a few of the same people — Mary Weismantel, or Blanca Muratorio?
BTW, looks like Devon 2.0 is now available.
[…] Devonthink for Historical Research, part II […]
Circus Pony new Notebook version has many problems . Aquaminds Notetaker is more versatile with Applescript support, applet plugins, library tab. It has also better sharing options . CP’s is much more beautiful with less features. I struggle between both until I decided both are superceded by a more flexible DTPO . NT remains useful perhaps for other things, a travel planner or collecting a single subject info per book, but none have the flexibility and power of Devon .
I use ‘Papers’ to store my PDFs; it’s got integrated searches on the web (and JSTOR), and indexes etc. But I’m currently wondering about an overall app to organise research and writing as well.
http://mekentosj.com/papers/
I gave Papers a brief shot a while back and soon gave up as it seemed to be centered exclusively on scientific research. All my research is in literature (most journals I use are not yet available as PDF for download). My PDF collection is mostly scanned from current or older journals I get through inter-libary loan.
Oliver, I’m curious: how has your experience been? Are you in the humanities?
Thank you for your helpful guide to organizing research with DevonThink. I’d like to ask for details about how you get the short citation note out of Bookends. If I drag a reference, it includes the author, year, number of pages, but not the database number. I assume you add the page number of the citation manually once the bracketed info comes into DT.
Nevermind. I found it in the Bookends Preference pane.
Just in case anyone else is interested– On the Bookends Preferences pane, under the Scan and Bib tab, I chose {} as the citation delimiters, and Author, Date, Unique ID from the Cite By drop-down menu.
Here’s a screen shot.
To insert the citation, I link Bookends to Devonthink, and when an entry is highlighted in Bookends, press ‘Shift-Cmd-Y’, which copies the short citation, takes me back to Devonthink (or Scrivener, or whatever app is linked to Bookends), and inserts the short citation. Then, yes, I add the page numbers, @2-3 for example.
Another question for you. Do you store the pdf in Bookends, in Devonthink, or outside of them in the standard file structure (where they are then indexed by DT)?
Thanks again.
Hi Chad- for my main research database, I store pdfs in the database. For my courses, I keep them in the file structure, and index the folders. I’m going to convert that to a standalone database before the fall that I can sync to dropbox and have available at any of my computers. I don’t store anything in Bookends, except citations.
I also use hazel to move pdfs out of my download folder and into a special pdf folder, which I periodically clean out, renaming the files and moving them to permanent home, etc.
Hi
great site. Do you sync your devonthink databases across macs using dropbox? If so any tips to avoid problems? I’m also a research scientist getting into DT.
Tx for any advice.
James
Hi James–
I have used Dropbox to sync databases across macs, in this case for courses I teach. One of those became corrupted and unrecoverable because I made the one mortal sin in keeping DT databases in dropbox– I inadvertently opened an already open database on two computers at once. That’s my tip NEVER EVER EVER EVER try to have a database open on more than one computer. In fact, I would suggest that you manually close each database before you close the application, as opposed to just closing DT.
Thanks. And if one doesn’t commit the mortal sin, have things run smoothly for you? I imaging you import rather than index – hence the need for syncing?
Thanks
Yes, I have generally imported. Although, I think that next semester I’ll probably try using an indexed dropbox folder for course files. Aside from the mortal sin, I haven’t experienced any problems with dropbox. I use it pretty extensively – and paid for extra storage. I imagine it would be a bad idea to try indexing while dropbox is updating. But, if you simply have mirror databases on different computers, and make sure to wait until dropbox is done updating before ever re-indexing, that should work.
Great blog about DevonThink in action for research. I am new to both DevonThink and Bookends. You mention creating a ‘hot link’ from DT to the reference in Bookends. How exactly is that done? I am a new grad student and have already accumulated a lot of research and want to start my use of both programs in the most efficient manner. I indexed my research folder on my mac with DT and want to attach files to references in Bookends, but don’t want duplicate copies of all my files on my drive. Your ‘hot link’ method sounds like it would solve my problem.
Thanks for any explanation you can provide.
Hi Cheryl–
I’ve actually stopped using Bookends since I wrote this post. That said, I think that Devonthink 2.0 has a script specifically for linking to a Bookends entry. What I used to do was to double click on a citation– you get a popup window– and then copy the ID# and put that at the end of the note. Or, and more likely, when taking notes on one source for a while, I would highlight that source in Bookends, use Cmd-K to copy a formatted short citation, and then just paste that at the end of every note. Ultimately, I abandoned Bookends as my citation manager in favor of zotero, which is soooooo much better at scraping citations off of the web– library sites, google scholar, google books, jstor, etc. And, i essentially just make sure the note has some kind of short citational reference in it. When I write, I generally like to manually do my footnotes much better than I like the process of entering citations with a manager that are later scanned and converted. That may simply be because that’s how I always did it. The other benefits of zotero are cloud storage of your entire library, syncing across computers in a seamless fashion, annotation of sources, the ability to easily publish bibliographies and participate in subject-based, or class-based groups, etc. I’m not going to abandoned DT for the simple note-taking features of zotero, but it’ the best (and free) citation software out there, IMO. Oh, and because it only runs as a firefox plugin, a browser I don’t use that much, I simply treat firefox as zotero.
[…] entries here and […]
Thanks so much for this helpful overview! Would you please say more about how you’re coming to use Zotero with DTPro? I started with Endnote, but find it cumbersome, plus the open source aspect of Zotero appeals to me. I’m starting a new project and want to build in good-habits, so I’m on the prowl for useful tidbits!
I’d also like to hear more about using Zotero and its relationship to DTPro. I’m still working in the windows world, but programs like DTPro and Scrivener have me thinking seriously about moving to a Mac. What do you find restrictive about Zotero for organizing research notes? I can see the advantage of a more customizeable and graphical program – Zotero’s note-management abilities are seeming increasingly limited.
There is no direct integration between Zotero and DT. That said, the Zotero team have just released an alpha version of stand-alone Zotero which a standard desktop app. It includes plugins to use Zotero with Chrome and with Safari. I would imagine that an enterprising individual could write a script to add bits to Zotero from DT, or vice versa– to import from Zotero to DT, but I haven’t looked very closely at it. Mainly, I’m not that concerned with it because while I find bibliographic programs nice for constructing bibliographies, and Zotero particularly nice for its ability to scrape info off of the web, when I write I like to manually construct my footnotes. I never used endnote, or any other program that integrates with a word processor. And, I’m increasing abandoning the wordprocessor altogether in favor of text editors.
All I want when I take a note is a short bit I recognize to tell me what the original source was. So, usually I’ll simply cut/paste the first time a parenthetical cite from Zotero over to whatever program I’m working in. Ultimately, my workflow changes from year to year, and as I’ve moved more and more into plain txt files, I’ve become less and less worried about the perfectly integrated solution.
Hi Chad, a question about transcriptions. When referring to a criminal suit do you transcribe the entire case into one RTF to TXT file in DT? I’m wondering about later on when I want to do searches, will it serve me better to break the transcription of the suit into various RTF files?
I’ve done a few different things in the past. I’ve maintained an “index” file, in which I have short entries for my cases, which I used to code the level and type of women’s participation. I toyed for a while with a case sheet, which was a form for each case that contained important information but not transcription. I also have tried the two different approaches you ask about– splitting the case into many individual notes/files, and transcribing the case in its entirety. I tend to prefer the latter myself, but in part that has to do with my desire to eventually share all of my transcriptions.
DT’s searching is plenty robust to find what you need regardless of the snippet size. It may be that if you have really long cases, the length would deteriorate the See Also functionality, but that hasn’t been my experience.
Finally, I’d also say you’re better off using plain text (.txt), and a simple markup syntax like multimarkdown than using .rtf. Rich text files are great, but there are more portability issues that there are with plain text files.
To be honest, in the past year I’ve moved increasingly away from devonTHINK until late stages of my research process. Now, I tend to do all the grunt work in a text editor, or the alternate version of Notational Velocity, and only index everything in devonTHINK at the end when I need the search capacity.
By the way, there are a number of great blogposts on digital workflow applicable to historians in this question on the DH Questions & Answers board.
Hi Chad,
I’m also a historian and regular DTPO user, so thanks for your blog. I’m looking to find a better workflow b/w DTPO and a bibliographic software and I’ve been playing around with Bookends and Zotero (I currently use Sente). I like that bookends will ‘leave’ the DT files in the DT and just create a pathway to them so that if you edit the file either in Bookends or DT it’s the SAME file. Can you do that with Zotero? I haven’t figured out if this is possible or not. If not, why did you switch?
Thanks, W
Hi W–
I’ve opted for Zotero for a couple of reasons, both pragmatic and philosophical. On the philosophical front, Zotero is open source software developed by the fantastic people at George Mason’s Center for History and the New Media. Whenever there is an open source alternative for a piece of software that offers the same functionality or fits my needs, I use it. In this case, I also find on the pragmatic front that Zotero is just better at the tasks I want out of a bibliographic application, and that is to be able to build bibliographies. In the 20-odd years I’ve been writing historical papers, I’ve never found the insert-citation, scan, replace w/full-note element of citation management software to be of much use to me. All I want is to know that I’m citing the right work. A few years ago, when i wrote this post, all I really wanted to do with Bookends was that– to put some kind of code in a note that pointed to the right citation. I never really cared about downstream integration with microsoft word, or the like.
I always, though, liked Zotero because it is just so good at scraping citations off of the web– the best I’ve experienced. And, if there is a site it doesn’t work with, it’s possible to write a small piece of code to make it work. I like that too. I held back for a long time from really committing to Zotero, though, because it was tied to firefox, a browser I’ve never really liked. Not anymore. Zotero is now standalone, and offers its scraping powers to Safari, Firefox, or Chrome (my browser of choice). Finally, Zotero added online syncing– which keeps my bibliographies consistent across all my computers, and accessible from online, and even shareable with others. All of these elements add up to a really great piece of software.
I would also add that my workflow has changed a bunch in the last year. I wrote about some of the changes here in March. I do all of my note-taking outside of DT now, and only use it towards the end of the project. I keep my citations in Zotero, and add them to note files either via drag-and-drop, or using textexpander snippets. The only reason I’m still holding on to DT is because its search capabilities are so good, but they work just as well whether I have an indexed database or a native one.
[…] Devonthink — later he blogged a version of this on BoingBoing. Others have since chimed in: Devonthink for Historical Research is a really involved essay, which also contains a link to how the author uses Devonthink for course […]
[…] https://parezcoydigo.wordpress.com/2008/11/13/devonthink-for-historical-research-part-ii/ […]
[…] Chad Black’s post I mentioned above also has a great list of digital tools. Black also has a post on using Devonthink to construct databases. […]