Electronic lab notebook question

[baratron]

Hello! I haven't written anything in livejournal for weeks. If you're wondering where I've been, the answer is mostly at home. If you're wondering why you haven't seen anything from me (and are wondering if I'm writing lots of secret things which are filtered away from you), the answer is that I genuinely have not posted anything since my public post of 22nd September. Why not? I've been busy with trying to sort out my body clock and get back into working...

I have a backlog of 5 or 6 posts to make when I find spoons for them (maybe this weekend?), but for now, I have a question for anyone who knows anything about postgraduate students and/or Macintosh computers.

I'm looking for lab notebook software that time-and-date-stamps everything you write. It's important to keep good records in case of disaster later, and it's important to have clear what you knew when. However, most of the programs labelled as "electronic lab notebooks" are designed for industry and are ridiculously expensive and feature-ful. Apparently, other postgraduate students have used Evernote, Google Notebook, Circus Ponies Notebook and wiki and blogging software - but I'm not sure how they're "getting away with it" since only wikis (to my knowledge) will datestamp everything. Even blogs are editable later, though I don't *think* you can fake the datestamp on the initial post.

Does anyone have any ideas that are easier to use than a wiki? Having to learn something akin to HTML simply to mark up my own lab notebook seems almost as much hassle as writing it out by hand, though there may be wiki editors like livejournal clients available now. (I still miss Semagic after switching from Windows to Mac - Xjournal is miles better than typing directly into the web interface, but awful compared to Semagic).

Feel free to point other people directly at this post - it's public, after all. And if you have any ideas for communities where I could ask questions like these, please let me know. The PHD Comic forums are down, and the old Papers (Mac research program) forum where you could ask anything has closed, so I'm pretty stuck for options. Thanks!

This post has been superceded by friends-only post here. If people who aren't on my trusted people list still want to comment, read this thread first.

Comments

[brooksmoses]

One possible option would be keeping your notebook files (in whatever format) in a version control repository of some sort. Those date-and-time stamp everything you put into them.

What sort of documents would you like to be putting into this notebook? Is plain text sufficient, or do you need more than that? Things that look like Word documents? Images?

[baratron.livejournal.com]

I definitely need something more complicated than plain text. Ideally, I want a program that is scrapbook-like, where I can write in plain text but also include tables, copy-pasted program outputs, and screenshots. If I could figure out a way to save images in some sort of 3D format that would be ideal.

I have quite a lot of handwavy ideas where I need to talk to chemists who have been doing computer modelling stuff for years, but my department is tiny. I need to be assertive and go talk to people at UCL and Imperial. Making the initial contact is REALLY hard, though.

[emperor]

I keep my log.txt as a text file, checked into CVS (but a more modern version control system would work fine too); when I commit a change, it posts it to LJ too.

[johnckirk.livejournal.com]

How sure do people have to be that you haven't faked the date? Any data file can be edited, and you could theoretically modify the date on your machine. I don't know anything about Macs, but I can see two extremes based on Windows machines:

1) In Notepad, press F5 and it will insert the current date/time. There may well be a similar option for other editors, in which case you don't need to buy/learn extra software.

2) When I digitally sign macros/files, I use a timestamp server (e.g. at Verisign). That way, my signature will still be valid once my certificate has expired at the end of the year, because Verisign (or whoever) have confirmed that I did the signing while my certificate was valid. If you have a separate data file for each day, you could sign each one, then it would be tamper-evident if you tried to modify them later.

[baratron.livejournal.com]

I could theoretically fake the date in a paper notebook (and indeed have on many occasions - I am crap at writing up what I've done each day, which is one reason why I'm looking for an alternative). I suppose though that there's some evidence in that no one writes exactly the same way each day, especially when making quick scribbled notes of what you've done.

And yes, I'd already thought of faking the date via changing the date on my machine. I assume though that if I were to upload files to university servers, that would count as acceptable, since I don't have root access to them.

I am waiting for the most IT-literate member of the Graduate Committee to get back to me with a ruling of some kind.

[skibbley.livejournal.com]

Depends on how secure you need it. My first thoughts are also to upload the stuff, in any format you like, to a trusted version control system (this can be done with a command or integrated into the editor for some editors).
At a university can it be uploaded to some uni controlled blackboard system?

[baratron.livejournal.com]

The impression I have been given by lecturers is that Blackboard is a right royal pain in the arse to use - and that includes the IT-literate person mentioned above. Also there is no way that the university would give a student access to it.

Definitely a university server of some kind seems like a good idea, though.

I don't want my data visible to anyone who doesn't have a password, which is the main issue I have about using blogging software - how do you set up a private blog? I know you *can* set up private wikis because I've seen one.

It would also be ideal if someone who isn't me was responsible for backups, because I don't want to lose everything due to a brain fart. The crystallography servers seem to be UNIX-based and have proper sysadmins (as opposed to the college servers as a whole, which seem dangerously close to Windows for my liking), so I think that should be okay.

[skibbley.livejournal.com]

How do students turn in pieces of work for marking?

[baratron.livejournal.com]

Mostly by printing it out and taking it into the office. Electronic submissions go by email.

I am totally happy to email my lab notebook to someone (e.g. my supervisor) every week. The problem with doing that is that each week the lab notebook will get bigger, so it will take up unnecessary space on a university hard drive. A version management system gets around that problem by only saving the changes to the file each time.

I suppose I could email one week's worth? Hmm.

I have a reply now from the most IT-literate person, which is totally unsatisfactory. Will include as a new post.

[otterylexa.livejournal.com]

If you were keeping the data in a version control system, they can usually produce a diff between two arbitrary versions.

[hoopycat (from livejournal.com)]

I generally use "normal" files (.odt, text, .pdf scans, etc) checked into a git repository, which is available on all of my machines and my usual thumbdrive. The commits are timestamped using the machine's local time(*) and depend on the entire revision history leading up to the commit, so changing something in the past is difficult(**).

The disadvantages of this approach are that it isn't an industry-standard solution (for your industry, at least), and it isn't perfect. Also, our lab machines don't have git installed.

I still use pen and a notebook out of habit, although I now use a cheap spiral-bound notebook if I don't need sequentially-numbered pages, and all of my *data* is stored electronically and committed into git. Life's too short to copy a 256-sample discrete Fourier analysis into a notebook by hand. :-)

Might also be worth taking a look at fossil(***)... Dawn has been experimenting with this for a different application, and I've been impressed by its usability. This might fall too close to "wiki" from a markup standpoint(****), though. -rt (from the Detroit Manual of Style department)

(*) Which can be changed, of course, so I commit frequently and monotonically out of habit.
(**) Probably not impossible, especially if you have access to all copies of the repository. Sharing it with other people makes this much more difficult.
(***) http://fossil-scm.org/ specifically http://fossil-scm.org/index.html/doc/trunk/www/event.wiki
(****) Although there are great strides being made on the ease-of-use front with the wiki concept... see http://sapling.rocwiki.org/ and for example http://sapling.rocwiki.org/Front_Page/_history/16...17 for LocalWiki's approach to markup. A WYSIWYWGILWF(******) editor on front of honest-to-goodness, god-fearing HTML5, with ... well, let's just say the differences-between part is a little gamey yet. Still, this is a good thing: there's a crapload of embedded rich text editors out there, so you shouldn't *have* to pound out your own HTML or whatnot.(*****)
(*****) I should have just numbered the footnotes, you know?
(******) What You See Is What You Would Get If Life Were Fair, the new benchmark for programmatic markup generation.
(*have*) Not a footnote.

[memevector.livejournal.com]

There exist services online specifically for date-stamping things for copyright proof/ legal record reasons. Small amounts of money and small amounts of faff.

Can say more if that sounds possibly useful - not an expert but have used a couple of them.

[baratron.livejournal.com]

Go on...? Richard found one online that was 40p per signature, which would be ridiculous over the course of an entire PhD, but let me know what you've used.

[memevector.livejournal.com]

I've timestamped a few things using this:

https://www.digiprove.com/secure/subscribe.aspx

which may or may not be unfeasibly expensive for you (if indeed it's still in the running after your official person's ruling). How the prices work out depends on how many you're doing.

I've also used Registered Commons, but their registration device was broken last time I looked.

I haven't had cause to call on either of them to verify anything, which I suppose is the acid test! But DigiProve does appear to be a professional service, and their software's not caused me any bother so far.