First, you will need a DocumentCloud account.  They are still in a private beta, so you will have to request an account for now.  Assuming you have an account, you probably want to start by looking at the help for their API.  We need to be able to make HTTP GET and POST requests – including the ability to send a file in multipart/form-data format.  This isn’t straightforward to do using the Python standard libraries, so I found this MultipartPostHandler library to help out.

Now the way I wanted this to work for our site is to have a model that represents a DocumentCloud document and allow staff to edit these documents directly through our admin site without having to log in the DocumentCloud site directly.  Since the HTTP requests may take a while or error out, it is a good idea to run them asynchronously so users do not have to wait for them to finish.  To do this I use celery.  Setting up celery is beyond the scope of this post, but the documentation on their site is good and you should be able to follow that.  Now for some code!

The model is just set up to mirror the options available for a DocumentCloud document, plus a ForeignKey to a FOIA Request model, which represents the original request on our website and a doc_id which is used to store the unique ID DocumentCloud assigns to this document:

# models.py
class FOIADocument(models.Model):
    """A DocumentCloud document attached to a FOIA request"""
 
    access = (('public', 'Public'),
              ('private', 'Private'),
              ('organization', 'Organization'))
 
    foia = models.ForeignKey(FOIARequest, related_name='documents')
    document = models.FileField(upload_to='foia_documents')
    title = models.CharField(max_length=70)
    source = models.CharField(max_length=70)
    description = models.TextField()
    access = models.CharField(max_length=12, choices=access)
    doc_id = models.SlugField(max_length=80, editable=False)

Now for the admin model to integrate it into the Django admin interface:

#admin.py
class FOIADocumentAdmin(admin.ModelAdmin):
    """FOIA Image Inline admin options"""
    readonly_fields = ['doc_id']
    list_display = ('title', 'foia', 'doc_id', 'description')
 
    def save_model(self, request, obj, form, change):
        """Attach user to article on save"""
        obj.save()
        # wait 3 seconds to give database a chance to sync
        upload_document_cloud.apply_async(args=[obj.pk, change],
                                          countdown=3)

We make doc_id a read-only attribute, as this will be set automatically.  On a save, we first save the object to store it to the database.  Then we make an asynchronous call to upload_document_cloud which is a celery task I have set up to do all the interesting stuff with the API.  There is a delay of 3 seconds to make sure the database has chance to save the object, as in development I would sometimes get errors that the object was not in the database.  It passes the primary key for the object, and whether or not this is a change (as opposed to a new object).

Lets take a look at the task now.

First we import the libraries we will be using, our DocumentCloud username and password (I put these in my settings.py), and the model from above.

# tasks.py
from celery.decorators import task
from django.core import management
from settings import DOCUMNETCLOUD_USERNAME, DOCUMENTCLOUD_PASSWORD
 
import base64
import json
import urllib2
from vendor import MultipartPostHandler
 
from foia.models import FOIADocument

The task itself starts with loading the recently saved model back form the database – we waited 3 seconds to make sure it was synced, but just to be sure we will retry if it still doesn’t exist. The default retry is 30 seconds later. We also return without doing anything if the model already has a doc_id but is not being changed – this should never happen, but it is always a good idea to code defensively.

@task(ignore_result=True)
def upload_document_cloud(doc_pk, change, **kwargs):
    """Upload a document to Document Cloud"""
 
    try:
        doc = FOIADocument.objects.get(pk=doc_pk)
    except FOIADocument.DoesNotExist, exc:
        # give database time to sync
        upload_document_cloud.retry(args=[doc_pk, change],
                                    kwargs=kwargs, exc=exc)
 
    if doc.doc_id and not change:
        # not change means we are uploading a new one -
        # it should not have an id yet
        return

Now we set up the parameters for our API call. They need to be coerced from unicode to regular strings due to the way they are encoded in the MultipartFormHandler – failing to due so caused encoding errors. If we are changing we will use the update API by making a PUT (we fake it using _method) to the documents doc_id. If this is a new document we upload the file using a POST to upload.

    # coerced from unicode to regular strings
    # in order to avoid encoding errors
    params = {
        'title': str(doc.title),
        'source': str(doc.source),
        'description': str(doc.description),
        'access': str(doc.access),
        'related_article': str('http://www.muckrock.com' +
            doc.foia.get_absolute_url()),
        }
    if change:
        params['_method'] = str('put')
        url = '/documents/%s.json' % doc.doc_id
    else:
        params['file'] = open(str(doc.document.path), 'rb')
        url = '/upload.json'

We perform the request here. Urllib2 does not allow you to use the http://username@password:example.com syntax for basic authentication, so we add the header manually. Also notice that we are using https so as not to allow snoopers to find our account’s password. If this is a first time upload, upon return of the call we will parse the JSON returned and assign the id attribute to the doucment and save it. We also catch any errors that may happen, such as timing out due to a bad network connection, and will retry the request. It will retry up to 3 times by default before giving up.

    opener = urllib2.build_opener(MultipartPostHandler.MultipartPostHandler)
    request = urllib2.Request('https://www.documentcloud.org/api/%s' % url, params)
    # This is just standard username/password encoding
    auth = base64.encodestring('%s:%s' % (DOCUMNETCLOUD_USERNAME, DOCUMENTCLOUD_PASSWORD))[:-1]
    request.add_header('Authorization', 'Basic %s' % auth)
 
    try:
        ret = opener.open(request).read()
        if not change:
            info = json.loads(ret)
            doc.doc_id = info['id']
            doc.save()
    except urllib2.URLError, exc:
        upload_document_cloud.retry(args=[doc.pk, change], kwargs=kwargs, exc=exc)

And that is it. If you have any questions or need help adapting any code for your particular use, feel free to contact me at mitch@muckrock.com

The viewer lets you easily scroll around and read through a document, download the original PDF, search through text and much, much more, and we’d love to hear your thoughts. A bit thanks to the DocumentCloud team for their incredible work, and for letting us take part in their own beta!

You can also see a progress bar up top that shows you how far you are along in actually getting that request finished up. You’ll actually find a similar progress bar on all request pages now (see here for a sample). That gives you a quick, visual way to see how far along your request is: Whether it’s waiting for a response, hasn’t been sent out yet, or is even still just a draft in your inbox. There’s also a new color-coding system for tracking these responses, so that if a response is coded green, everything’s smooth sailing; if it’s coded red, the request has been denied, the government has failed to respond within the statutory limits, or something else has gone wrong; and if it’s coded yellow, a user response is needed, such as finishing up a draft or OK’ing some proposed compromise that the government agency is requesting. You can even quickly see the status of all current requests with this system on the FOI request homepage, which is right now almost entirely green.

Released by Wikileaks, the documents provide vivid descriptors of efforts to contain the Taliban, and, most strikingly, reveal a potential alliance between al Qaeda and Pakistani intelligence.

But for all of the clamoring by government officials to diffuse attention from the information contained in these reports, commentators almost unanimously agree on one thing — we already knew this. These thousands of documents only give voice to what the government and media has already told us: That winning a war against terrorists is a near impossible task with few victories and many failures.

So was it worthwhile to release all of this information? In a word, yes. Not only does it confirm that the news we have been fed about the war in Afghanistan is, for the most part, accurate, but it also gives us the information with almost zero bias.

Yes, it’s possible that Wikileaks and the publications it partnered with to release the documents picked and chose what to make public. And no, it is likely not possible to confirm the accuracy of every piece of information contained in these volumes of history. But for the most part, these newly public facts stand alone – apart from the partisan bickering that can obscure the truths of war, and apart from the news media that sometimes fails to capture world events with infallible accuracy.

With a compendium of knowledge out in the open, the public and officials on both sides of the aisle can look inward and revise their own visions of the war on Afghanistan. These documents could force the president to make a swift decision to either reassert his policy in Afghanistan or come up with an exit strategy. And they could force the public to take apolitical views on a war that had become fodder for policy wonkery.

Open information, whether or not it provides a smoking gun revelation, has the power to refocus interests and align people in working towards a better strategy. Whether or not a solution emerges depends now on how long we can lift the fog of war for.

As a Boston Herald editorial summarized:

Just in case anyone thought the Patrick administration’s health insurance rate cap was anything other than an exercise in raw political pandering, along comes a set of e-mails that proves the point.

Thanks to a Freedom of Information request made by State House News Service, we now have a glimpse into how the administration crafted and rallied to defend what has turned out to be an indefensible policy.

Many of the e-mails were in response to a column by Mike Widmer, head of the Massachusetts Taxpayers Foundation, who pointed out what a dreadful idea the rate cap was. But the oped, which ran in The Boston Globe on March 17, did nothing to prevent the administration from going ahead with the cap officially on April 1. The e-mails showed administration officials were less concerned about the policy than about the spin!

The Associated Press had apparently filed its own  request for the e-mails discussing the rate cap, but one thing is conspicuously missing from both reports: The actual e-mails themselves.  Not surprising, but a bit obnoxious: Why can’t we see for ourselves how the Patrick administration was discussing the proposed internally? Soon, state willing, we will. MuckRock has filed our own request (click the link if you’re a registered member), which we’ve reproduced below the fold.

Our Request, filed via the Consumer Affairs and Business Regulation contact form

To Whom It May Concern:

Pursuant to the state Freedom of Information Act, G.L.M. c.4, €7, clause 26th, I hereby request the following records:

Any and all e-mails and related materials provided to the State House News Service and/or the Associated Press in response to Freedom of Information requests for e-mails regarding the proposed health insurance rate cap instituted April 1st, as referenced at the following URLs:

http://www.bostonherald.com/news/opinion/editorials/view/20100715policy_vs_the_spin/

http://www.businessweek.com/ap/financialnews/D9GUQG880.htm

and including but not limited to e-mails from and to Barbara Anthony, a government employee serving as the undersecretary of consumer protection.

I also request the initial requests filed by the State House News Service and/or the Associated Press that lead to the e-mails release.

I agree to pay reasonable search and duplication fees for the processing of this request in an amount not to exceed $5.

In the event that fees do exceed $5, I would be grateful if you would inform me of the total charges in advance of fulfilling my request. I would prefer the request filled electronically, via e-mail attachment or CD-ROM.

I also request that, if appropriate, fees be waived as we believe this request is in the public interest, as suggested but not stipulated by the recommendations of the Massachusetts Supervisor of Public Records. The requested documents will be made available to the general public free of charge as part of the public information service at MuckRock.com.

Thank you in advance for your anticipated cooperation in this matter. I look forward to receiving your response to this request within 10 business days, as the statute requires.

Sincerely,

Michael Morisy
Filed via MuckRock.com
185 Beacon St. #3
Somerville, MA 02143

Daytime: (857) 488- 3081
E-mail: requests@muckrock.com

A Modest Proposal:

After having filed what I guess are 2,000 open records requests in my 40 odd years of existence, one of the things that I am most often struck by (and annoyed by) are government institutions that go to the mat to prevent release of public domain information.

Again and again, I find myself denied information under FOI that is readily available from public sources OTHER THAN open records laws. In my kind of requests, it is generally hyper-exaggerated notions of intellectual property that are the problem.

Why is it that government agencies so often fight release of information that is already public?  Of course, there are frequent “our hands are tied” arguments; but these are usually exaggerated and can be negotiated around. No, it really seems to me to be most often a matter of resistance to FOI in general more than some particular odd provision in FOI law that exempts already public information.

Why couldn’t honesty be made a systematic metric of FOIA?  Isn’t it actually a potentially better measure than some others currently in use?

I suspect that like a lot of other people, I routinely file open records requests towards the beginning of researching an issue.  Sometimes, I answer my own question before I get a FOI response.

Unfortunately, the FOI response is often that the information I have found elsewhere is exempt from disclosure.  Then I find myself in the nonsensical position of being denied the very information I already know.

For example, my current round of this problem relates to Texas A&M University and some plant varieties that they think are valuable.  They are fighting to avoid telling me where these plant varieties come from because it is, allegedly, Texas A&M intellectual property.  But I already know the answer.  I know where they were collected, when, from whom, and how they got around the world to College Station, Texas. All of this was found in public documents – almost exclusively scientific publications.

Nevertheless, Texas A&M is ‘valiantly’ fighting on to stop me from knowing.

If it’s not plant varieties, then it’s some other issue.

So why can’t the FOI community turn this problem on its head?  Why not do honesty checks?  Systematically file FOI requests for already public information relating to a particular agency, and then evaluating that agency’s reply.  There would be a number of practical issues to work through in order to get comparable data; but I don’t see any intrinsic show stoppers.

I like having an “honesty” gauge too.  Every ethical person – on all sides – wants to be honest.  So where institutional dishonesty arises, I think FOI officers and others may very well be potential allies in fixing it.

Edward Hammond

As one reporter responded on the mailing list:

Definitely need a period of exclusivity to encourage use of open-records laws. If there is none, requestors will be less willing to expend time and money to fight for the release of withheld records.

I once fought a lengthy Access to Info battle for some federal records only to have the department release them publicly and announce it with a press release. When I got home that day, I found that a courier had been and left them the documents requested on my doorstep.

So while government officials don’t always like the results of freedom of information requests, by law they can only stall so much, so often and they’ve turned to transparency as an obnoxious counter-measure against requests.

For a case study, see Chicago:

CHICAGO, May 13 (UPI) — Anyone filing Freedom of Information Act requests with the city of Chicago will now find their names on the city’s Web site, Mayor Richard Daley said Thursday.

The mayor says he’s just trying to be totally transparent, not get back at nosy news reporters, who now may find competitors privy to what story leads they are investigating.

What he’s finding, however, is a very mixed picture, particularly in terms of police department openness:

Time after time journalists are raising this issue: They can’t get anything out of police anymore. As I do sessions I ask the old timers to describe what it was like to cover cops 20 years ago. Then I ask a new reporter to describe what it is like today. Here is how it goes:

20 years ago: Walk into the police station and go to the incident reports, kept in a basket or clipboard. Flip through all the reports for the past 24 hours, with no redactions. Everything is there – name of suspects, full address, name of victims – the works. If you had a question you asked the sarge on duty, or even called the officer who handled the call. If we heard something on the scanner we could ask about it. We got news out fast and we got it complete.

Today: You walk into a police station and talk to a PIO, who tells you what the police think is newsworthy, sanitized and little detail. No looking at incident reports. No interviewing the officer or getting information from a sarge in charge. Some agencies are encrypting their scanner channels so nobody can hear what is happening. We are at the mercy of what a PIO wants to tell us, or not tell us. Secret police.

Read on for other great tips and his thoughts on the state of FOI.

The 132-page document includes:

• Saddam’s thoughts on his greatest accomplishment: “The social programs for the citizens of Iraq and improvements in other sectors of the economy including enhancements to education, the health care system, industry, agriculture, and other areas that generally enhanced the way of life for Iraqis.”
• When asked about his own mistakes, he told the interviewer that “All humans make mistakes, and only God is free of error.” But that he would not identify mistakes to an enemy, and the American system of government was his enemy.
• Saddam wishes that both America and Iraq advance in all areas, “financial, religious, etc.”.
• Pages and pages of details about coups, the Ba’ath uprising, the early days of the Iraqi revolution, and more.

The document release is 132 pages, but unfortunately is not searchable until somebody takes some OCR software to it. You can download it directly from the FBI (warning: PDF).

But with bills generally seeing multiple revisions, it’s easy to lose track of what exactly was or is included in a bill and when it was added, taken out, or even re-added. The results can be disastrous when it comes to reasoned discourse, as the League of Technical Voters’ founder Silona Bonewald notes in a recent piece for O’Reilly Radar:

Commenting on these types of documents as they are currently implemented is extremely challenging. Pointing a finger at that big pond and telling someone that you swear you saw a fish isn’t very effective. It’s even worse when someone swears they saw a fish that isn’t really there and it is effective because no one is willing to refute them. No one has time to wade around themselves and so they take it on faith. The recent “killing grandma” scare is an excellent example.

So the league has created an advanced citability solution, Citability.org, that tracks changes and makes sure that it’s easy to see how your political sausage is made, and who is putting in what ingredients.