Progress rarely comes in a straight line, particularly when it comes to public records and transparency. This week, some stories about the ups and downs after reforms happen, including challenges at the state, local, and federal level.

Send over your favorite FOIA stories via email, on Twitter, or on Facebook, and maybe we’ll include them in the next roundup.

Piecing together America’s broken hate crime tracking systems

ProPublica’s Ken Schwencke used public records and jarring examples to show how federal requirements around hate crime reporting fall short in sharing vital information about where and how often hate crimes occur.

The story raises important questions that local reporters and watch dogs can dig into more deeply:

But to understand why so many cases that are reported to authorities still fall through the cracks, ProPublica requested incident reports or aggregate data from more than 350 law enforcement agencies in 48 states, including the 50 largest agencies nationwide, on the bias-motivated crimes they had investigated since 2010.

More than 280 agencies responded, but in many cases only to say they hadn’t investigated any such incidents, or had no records, or that their records were bad. When we followed up with agency public information officers, they acknowledged that investigators frequently did not mark down incidents as motivated by bias, even if there was evidence suggesting this was so — a spray-painted swastika, for example, might be classified simply as vandalism and not also as a hate crime.

The FBI and some larger agencies champion a two-tiered process in which incidents classified by the first responding officer as potentially bias-motivated are re-evaluated by a second investigator, who determines if the incident should be counted officially as a hate crime. Few of the agencies that responded to our records request appeared to follow this procedure.

Read the full story here. ProPublica used MuckRock to help file and manage the requests involved, but given the sensitive nature of the requests is keeping the requests embargoed.

Lobbyists refuse to disclose how much they’re making

Elizabeth Findell at the Austin American-Statesman has a fascinating story about local attorney Fred Lewis’ fight to make city hall lobbyists follow the law. Last year, the city brought its laws in line with similar state and federal regulations in requiring lobbyists to disclose how much they’re making from their clients in exchange for lobbying work.

So far, 17 lobbyists have refused - stating that disclosing such payments would violate lawyer-client confidentiality. As Findell reports:

Their refusal to disclose compensation to City Hall appears to have been coordinated. Fourteen of the 17, from five different law firms, wrote the exact same sentence on their forms: “Disclosure of client compensation is not provided because such disclosure would violate applicable state law in Texas Disciplinary Rule 1.05 and Chapter 81 of the Texas Government Code.”

Transparency growing pains can strike in the most unexpected places.

Massachusetts public records reform helps slash fees. Now will agencies comply?

Speaking of transparency growing pains, Massachusetts public records watchdog and Boston Globe reporter Todd Wallack shared some promising developments in Bay State sunshine: The Supervisor of Public Records has quashed a high price tag for what should be a basic document from a notoriously secretive agency.

The Massachusetts State Police wanted to charge $2,475 to review and redact their #FOIA log. The state Supervisor of Records just ruled the fee is not allowed. pic.twitter.com/GfdRHjXywW

— Todd Wallack (@TWallack) December 7, 2017

Just to clarify, the log is just a list of public records requests, including the name of the requester, the date and a brief description of the request. (I did not ask for any columns with contact information or internal notes.)

— Todd Wallack (@TWallack) December 7, 2017

Some context: Massachusetts State Police was a Golden Padlock winner in 2015 after they tried charging $42,750 for the same document and assessed exorbitant estimates for a number of other requests.

It will be interesting to see the state police’s next move.

Seen a great FOIA-based news story? Let us know and maybe we can include it in our next round up! Send it over via email, on Twitter, or on Facebook.

Image via Warner Bros