Public Safety Assessment - Code, policies, procurement, financial, and other records (Utah Administrative Office Of The Courts)

Ms. Lipton,

Thank you for your request. Below is my response:

1. Attached is the fully executed MOU between Arnold Ventures and the
Utah Administrative Office of the Courts authorizing use of the PSA.
2. The Utah Administrative Office of the Courts (AOC) calculates,
generates, and uses the PSA exactly as it was created and validated by
Arnold Ventures.
3. Material related to the function, algorithm, and inner workings of
the PSA is all publicly available. You can find that information in the
following locations:
- Arnold Ventures'
<https://www.arnoldventures.org/work/pretrial-justice/> website
- PSA.org <https://www.psapretrial.org/> website
- Utah Courts' <https://www.utcourts.gov/resources/reports/psa/>
website
4. Input data: Utah has a unified court system and an in-house case
management system used by all courts. Our system connects with other
criminal justice databases to gather the data used to score a PSA. See the
PSA FAQs here <https://www.utcourts.gov/resources/reports/psa/faq.html>.
Training information on how data is input (how a PSA is scored) can
also be found on the Court's website here
<https://www.utcourts.gov/resources/reports/psa/docs/PSA%20Manual%20Calculation%20Guide.pdf>.
We make all PSAs publicly available on Xchange
<https://www.utcourts.gov/xchange/subscribe/>. "Inputs" come from our
case files and state and federal criminal history databases. "Outputs"
would be the PSAs themselves.
- I'm not sure exactly what you're looking for. This is not
artificial intelligence. Judges have complete discretion when making
release decisions. The PSAs are presented alongside other relevant
information for consideration by judges, but we do not require their use
nor do we require judges to follow the recommendations included in PSAs.
PSAs are simply another piece of information.
- We use an individual's State Identification (SID) number, which is
specifically tied to their fingerprints, to pull the criminal history and
case information needed to calculate a PSA. If we do not have
that unique
identifier, we do not generate a PSA because we do not want to risk
presenting a judge with incorrect risk information. The same
applies if we
get an NLETs hit back from the NCIC database that our system doesn't
recognize. In those instances, we do not generate or present a PSA to a
judge.
5. The AOC identified which Utah offenses would constitute violent
offenses, and which would affect release recommendations. Using the PSA
and Risk Factor Definitions
<https://www.utcourts.gov/resources/reports/psa/docs/PSA%20Risk%20Factor%20and%20Outcome%20Definitions%20-%20May%202018.pdf>,
a working group made up of defense attorneys, prosecutors, judges, and
staff attorneys created three lists: Violent Offenses, Step-Up Offenses,
and Hold or Max Offenses.
- Those lists can be found on the Courts' website
<https://www.utcourts.gov/resources/reports/psa/>. Download the
Excel spreadsheet titled “PSA Calculator with Report Capability and Lists
(May 2018).” The lists can be found in the tabs along the bottom of the
spreadsheet.
6. Validation: The PSA was validated nationally by Arnold Ventures.
Information pertaining to Arnold Ventures' validation study will need to be
obtained from them. The PSA has not been validated locally in Utah.
Harvard Law School's Access to Justice Lab (A2J Lab) is currently
conducting randomized control trials to assess the effectiveness of the PSA
in Utah. Information related to that study is outlined in the MOU and
training materials attached.
7. Reports: All Utah reports related to the PSA can be found on our
website here <https://www.utcourts.gov/resources/reports/psa/> and I've
attached a recent summary update that I haven't had a chance to post yet.
8. This isn't "software"
9. Funding and Procurement: Arnold Ventures provided use of the PSA at
no cost, however, they required the AOC to use a specific organization for
technical assistance to ensure we were properly trained and the PSA was
implemented accurately. Attached is a copy of the MOU, contract, and scope
of work for those services.
10. Training Materials - attached.
11. We do not market the PSA.

If you have any questions, I'm happy to discuss them with you.

Sincerely,
--
Keisa Williams
Associate General Counsel
Administrative Office of the Courts
450 South State Street
P.O. Box 140241
Salt Lake City, UT 84114-0241
Work Cell: 385-227-1426
Email: keisaw@utcourts.gov

---------- Forwarded message ---------
From: <requests@muckrock.com>
Date: Mon, Nov 25, 2019 at 9:28 AM
Subject: Government Records Access and Management Act Request: Public
Safety Assessment - Code, policies, procurement, financial, and other
records (Utah Administrative Office Of The Courts)
To: <jeniw@utcourts.gov>

Utah Administrative Office Of The Courts
GRAMA Office
P. O. Box 140241
450 South State
Salt Lake City, UT 84114-0241

November 25, 2019

To Whom It May Concern:

Pursuant to the Government Records Access and Management Act, I hereby
request the following records:

Records and materials related to the solicitation, acquisition, evaluation,
trial, pilot, and use of the Public Safety Assessment (PSA), created by the
Laura and John Arnold Foundation (now known as Arnold Ventures) or any
equivalent risk assessment system used by your agency or associated
criminal justice offices.

This "software" or services may be provided by the Arnold Foundation,
though this request is applicable to this, as well as any other company
providing predictive risk assessment tools related to the criminal justice
system of this jurisdiction or under consideration or contract with this
agency for similar services.

Materials responsive materials include but are not limited to:

1. ALGORITHM or CODE
Any algorithm or code developed for implementation of or as part of this
"software" or program or any related system that conducts calculations or
decision making. This includes any algorithm or software developed by or
with, given to, used by, purchased or licensed to this agency for
implementation of the above referenced program, as well as any algorithm or
source code in use by this program or by which this system conducts
calculations or decision making. This would include original source code,
any compiled binaries (also known as executables), specification documents,
spreadsheets, program scripts, and other digital materials used to
calculate any data in the above program.

2. INPUT DATA
Any input training data for the "software" or related machine learning
algorithms or programs.

For the aspects of the software that require an input (for example, to
compute a value), please provide:
• a copy of the five most recent sets of data that were used for input, as
well as
• the five most recent outputs of the software

in whatever their native format is.

If these inputs or outputs include exempt information, please provide all
material save for specific information that is specifically exempted by
law. Please also provide a description of input and output data fields that
will aid in understanding the type of information that is submitted to the
software and that is produced by the software.

3. AUDITS, REPORTS, and VALIDATION STUDIES
A copy of all reports concerning or mentioning the "software," including
audit records, annual reports that mention the use of the "software,"
reports to legislative bodies, misuse reports, reports to oversight bodies.

This also includes all impact and validation studies. Please provide a copy
of any validation studies conducted with respect to the program or with
respect to any software or algorithms used in connection with the program.
Please also provide a copy of any records pertaining to inquiries for the
need of validation studies or discussion of potential or actual validation
studies. A “validation study” in this context is any study designed to
assess how accurate the program is in predicting what it is supposed to
predict, used to assess whether the program may err in the direction of
either under- or overestimating likelihoods or predicted outcomes, or
intended to evaluate whether the software does or will produce any results
that are biased or unexpected in any way.

4. AGREEMENTS and CONTRACTS
Any and all agreements related to the acquisition and use of this
"software," algorithm, or program.

This includes (but is not limited to):
• all contracts (as well as associated amendments, attachments, and
exhibits),
• data sharing agreements
• insurance agreement
• intergovernmental services agreements
• licensing agreement
• memorandums of understanding, and
• nondisclosure agreements

regarding or relevant to this software or program.

Please provide a copy of any informal agreements, insurance agreements,
liability waivers, and warranties that reference this technology or
otherwise guide its use or the relationship between this agency, the
provider or developer of the "software," or any other partner or entity
that may use or access it.

5. BIDDING and PROCUREMENT
A copy of any available or relevant Requests for Expressions of Interest,
Requests For Proposal (or equivalent calls for bids), responses to Requests
for Expressions of Interest, letters of interest, responses to Requests for
Proposal, sole source or limited source justification and approval
documentation, documentation of selection, and other materials generated in
the consideration and selection of the technology in question.

6. FINANCIAL and FUNDING MATERIALS
Any records related to the financing or funding of this "software,"
including a copy of any related funding opportunity announcements, grant
applications, grantor status/progress reports, purchase orders, invoices,
and other memoranda and documentation related to the payment or cost (or
lack thereof) of the "software," related technology, personnel, equipment,
or other elements concerning this "software" or program.

7. INSTRUCTIONAL MATERIALS and TRAINING
All instructional materials, presentations, and presentation materials
(including recorded video and audio, PowerPoint files, prepared remarks,
and slides formats), and other guidance on the use of "the software."

This includes any notes taken during meetings that discussed the use of the
software, any explanations (whether generated internally or externally) of
how the software works, and any other document that has been used to help
explain the use of "the software" to any party, including internal
documentation, public relations materials, and executive summaries.

This also includes training material governing the use, sharing, or access
to the "software" or any data related to or collected by the face
recognition software/technology, including the legal standard that is
required before using the technology.

This also includes any description of input and output data fields that
will aid in understanding the type of information that is submitted to the
software, and that is produced by the software.

8. PRIVACY ASSESSMENTS
A copy of any data retention guidelines or policies, data security
policies, data security specifications, privacy impact assessments,
security audits, or other materials evaluating or guiding the security of
the "software" or the privacy of the data involved

9. USE POLICIES
Please provide a copy of any policy directives, guidance documents,
memoranda, training materials, or similar records governing the use and
function of this technology for immigration, law enforcement, or any
purpose, including all those related to data retention, permissible and
impermissible use, and security standards. This would include materials
that describe the application, function, and use of the "software,"
including advertisements, emails, handouts, usage policies, PowerPoint
presentations, specification documents, or standard operating procedures.

10. COMMUNICATION and MARKETING MATERIALS
All communications or marketing materials relevant to or mentioning this
"software" or program.

This includes any correspondence between any staff at this agency with any
other organization or vendor relevant to the use of the aforementioned
technology or program, including records related to meetings or follow-up
actions with any vendors, companies, or other private entities marketing
this technology for immigration, intelligence, law enforcement, or any use.
This also includes correspondence in the form of emails, email attachments,
paper letters, text message, and any other form of correspondence, as well
as all marketing materials - unsolicited, requested, or otherwise -
acquired from vendors of this technology.

____

If your office has questions about this request, please feel free to direct
them to the address associated with this request or call the MuckRock
office at 617-299-1832.

The requested documents will be made available to the general public, and
this request is not being made for commercial purposes.

In the event that there are fees, I would be grateful if you would inform
me of the total charges in advance of fulfilling my request. I would prefer
the request filled electronically, by e-mail attachment if available or
CD-ROM if not.

Thank you in advance for your anticipated cooperation in this matter. I
look forward to receiving your response to this request within 10 business
days, as the statute requires.

Sincerely,

Beryl Lipton

Filed via MuckRock.com
E-mail (Preferred): requests@muckrock.com
Upload documents directly:
https://accounts.muckrock.com/accounts/login/?url_auth_token=AABpM2G2zRs6P6Dmut8MaeKQJHE%3A1iZHDa%3AjeqBzEwKjDBPQJMNlZU0M9G4twg&next=https%3A%2F%2Fwww.muckrock.com%2Faccounts%2Flogin%2F%3Fnext%3D%252Faccounts%252Fagency_login%252Futah-administrative-office-of-the-courts-19196%252Fpublic-safety-assessment-code-policies-procurement-financial-and-other-records-utah-administrative-office-of-the-courts-83746%252F%253Femail%253Djeniw%252540utcourts.gov
Is this email coming to the wrong contact? Something else wrong? Use the
above link to let us know.

For mailed responses, please address (see note):
MuckRock News
DEPT MR 83746
411A Highland Ave
Somerville, MA 02144-2516

PLEASE NOTE: This request was filed by a MuckRock staff reporter. Also note
that improperly addressed (i.e., with the requester's name rather than
"MuckRock News" and the department number) requests might be returned as
undeliverable.