Allegheny Family Screening Tool - RFP responses and development materials - Allegheny County Department of Human Services
| Tracking # |
W004725-032519 |
| Submitted | March 23, 2019 |
MuckRock users can file, duplicate, track, and share public records requests like this one. Learn more.
Communications
From: Beryl Lipton
To Whom It May Concern:
Pursuant to the Pennsylvania Right to Know Act, I hereby request the following records:
• The initial 15 responses to the Request for Proposal issued by the Allegheny County Department of Human Services (DHS) to design and implement a system of decision-support tools and predictive analytics for human services, also known as the Allegheny Family Screening Tool
• All review reports, notes, evaluations, and associated materials created by the RFP evaluation committee.
• All deliverables provided by the winning applicants: Rhema Vaithianathan (AUT) and researchers from Auckland University of Technology (AUT), University of Southern California (USC), University of California-Berkeley and University of Auckland.
• All minutes for meetings held to solicit community input on the creation of the tools, including the six project-specific meetings, any held at early stages of the project to collect feedback from key external stakeholders and funders, and the three open community meetings where over 30 stakeholder groups (including the Courts and the ACLU) were invited to discuss the work to date, implementation timeline and results. Please include all notes on the work to date, implementation timeline, and results. Please also include all project updates DHS shared with existing community networks including the Children’s Cabinet and the Children, Youth and Families Advisory Board, and through the DHS Speaker Series.
• All responses to the RFP for two independent evaluations of the AFST (process and impact) that was issued in 2015 and for which Hornby Zeller Associates was selected to conduct an process evaluation and Stanford University was selected to conduct an impact evaluation.
• The process evaluation and the impact evaluation focused on whether the AFST increases the accuracy of decisions, reduces unwarranted variation in decision-making and reduces disparities, and overall referral rates and workload.
• A list of sources/datasets - including child protective services, publicly funded mental health and drug and alcohol services, and bookings in the County jail - used by the AFST.
• The current percentage of incoming referrals were not generating an AFST score.
• The number of overrides conducted on “mandatory screen-ins”
• Any correspondence and contract documents relating to the development and implementation of the program.
• Records that show how the County's Integrated Data System is used in the program, including records that identify how risk factors are generated and the weights that are given to various risk factors.
• Any software and algorithms developed for the implementation of the program. I'm particularly interested in any algorithms that have a public policy outcome, such as providing guidance on a policy or an assessment of an individual. More generally, I would like any software that was developed by or with, given to, used by, purchased or licensed to this agency for implementation of the above referenced program. This would include original source code, any compiled binaries (also known as executables), spreadsheets, program scripts, and other digital materials used to calculate any data in the above program. It would also include the input training data for machine learning algorithms. Collectively, these responsive materials will be referred to as "the software" in the following paragraphs.
• All instructional materials, presentations and presentation materials (including recorded video and audio, PowerPoint files, prepared remarks, and slides formats), and other guidance on the use of "the software." This includes any notes taking during meetings that discussed the use of the software, any explanations (whether internally or externally generated) of how the software works, and any other document that has been used to help explain the use of "the software" to any party, including internal documentation, public relations materials, and executive summaries. This also includes any description of input and output data fields that will aid in understanding the type of information that is submitted to the software, and that is produced by the software.
• A copy of any privacy impact assessments, use policies, standard operating procedures, data retention policies, legal opinions, warranties, non-disclosure agreements, contracts, liability waivers, insurance agreements, Requests for Proposals, Responses to Requests for Proposal, Memorandums of Understanding, Letters of Interest, usage policies, or informal agreements that reference "the software" or otherwise guide it use or the relationship between this agency and the provider or developer of the software.
• A copy of any funding opportunity announcements, requests for proposals, grant applications and grantor status/progress reports, reports to legislative bodies, annual reports that mention the use of the Software, as well as audit records, including but not limited to security audits of the software, misuse reports, and reports to oversight bodies.
• A copy of any validation or ethical review studies conducted with respect to the program or with respect to any software or algorithms used in connection with the program, and a copy of any records pertaining to inquiries for the need of validation studies or ethical reviews, or discussion of potential or actual validation studies or ethical reviews. By “validation study,” I mean any study designed to assess how accurate the program is in predicting what it is supposed to predict, or to assess whether the program may err in the direction of either under- or overestimating likelihoods or predicted outcomes, or may produce any results that are biased or unexpected in any way. By "ethical review," I mean a review of how the design and/or implementation of the predictive algorithm conforms with ethical standards of fairness and privacy.
• For the aspects of the software that require an input, for example, to compute a value, we request a copy of the five most recent sets of data that were used for input, as well as the five most recent outputs of the software, in whatever their native format is. If these inputs or outputs include exempt information, we ask that you exempt only that information which is specifically exempted from law, and provide the other material.
• If any of the records requested are protected by a nondisclosure agreement, I request that you provide documentation of that agreement.
The requested documents will be made available to the general public, and this request is not being made for commercial purposes.
In the event that there are fees, I would be grateful if you would inform me of the total charges in advance of fulfilling my request. I would prefer the request filled electronically, by e-mail attachment if available or CD-ROM if not.
Thank you in advance for your anticipated cooperation in this matter. I look forward to receiving your response to this request within 5 business days, as the statute requires.
Sincerely,
Beryl Lipton
From: Allegheny County Department of Human Services
Dear Beryl Lipton,
We have received your RTK Request.
Sincerely,
Allegheny County
From:
Ms. Lipton,
Please see the attached in response to your RTK request.
Thank you,
Kathy Colosimo
County of Allegheny
Department of Administrative Services
Office of Open Records - Executive Branch
436 Grant Street - Room 202 Courthouse
Pittsburgh, PA 15219
412-350-6109
Confidentiality Notice: This e-mail message, including any attachments, is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure, forwarding, or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of the original message.
From: Allegheny County Department of Human Services
A copy of documents responsive to the request.
-
November2018_variables_weights.xlsx
There are too many files to display on this communication. See all files
From: Allegheny County Department of Human Services
Hello, Mrs. Lipton.
I hope this finds you well.
At the time you submitted your Right-to-Know request to Allegheny County for records about the Allegheny Family Screening Tool, the Impact Evaluation was not finalized. However, the evaluation is now complete and listed below in our recent press release announcing the key findings.
Take care,
Elaine
Elaine Plunkett
Communications Specialist/Media
Allegheny County Department of Human Services, Office of Community Relations
Phone 412.350.6897 || Email elaine.plunkett@alleghenycounty.us<mailto:elaine.plunkett@alleghenycounty.us>
_______________________________________
One Smithfield Street, Pittsburgh, Pa 15222
[cid:image001.jpg@01D5001B.79CA0F60]
FOR IMMEDIATE RELEASE
May 1, 2019
Contact:
Elaine Plunkett
412-350-6897
Elaine.Plunkett@AlleghenyCounty.us<mailto:Elaine.Plunkett@AlleghenyCounty.us>
Mark Bertolet
412-350-3439
Mark.Bertolet@AlleghenyCounty.us<mailto:Mark.Bertolet@AlleghenyCounty.us>
Impact Evaluation on Allegheny Family Screening Tool Released
PITTSBURGH - An evaluation of the Allegheny Family Screening Tool (AFST)<https://www.alleghenycountyanalytics.us>, a predictive risk modeling tool designed to improve the accuracy of child maltreatment hotline call screening decisions, has been completed by Stanford University (principal investigator: Jeremy Goldhaber-Fiebert).
Key among its findings, the report concludes that implementing the AFST and accompanying policy changes improved the accuracy of maltreatment referral screening decisions (i.e., which referrals to 'screen-in' for investigation) and reduced racial disparities in case openings between black and white children.
The design and implementation of the Allegheny Family Screening Tool was a multi-year project that included community and stakeholder meetings to solicit input, an intentional procurement process, an ethical review, a validation study, and independent and rigorous process and impact evaluations.
This new report, Impact Evaluation of a Predictive Risk Modeling Tool for Allegheny County's Child Welfare Office, is the result of an 18-month-long evaluation of the tool's indirect and direct effects on the maltreatment screening process, including decision accuracy, workload, and consistency.
The Stanford Research team found that implementation of the tool saw no adverse consequences and increased the accurate identification of children who needed further intervention services, without increasing the workload on investigators. Researchers also documented reductions in racial disparities in case openings following investigations, resulting from modest but statistically significant increases in cases opened for white children coupled with declines in the rate at which black children were screened in for investigation.
"We implemented the AFST at the end of a lengthy community engagement process in the hope that we could improve accuracy of decision-making at our child welfare's front door, when call screening staff decide whether to investigate a situation that has come to our attention," said Marc Cherna, Director of the Department of Human Services (DHS). "We are encouraged that the AFST has shown positive results by increasing accuracy, while preserving clinical judgement, and we believe that it has great potential as we continuously strive to improve our ability to keep children safe."
The findings of the impact evaluation will help to guide DHS in making additional improvements to the AFST and adaptations to its implementation. In November 2018, an updated version of the AFST (AFST Version 2) was released with significant enhancements to the original tool and accompanying changes to policy and practice. The county will continue to fund an independent evaluation by Stanford University, which will observe AFST Version 2, follow its impact on practice, extend the results in time, and expand the outcomes evaluated to include home removals.
"Allegheny County committed to an independent evaluation of this tool from the start, demonstrating their commitment to knowledge building and transparency," said Rhema Vaithianathan, Co-Director of the Centre for Social Data Analytics, Auckland University of Technology, Auckland, New Zealand, which, through a competitive process, was awarded a contract to develop the AFST for the county. "For the child welfare field at large, the evaluation offers invaluable information and guidance, as more and more agencies consider predictive analytics to support better decisions. We often hear about concerns that predictive analytics might worsen racial disparities, so we are pleased the evaluation finds that the AFST can support decisions that actually reduce racial disparities in the child welfare system."
A summary of the Impact Evaluation can be found here: Impact Evaluation Summary of the Allegheny<http://www.alleghenycountyanalytics.us/wp-content/uploads/2019/05/Impact-Evaluation-Summary-from-16-ACDHS-26_PredictiveRisk_Package_050119_FINAL-5.pdf>
Family Screening Tool<http://www.alleghenycountyanalytics.us/wp-content/uploads/2019/05/Impact-Evaluation-Summary-from-16-ACDHS-26_PredictiveRisk_Package_050119_FINAL-5.pdf>
The full Impact Evaluation can be found here: Impact Evaluation of a Predictive Risk Modeling Tool for Allegheny County's Child Welfare Office<http://www.alleghenycountyanalytics.us/wp-content/uploads/2019/05/Impact-Evaluation-from-16-ACDHS-26_PredictiveRisk_Package_050119_FINAL-6.pdf>
A comprehensive packet featuring the impact evaluation and other reports relevant to the development and implementation of the tool can be found here: Developing Predictive Risk Models to Support Child Maltreatment Hotline Screening Decisions <https://www.alleghenycountyanalytics.us/index.php/2019/05/01/developing-predictive-risk-models-support-child-maltreatment-hotline-screening-decisions/>
Section links for additional information:
Allegheny Screening Tool - Frequently Asked Questions and Answers<http://www.alleghenycountyanalytics.us/wp-content/uploads/2019/05/16-ACDHS-26_PredictiveRisk_Package_050119_FINAL-2.pdf#page=216>
Allegheny County Predictive Risk Modeling Tool Implementation: Process Evaluation<http://www.alleghenycountyanalytics.us/wp-content/uploads/2019/05/16-ACDHS-26_PredictiveRisk_Package_050119_FINAL-2.pdf#page=60>
Ethical Analysis: Predictive Risk Models at Call Screening for Allegheny County<http://www.alleghenycountyanalytics.us/wp-content/uploads/2019/05/16-ACDHS-26_PredictiveRisk_Package_050119_FINAL-2.pdf#page=48>
Ethical Analysis: Predictive Risk Models at Call Screening for Allegheny County - DHS Response<http://www.alleghenycountyanalytics.us/wp-content/uploads/2019/05/16-ACDHS-26_PredictiveRisk_Package_050119_FINAL-2.pdf#page=57>
Developing Predictive Models to Support Child Maltreatment Hotline Screening Decisions: Allegheny County Methodology and Implementation, Version 1<http://www.alleghenycountyanalytics.us/wp-content/uploads/2019/05/16-ACDHS-26_PredictiveRisk_Package_050119_FINAL-2.pdf#page=4>
Allegheny Family Screening Tool: Methodology, Version 2<http://www.alleghenycountyanalytics.us/wp-content/uploads/2019/05/16-ACDHS-26_PredictiveRisk_Package_050119_FINAL-2.pdf#page=194>
# # #
Marc Cherna, Director
Department of Human Services - Office of Community Relations
101 Human Services Building │ One Smithfield Street │ Pittsburgh, PA 15222
www.alleghenycounty.us<http://www.alleghenycounty.us>
Follow us: allegheny alerts<http://www.alleghenycounty.us/alerts> | facebook<http://www.facebook.com/ACDHS> | instagram<http://www.instagram.com/alleghenycounty> | linkedin<http://www.linkedin.com/company/allegheny-county> | twitter<http://twitter.com/ACDHS> | vimeo<http://vimeo.com/acdhs>
Confidentiality Notice: This e-mail message, including any attachments, is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure, forwarding, or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of the original message.
-
image001
From: Beryl Lipton
Hello Elaine,
Thank you so much for this. I'll be sure to let you know if I have any questions.
Best wishes,
Beryl
Files
pages
