![](\"https:\/\/images-na.ssl-images-amazon.com\/images\/M\/MV5BMDgzNDk3ZDctYTUxMC00MDQzLWI1MTctMmQxYWE0MGUzYzIxXkEyXkFqcGdeQXVyNTIzOTk5ODM@._V1_SY1000_CR0,0,675,1000_AL_.jpg\")
One of the things that we in technology understand is that we make mistakes, a truth we don\u2019t like to admit to customers.\u00a0 What happens, however, when a mistake can lead to tragic consequences?<\/p>\nOne of the things that we in technology understand is that we make mistakes, a truth we don\u2019t like to admit to customers.\u00a0 What happens, however, when a mistake can lead to tragic consequences?<\/p>\n
Yesterday\u2019s New York Times reports about a case<\/a> that the U.S. Supreme Court may soon hear, involving a man who received a six year jail sentence, in part due to a computer program.\u00a0 The software known as Compas was supposedly developed by Northpointe Inc. (although a search seems to redirect to a Equivant) to provide a risk assessment of a person\u2019s reentry into society.\u00a0 Such a data-driven analysis is vaguely reminiscent of the movie, Minority Report<\/a>.\u00a0 In this case, the defendant Eric L. Loomis was not allowed to examine the software that assessed that he was a significant risk to the community, even though at least one analysis showed that the software may be programmed with some form of racial bias.\u00a0 The company argues that the algorithm used to make the sentencing recommendation is proprietary, and so should not be subject to review, and that if they release their algorithm to scrutiny they will essentially be giving away their business model, and they may have a point.\u00a0 Patents on such technology may be flimsy, and they eventually do come to a halt.\u00a0 To protect themselves, they make use of another legal tool, the trade secret, which has no fixed term of protection.<\/p>\n One can\u2019t say that a mistake is being made in the case of Mr. Loomis, nor can one authoritatively state that the program is formally correct.\u00a0 The Wisconsin Supreme Court argued creatively that much like college admissions, so long as the software is one input combined with others, the software can be used.\u00a0 Is it, therefore, any different from a potentially flawed witness giving evidence?\u00a0 The question here is whether those who wrote the software can be cross-examined, to what extent they may be questioned, and whether the software itself can be examined.\u00a0 Mr. Loomis argues that to deny his legal team access to the source is a violation of his 14th<\/sup> Amendment<\/a> right to due process.<\/p>\n We know from recent experience that blind trust in technology, and more precisely, those who create and maintain it, can lead to bad outcomes.\u00a0 Take for instance the over 20,000 people whose convictions were overturned<\/a> because a chemist falsified hair analysis results, or other examples where the FBI Crime Lab just flat got it wrong<\/a>.\u00a0 Even Brad D. Schimel, the Wisconsin attorney general, conceded before the appeals court that, \u201cThe use of risk assessments by sentencing courts is a novel issue, which needs time for further percolation.\u201d\u00a0 But what about Mr. Loomis and those who may suffer tainted results if there is a software problem?<\/p>\n While the Supreme Court could rule soon on the matter, they will only have very limited avenues, such as permitting or prohibiting its use.\u00a0 Congress may need to get involved in order to provide other alternatives.\u00a0 One possibility would be to provide the company some new intellectual property protection, such as an extended patent with additional means of enforcement (e.g., higher penalties against infringement or lower thresholds for discovery) in exchange for releasing the source.\u00a0 Even if they do, one question would be whether or not defendants could then game the system so as to score better on sentencing.\u00a0 How great a risk that is we can\u2019t know without knowing what the inputs to the algorithm are.<\/p>\n It is probably not sufficient for the defendant and his legal teams to have access to the source, precisely because more research is needed in this field to validate the models that software like Compas uses.\u00a0 That can\u2019t happen unless researchers have that access.<\/p>\n","protected":false},"excerpt":{"rendered":" Could you spend a long time in prison due to a software bug and not have the right to examine the software? Possibly.<\/p>\n","protected":false},"author":172,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[30,4],"tags":[566,567],"class_list":["post-2218","post","type-post","status-publish","format-standard","hentry","category-humanity","category-politics","tag-jail","tag-software-bugs"],"_links":{"self":[{"href":"https:\/\/ofcourseimright.com\/index.php?rest_route=\/wp\/v2\/posts\/2218","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/ofcourseimright.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/ofcourseimright.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/ofcourseimright.com\/index.php?rest_route=\/wp\/v2\/users\/172"}],"replies":[{"embeddable":true,"href":"https:\/\/ofcourseimright.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=2218"}],"version-history":[{"count":3,"href":"https:\/\/ofcourseimright.com\/index.php?rest_route=\/wp\/v2\/posts\/2218\/revisions"}],"predecessor-version":[{"id":2221,"href":"https:\/\/ofcourseimright.com\/index.php?rest_route=\/wp\/v2\/posts\/2218\/revisions\/2221"}],"wp:attachment":[{"href":"https:\/\/ofcourseimright.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=2218"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/ofcourseimright.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=2218"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/ofcourseimright.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=2218"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}