The (AI2) and 天美影视传媒 researchers have created an artificial intelligence (AI) system that can solve SAT geometry questions as well as the average American 11th-grade student, a breakthrough in AI research.
This system, called , uses a combination of computer vision to interpret diagrams, natural language processing to read and understand text and a geometric solver to achieve 49 percent accuracy on official SAT test questions. If these results were extrapolated to the entire Math SAT test, the computer roughly achieved an SAT score of 500 (out of 800), the average test score for 2015.
A paper outlining the research, “,” was a joint effort between the UW Computer Science & Engineering department and AI2.
These results, presented at the (EMNLP) in Lisbon, Portugal, were achieved by GeoS solving unaltered SAT questions that it had never seen before and that required an understanding of:
* Implicit relationships
* Ambiguous references
* The relationships between diagrams and natural-language text
鈥淯nlike the Turing Test, standardized tests such as the SAT provide us today with a way to measure a machine鈥檚 ability to reason and to compare its abilities with that of a human,鈥� said , CEO of AI2. 鈥淢uch of what we understand from text and graphics is not explicitly stated, and requires far more knowledge than we appreciate. Creating a system to be able to successfully take these tests is challenging, and we are proud to achieve these unprecedented results.鈥�
Said , senior research manager for Vision at AI2 and UW assistant professor of computer science and engineering, “We are excited about GeoS’s performance on real-world tasks. Our biggest challenge was converting the question to a computer-understandable language. One needs to go beyond standard pattern-matching approaches for problems like solving geometry questions that require in-depth understanding of text, diagram and reasoning.”
How GeoS Works
GeoS is the first end-to-end system that solves SAT plane geometry problems. It does this by first interpreting a geometry question by using the diagram and text in concert to generate the best possible logical expressions of the problem, which it sends to a geometric solver to solve. Then it compares that answer to the multiple-choice answers for that question.
A demonstration of the system鈥檚 problem-solving .
This process is complicated by the fact that SAT questions contain many unstated assumptions.
For example, in the SAT problem at right, there are several unstated assumptions, such as the fact that lines BD and AC intersect at E, that 鈥渃ircle O has a radius of 5鈥� is the same as 鈥渃ircle O radius equals 5鈥� and that the drawing may or may not be to scale.
GeoS had a 96 percent accuracy rate on questions it was confident enough to answer, which is an important dimension of learning. Today, GeoS can solve plane geometry questions; AI2 is moving to solve the full set of SAT math questions in the next three years.
As part of AI2鈥檚 commitment to sharing its research for the common good, all data sets and software are .
AI2 is also building systems that can tackle science tests, which require a knowledge base that includes elements of the unstated, common-sense knowledge that humans generate over their lives. This Aristo project .
Co-authors include lead author , a UW computer science and engineering doctoral student, UW electrical engineering assistant research professor , and former UW undergraduate student Clint Malcolm.
About AI2
was founded in 2014 with the singular focus of conducting high-impact research and engineering in the field of artificial intelligence, all for the common good. AI2 is the creation of Paul Allen, Microsoft cofounder, and is led by Dr. Oren Etzioni, a renowned researcher in the field of AI. AI2 employs more than 35 top-notch researchers and engineers, attracting individuals of varied interests and backgrounds from across the globe. AI2 prides itself on the diversity and collaboration of this team, and takes a results-oriented approach to complex challenges in AI.
About 天美影视传媒 Computer Science & Engineering (UW CSE):
educates tomorrow’s innovators, conducts high-impact research, transfers new discoveries to society and creates opportunities for faculty and students to push the boundaries of a rapidly expanding field while developing solutions to humanity’s greatest challenges.
For more information, contact Hamilton McCulloh at 206-957-4260 or hamiltonm@greenrubino.com or Jennifer Langston at 206-543-2580 or jlangst@uw.edu.