Education Software

Proposal

PREPARED BY
Neil Aylor and Devon Schwartz
1. Letter of Intent
5/9/2021
To whom it may concern,
In Stratford High School’s Academy of Science and Engineering, we have
learned the fundamentals of engineering, from a conceptual level to pragmatic
applications. Our curriculum has covered structural engineering in the contexts of both
terrestrial and aquatic design, as well as forensics and the development of reusable
energy. However, the engineering soft skills that the program has taught us are more
important than the content of our curriculum. Our projects throughout the class have
taught us to become comfortable with the workflow of getting from a problem to an
idea to a testable prototype to a constructed solution. In addition, they have taught us
how to work in teams with fellow engineers even when placed in an imperfect team.
All of these acquired skills and experiences culminate into a final project developed
over the course of an entire year (2021-2022). Because this project is extremely
significant to the program, we feel that the project ought to be significant to the
community that has brought us the program. Therefore, we have decided that our
senior project for this program will focus on easing and improving school teachers’
abilities to provide high quality education through AI-empowered software.
The problem we have identified is that teachers in the United States have been
spread too thin. For example, teachers in the U.S spend more time during the school
day (46 hours per week) teaching than teachers in any other part of the world.1 In
addition, there have been an increase in the number of failing students, particularly
within the Houston-area, as a result of the hardships of distance learning. For example,
there are districts reporting nearly half their middle school and high school students
receiving at least one failing grade in class. This has created an increased need for
intensive support to help these students pass classes.2 Both of these developments are
combined with the fact that the rote nature of school has left students bored, as two
out of three students in a large sample survey report being bored mainly because the
material taught is not intriguing enough.3
Our solution to these problems within the education system is to make artificial
intelligence software that allows teachers to easily convert a text-description of a
lesson plan into visual diagrams, thereby making it easier for educators to create

1
[Link]
are-overworked-feel-underappreciated
2
[Link]
-[Link]
3
[Link]
interesting lessons that can help make classes more intriguing and decrease the
number of students failing at least one class. Although visual diagrams are not a
solution to every problem in the classroom, according to a study by researchers at
Carnegie Mellon University, visual representations when used appropriately will
enhance learning for “low performing students.”4 This is explained by another study by
a psychology professor at Washington University in St. Louis found that visual aids
helped students remember the lecture better.5 Although teachers could create visuals
on their own without the software, they do not do so in the status-quo because
creating visual representations would take time and effort that is already consumed by
their more-vigorous-than-average work schedule. In short, artificial intelligence will
make visual representations far more ubiquitous in classrooms because it makes
creating these representations far easier and less time consuming.
Teachers will interact with this software using a website interface so that even
those unfamiliar with technology can utilize it. On the website, teachers will be
prompted to write an English description of the diagram they want to include in their
lesson. An artificial intelligence program will then generate up to 5 diagrams based on
the teacher’s description, similar to those found in a textbook, and present them to the
user. Then, the user will be able to request modifications to the images in the event
that the AI made slight errors. When the user reports they are happy with their
generated image, they will be given the opportunity to export it to various image
formats, or to copy it directly to their clipboard for direct pasting into their
presentation.
The way we will create this artificial intelligence is through a combination of
multiple machine learning techniques. We begin by constructing a dataset for the
machine learning program to train off of. To do this, we will collect thousands of
diagrams that have been labeled with a plain English description. We must encode
these both into numerical representations because machine learning algorithms are
purely mathematical and therefore cannot interpret images nor words. To encode our
images, we will use a machine learning algorithm called a Variational Auto-Encoder
(VAE). A VAE is a machine-learning assisted compression program that takes an image
as an input, and outputs a numerical representation of the image that only contains
key features of the original (as opposed to containing data about every single pixel).
The VAE can also use this numerical representation to recreate the image. This is
hugely beneficial for this sort of training and image generation, because it allows the

4
[Link]
nt_Representations.pdf
5
[Link]
h-visual-aids/2015/06
model that is generating the image to associate certain words to certain features.
Because these numerical representations of the images have less extraneous data, the
AI outputs less data to generate an accurate image, easing the burden of how much
the AI has to get “right.” Encoding the text is a simpler process: we map groupings of
characters to specific integer identifiers. When we combine these two encodings, we
are able to train a machine learning algorithm to map the relationships between the
words and the objects depicted in an image. After training this machine learning model
to understand these relationships, we can give it a new sentence it has never seen
before. It will generate a new numerical representation of the objects because of its
understanding of the relationship between the inputted words and image features. We
can then take this output and “decompress” it with the VAE to give us an image
representation of the text description, which is what will be outputted to the end user.
Additionally, because the machine learning algorithm is able to take an image as a
prompt in addition to a sentence, we can have the user request modifications to the
original diagram generated by the AI. This machine learning “pipeline” will be accessed
by the users through a website interface hosted on the Internet, and the machine
learning algorithm itself will be hosted on a backend server. This ensures that even
teachers who are not accustomed to technology will be able to utilize this product.
We believe that we can achieve this project given both of our skillset. Both
teammembers of this project, Neil and Devon, have multiple years of experience in
artificial intelligence, including image generation and natural language processing. In
fact, this is not the first time we have worked together on a machine-learning-centric
project. During the summer of 2020, we collaborated to make a
machine-learning-assisted research program for the Stratford debate team. This
program involves natural language processing and artificial intelligence syntenthis of
data, both of which are the main artificial intelligence components of this project.
Furthermore, both team members have been programming for more than 6 years
each, and they are heavily involved in the computer science community. Both are
employees of CodeNinjas, where they teach computer programming to elementary
and middle school students. Neil and Devon are also president and vice president of
Stratford’s computer science club respectively, and we have founded and are deploying
an entirely student-led district-wide computer science summer program that teaches
programming to middle schoolers, of which Devon is president and Neil is
vice-president.
The timeline for this project is in phases of development and testing. First, we
will collect thousands of training images from the Internet, which are diagrams labeled
with plain English descriptions. We have already identified several datasets that are
viable. There are a variety of techniques we could use within the techniques described
above, and there is no way of knowing which will work best until they are each tested,
and therefore we must try all of them to ensure the best possible results. Firstly, we
will be experimenting with different Variational AutoEncoders, or VAEs. We will begin
by developing our own that is specifically designed for illustrations, so that it will be
able to more accurately represent the features depicted in images. Then, we will
evaluate its results by using various VAEs that have been released on the Internet, or
described in research papers, that have been designed for these same generative
processes in its place and comparing outputs. As VAEs are extremely complex, there
are many opportunities to optimize them. The same can be said for the machine
learning program that maps the encoded text to encoded output numerical
representations. We will begin with making our own autoregressive transformer, and
compare outputs to other models described in academic papers that are used for
similar generative tasks. Based on the output of our model, we will modify it until we
get desired outputs. . Finally, we will write a website for this machine learning program
to make it easily accessible to teachers.
The costs of the actual building of the project are none, as all that is needed is a
computer and an internet connection, which both of us already have. Although the
frontend, which is the web interface, can be hosted for free through Github Pages, we
will need a domain name, which costs roughly $10. We will also need to purchase
server space through Amazon Web Services (AWS) to host the machine learning
algorithm, which we estimate will cost us a maximum of $30 / month. However,
because deploying the website is the last step of our timeline, these costs will not
occur until late in the project development process. Additionally, if these costs become
too much, we have the capabilities to set up our own server on one of our home
networks via port forwarding, removing all server side hosting costs. However, our
largest cost is for the use of GPT-3, a state of the art autoregressive transformer model
that is used for generating the image vector when given the text prompt. While we
have other models we plan to test, we believe that the complexities of this
machine-learning task necessitate experimenting with the most powerful technology
we have access to. First, we will utilize GPT-2, the much less powerful but free
predecessor of GPT-3, to test generation and try to determine if we believe we can get
accurate results with GPT-3, as both GPT-2 and GPT-3 use the same fundamental
algorithm. GPT-3 is a premium model, so we have determined that we require about
$7.76 per image generation if we are to use 100 training images and the best version of
GPT-3, but this cost is only if GPT-3’s results by far supersede any other recurrent
neural network we try for generation, and costs can be reduced by using lower cost (at
the cost of somewhat lower power) versions of GPT-3 and by reducing the number of
training images. We will always prefer the least expensive solution that produces
satisfactory results. For the funding of both website hosting costs and the usage of
GPT-3, we request $500 in funding.
The quality of education is undoubtedly one of the most important parts of
modern society, which is why it is important that society augments education with new
and innovative technology to increase the learning of students. Although visual
diagrams are not a permanent solution to the woes of the U.S education system,
enabling the creation of these diagrams with AI, which are proven to increase learning
in the classroom, will make them far more ubiquitous and easy to implement in
classrooms. With little cost to design, implement, and support, this AI can transform
learning in classrooms across the country.

Sincerely,
Neil Aylor and Devon Schwartz

2. Summary
Our project is an artificial intelligence that will convert a simple text description

of a diagram into an image. The goal is to make visual representations more ubiquitous

in the classroom, since the AI will make creating these visual representations easier for

teachers. There is a body of research supporting the notion that visual representations

help students better comprehend lessons, therefore our software will help students

who struggle and risk failing their classes.

First, we construct a dataset for the machine learning pipeline to train from.

This will be done by

1. Collecting thousands of diagrams labelled in plain English - The diagram

will be our desired-output variable, and the labels will be our input

variable in the dataset.

2. Converting the diagrams into numerical representations using a

Variational Auto-Encoder (VAE). We will adapt a VAE from academic

an image to an integer vector in a way that only the key features that

are absolutely necessary for image reconstruction are encoded, and

extraneous information is discarded. In our case these key features are

the objects depicted in the image. This application of the VAE ensures

the computer associates certain words with certain features of the

diagram.

3. Converting the labels into numerical representations by map-grouping

characters into specific integer identifiers. We can use built-in functions

in Python to do this.

After we construct the data set, we will feed it into an autoregressive

transformer. An autoregressive transformer is a machine learning program that solves

a very simple problem: given a sequence of tokens, predict the token that comes next.

This type of machine learning algorithm is able to determine complex relationships

extremely well, to a greater extent than any human could. We can apply this model to

our training data by giving the model the encoded prompt entered by the user, and

requesting it to predict the integer in the image vector that comes next, a process we

will repeat until the vector has been entirely written. This is the best model for the

project because of the abstraction required to derive an image from text, and

additionally because of its one-shot capabilities, or ability to generate from a prompt it

has never seen before. This type model is by far the most forgiving model when it

comes to textual errors and vague descriptions, as it is still able to predict future tokens
no matter how many grammatical errors or misspellings a sentence has, because these

the next-token predictions are not an exact function of any one word or grammar

structure. Although it may take large amounts of training data to get next-token

predictions to be as accurate as we want, academic papers describing similar

generative applications of this exact same technique have proven that it is more than

capable of the results we want.

Once we feed our inputs into the autoregressive transformer, it will give us an

encoded version of the generated visual representation. We can then “decompress”

this generated encoding using our VAE and turn it into an actual reconstructed image

of the AI generated representation. This will be continually trained to match the

desired-output of the model, which is the VAE encoded image. By doing this to

thousands of images, we can train our model to accurately generate what the user

wants from a given text-input.

Once the AI has been trained, we will test the AI by giving a new piece of

information it has never seen before, and it will ideally create a diagram closely

resembling what the user requested. If our output is not exactly what we wanted, we

can call for modifications to the image since the algorithm can also take an image as a

parameter, alongside the text prompt.

Once we have thoroughly tested the algorithm ourselves, we will create a web

interface for our end-users, teachers, to access the algorithm and create their own

diagrams. The front-end of this website, which will be designed in the ReactJS

programming framework, will be hosted on GitHub pages for free. The backend of the
website, which accesses the machine learning pipeline, will be hosted on Amazon Web

Services for approximately $30.00 per month (this is a predicted cost and could vary

depending on the usage of the site and calls made to the pipeline).

2. Introduction
Neil Aylor and Devon Schwartz are both students enrolled in the Academy of

Science and Engineering (ASE) program at Stratford High School, which is a program

for students who excel in STEM. In addition, we are founders of a district-wide virtual

coding camp Let’s Code Spring Branch, we are also instructors at CodeNinjas, a

company that teaches 7-14 year olds how to code, we have both taken and excelled in

AP Computer Science, and we are both experienced in programming that machine

learning and natural language processing.

For our ASE senior project, we are trying to increase the quality of education by

using a machine learning program to convert teacher lesson descriptions into visual

diagrams so that visual diagrams, which are proven in certain situations to be more

effective at teaching students than written notes, will be more ubiquitous and

accessible in classrooms. First, we will collect thousands of diagrams and descriptions

of those diagrams as our training data. Then to train our AI, we create a

machine-learning enabled algorithm called a Variational Auto-Encoder (VAE), adapted

from commonly-used designs and academic papers, to convert the images and text

into numerical representations that the computer can understand. We will pass these

representations into an autoregressive transformer provided by Open AI, which will

Because we can also use our VAE to decode the representation too, the algorithm will

compare the produced diagram to the original one and adjust accordingly to create

better predictions (a commonly used process AI programs). After all the training data

has been passed, when we provide a new description into the algorithm, without an

original image, it will produce 4-5 diagrams similar to what the teacher wants, allowing

them to adjust as necessary. We will host this algorithm on a website so that teachers

can access our machine learning “pipeline” for free.

3. Statement of Need
The education system has struggled with a litany of inherent issues that have

prevented it from ubiquitously implementing visual diagrams and representations on a

wide scale to augment the existing curriculum. One significant issue is that teachers in

the United States are more overworked (46 hours per week) compared to teachers in

other parts of the world.6 This has naturally prevented teachers from integrating more

effective learning strategies into their lessons because adding these strategies would

place an undue burden in tandem with the existing obligations of teachers. However,

there is an increase in failing students, particularly in the Houston area, as a result of

COVID-19-induced distance learning; for example, districts are reporting nearly half

their middle school and high school students receiving at least one failing grade.7

6
[Link]
are-overworked-feel-underappreciated

7
[Link]
-[Link]
 Although visual diagrams are not a total solution to these issues, our AI will

make them easier to create for teachers and help students comprehend more of their

material. Firstly, the ease-of-use of our machine learning program, which will be

provided at no cost to teachers, will make converting written lessons into diagrams a

simple task. Rather than taking an hour, teachers will only have to spend minutes to

create visual representations. As for students, a study by researchers at Carnegie

Mellon demonstrate that visual diagrams, when used appropriately, will enhance the

learning of low-achieving students.8 According to a researcher at Washington

University in [Link], this is probably because visual aids help students remember

their lectures better.9

Since our program is free for users and easy to use, it is a strong first step

towards solving the problem of low-achievement among struggling students while

adapting to the undue burdens placed on teachers in the United States.

4. Objective
Our goal is to create a machine-learning-enabled program that will convert

English text-based lesson descriptions into visual diagrams because visual diagrams are

more effective at helping students retain information from their lectures. To do this, we

will need to scrape the the Internet for a database with visual diagrams and text

8
[Link]
nt_Representations.pdf

9
[Link]
h-visual-aids/2015/06
descriptions of those diagrams as training data for the machine learning algorithm.

Then we will need to develop a Variational Auto-Encoder to convert both the text and

diagrams into numerical representations that the computer can understand. We will

then pass this data through two auto-regressive transformers, the GPT-2 and GPT-3

(the GPT-3 costs money to use), and determine which is more effective in terms of cost

and result. Once we modify our process and create the final product, we will host the

machine learning “pipeline” on a website that is free to use. Our beta version will be

tested at Stratford, and our alpha version will be released for all teachers once we

adjust the algorithm to accommodate feedback from our initial users.

The alpha version will ideally be distributed by contacting officials from the

career and technology departments of school districts throughout the Houston area.

We will present our idea and algorithm to them and market it as a viable means of

enhancing education while accommodating the inherent issue of overworked teachers.

By the end of school year in 2022, our goal is for our machine learning algorithm to be

used in at least 1 school district in the Houston area.

5. Timeline
Month Objective

June 2021 - Start scraping training data and begin designing VAE

July 2021 - Finish scraping training data and continue designing VAE

August 2021 - Finish the first version of the VAE and design total algorithm

September 2021 - Begin passing data through algorithm (passing VAE encoded data to auto-regressive
transformer)
- Adjust VAE and overall algorithm accordingly
October 2021 - Continue modifying algorithm, including the VAE

November 2021 - Continue modifying algorithm, including the VAE

December 2021 - The algorithm will be functional

January 2022 - Start creating website (frontend and backend)

February 2022 - Finish the website and test the machine learning pipeline - this is the beta version

March 2022 - Start asking teachers at Stratford High School to test the algorithm in their classes and
collect feedback

April 2022 - Adjust algorithm/website to adjust to the feedback accordingly

May 2022 - Adjust the alpha version based upon a feedback given through the website

6. Methods
We will be using the “agile method” of software development to create our

product. This process starts off with establishing the requirements of a certain part of

the application, designing that part, implementing it in computer code, and verifying

the effectiveness it. We will apply this process to all parts of the application, and

maintain those parts once the final application is released. When we design a certain

part of the application, we will do it using pseudo code and flowcharts, and when we

implement it we will use the Python programming language.

This process would be applied to the development of the Variational

auto-encoder, the modifications made to the auto-regressive transformer, the

development of the machine learning algorithm, and the development of the website.

When we verify the effectiveness of the website, we will beta test it with our own
inputs and through allowing teachers at Stratford High School to test out the

application. Once the product is thoroughly tested and all the bugs are sorted out, we

will release an alpha version that will be available for all school teachers to use.

The final part of the development process is maintenance, which will happen

once the entire application is released. To ensure that our application consistently

improves, we will adjust parts of the program post-release in response the user

feedback that we will get on the website by releasing small updates accordingly.

7. Evaluation
For this project to be a success, we must complete all of the following: an

autoregressive transformer that accurately generates images, a variational

autoencoder that successfully converts images and text into numerical representations

that can effectively be used by the transformer, and a frontend web interface that

allows users with limited technology skills to still make use of the project. Additionally,

the images that are produced by the product must be accurate representations of the

prompt, and the images must be able to be used in presentations as teaching tools.

When we determine the accuracy and quality of the images, we will do it based upon

an evaluation by us, the engineers, and teachers, who are our end-users. Once we are

satisfied with the project, we will roll out a phase of beta testing with Stratford

teachers to ensure they are satisfied with the results, as the perceived quality and

accuracy of the outputted images is entirely subjective. Additionally, a beta testing

phase allows us to test the AI’s natural language processing abilities when it is exposed
to a variety of writing styles. A final criteria of our project is that the AI creates a

significant positive impact for teachers. We will determine the positive impact for

teachers by including a survey for all the users to fill out, which will ask questions

concerning the AI’s significance in the classroom and the AI’s effect on the students’

comprehension of classroom material.

8. Budget
The budget for this project is relatively small. Although the GPT-2 auto

regressive transformer is completely free to use, we will allocate $500 towards testing

out the GPT-3 auto regressive transformer to see if the GPT-3 is significantly better than

the GPT-2. Starting in February, we will start spending approximately $30 a month for

Amazon Web Services to host the machine learning algorithm. Finally, we will dedicate

$10 towards extraneous costs, namely hiring a data entry person on Fiverr to help

enter the training data needed to train the auto-regressive transformer.

9. Conclusion
This project will make diagram learning more ubiquitous by creating a

free-to-use machine learning program for teachers that will convert text-based lessons

into visual diagrams since visual diagrams have been proven to be more effective at

helping students retain information from class.

The machine learning program will work by converting training data (diagrams

and their descriptions) into numerical representations the computer can understand

using a Variational Auto-Encoder (VAE) adapted from the Internet and academic
papers. This encoded data will be passed through an auto-regressive transformer to

train the machine learning algorithm. The machine transformer will produce an

encoded version of the diagram, which will be decoded using the same VAE. After the

algorithm is trained, a new text input will be given and 5 diagrams will be generated for

the user to choose from. The user will be able to make modifications to their generated

diagrams so that they get their desired output.

Once we develop and test the algorithm, we will implement it on a website

hosted on Amazon Web Services. We will then beta test the application by allowing

Stratford teachers to use the application in their classrooms, and we will change the

application accordingly based upon their feedback. Once this is completed, we will

release an alpha version of the application available for all school teachers to use. We

will have a feedback form on the website so that we can make improvements as

necessary.

Ideally, this application will be used consistently by one school district in the

Houston area. We will publicize this application by speaking with the Career and

Technology departments of local school districts.

10. Appendix
(Blog link - [Link]
 Detailed Budget

Part Link Cost Purpose

Data Entry [Link] $10 The data-entry person will be hired on

Amazon Web [Link] $30 This will host the backend of our
Services [Link] per application, where the machine learning
/ month pipeline will operate. Costs will vary
(estim depending on usage, but $30 is a typical
ated) estimate.

Open AI GPT-3 [Link] $0.06 We will test the effectiveness of the GPT-3
[Link]/ per Da Vinci Engine against the effectiveness
pricing 1000 of the free-to-use GPT-2 to determine if
tokens the GPT-3 is significantly better.
$500
total
estima
te
View Generated Images