Extraocular Image Processing for Optoelectronic/Optogenetic Retinal Prosthesis

CHAPTER 1
INTRODUCTION AND MOTIVATION

1.1 INTRODUCTION
The restoration of vision to patients suffering from incurable blinding diseases is a
remarkable challenge being addressed by several research groups across the world
Recent patient trials have demonstrated that intraocular electrical stimulation using
retinal implants can return a form of basic vision. based on phosphene percept
[1], [2]. This has been sufficient to enable subjects to perform simple tasks such as
reading single, large, and high contrast letters and shapes. Such demonstrations
provide hope that one day, functional vision can be returned with this technique.
Retinal prosthesis is primarily aimed at patients blinded by outer retinal diseases,
particularly retinitis pigmentosa (RP), a class of hereditary disorders affecting
∼1:4000 [3]. It causes dysfunction of the rod photoreceptors, resulting in night
blindness, tunnel vision, and, eventually, for some patients total blindness. Numerous
treatments are under investigation for RP including neurotropic factors for
photoreceptor protection, retinal
transplant, stem cell, treatment for macular oedema. Perhaps the most promising is
gene therapy [4]. So far, efficacy has been shown for single gene defects such as
RPE65 [5], but RP has clusters of involved genes that make it much more difficult to
treat [6]. Consequently, there is currently no effective treatment available for this
condition. In the 1990s, Stone et al. [7] discovered that the retinal ganglion cells
(RGCs) were still intact in RP patients, even after the onset of full blindness. This
formed the basis for subsequent work on electrical retinal prosthesis targeting RGCs.
The quality of prosthetic vision is determined by the number of pixels, where they are
placed, and their ability to stimulate a signal the brain can understand. Simulations of
pixelized vision have suggested that at least several hundred electrodes are needed to
perform useful visual functions such as navigation or face recognition [8]. In
particular, Cha et al. [9], [10] suggested, from measurement of simulated pixelized
vision with normally sighted subjects, that a 25×25 pixel array produced a vision with
field of view of 1.7◦ and 6/9 visual acuity. The retina is structured so that the central
fovea is dedicated toward high-resolution spatial imaging and the periphery more

ME E&TC (Signal Processing), Sem –III, DYPSCOEA, Ambi. Savitribai Phule Pune
University
1
 Extraocular Image Processing for Optoelectronic/Optogenetic Retinal Prosthesis

toward spatiotemporal imaging. The type of vision returned is then dependent on

where the electrode is placed. In practice, numerous issues have impeded the
realization of functional retinal prosthesis. Flat implanted chips need to be small
within the curvature of the eye, resulting in a tunnel vision. The RGCs connected to
the high resolution fovea are bunched up around the macula, meaning that it will be
very difficult to return foveal vision. As the number of electrodes starts to scale
toward useful vision, power dissipation in the retina can start to become a significant
problem [11]. This, in turn, can cause degradation problems in the electrodes
themselves. Such problems are the basis for significant investigation, and the general
perspective is that rudimentary but increasingly better functioning systems will be
available in the coming years. Recently, we have proposed a novel, optogenetic/
optoelectronic retinal prosthesis, which offers a solution to some of these challenges
[12]. This technology is based on the photosensitization of neurons using light-gated
ion channels and pumps, which have been successfully expressed in RGC’s [13],
bipolar cells [14], and photoreceptors [15]. Such an approach requires high brightness
illumination, and we have, therefore, previously presented LED microarray
stimulators with up to 4096 pixels [12] with the capacity to scale further. Optogenetic
neural stimulation has, thus far, been demonstrated in primates [16] without negative
immune responses. Nevertheless, as RGCs project to the visual cortex, a potential
inflammatory response there could have serious negative consequences for the
patient. It is, therefore, most likely that early human trials in the coming years will
focus on bipolar cells [14] and persisting cones that have lost light sensitivity [15].
Nevertheless, as RP is a progressive disease which also damages these layers, long-
term implementation in the RGC layers may also be necessary. Whichever approach
is used, electronic or optogenetic, it is unlikely that we will return perfect vision in the
near future. More likely, the resulting vision from a higher resolution retinal
prosthesis is likely to have characteristics of both early stage RP patients and age-
related macular degeneration (AMD) patients (i.e., tunnel vision with very poor visual
acuity). Previously, we have shown that effective contrast enhancement algorithms,
such as cartoonization, can improve visual recognition in visually impaired but not
blind patients with retinal degenerative disorders [17]. We also demonstrated a seam-
assisted shrinkability (SAS) algorithm [18] to spatially compress the non important

ME E&TC (Signal Processing), Sem –III, DYPSCOEA, Ambi. Savitribai Phule Pune
University
2
 Extraocular Image Processing for Optoelectronic/Optogenetic Retinal Prosthesis

features of the visual scene, increasing the effective field of view for those having
tunnel vision problems. Additionally, depending on the target retinal layer to be
stimulated, some of the retinal processing will have to be performed. Although the
retina is complex, the full coding of the exiting different types of RGCs is still under
investigation [19]–[21]. However, reduced spatiotemporal models that can be
processed in real time on portable platforms may prove sufficient when combined
with the plasticity of the brain to adapt interpretation. In this paper, we present the
processing platform for a high-resolution optogenetic retinal prosthesis. The scene
enhancement algorithms have been incorporated into the system to improve the ability
of the patient to use the prosthetic vision to perform visual tasks. To optimize the
quality of this prosthetic vision, the platform also includes algorithms that mimic
retinal processing to reproduce the ganglion cell spike patterns from healthy retina,
taking into account the target neurons and the biophysics of the channel rhodopsin
[14]. To ensure that the platform is scalable, we also present a control algorithm to
meet the power consumption limitations. As proof of concept, we implemented the
platform in a system including camera acquisition and optoelectronic stimulator and
demonstrate its use with ex vivo preparations.
1.2 AGE-RELATED MACULAR DEGENERATION (AMD)
AMD is a common eye condition and a leading cause of vision loss among people age
60 and older. It causes damage to the macula, a small spot near the center of the retina
and the part of the eye needed for sharp, central vision, which lets us see objects that
are straight ahead. In some people, AMD advances so slowly that vision loss does not
occur for a long time. In others, the disease progresses faster and may lead to a loss of
vision in one or both eyes. As AMD progresses, a blurred area near the center of
vision is a common symptom. Over time, the blurred area may grow larger or you
may develop blank spots in your central vision. Objects also may not appear to be as
bright as they used to be AMD by itself does not lead to complete blindness, with no
ability to see. However, the loss of central vision in AMD can interfere with simple
everyday activities, such as the ability to see faces, drive, read, write, or do close
work, such as cooking or fixing things around the house.

ME E&TC (Signal Processing), Sem –III, DYPSCOEA, Ambi. Savitribai Phule Pune
University
3
 Extraocular Image Processing for Optoelectronic/Optogenetic Retinal Prosthesis

1.2.1 The Macula

The macula is made up of millions of light-sensing cells that provide sharp, central
vision. It is the most sensitive part of the retina, which is located at the back of the
eye. The retina turns light into electrical signals and then sends these electrical signals
through the optic nerve to the brain, where they are translated into the images we see.
When the macula is damaged, the center of your field of view may appear blurry,
distorted, or dark.

1.2.2 Major Causes Of AMD

Age is a major risk factor for AMD. The disease is most likely to occur after age 60,
but it can occur earlier. Other risk factors for AMD include:
Smoking: Research shows that smoking doubles the risk of AMD
Race: AMD is more common among Caucasians than among African-Americans or
Hispanics/Latinos.
Family history: People with a family history of AMD are at higher risk.

1.2.3 Detection Methods Of AMD

The early and intermediate stages of AMD usually start without symptoms. Only a
comprehensive dilated eye exam can detect AMD. The eye exam may include the
following:
[Link] acuity test: This eye chart measures how well you see at distances .

ME E&TC (Signal Processing), Sem –III, DYPSCOEA, Ambi. Savitribai Phule Pune
University
4
 Extraocular Image Processing for Optoelectronic/Optogenetic Retinal Prosthesis

[Link] eye exam: Your eye care professional places drops in your eyes to widen or
dilate the pupils. This provides a better view of the back of your eye. Using a special
magnifying lens, he or she then looks at your retina and optic nerve for signs of AMD
and other eye problems.

[Link] grid: Your eye care professional also may ask you to look at an Amsler grid.
Changes in your central vision may cause the lines in the grid to disappear or appear
wavy, a sign of AMD

Here is what asmbler is normally This is what asmbler grid might

looks like looks like to someone with AMD

[Link] angiogram: In this test, which is performed by an ophthalmologist, a

fluorescent dye is injected into your arm. Pictures are taken as the dye passes through
the blood vessels in your eye. This makes it possible to see leaking blood vessels,
which occur in a severe, rapidly progressive type of AMD (see below). In rare cases,
complications to the injection can arise, from nausea to more severe allergic reactions

Optical coherence tomography. You have probably heard of ultrasound, which uses
sound waves to capture images of living tissues. OCT is similar except that it uses
light waves, and can achieve very high-resolution images of any tissues that can be
penetrated by light such as the eyes. After your eyes are dilated, you’ll be asked to
place your head on a chin rest and hold still for several seconds while the images are
obtained. The light beam is painless.
During the exam, your eye care professional will look for drusen, which are
yellow deposits beneath the retina. Most people develop some very small drusen as a

ME E&TC (Signal Processing), Sem –III, DYPSCOEA, Ambi. Savitribai Phule Pune
University
5
 Extraocular Image Processing for Optoelectronic/Optogenetic Retinal Prosthesis

normal part of aging. The presence of medium-to-large drusen may indicate that you

have AMD. Another sign of AMD is the appearance of pigmentary changes under
the retina. In addition to the pigmented cells in the iris (the colored part of the eye),
there are pigmented cells beneath the retina. As these cells break down and release
their pigment, your eye care professional may see dark clumps of released pigment
and later, areas that are less pigmented. These changes will not affect your eye color.
1.2.4 Stages Of AMD
There are three stages of AMD defined in part by the size and number of drusen under
the retina. It is possible to have AMD in one eye only, or to have one eye with a later
stage of AMD than the other.

[Link] AMD: Early AMD is diagnosed by the presence of medium-sized drusen,

which are about the width of an average human hair. People with early AMD
typically do not have vision loss.

[Link] AMD:People with intermediate AMD typically have large drusen,

pigment changes in the retina, or both. Again, these changes can only be detected
during an eye exam. Intermediate AMD may cause some vision loss, but most people
will not experience any symptoms.

[Link] AMD:In addition to drusen, people with late AMD have vision loss from
damage to the macula. There are two types of late AMD:
In geographic atrophy (also called dry AMD), there is a gradual breakdown of the
light-sensitive cells in the macula that convey visual information to the brain, and of
the supporting tissue beneath the macula. These changes cause vision loss.
In neovascular AMD (also called wet AMD), abnormal blood vessels grow
underneath the retina. (“Neovascular” literally means “new vessels.”) These vessels
can leak fluid and blood, which may lead to swelling and damage of the macula. The
damage may be rapid and severe, unlike the more gradual course of geographic
atrophy. It is possible to have both geographic atrophy and neovascular AMD in the
same eye, and either condition can appear first.

ME E&TC (Signal Processing), Sem –III, DYPSCOEA, Ambi. Savitribai Phule Pune
University
6
 Extraocular Image Processing for Optoelectronic/Optogenetic Retinal Prosthesis

CHAPTER 2
OBJECTIVE AND PROBLEM STATEMENT
2.1 OBJECTIVE
To implement method to simplify the scene, perform spatial image compression, and
then apply spike coding. Show the potential for translation on standard consumer
processors. The algorithms are applicable to all forms of visual prosthesis, but it
particularly focus on optogenetic approaches.

2.2 PROBLEM STATEMENT

The field of retinal prosthesis has been steadily developing over the last two decades.
Despite the many obstacles, clinical trials for electronic approaches are in progress
and already demonstrating some success. Optogenetic/optoelectronic retinal
prosthesis may prove to have even greater capabilities. Although resolutions are now
moving beyond recognition of simple shapes, it will nevertheless be poor compared to
normal vision. If we define the aim to be to return mobility and natural scene
recognition to the patient, it is important to maximize the useful visual information
we attempt to transfer.

ME E&TC (Signal Processing), Sem –III, DYPSCOEA, Ambi. Savitribai Phule Pune
University
7
 Extraocular Image Processing for Optoelectronic/Optogenetic Retinal Prosthesis

CHAPTER 3
LITERATURE SURVEY
Humayun, M.S[1]: The Argustrade II 60 channel epiretinal prosthesis has been
developed in order to provide partial restoration of vision to subjects blinded from
outer retinal degenerative disease. To date the device has been implanted in 21
subjects as part of a feasibility study. In 6 month post-implantation door finding and
line tracking orientation and mobility testing, subjects have shown improvements of
86% and 73%, respectively, for system on vs. system off. In high-contrast square
localization tests using a touch screen monitor 87% of tested subjects performed
significantly better with the system on compared with off.
These preliminary results show that the Argus II system provides some functional
vision to blind subjects.

Zrenner, E.[2]: Eleven patients received subretinal implants, powered and controlled
via a subdermal cable ending in a thin intraocular foil, placed transsclerally between
the retinal pigment epithelium and the neuroretina. The tip of this foil carries two
distinct arrays, a Multiphotodiode Array (MPDA) with 1500 electrodes, each
electrode being controlled by an adjacent photodiode and an amplifier within a
3times3times0.1 mm chip, as well as a second array with 16 electrodes, for direct
stimulation (DS). Subretinalmultielectrode implants with currents close to recognition
threshold (10 to 27 nC/electrode) produce retinotopically correct patterns
that allow for the first time recognition of individual letters (8 cm high, viewed in
appr. 62 cm distance) even at low luminance levels. Stripe patterns of moderate
luminance can be resolved up to 0.35 cycles/deg via the subretinal chip. This clearly
supports the feasibility of light sensitive subretinal multi electrode devices for
restoration of useful visual percepts in blind patients.

[Link][25]:This paper mainly discusses two approaches in image

processing which reduces the size of the image without loss of the object detection
rate to that of the original image. One is about the related image processing

ME E&TC (Signal Processing), Sem –III, DYPSCOEA, Ambi. Savitribai Phule Pune
University
8
 Extraocular Image Processing for Optoelectronic/Optogenetic Retinal Prosthesis

algorithms include image resizing, color erasing, edge enhancement and edge
detection. Second one is to generate the saliency map for an image.
McGovern, B.[34]: Here, we demonstrate the use of a micro light emitting diode
(LED) array as a powerful tool for complex spatiotemporal control of photosensitized
neurons. The array can generate arbitrary, 2-D, excitation patterns with millisecond
and micrometer resolution. In particular, we describe an active matrix control address
system to allow simultaneous control of 256 individual micro LEDs. We present the
system optically integrated into a microscope environment and patch clamp
electrophysiology. The results show that the emitters have sufficient radiance at the
required wavelength to stimulate neurons expressing channelrhodopsin-2 (ChR2).

Asher, A[40]:In an effort to restore visual perception in retinal diseases such as age-
related macular degeneration or retinitis pigmentosa, a design was recently presented
for a high-resolution optoelectronic retinal prosthesis having thousands of
electrodes. This system requires real-time image processing fast enough to convert a
video stream of images into electrical stimulus patterns that can be properly
interpreted by the brain. It present image-processing and tracking algorithms for a
subretinal implant designed to stimulate the second neuron in the visual pathway,
bypassing the degenerated first synaptic layer. For this task, he has developed and
implemented: 1) A tracking algorithm that determines the implant's position in each
frame. 2) Image cropping outside of the implant boundaries. 3) A geometrical
transformation that distorts the image appropriate to the geometry of the fovea. 4)
Spatio-temporal image filtering to reproduce the visual processing normally
occurring in photoreceptors and at the photoreceptor-bipolar cell synapse. 5)
Conversion of the filtered visual information into a pattern of electrical current.
Methods to accelerate real-time transformations include the exploitation of data
redundancy in the time domain, and the use of precomputed lookup tables that are
adjustable to retinal physiology and allow flexible control of stimulation parameters.
A software implementation of these algorithms processes natural visual scenes with
sufficient speed for real-time operation. This computationally efficient algorithm
resembles, in some aspects, biological strategies of efficient coding in the retina and
could provide a refresh rate higher than fifty frames per second on our system

ME E&TC (Signal Processing), Sem –III, DYPSCOEA, Ambi. Savitribai Phule Pune
University
9
 Extraocular Image Processing for Optoelectronic/Optogenetic Retinal Prosthesis

Yuexian Zou[41]:Simulations of artificial vision suggest that thousands of electrodes

may be required to restore vision for ones with diseases of the outer retina. With the
development of MEMS fabrication process for the stimulation electrode array,
extraocular image processing is becoming more and more important for the
retinal prosthesis systems. A Digital Signal Processor (DSP) based
extraocular image processing system (EIPS) for a retinal prosthesis has been
developed in this paper. The system mainly consists of a CMOS image sensor and a
DSP processing system, which provides the capability of implementing the real-
time image processing with low power consumption. Furthermore, this system offers
the flexibility of realizing various image processing algorithms with different
specification requirements on the DSP by programming, such as different frame rate,
resolution and throughput data rate. The related image processing algorithms include
the image resizing, color erasing, edge enhancement and edge detection. Finally, the
speed of different DSPs in the market has been evaluated and compared for achieving
better performance.

Hao Wu[42]:Retinal prosthesis provides a promising solution for retinitis pigmentosa

(RP) and aged-related macular degeneration (AMD), which are the two
major retinal degenerative diseases that cause a loss of vision. Clinical trials reported
different irregular characteristics of phosphene map elicited by retinal prosthesis
including irregular shaped phosphene, distorted phosphene array and dropout
phenomenon. Psychological experiments under simulated prosthetic vision can be
carried out to evaluate the ability of performing daily activities for prosthesis wearers.
A prosthetic vision simulating system for retinal prosthesis is established based on a
digital signal processor (DSP) TMS320DM642. In this system,
external images captured by a microcamera are processed on the DSP and the
simulation results are presented in a head-mounted display (HMD) after
several image processing steps. Mathematic models are used to create phosphene
maps with irregular characteristics including irregular shaped phosphene, distortion
and dropout. This system can achieve a processing speed of 24 frames per second
which can meet the requirement of the real-time processing. The portability and the

ME E&TC (Signal Processing), Sem –III, DYPSCOEA, Ambi. Savitribai Phule Pune
University
10
 Extraocular Image Processing for Optoelectronic/Optogenetic Retinal Prosthesis

real-time processing ability allow extensive psychological experiments to be

conducted based on this system.

Parikh, N.J.[43]:Retinal prosthesis recipients may still have degraded vision, such that
additional information about their surroundings may help them perform certain tasks.
He evaluate a system that provides cues that point towards important objects. Using a
simulated vision grid of 6times10 pixels, subjects perform object location and
mobility tasks with and without the help of cues. The velocity of head movement in
degrees per second and the time taken by subjects to finish the tasks are recorded.
Results show that a cueing system may help to reduce and organize the head
movements of the subjects, whereas a time benefit exists in object location but not in
mobility tasks.

Yuexian Zou[44]: The image processing of retinal prosthesis system converts the
original images from the camera to the stimulus pattern that can be properly
interpreted by the brain. Practically, the original images are with much high resolution
(256×256) than that of the stimulus pattern (such as 25×25), which causes a technical
challenge to extract the stimulus pattern from the original image. In this paper, focus
is on developing an efficient stimulus pattern extraction algorithm by using the single
cue saliency map, where the salient objects in the image with an optimal trimming
threshold are extracted. Experimental results show that the proposed stimulus pattern
extraction algorithm performs quite well for different scenes in terms of the
perception of the stimulus pattern. Some suggestions are also given on trimming
threshold selection for different scenes.

Ayton, L.N.[ 45]:Recent advances in the field of visual prostheses or “bionic eyes”
have shown that it is possible to use electrical stimulation to produce basic
phosphenised vision to patients who are profoundly vision impaired or blind. In
particular, retinal prostheses have been implanted in a number of clinical trials for a
degenerative eye disease known as retinitis pigmentosa. To date, the visual
improvements in these trials have been small and not easily quantified. The aim of
this paper is to highlight the inherent complexities in the assessment of visual function

ME E&TC (Signal Processing), Sem –III, DYPSCOEA, Ambi. Savitribai Phule Pune
University
11
 Extraocular Image Processing for Optoelectronic/Optogenetic Retinal Prosthesis

in the profoundly vision impaired, and discuss the potential for improvement in
outcomes using image processing technology.

Kyle D. Slater*[3]: A system is presented which can deliver charge-balanced,

constant-current biphasic pulses, with widely adjustable parameters, to arbitrary
configurations of output electrodes. This system is shown to be effective in eliciting
visual percepts in a patient with approximately 20 years of light perception vision
only due to retinitis pigmentosa, using an electrode array implanted in the
suprachoroidal space of the eye. The fexibility of the system also makes it suitable for
use in a number of other emerging clinical neuro-stimulation applications, including
epileptic seizure suppression and closed-loop deep brain stimulation. Clinical trial
registration number NCT01603576 ([Link]).

ME E&TC (Signal Processing), Sem –III, DYPSCOEA, Ambi. Savitribai Phule Pune
University
12
 Extraocular Image Processing for Optoelectronic/Optogenetic Retinal Prosthesis

CHAPTER 4
EXISTING TREATMENTS FOR AMD

Retina International and its members support extensive multi-disciplinary research

programmes in an effort to find the causes prevention, and possible treatment for
Macular Degeneration and other forms of retinal degenerations. Researches studying
retinal degenerative diseases may contribute to the understanding of others so that, for
example, research related to RP may also benefit those with Macular Degeneration.
Information from research projects is shared with others in the field in order to
advance the goal of understanding all forms of retinal degeneration. Listed below are
details of some of the latest advances in research and treatments for AMD.
Dry AMD:Although several new drugs are being investigated and approved by
regulatory agencies around the world for the treatment of the exudative (wet) type of
AMD, aside from cessation of smoking and a healthy diet of dark green leafy
vegetables and fruits supplemented by zinc and anti-oxidant vitamins (Vitamins E, C,
and beta carotene), very little is available to help patients with atrophic or "dry" AMD
to prevent progression to more serious stages of debilitating disease. A blood filtration
process, called Rheopheresis, is being marketed in Canada as a treatment for dry
AMD by Occu Logix Inc. Little is known about what causes the conversion from dry
to wet AMD, and this is the subject of ongoing research studies. Check back regularly
for updates to this website as research studies are published.
Wet AMD:At present people with macular degeneration have three possible treatment
options: thermal (heat laser); Photodynamic Therapy; or anti-VEFG drugs.

4.1 LASER PHOTOCOAGULATION

Laser photocoagulation is a surgical procedure involving the application of a hot laser
to seal and halt or slow the progression of abnormal blood vessels. In the 1990's laser
treatment was the only therapy available for AMD.

ME E&TC (Signal Processing), Sem –III, DYPSCOEA, Ambi. Savitribai Phule Pune
University
13
 Extraocular Image Processing for Optoelectronic/Optogenetic Retinal Prosthesis

Through the use of a high-energy light that turns to heat when it hits the parts of the
retina to be treated, laser photocoagulation seals the choroidal neovascularization
(CNV) and inhibits the leaky blood vessels growth, preventing further vision
deterioration. A scar forms as a result of the treatment, and this scar creates a
permanent blind spot in the field of vision. Vision does not usually improve after laser
treatment and may even be somewhat worse. However, loss of vision following laser
treatment, though immediate, is generally less severe than the eventual loss of vision
that usually occurs if laser treatment is not done. In many cases, some visual
distortion will disappear after laser treatment.

4.2PHOTO DYNAMIC THERAPY (PDT)

Photodynamic Therapy (PDT) (trade name Visudyne) uses a non-thermal (or cold)
laser with an intravenous light-sensitive drug to seal and halt or slow the progression
of abnormal retina blood vessels. This treatment does not produce a blind spot on the
retina. The light is shone directly at the targeted tissue and the drug accumulates in
these cells. It therefore reduces damage to normal surrounding tissue and allows the
treatment to be given again as needed. . However, early diagnosis of AMD is key,
because once vision is lost due to of the growth of abnormal blood vessels, it cannot
be reclaimed by either treatment.

4.3 ANTI-ANGIOGENESIS THERAPIES

As of February 2006, pegaptanib sodium (trade name Macugen) is approved for use in
Canada, the United States and Europe. The United States Food and Drug
Administration (FDA) approved Macugen for treatment of neovascular (wet) age-
related macular degeneration. FDA approval came following successful clinical trials
demonstrating that the drug reduced vision loss in 70 per cent of clinical trial patients.
It is also very encouraging that the drug is effective for all kinds of wet AMD,
whether in the early or late stages.
Pegaptanib sodium (trade name Macugen) is what researchers call an anti-VEGF
drug, or in other words, a drug which works by targeting the proteins which act to
trigger abnormal blood vessel growth and leakage. Anti-VEGF drugs are delivered
directly to the eye by an injection, which is repeated every four to six weeks.

ME E&TC (Signal Processing), Sem –III, DYPSCOEA, Ambi. Savitribai Phule Pune
University
14
 Extraocular Image Processing for Optoelectronic/Optogenetic Retinal Prosthesis

Other anti-VEFG drugs on the horizon include ranubizimab (trade name Lucentis),
from Genetech and Novartis. On June 30, 2006, the US Food and Drug
Administration (FDA) announced approval of Lucentis (Ranibizumab). AMD
Alliance International (AMDAI) loudly applauded the decision, which effectively
makes available in the USA a ground breaking treatment for wet age related macular
degeneration. This approval is based on the evidence presented from several years of
rigorous clinical trials, in which Lucentis was shown to maintain vision in 95% of
trial participants, and improve vision in approximately 30 to 40% of trial participants.
This decision means that treatment with Lucentis will now be widely available in the
USA through retinal specialists. The FDA approval of course only covers the USA.
Introduction of Lucentis in Europe is expected to follow in the coming year. Fighting
Blindness and the AMD Alliance International of which it is a member, applauds the
introduction of new treatments, which bring hope and help to those with macular
degeneration.

4.4ANGIOSTATIC THERAPIES
In other research developments, a completely different class of AMD drugs, called
angiostatic therapies, is showing promise. This class of drugs propose yet another
approach to treatment of AMD, in this case by administering a type of steroid to stop
the abnormal growth of blood vessels in the eye. Unlike the anti-VEGF treatments,
angiostatic drugs are delivered through a canula, to the back of the eye.

One possible angiostatic treatment is anecortave acetate (Retaane), from Alcon

Laboratories. Although early clinical results were not as stellar as hoped, scientists
working on the treatment believe this may be a result of drug delivery problems, not
the drug itself and are making adjustments. On May 24, 2005, the USA Food and
Drug Administration released what is called an "approvable" letter, basically meaning
that the drug is approvable but some further study is required. Alcon recently reported
that their researchers and officials will "meet with the FDA to discuss the approvable
letter, the clinical studies submitted with the NDA and other ongoing clinical studies
for RETAANE® suspension to determine the steps necessary to gain final approval
for the wet AMD indication." Retaane has received market approval for use in

ME E&TC (Signal Processing), Sem –III, DYPSCOEA, Ambi. Savitribai Phule Pune
University
15
 Extraocular Image Processing for Optoelectronic/Optogenetic Retinal Prosthesis

Australia. In early March 2006, a request for market approval in Europe was
withdrawn, by Alcon, from regulatory consideration.

4.5COMBINATION THERAPIES
Other investigations are also showing promise, including combination therapies,
which combine traditional PDT therapy with new drugs to increase the effectiveness
of PDT. Combination treatments pair one or more existing or new AMD treatments to
see if the end result might be greater than what could be achieved individually. More
and more medical practitioners believe that combination methods are the way of the
future for wet AMD treatment. Usually the idea is that one kind of treatment will take
care of existing AMD in the patient, and the other will help to prevent any future
developments.

4.6 POSITION STATEMENT ON AVASTIN (BEVACIZUMAB)

The role, efficacy, and safety of anti-vascular endothelial growth factor (VEGF)
therapies for use in the treatment of age-related macular degeneration (AMD) were
first established by clinical trials of pegaptanib sodium, (Macugen, [OSI]
Eyetech/Pfizer) and later by clinical trials for ranibizumab (Lucentis, Genentech, Inc.)

4.7PEGAPTANIB SODIUM
Phase 3 clinical trials for pegaptanib sodium demonstrated that after 1 year of
treatment, individuals who were treated with 0.3 mg and 1 mg pegaptanib sodium
experienced less vision loss than those who were treated with a placebo. Individuals
who were treated with pegaptanib sodium experienced lasting results for 2 years. The
most common side effect (occurring in approximately 1.3% of cases) was
endophthalmitis, which was caused by the injection.

4.8RANIBIZUMAB
Phase 3 clinical trials for ranibizumab demonstrated superior results after 1 year of
treatment, and showed that the majority of individuals who were treated with
ranibizumab improved or maintained vision 2 years later. The improvement in visual
acuity endpoints in the ranibizumab-treated groups (0.3 mg and 0.5 mg) was

ME E&TC (Signal Processing), Sem –III, DYPSCOEA, Ambi. Savitribai Phule Pune
University
16
 Extraocular Image Processing for Optoelectronic/Optogenetic Retinal Prosthesis

maintained at year 2, while individuals in the control group continued to experience

vision deterioration. At 2 years, at least 90% of individuals who were treated with
ranibizumab maintained or improved vision compared to approximately 53% of
individuals who were treated with sham injections. Treatment side effects were mild
to moderate, affected less than 3% of individuals, and included conjunctival
hemorrhage, increased IOP, vitreous floaters, and endophthalmitis.

4.9BROADENING THE ANTI-VEGF THEORY

Ranibizumab was developed by Genentech, Inc. The company had previously

developed bevacizumab (Avastin, Genentech, Inc.) an anti-VEGF drug that is
currently approved by the Food and Drug Administration (FDA) as an intravenous
therapy for metastatic colorectal cancer patients. Bevacizumab for use in cancer
therapy is currently being investigated. Ranibizumab is a molecular fragment of an
antibody, and bevacizumab is a full-length antibody. They are both thought to work
by a similar principle - the drug blocks the production of VEGF. VEGF, which is also
produced by cancer cells, prompts the abnormal growth of blood vessels, also known
as angiogenesis. Bevacizumab binds with VEGF and interferes with its ability to
stimulate blood vessel growth.

In early 2004, Philip Rosenfeld, MD, PhD, and colleagues at the Bascom Palmer Eye
Institute in Miami, Fla, initiated the use of bevacizumab in the treatment of AMD.
Their first study was called Systemic Avastin for Neovascular AMD (SANA). In this
and subsequent studies, which consisted of intravitreal injections of bevacizumab,
individuals who were clinically followed reported improvements in visual acuity
comparable to ranibizumab with no serious adverse events. It is important to note that
these clinical studies were not conducted as randomised clinical trials.4,5 Based on
these results, the use of bevacizumab for the treatment of AMD appears to have been
broadly accepted by retinal specialists around the world.

The use of bevacizumab in the eyes, an indication for which it is not approved, is
called off-label use. It is reasoned conjecture on the part of the AMD Alliance

ME E&TC (Signal Processing), Sem –III, DYPSCOEA, Ambi. Savitribai Phule Pune
University
17
 Extraocular Image Processing for Optoelectronic/Optogenetic Retinal Prosthesis

International that the off-label use of bevacizumab was first suggested for reasons of
economy and availability in the face of a significant unmet need. Until the June 30,
2006 FDA approval in the USA, treatment with ranibizumab was not available unless
an individual was registered in a clinical trial, or, as is possible in some European
countries, receives the treatment on what is called a 'named-patient' basis.

4.10ANTIOXIDANTS AND VITAMIN THERAPIES

One working hypothesis is that a cause or contributing factor to Macular

Degeneration involves the formation of chemicals in the body called free radicals.
Free radicals are thought to result, in part, from exposure to sunlight and other forms
of ultraviolet light. They cause cellular damage by taking electrons from molecules in
healthy cells. This process, called oxidation, has been linked to a variety of health
problems including heart disease and cancer. Substances called antioxidants may
counteract the oxidation process; the body produces its own antioxidants, and these
are helped by antioxidants that we ingest through food or vitamin supplements.
Vitamins C, E nd carotenoids, including beta-carotene, are examples of potent
antioxidants. However, which of these is helpful to AMD is not yet known.

ME E&TC (Signal Processing), Sem –III, DYPSCOEA, Ambi. Savitribai Phule Pune
University
18
 Extraocular Image Processing for Optoelectronic/Optogenetic Retinal Prosthesis

CHAPTER 5
OVERVIEW OF THE PROPOSED METHOD
The visual processing is divided into two main parts. In the first part, there are the
image enhancement algorithms, which have been developed to enhance the vision of
the patient given the small field of view and limited visual acuity.

Fig. 1. System [Link] stages of the approach for stimulating different retina targets:
resensitized cone, bipolar cell (BC), and RGC stimulation. In the case of cone stimulation, the
retargeted simplified cartoon scene will be transmitted. For BC stimulation, both of the
retargeted ON and OFF retinal spatiotemporal derivative maps of the scene will be
transmitted. Finally, for the RGC stimulation, the SCMs from the spatiotemporal derivatives
will beused for stimulation. RET ˙∇ (x,y ) represents the retargeted spatiotemporal derivative
map.

ME E&TC (Signal Processing), Sem –III, DYPSCOEA, Ambi. Savitribai Phule Pune
University
19
 Extraocular Image Processing for Optoelectronic/Optogenetic Retinal Prosthesis

1) Scene simplification: We use anisotropic filtering to simplify the scene, and

overlay a negative spatial derivative to complete the effect of cartoonization.
2) Spatial scene compression: We develop an importance map from which we
dynamically shrink less important features with respect to the important
features. Second, there are the retinal processing and spike coding algorithms.
3) Retinal processing: The algorithms reproduce the ON and (if required) OFF
signal pathways in the retina.
4) Spike coding: As optogenetic stimulation of RGCs allows control of
individual action potentials, we generate a spike coding protocol for both the
ON and OFF pathways.
These image processing components are described in the flowchart of Fig. 1. In all
cases the first two image enhancement algorithms are applied. The configuration of
the rest of the processing platform depends on the target neurons as shown. From the
photoreceptors to the optic nerve, signal processing in the retina can be considered to
have two main stages, in the outer plexiform layer (OPL) with the interaction of the
photoreceptors, the horizontal cells and the bipolar cells, and the inner plexiform layer
(IPL) where information from the bipolar cells is processed by the bipolar, amacrine,
and RGCs, and transmitted to the brain via the optic nerve. In the case of optogenetic
photosensitization of the degenerate cones, where the light sensitive outer segment is
nonfunctional or missing but the inner segment retains viability, only the image
enhancement algorithms are needed. However, when the bipolar cells are targeted,
replication of the OPL retinal processing is needed. Targeting RGC, which has been
the approach adopted in prosthesis based on electronic implants, requires
theadditional IPL processing as well as spike coding. Implementation concerns must
also be addressed as the algorithms need to be realized in real time with sufficiently
low latency to enable the patient to function effectively, while remaining viable for a
portable headset solution and scalable to high resolution.
5.1 IMAGE ENHANCEMENT ALGORITHMS

ME E&TC (Signal Processing), Sem –III, DYPSCOEA, Ambi. Savitribai Phule Pune
University
20
 Extraocular Image Processing for Optoelectronic/Optogenetic Retinal Prosthesis

5.1.1Scene Simplification: Image simplification is an important step before

performing the retinal processing. It is also the primarily useful strategy for
stimulating optogenetically resensitized cones. The purpose is to suppress non
important high frequency noise and textures [22]. As Gaussian smoothing and median
filters are not efficient in preserving the significant boundaries in the scene intact,
instead, we use an anisotropic diffusion filter. It eliminates noise and low importance
textures, while avoiding smoothing across object boundaries. It is an iterative process
that progressively smoothes the image while maintaining the significant edges by
reducing the diffusivity at those locations having a larger likelihood to be edges [23].
The discrete equation of the anisotropic diffusion filter is
In+1 = In +Δt[Δ (C · ΔIH) + Δ (C · ΔIV ) + Δ (C · ΔID1)+ Δ (C · ΔID2 )]
where Δ is the gradient operator and C is the diffusion coefficient or the diffusivity of
the equation. n denotes the iteration number, Δt is the time step (it controls the
accuracy and the speed of the smoothing), and ΔIH , ΔIV , ΔID 1 , and ΔID 2
represent the horizontal (H), vertical (V), and two diagonal (D) gradients of the image.
Although there are different methods to calculate the gradients [24], we use Sobel
operators for their simplicity in implementation and, thus, processing time/power
consumption. Directionless Laplacian filters would require less processing than
calculating all four gradients. However, it is more sensitive to fine (irrelevant)
textures and produces very thin edges. It is mainly useful in sharpening process. The
diffusion coefficient C is calculated from the following equation:
C= 1/(1 +√( ∇ I2H+ ∇ I2V + ∇I2D1 + ∇ I2D2)

To increase the visual distinctiveness of high contrast regions in the scene, we overlay
a negative spatial derivative over simplified scene that gives a notable edge
enhancement giving the image a cartoon like effect. The overlaid spatial derivatives
arecalculated as follows:
∇IX =∇IH +( ∇ ID1 + ∇ ID2)/2

∇ IY = ∇ IV +( ∇ ID1 − ∇ ID2)/2
Then, gradient’s amplitude is
∇ = √ (∇I2X+ ∇ I2Y )

ME E&TC (Signal Processing), Sem –III, DYPSCOEA, Ambi. Savitribai Phule Pune
University
21
 Extraocular Image Processing for Optoelectronic/Optogenetic Retinal Prosthesis

Patients can control the smoothness or simplification level through increasing or

decreasing the number of iterations n.
5.1.2 Spatial Scene Retargeting: To tackle the tunnel vision problem and the limited
resolution of current prosthetic vision systems, we desire to compress the visual scene
into the effective field of view. Previous works targeted the process of image
compression for retina prosthesis devices depended on ordinary bilinear resizing
approach which makes the key features seem further away [25]. This makes the object
identification process more difficult at lower spatial [Link], therefore, need to
nonlinearly retarget the scene into a smaller size, thus expanding the effective field of
vision. To do that, we have developed a nonlinear scene retargeting technique, which
generates an importance matrix WST; this involves how much each pixel in the input
scene should be nonlinearly shrunk (comprised). It is a combination of two measures:
a local saliency gradient map and a temporal motion map. The saliency map comes
from (5) and the motion map comes from a temporal derivative of the visual scene
WST = ∇(x, y, t) +1nnn=1∇(x, y, t)−∇(x, y, t − n).
As multiple frames need to be stored in memory and reprocessed, this temporal
derivative can be processor intensive. We, therefore, typically use n = 1, i.e., the
equation then becomes a simple frame difference.
To give higher importance values for the foreground objects over the saliency and
background objects, we modify the importance matrix by giving background areas
very low importance weights. The modification process starts by looking for seams
with lowest cumulative energy values [17], [18]. A seam is defined to be a connecting
path of pixels (one in each row, in case of vertical compression) with minimum
energy values. This cumulative energy mapM(i,j) is generated from the
spatiotemporal importance map in the following
M(i, j) = WST(i, j) + min[M(i − 1, j − 1),M(i, j − 1), M(i + 1, j − 1)].
To find the lowest seam energy path, starting from the last row in the cumulative
energy matrix M, we search for the minimal cumulative pixel. After that, we work
backward from this pixel to obtain an optimal vertical seam by finding the minimum
of the three neighboring pixels of this pixel in the previous row and then store this
pixel to the seam path. We do the process of searching for the lowest seam energy
path K times, as K is the number of columns to be shrunk. Then, we generate another

ME E&TC (Signal Processing), Sem –III, DYPSCOEA, Ambi. Savitribai Phule Pune
University
22
 Extraocular Image Processing for Optoelectronic/Optogenetic Retinal Prosthesis

matrix of size N × K, which contains the x and y positions of the seams. By knowing
the locations of lowest energy pixels, we rescale all the pixels of the importance
matrix along the path of all the seams to very low importance values.
In order to preserve continuity between rows,WST(i,j) should be equal to WST(i−1,j),
in case of vertical retargeting. For that, we assign a fixed importance value for each
column by applying a moving average window of size L and overlapping by L/2 on
each column in the importance matrix [26]; then, we take the maximum value for
each column, so that the updated importance matrix is
W(0: imax, j) = 2N/L maxr=1{ Wr (j)}
where Wr (j) is an array containing the moving average values of the importance
values in column j. For dynamic scene retargeting, calculating the importance map for
each frame individually generates jittering artifacts inthe retargeted video sequence.
To counteract this, movement of seams from one scene to the next need to be
constrained. We, therefore, calculate the seams of the first couple of frames, and then,
the actual locations of these seams are stored into an arbitrarymatrix T to be used in
calculating the seams for the next frame. These seams’ locations are adapted, for the
forthcoming frames, if there are dynamics in the scene. If the objects in the frame are
static, then the locations of the seams will be the same, but if these objects move, then
the seam locations within the same areas through which the objects move around will
also move to avoid crossing the moving objects. The adaptation process of the seams
locations is fully discussed in our previouspaper [18]. As discussed previously, the
importance matrix defines which pixels in the source image are significant and should
be preserved in the retargeted image. In contrast, the shrinkability matrix defines the
relative extent to which pixels in the source image should be shrunk to retarget the
image by K columns. The shrinking value of each pixel S(j) in any row, considering
retargeting the whole image by only one column, can be calculated from
S(j) =1/(W(j) ∗∑Mj=1 (1/W(j)))
The summation of S(j) over j columns equals 1 if K is 1. Then, we rescale this
shrinkage map so that the summation equals to K for higher values of K. This scaled
cumulative shrinkability map is used to retarget the difference of Gaussian (DoG)
image to the desired dimension by using an algorithm suggested by Fant [27]. The
algorithm is a 1-D method used in separable transformations defined in terms of

ME E&TC (Signal Processing), Sem –III, DYPSCOEA, Ambi. Savitribai Phule Pune
University
23
 Extraocular Image Processing for Optoelectronic/Optogenetic Retinal Prosthesis

forward mapping functions. It maps a limited line of discrete input pixel intensity
values into a limited line of discrete output pixel intensity value.

5.2 RETINAL PROCESSING

5.2.1Outer Plexiform Layer Processing: The core spatial processing of the
horizontal and bipolar cells is the spatial derivative. As the inputs of central cone cells
are subtracted from surrounding cone cells, this is classically modeled as a DoG
functions. The positive values correspond to the ON pathway and the negative values
to the OFF pathway. Each pathway looks similar to a simple edge filtering of the
image. We have found slightly better results by combining vertical and horizontal
Sobel operators rather than a DoG or Laplacian filter. The resultant time varying
spatiotemporal derivative is the one used in (6). However, digitally, the derivative is
scaled as an unsigned integer to between−128 and+128. Then, theONsignal is, thus,
WST > 0 and the OFF signal is |WST < 0|. 2) Spike Coding: In traditional electronic
forms of epiretinal prosthesis, the spiking output from the RGCs has been modulated
by stimulation amplitude [28]. Although coding matching the natural output of the
retina has been neither attempted nor achieved, patients have seen phosphenes with
amplitude roughly correlating to stimulation intensity and, thus, firing frequency. In
the case of optogenetic approaches, it is now possible to improve upon this by
determining the firing code through pulses of direct optical stimulation. The coding
mathematics of the spike trains that carry the visual information is still under
investigation [20], [21]. Primate retina has ~17 RGCs types and the encoding
strategies for each of these types are different. Different encoding strategies have been
reported in the literature [21]. Generally, these can be classed into rate coding
(intensity α number of spikes per second), correlated spike timing (intensity α latency
of a spike from an event or other spike), and population coding (information is
correlated with neighboring spike patterns) [29]. For example, in response to a step
change in local luminance, the firing rate of the midget and parasol cells increases
[30]. Meister and Gollisch [20] revealed howthe first spike latency on four types of

ME E&TC (Signal Processing), Sem –III, DYPSCOEA, Ambi. Savitribai Phule Pune
University
24
 Extraocular Image Processing for Optoelectronic/Optogenetic Retinal Prosthesis

RGCs (particularly fast OFF, biphasic OFF, slow OFF, and fast ON cells) relative to
postsaccadic eye movement could encode information. Correlations between multiple
ganglion cells spike may constitute another population code, conveying information
on the visual stimulus that is not present if ganglion cells acted as independent
encoders [31]. However, such correlations in spike timing occur mainly between
neighboring ganglion cells which suggests that such synchronization is a local
phenomenon. This may be due to shared synaptic inputs [31], [32]. Indeed, other
coding schemes could be used; for example, a tonic change in the firing rate of a
neuron might represent its response for a small spot. However, in the case of a large
spot, the same neuron might respond with oscillatory type firing [29]. Indeed, most of
the studies found that the ganglion cells firing rate provides useful information about
these stimulus parameters, while the timing of the first action potential provides
almost as much information [20], [33]. As spatial synchronization between external
μLEDs and a moving eye will be difficult, it will only be possible to chromatically
distinguish cell types. Furthermore, with current eye tracking technology, it would be
difficult to implement latency coded to saccadic movements. We, thus, implement a
rate-based coding approach. First, we decoded the image intensities into frequencies
with an exponentially decaying function. This is defined in the following relation:
F(x, y, t) = F0 (x, y) · et/S (10)
where F(x,y,t) is the coded frequency for each pixel I(x,y). S sets the decay slope
profile, which we chose to be S = 20 in this study. It can be changed according to the
decaying profile of the spike pattern for different RGCs. F0 is given by
F0(x, y, t) = fmax/ImaxR(x, y, t)
where fmax is the maximum firing rate of the RGCs, Imax is the maximum intensity
of the pixels, and R(x,y,t) is the retargeted ON or OFF intensity matrix. Typically, we
set fmax = 40 Hz which represents the maximum firing frequencies of the tonic
firing RGCs that we have previously achieved [12] and Imax =255 which is the
maximum unsigned gray level value for any pixel. Then, F(x,y,t) will be a matrix that
includes the positions of spikes for each pixel over a time period which we set
arbitrarily to 1000 ms. We assume that relaxation of any temporal decay effects will
be complete in this time period. The final spike coded map (SCM) will be a binary
matrix that has ones at the position of each spike.
SCM(F(x, y, t)) = 1∀ 1/F(x, y, t)< 1000 ms.

ME E&TC (Signal Processing), Sem –III, DYPSCOEA, Ambi. Savitribai Phule Pune
University
25
 Extraocular Image Processing for Optoelectronic/Optogenetic Retinal Prosthesis

5.2.3Scalability—Power Control To stimulate RGC’s, we use a GaN LEDs that are

capable of emitting high brightness, i.e., emitting over 10 W/cm2 [34]. As from our
previous work [12], we know that the individual microLED emitters operate at around
5mA to produce enough light to cause an action potential. As 5W is commonly used
as the threshold for fan-assisted cooling, and our operational voltage range is 10V, we
will, therefore, not drivemore than 1A through the chip. This means that each
microLED can consume 50 mJ/s; hence, 100 LEDs can be driven simultaneously with
such [Link] calculate that at the resultant irradiance, we would need to pulse
for 10 ms to produce an action potential. Therefore, we can drive a total of 10 000
action potentials per second over an array of neurons. A feasible 1-cm2 chip could
provide around 36 k pixels (192 × 192 pixels), which was typical for early color
mobile telephone screens. The limiting factor on the technological development will
be the power requirement, which ultimately limits the number of LEDs that can be
illuminated. One method of limiting the number of pixels to be displayed is to
threshold the ON and/or OFF signals. We developed a control algorithm that limits
the number of illuminations to 10 000 per second (assuming 10 ms illumination
periods). The algorithm limits the number of spikes through exponentially controlling
the dynamic range of the scene pixels and by thresholding the pixel intensity value. A
set of 15 images ranging from simple to complex scenes has been chosen, and from
them we generated two equations for the exponential factor and the threshold value
that limit the summation of the spikes to 10 000
E = AE eαS − BE eβS − CE
T = AT eαS − BT eβS − CT
where E and T are the exponential factor and the intensity threshold values,
respectively. S represents the summation of the preprocessed spatiotemporal
derivative intensities, and the constants have the following values:

Then, for a given point in time, the dynamically scaled and retinally processed and
thresholded image will be Scaled ˙
∇ (x,y) = eE·˙∇ (x , y )
Thresholded ˙∇ (x,y) = Scaled ˙∇ (x,y) if > T

ME E&TC (Signal Processing), Sem –III, DYPSCOEA, Ambi. Savitribai Phule Pune
University
26
 Extraocular Image Processing for Optoelectronic/Optogenetic Retinal Prosthesis

0 if < T
1
As aforementioned, from previous data, the maximum spike rate that can be reliably
generated at the RGC’s is 40 Hz. Given that the RGCs need at least two spikes to
recognize the pixel value at a certain frame, the maximum frame rate that can be
sent to these cells is 10 frames/s. This means that the minimum gray scale intensity
value that we can encode at each frame is 104 (if we have 256 gray scale levels),
which represents a spike with frequency of 99 Hz. To encode the video stream, we
encode the pixels of each frame over a period of 100 ms at maximum and the
position of last spike for each pixel is stored to be the starting point from which the
spikes for the next frame will start.
D. Implementation
Key to this long-term system is its implementability into a portable and wearable
head-mounted display (HMD) system. The prototyping platform that can be
connected to the LED unit is implemented on a MATLAB platform on a desktop
computer; with a 2.8-GHz Intel core quad processor, 8-Gb memory, and a GTX285
graphics card. To utilize the parallel processing power of the graphics card, the scene
simplification, retina processing, and spike coding processes are implemented using
the Acceler Eyes graphics processing unit (GPU) jacket plug-in [35] for MATLAB.
The scene retargeting process which is a more linear operation is processed on the
CPU/floating point unit. As our target is to implement this system on a portable
platform, we implemented the scene simplification and spatiotemporal derivative
processes on a Tegra I development platform from NVidia [6]. The Tegra I is a
system-on-a-chip platform with an ARM CPU core and a graphics processor with
eight processing cores. There is a feed from the camera to the GPU allowing power
efficient parallel processing. The specification of the Tegra I is low compared to
laptop or desktop processors. However, the CPU/GPU operations can be performed at
sub-1W power consumption which is desirable for portable/wearable processing. We
also calculated estimated improvements for speed for the Tegra II and III platforms
according to NVidia’s specifications.

ME E&TC (Signal Processing), Sem –III, DYPSCOEA, Ambi. Savitribai Phule Pune
University
27
 Extraocular Image Processing for Optoelectronic/Optogenetic Retinal Prosthesis

5.1 ALGORITHM
1. Create training dictionary.
2. Apply Local anchor embedding and save features.
3. Apply Local independent projection based classification and calculate
reconstruction error(E).
4. Select testing sample and apply median filtering. Repeat step 2 and 3.
5. Detect patch whose intensity is greater than threshold value.
6. Divide patch into edema and tumour part.
7. Analyze tumour part.

5.2 PREPROCESSING
5.2.1 Image Filtering by using Median Filter
Median filtering and averaging filter are similar, In averaging filter each pixel of
output is group of an average of the values pixel in the corresponding input pixel of
neighborhood and In median filtering, each pixel of output is analyze by the median

ME E&TC (Signal Processing), Sem –III, DYPSCOEA, Ambi. Savitribai Phule Pune
University
28
 Extraocular Image Processing for Optoelectronic/Optogenetic Retinal Prosthesis

of the neighborhood pixels, not the mean. The main difference in between mean and
median is mean which is more sensitive than the median to extreme values. Therefore
Median filtering is better to remove these outliers with maintaining the same
sharpness of the image. The medfilt2 function is used to implement median filtering.
Also PSNR, MSE, contrast and correlation values are calculated.
The comparison between average and median filter is shown in below image. They
are used to remove salt and pepper noise. This type of noise consists of random pixels'
being set to white or black. In both cases the size of the 3-by-3 image is used for
filtering.
1. Read in the image and display it.
I = imread('[Link]');
imshow(I)
2. Add noise to it.
J = imnoise(I,'salt and pepper',0.02);
figure, imshow(J)

Figure 5.2: Original image

ME E&TC (Signal Processing), Sem –III, DYPSCOEA, Ambi. Savitribai Phule Pune
University
29
 Extraocular Image Processing for Optoelectronic/Optogenetic Retinal Prosthesis

Figure 5.3: Noisy image

3. Filter the noisy image with an averaging filter and display the results.
K = filter2(fspecial('average',3),J)/255;
figure, imshow(K)

Figure 5.4: Filter the noisy image with an averaging filter

4. Now use a median filter to filter the noisy image and display the results.
Notice that medfilt2 does a better job of removing noise, with less blurring of
edges.
L = medfilt2(J,[3 3]);
figure, imshow(L)

ME E&TC (Signal Processing), Sem –III, DYPSCOEA, Ambi. Savitribai Phule Pune
University
30
 Extraocular Image Processing for Optoelectronic/Optogenetic Retinal Prosthesis

Figure 5.5: Image after median filtering

5.3 FEATURE EXTRACTION
MRI images Image intensities in do not have a fixed meaning and mostly vary within
or between subjects so before extracting image features, intensity normalization and
image inhomogeneity correction should be performed. In this project, we are using a
patch-based technique for extracting the image feature. The patch intensity values in
around a voxel v were determined and rearranged as a feature vector. The intensity
values in patch are greater than threshold value. All intensity values in patch are
plotted on graph.
From the study it has been found that there are some metrics which can be used to
check the performance of particular approach. So we considered some metrics for the
evaluation of their approach.

CHAPTER 6

ME E&TC (Signal Processing), Sem –III, DYPSCOEA, Ambi. Savitribai Phule Pune
University
31
 Extraocular Image Processing for Optoelectronic/Optogenetic Retinal Prosthesis

SOFTWARE TOOL USED: MATLAB R2010a

The MATLAB programming language is come from commercial MATLAB software
that is often known and works in research and industry. This is also example of a
high level \scripting" or new \4th generation" language. The most striking differs in
C and other compiled languages is the code interpreted while the program is executed
(interpreter program reads the source codes line by line and translates it into machine
instructions on the y), in other words no compilation is required. While this reduces
the execution speed, it frees the programmer out of memory management and gives
dynamic typing and interactive sessions. It's also take highly expressive less time to
code and debug because its worth referring that programs written in scripting
languages are always highly expressive shorter than equal programs written in
compiled languages.
Shortly, trade off in between the execution time and the development time. A
momentous feature for teaching point of view is the sufficient of MATLAB to have
act on each other sessions. The user able to type one or several commands at the
command prompt and these commands are executed immediately after pressing
return. This gives act on each other testing of small parts of the code and promotes
experimentation. Using the act on each other prompt, interpreted languages also tend
to be not dificult to debug than
Compiled executable with sophisticated libraries for matrix The MATLAB package
comes operations, general numeric methods and plotting of data. So we use
MATLAB as our Software Tool for this Project.
6.1 FEATURES OF MATLAB
1. MATLAB is a high-level language for visualization, numerical computation and
application development.
2. MATLAB also provides an interactive environment for problem solving, iterative
exploration and design.
3. MATLAB provides large library of mathematical functions for Fourier analysis,
linear algebra, filtering, statistics, numerical integration, optimization and solving
ordinary differential equations.
4. MATLAB provides built-in graphics for creating custom plots it provide tool sand
for visualizing data.

ME E&TC (Signal Processing), Sem –III, DYPSCOEA, Ambi. Savitribai Phule Pune
University
32
 Extraocular Image Processing for Optoelectronic/Optogenetic Retinal Prosthesis

5. MATLAB's programming interface gives development tools for maximizing per

formance and improving code quality and maintainability.
6. MATLAB provides tools for building applications.
7. MATLAB provides functions for integrating MAT LAB based algorithms with
external applications and languages such as Java,C, .NET and Microsoft Excel.
6.2 USES OF MATLAB
MATLAB is mostly used as a computational tool in engineering and science en-
compassing the fields of math, physics, chemistry and all engineering streams. It is
employed in a range of applications including:
1. Signal Processing and Communications
2. Image and Video Processing
3. Control Systems
4. Test and Measurement
5. Computational Finance
6. Computational Biology
6.3 IMAGE PROCESSING TOOLBOX FUNCTIONS USED IN BRAIN TU-
MOUR DETECTION AND SEGMENTATION
imread-Read image from graphics file
imwrite-Write image to graphics file
imfinfo-Information about graphics file
mat2gray-Convert matrix to grayscale image
rgb2gray-Convert RGB image or colormap to grayscale
ind2rgb-Convert indexed image to RGB image
imcrop-Crop image
imresize-Resize image
imrotate-Rotate image
imtranslate-Translate image
medfilt2 -2D median filtering
bwboundaries-Trace region boundaries in binary image
bwtraceboundary-Trace object in binary image
visboundaries-Plot region boundaries
edge-Find edges in intensity image

ME E&TC (Signal Processing), Sem –III, DYPSCOEA, Ambi. Savitribai Phule Pune
University
33
 Extraocular Image Processing for Optoelectronic/Optogenetic Retinal Prosthesis

corner-Find corner points in image

immse-Mean squared error
psnr-Peak Signal-to-Noise Ratio (PSNR)
activecontour-Segment image into foreground and background using active contour
imsegfmm-Binary image segmentation using Fast Marching Method
imseggeodesic-Segment image into two or three regions using geodesic distance-
based
color segmentation
gradientweight-Calculate weights for image pixels based on image gradient
graydifiweight-Calculate weights for image pixels based on grayscale intensity
difference
graythresh-Global image threshold using Otsu's method
multithresh-Multilevel image thresholds using Otsu's method

CHAPTER 7
LOCAL INDEPENDENT PROJECTION BASED
CLASSIFICATION

ME E&TC (Signal Processing), Sem –III, DYPSCOEA, Ambi. Savitribai Phule Pune
University
34
 Extraocular Image Processing for Optoelectronic/Optogenetic Retinal Prosthesis

7.1 THE BASIC PRINCIPLE OF LIPC

The segmentation of brain tumour is considered as a multiclass classification problem.
To explain this issue, a one-versus-all (OvA) strategy can be used. In our proposed
method One versus all strategy, a classifier is trained per class to different a class
from all others. Before the proposed LIPC was introduced, the following assumption
was considered as the base for LIPC:
Assumption I : Every Sample is located on different non-linear sub manifolds
according to their classes, and a sample may be nearly represented as a linear
combination of several closest neighbors from its corresponding sub manifold.
7.2 LIPC IMPLEMENTATION
7.2.1 Dictionary construction
By means of manually labeled original samples in a training group (set) Dictionary is
constructed. However, number of original training samples produces large D [29],
which increases computational costs and memory. In the present study, more samples
for each class are available for learning but when we are going to implement this that
time this process becomes impractical. For learning a compact presentation of the
original training samples, it is necessary to apply a dictionary learning method. The k-
means method is used in this proposed method. Clustering is the process of
partitioning a group of data points into a small number of clusters.
For construction D, the manually labeled original samples in a training set are used.
Therefore, numerous original training samples perhaps produce a large D, which
strikingly increases computational and memory costs. In our project study, more than
halfa million samples are available for training for each class. Thus, specified
processes are not practical when conducted traditionally. We are going to apply a
dictionary learning method is dispensable in learning a compact presentation of the
original training
samples. In proposed method the k-means method can get the structures of the
original sample space; and this method is used in the current study to get knowledge
of a compact presentation of the original training samples of every class.
1. Patch Extraction: In learning process, training images should be pre-processed to
get effective dictionary construction. Firstly, each HR image IH is sub-sampled and

ME E&TC (Signal Processing), Sem –III, DYPSCOEA, Ambi. Savitribai Phule Pune
University
35
 Extraocular Image Processing for Optoelectronic/Optogenetic Retinal Prosthesis

blurred to create a LR image IL. After that, IL is again up-scaled using simple linear
insertion for that we use bi-linear insertion or cubic convolution to create an image
IUP whose resolution is same as IH. The HF details which are lost by image
degradation process are possessed by dictionary and clearly define features to index
them. The HF image IHF is generated by removing IUP from IH, and high pass
filtered version of IUP is mid frequency image IMF. IMF is used as the features for
indexing. Note that the proposed algorithm recovered HF layer IHF. They show lost
HF and MF layers for forecasting them, respectively. As an output, we extract and
store LR and HR patches from IMF and IHF, respectively. Those patches are partly
covered with neighbour patches for local smoothness. Without loss of generality, we
are assuming that the relationship in IHF and IMF is not depending on the local image
contrast. Finally, so it is called primitive patches including textures are chosen or
edges and they only belong to the dictionary.
2. Dictionary Size Reduction: Now, there is need to effectively minimize the number
of LR-HR patch pairs so as to decrease computational burden and memory cost in
synthesis. This process is highly expressive in that the number of training examples
generally controls the performance of learning-based SR. Generally, the small number
of the samples can make better practicality of the proposed SR algorithm.

Figure 7.1: Pre-processing for dictionary construction

So, we adjacent the group LR-HR patch pairs into different patch pair. Togather
similar patches we adopt K-means clustering. Figure shows clustering process.

ME E&TC (Signal Processing), Sem –III, DYPSCOEA, Ambi. Savitribai Phule Pune
University
36
 Extraocular Image Processing for Optoelectronic/Optogenetic Retinal Prosthesis

Consider L2-norm distances, according to that LR patches which are close each other
are clustered into a one group and similarly to that respective HR patches are
clustered. At last, the center points of each cluster become new HR and LR patches
connected to the ordinary dictionary. Practically, we can determine memory cost and
computational complexity.

Figure 7.2: Dictionary size reduction

3. Algorithm:
(a) Initialize the center of the clusters.
(b) Attribute the closest cluster to each data point.
(c) Set the position of each cluster to the mean of all data points belonging to
that cluster.
(d) Repeat steps 2-3 until convergence.

Figure 6.3: Example of K-mean clustering

7.2.2 Locally linear representation
ME E&TC (Signal Processing), Sem –III, DYPSCOEA, Ambi. Savitribai Phule Pune
University
37
 Extraocular Image Processing for Optoelectronic/Optogenetic Retinal Prosthesis

There are several methods are available for representation of sample which is linearly
based on training sample. These methods are sparse coding [29], locally constrained
linear coding and local anchor embedding. Sparse coding attempts to use the lowest
number of training samples in reconstruction. LLC and LAE approaches focus on
locality. To obtain solution for LAE, firstly select K nearest neighbors from
dictionary. Here we can vary value of K from 5 to 100 and observe effect of this on
reconstruction error.
Three steps were performed to obtain the solution of LAE.
1. Select k nearest neighbors of x from D and construct Nx(k)
2. For the samples that do not belong to Nx(k),associated ajs were set to 0
3. For samples that belong to Nx(k),calculate ajs
1. KNN Method
The given training examples are vectors each with a class label. The phase of the
algorithm contains only class labels of the training samples and storing the feature
vectors. K is a user-defined constant in the classification phase; also vector which is
unlabeled is classified by designating the label which is most occurring among the k
samples nearest to that query point.
Euclidean distance is commonly used distance metric for continuous variables. For
discrete variables overlap metric is used, such as for text classification. In the part of
text of gene expression microarray data, which is given in example, k-NN has also
been used with correlation coefficients. Many times, the classification accuracy of k-
NN can be increased to significant extent if the distance metric is employed with
expert algorithms such as Large Margin Nearest Neighbor or Neighborhood
components analysis.

Figure 6.4: Example of k-NN classification

ME E&TC (Signal Processing), Sem –III, DYPSCOEA, Ambi. Savitribai Phule Pune
University
38
 Extraocular Image Processing for Optoelectronic/Optogenetic Retinal Prosthesis

The given test sample which is shown by green circle should be classified into the
class no.1 of blue squares or to the class no.2 of red triangles. If k = 3 it is allotted to
the second class because there are 2 triangles and only 1 square inside the inner circle.
If k = 5 it is allotted to the first class.
7.2.3 Classification score computation
Softmax regression model is used for computation. Softmax regression model uses
Relation between Data distribution and reconstruction error. If distribution is uniform
and noise is low then classification may be performed well. The logistic regression is
binary [Link] classification problems in which class label y can take more than
two possible values. To ask that in logistic regression, we have a training set of m
labeled examples. In logistic regression, we used it in the binary classification setting,
so the labels were y(i) 2 0; 1

Figure 7.5: Figure shows how a testing sample is classified wrongly when the data
distributions on unlike submanifolds are not considered.

ME E&TC (Signal Processing), Sem –III, DYPSCOEA, Ambi. Savitribai Phule Pune
University
39
 Extraocular Image Processing for Optoelectronic/Optogenetic Retinal Prosthesis

Figure 7.6: Figure shows Learning a softmax regression model using the
reconstruction error
norms to divide the data into different classes.

CHAPTER 8
POSTPROCESSING & ANALYSIS OF
TUMOUR
In LIPC step we separated tumour and edema part. In post processing we analyze
tumour part. Here we calculated area and perimeter of tumour in pixels. Also we can
find in which lobe tumour is present and its stage. For stage detection we used SVM
algorithm. Our algorithm carries on a candidate Support Vector set. It begins the set
with the closest pair of points from opposite classes like the Direct SVM algorithm.
As soon as the algorithm finds a breaking point in the dataset it greedily adds it to the

ME E&TC (Signal Processing), Sem –III, DYPSCOEA, Ambi. Savitribai Phule Pune
University
40
 Extraocular Image Processing for Optoelectronic/Optogenetic Retinal Prosthesis

candidate set. It may so happen that addition of the breaking point as a Support Vector
may be prevented by other candidate Support Vectors already present in the set. We
are going to prune away all such points from the candidate set. To make sure that the
KKT conditions are satisfied we make again passes through the dataset until no
breaker can be found. We use the quadratic penalty formulation to ensure linear
division of the data points in the kernel space.

8.1 ALGORITHM
candidateSV = closest pair from opposite classes
while there are violating points do
Find a violator

candidateSV = candidateSV U violator

if any αp < 0 due to addition of c to S then
candidateSV = candidateSV \P
repeat till all such points are pruned
end if
end while

CHAPTER 9
9.1 CONCLUSION
This method is proposed to solve segmentation problem of brain tumour. Here we
apply local independent projection based classification. Finally features are extracted
by using threshold value and patch is detected which contain tumour and edema part.
After analyzing tumour part we get area and perimeter of the tumour region as well as
we detect exactly in which lobe tumour is present and its stage of development. The
information of stage of tumour is important for doctors. According to this information
doctor starts treatement. So this project is most useful for doctors.
9.2 LIMITATIONS

ME E&TC (Signal Processing), Sem –III, DYPSCOEA, Ambi. Savitribai Phule Pune
University
41
 Extraocular Image Processing for Optoelectronic/Optogenetic Retinal Prosthesis

For the brain tumour segmentation task the patch feature may be not enough to
discriminate because of complex characteristics of brain MRI images, so we need to
add more features to increase classification accuracy. This project works on 2D MRI
images. It can not give results for 3D MRI images i.e. Here we cannot calculate depth
of tumour.

9.3 FUTURE SCOPE

Future research in the segmentation of medical images will lead towards improving
the accuracy, exactness, and computational speed of segmentation approaches, as well
as minimizing the amount of manual interaction. These can be improved by uniting
discrete and continuous based segmentation methods. Computational effectiveness
will be extremely significant in real time processing applications. Segmentation
methods have shown their quality in research areas and are now stressing on increased
use for automated diagnosis and radiotherapy. These will be most important in
applications such as computer integrated surgery, where imagination of the anatomy
is a significant component.

ME E&TC (Signal Processing), Sem –III, DYPSCOEA, Ambi. Savitribai Phule Pune
University
42
 Speech Enhancement using
HNM

43