Deep Learning (AIC701)

Module 6- Recent Trends and

Applications

Devanand K. Bathe
Asst. Professor
 Contents
• 6.1 Generative Adversarial Network: GAN
Architecture.
• 6.2 Applications: Image Generation, Deep Fake
[ REFER PAPERS GIVEN IN CLASSROOM FOR THESE APPLICATIONS]
 GAN
• A Generative Adversarial Network (GAN) is a deep learning architecture that
consists of two neural networks competing against each other in a zero-sum
game framework. The goal of GANs is to generate new, synthetic data that
resembles some known data distribution.
What is a Generative Adversarial Network?
• Generative Adversarial Networks (GANs) are a powerful class of neural networks
that are used for unsupervised learning. It was developed and introduced by Ian
J. Goodfellow in 2014. GANs are basically made up of a system of two competing
neural network models which compete with each other and are able to analyze,
capture and copy the variations within a dataset.
Why were GANs developed in the first place?
• It has been noticed most of the mainstream neural nets can be easily fooled into
misclassifying things by adding only a small amount of noise into the original
data. Surprisingly, the model after adding noise has higher confidence in the
wrong prediction than when it predicted correctly. The reason for such an
adversary is that most machine learning models learn from a limited amount of
data, which is a huge drawback, as it is prone to overfitting. Also, the mapping
between the input and the output is almost linear. Although, it may seem that
the boundaries of separation between the various classes are linear, but in
reality, they are composed of linearities, and even a small change in a point in
the feature space might lead to the misclassification of data.
• How do GANs work?
• Generative Adversarial Networks (GANs) can be broken down into three
parts:
Generative: To learn a generative model, which describes how data is
generated in terms of a probabilistic model.
Adversarial: The training of a model is done in an adversarial setting.
Networks: Use deep neural networks as artificial intelligence(AI)
algorithms for training purposes.
• In GANs, there is a Generator and a Discriminator. The Generator
generates fake samples of data(be it an image, audio, etc.) and tries to fool
the Discriminator. The Discriminator, on the other hand, tries to distinguish
between the real and fake samples. The Generator and the Discriminator
are both Neural Networks and they both run in competition with each
other in the training phase. The steps are repeated several times and in
this, the Generator and Discriminator get better and better in their
respective jobs after each repetition. The work can be visualized by the
diagram given:
• Here, the generative model captures the distribution of data and is trained
in such a manner that it tries to maximize the probability of the
Discriminator making a mistake. The Discriminator, on the other hand, is
based on a model that estimates the probability that the sample that it got
is received from the training data and not from the Generator. The GANs
are formulated as a minimax game, where the Discriminator is trying to
minimize its reward V(D, G) and the Generator is trying to minimize the
Discriminator’s reward or in other words, maximize its loss. It can be
mathematically described by the formula below:

where,
G = Generator , D = Discriminator
Pdata(x) = distribution of real data
P(z) = distribution of generator , x = sample from Pdata(x), z = sample from P(z) ,
D(x) = Discriminator network , G(z) = Generator network
• Generator Model
• The Generator is trained while the Discriminator is idle. After the
Discriminator is trained by the generated fake data of the
Generator, we can get its predictions and use the results for
training the Generator and get better from the previous state to
try and fool the Discriminator.

• Discriminator Model
• The Discriminator is trained while the Generator is idle. In this
phase, the network is only forward propagated and no
back-propagation is done. The Discriminator is trained on real data
for n epochs and sees if it can correctly predict them as real. Also,
in this phase, the Discriminator is also trained on the fake
generated data from the Generator and see if it can correctly
predict them as fake.
• Different Types of GAN Models
• Vanilla GAN: This is the simplest type of GAN. Here, the Generator
and the Discriminator are simple multi-layer perceptrons. In vanilla
GAN, the algorithm is really simple, it tries to optimize the
mathematical equation using stochastic gradient descent.
• Conditional GAN (CGAN): CGAN can be described as a deep
learning method in which some conditional parameters are put
into place. In CGAN, an additional parameter ‘y’ is added to the
Generator for generating the corresponding data. Labels are also
put into the input to the Discriminator in order for the
Discriminator to help distinguish the real data from the fake
generated data.
• Deep Convolutional GAN (DCGAN): DCGAN is one of the most
popular and also the most successful implementations of GAN. It is
composed of ConvNets in place of multi-layer perceptrons. The
ConvNets are implemented without max pooling, which is in fact
replaced by convolutional stride. Also, the layers are not fully
connected.
• Laplacian Pyramid GAN (LAPGAN): The Laplacian pyramid is a
linear invertible image representation consisting of a set of
band-pass images, spaced an octave apart, plus a low-frequency
residual. This approach uses multiple numbers of Generator and
Discriminator networks and different levels of the Laplacian
Pyramid. This approach is mainly used because it produces very
high-quality images. The image is down-sampled at first at each
layer of the pyramid and then it is again up-scaled at each layer in
a backward pass where the image acquires some noise from the
Conditional GAN at these layers until it reaches its original size.
• Super Resolution GAN (SRGAN): SRGAN as the name suggests is a
way of designing a GAN in which a deep neural network is used
along with an adversarial network in order to produce
higher-resolution images. This type of GAN is particularly useful in
optimally up-scaling native low-resolution images to enhance their
details minimizing errors while doing so.
Selected Applications in Object Recognition
and Computer Vision
• Tremendous progress has been made in
applying deep learning techniques to
computer vision
• Object recognition in computer vision has
been relying on hand-designed features
• SIFT (scale invariant feature transform) and
HOG (histogram of oriented gradients)
• Capture low level but mide level,High level is
problem
• Deep learning aims to overcome such challenges
• Deep learning methods applied to computer
vision
(1) unsupervised feature learning where the deep
learning is used to extract features only, which
may be subsequently fed to relatively simple
machine learning algorithm for classification or
other tasks
(2) supervised learning methods where end-to-end
learning is adopted to jointly optimize feature
extractor and classifier components of the full
system when large amounts of labeled training
data are available.
 Unsupervised or generative feature
learning
• labeled data are relatively scarce
• Earlier work in ImageNet 2012
• Unsupervised deep autoencoder that exploits
DBN pre-training was developed
• Demonstrated by Hinton and Salakhutdinov
with success on the image recognition and
dimensionality reduction (coding) tasks of
MNIST with only 60,000 samples in the
training set;
• Deep learning methods have a rich family,
including hierarchical probabilistic and
generative models
• facial expression datasets
 Supervised feature learning and
classification
• convolutional neural networks (CNNs) in the early
90
• The CNNbased architectures in the supervised
learning mode have captured intense interest in
computer vision
• CNN-based deep learning methods have
demonstrated the same in a set of computer
vision benchmark tasks including category-level
object recognition, object detection, and semantic
segmentation.
• The task is to train a model with 1.2 million
high-resolution images to classify unseen
images to one of the 1000 different image
classes.
• On the test set consisting of 150k images
 Selected Applications in Language
Modeling and Natural Language
Processing
• Applications of deep learning to this area
started with language modeling (LM)
• Probability to any arbitrary sequence of words
or other linguistic symbols (e.g., letters,
characters, phones, etc.).
 Language modeling
• Applications
• Traditional techniques for estimating the
parameters in LMs are based on N-gram
counts
 Natural language processing
• Machine learning has been a dominant tool in
NLP for many years
• limited to numerical optimization of weights
for human designed representations and
features from the text data
• The goal of deep or representation learning is
to automatically develop features or
representations from the raw text material
appropriate for a wide range of NLP tasks.
 NLP tasks
• Language modeling
• Machine translation
• Part-of-speech tagging
• Named entity recognition
• Sentiment analysis
• Paraphrase detection
 Deep Fakes
● Deepfakes are a synthetic media created by machine-learning algorithms
named for the deep-learning methods used in the creation process and the
fake events they depict.
● Deepfake methods intersect disciplines and industries from computer
science and programming to visual effects, computer animation, and even
neuroscience.
● They can be convincingly realistic and difficult to detect when done well
and with the aid of sophisticated and powerful technologies.
● But ultimately, machine learning is a foundational concept for data
scientists, and as such, it offers an interesting area of study in the context
of deepfakes and the predictive models used to create them.
● The training methods, algorithmic structures and synthetic output of these
models offer insight into deep learning and data.
The following are some specific approaches to creating deepfakes:

● Source video deepfakes. When working from a source video, a neural

network-based deepfake autoencoder analyzes the content to understand relevant
attributes of the target, such as facial expressions and body language. It then
imposes these characteristics onto the original video. This autoencoder includes an
encoder, which encodes the relevant attributes and a decoder, which imposes these
attributes onto the target video.
● Audio deepfakes. For audio deepfakes, a GAN clones the audio of a person's
voice, creates a model based on the vocal patterns and uses that AI model to make
the voice say anything the creator wants. Video game developers commonly use
this technique.
● Lip syncing. Lip syncing is another common technique used in deepfakes. Here,
the deepfake maps a voice recording to the video, making it appear as though the
person in the video is speaking the words in the recording. If the audio itself is a
deepfake, then the video adds an extra layer of deception. This technique is
supported by recurrent neural networks.
Technology required to develop deepfakes

The development of deepfakes is becoming easier, more accurate and more prevalent as the

following technologies are developed and enhanced:

● GAN neural network technology uses generator and discriminator algorithms to

develop all deepfake content.

● Convolutional neural networks analyze patterns in visual data. CNNs are used for

facial recognition and movement tracking.

● Autoencoders are a neural network technology that identifies the relevant attributes of

a target such as facial expressions and body movements, and then imposes these

attributes onto the source video.

● Natural language processing is used to create deepfake audio. NLP algorithms

analyze the attributes of a target's speech and then generate original text using those

attributes.

● High-performance computing is a type of computing that provides the significant

necessary computing power deepfakes require.

● Video editing software isn't always AI-based, but it frequently integrates AI

technologies to refine outputs and make adjustments that improve realism.

● According to the U.S Department of Homeland Security's "Increasing Threat of Deepfake Identities"

report, several AI tools are commonly used to generate deepfakes in a matter of seconds. Those

tools include Deep Art Effects, Deepswap, Deep Video Portraits, FaceApp, FaceMagic, MyHeritage,

Wav2Lip, Wombo and Zao.

● The Gyan Management Journal also highlighted several deepfake applications, such as Datagrid,

which enables the creation of full-body personas from scratch, and Impressions, a desktop app

designed for creating celebrity videos using mobile phones.

How are deepfakes commonly used?

● Art. Deepfakes are used to generate new music using the existing bodies of an artist's work.

● Blackmail and reputation harm. Examples of this are when a target image is put in an

illegal, inappropriate or otherwise compromising situation such as lying to the public,

engaging in explicit sexual acts or taking drugs. These videos are used to extort a victim, ruin

a person's reputation, get revenge or simply cyberbully them. The most common blackmail or

revenge use is nonconsensual deepfake porn, also known as revenge porn.

● Caller response services. These services use deepfakes to provide personalized responses

to caller requests that involve call forwarding and other receptionist services.

● Customer phone support. These services use fake voices for simple tasks such as

checking an account balance or filing a complaint.

● Entertainment. Hollywood movies and video games clone and manipulate actors' voices for

certain scenes. Entertainment mediums use this when a scene is hard to shoot, in

post-production when an actor is no longer on set to record their voice, or to save the actor

and the production team time. Deepfakes are also used for satire and parody content in

which the audience understands the video isn't real but enjoys the humorous situation the

deepfake creates. An example is the 2023 deepfake of Dwayne "The Rock" Johnson as Dora

the Explorer.
● False evidence. This involves the fabrication of false images or audio that can be used as evidence

implying guilt or innocence in a legal case.

● Low-cost video campaigns. Marketers using deepfakes can cut video campaign costs by licensing an

actor's likeness and using existing digital recordings along with script dialogue to create new content

without needing in-person actors.

● Fraud. Deepfakes are used to impersonate an individual to obtain personally identifiable information,

such as bank accounts and credit card numbers. This can sometimes include impersonating executives

of companies or other employees with credentials to access sensitive information, which is a major

cybersecurity threat.

● Hyperpersonalization and inclusivity. Deepfake technology enhances brand personalization by

adjusting elements such as ethnicity and skin tone to better reflect diverse customer demographics. This

is used to foster inclusivity and broaden campaign reach.

● Misinformation and political manipulation. Deepfake videos of politicians or trusted sources are used

to sway public opinion and, in the case of the deepfake of Ukrainian President Volodymyr Zelenskyy,

create confusion in warfare. This is sometimes referred to as spreading fake news.

● Stock manipulation. Forged deepfake materials are used to affect a company's stock price. For

instance, a fake video of a chief executive officer making damaging statements about their company

could lower its stock price. A fake video about a technological breakthrough or product launch could raise

a company's stock.

● Texting. The U.S. Department of Homeland Security's "Increasing Threat of Deepfake Identities" report

cited text messaging as a future use of deepfake technology. Threat actors could use deepfake
How are deepfakes dangerous?

Deepfakes pose significant dangers despite being largely legal, including the following:

● Blackmail and reputational harm that put targets in legally compromising situations.
● Political misinformation such as nation states' threat actors using it for nefarious purposes.
● Election interference, such as creating fake videos of candidates.
● Stock manipulation where fake content is created to influence stock prices.
● Fraud where an individual is impersonated to steal financial account and other PII.
● Deepfake technology can fuel unethical actions such as creating revenge porn, where
women are disproportionately harmed.
● Raising awareness and educating people about deepfakes could erode trust in genuine
videos, causing an intellectual crisis in video evidence.
● Deepfakes can be used to deceive security measures or gain unauthorized access to
systems. For instance, a deepfake could potentially bypass facial recognition systems used
for authentication or access control.
Methods to detect deepfakes

● Unusual or awkward facial positioning.

● Unnatural facial or body movement.

● Unnatural coloring.

● Videos that look odd when zoomed in or magnified.

● Inconsistent audio.

● People who don't blink.

● Tiny deviations in the reflected light in the eyes of the subject.

● The aging of the skin doesn't match the aging of the hair and eyes.

● Glasses either have no glare or have too much and the glare angle stays the same despite

the person's movements.

● Sentences that don't flow naturally.

● Suspicious source email addresses.

● Phrasing that doesn't match the supposed sender.

● Out-of-context messages that aren't relevant to any discussion, event or issue.

How to defend against deepfakes
Companies, organizations and government agencies, such as the U.S. Department of Defense's Defense Advanced
Research Projects Agency, are developing technology to identify and block deepfakes. Some social media companies use
blockchain technology to verify the source of videos and images before allowing them onto their platforms. This way, trusted
sources are established and fakes are prevented. Along these lines, Meta and X, formerly known as Twitter, have both
banned malicious deepfakes.

Many organizations offer deepfake protection software, including the following companies:

● Adobe provides a system that lets creators attach a signature to videos and photos with details about their
creation.
● Intel FakeCatcher prioritizes speed and efficiency by analyzing subtle physiological details such as pixel
variations in blood flow to achieve high accuracy in real-time detection.
● Microsoft offers AI-powered deepfake detection software that analyzes videos and photos to provide a
confidence score that shows whether the media has been manipulated.
● Operation Minerva uses catalogs of previously discovered deepfakes to tell if a new video is simply a
modification of an existing fake that has been discovered and given a digital fingerprint.
● Sensity AI offers a detection platform that uses deep learning to spot indications of synthetic media in the same
way antimalware tools look for virus and malware signatures. Users are alerted via email when they view a
deepfake.
● Sentinel is a cloud-based option that offers real-time deepfake detection by using various technologies including
temporal consistency checks, facial landmark analysis and flicker detection to gauge manipulated media.
Notable examples of deepfakes
There are several notable examples of deepfakes, including the following:

● Facebook founder Mark Zuckerberg was the victim of a deepfake in 2019 that showed him boasting about how Facebook
"owns" its users. The video was designed to show how people can use social media platforms such as Facebook to deceive
the public.
● Concerns were raised back in 2020 over the potential to meddle in elections and election propaganda. U.S. President Joe
Biden was the victim of numerous deepfakes showing him in exaggerated states of cognitive decline meant to influence the
presidential election.
● Presidents Barack Obama and Donald Trump have also been victims of deepfake videos, some to spread disinformation and
some as satire and entertainment. During the Russian invasion of Ukraine in 2022, a video of Ukrainian President Volodymyr
Zelenskyy was portrayed telling his troops to surrender to the Russians.
● In early 2024, authorities in Hong Kong claimed that a finance employee of a multinational organization was tricked into
handing over $25 million to con artists posing as the business's chief financial officer over video conference calls, using
deepfake technology. According to the police, the employee was duped into entering a video call with numerous other
employees, but they were all deepfake impersonations.
● There's a TikTok account dedicated entirely to Tom Cruise deepfakes. While there's still a hint of the uncanny valley about
@deeptomcruise's videos, his mastery of the actor's voice and mannerisms, along with the use of rapidly advancing
technology, has resulted in some of the most convincing deepfake examples.
THANK
YOU