Mobile Interaction

Introduction

Prof. Dr. Michael Rohs

[Link]@[Link]
 Mobile Interaction is Usage in Context

[Link]

Mobile Interaction – Summer 2014 2

 Inhalt

In dieser Vorlesung werden die Besonderheiten der mobilen Mensch-Computer-

Interaktion, wie Aufenthaltsort und Einfluss von Umgebungsfaktoren, behandelt.
Es werden mobile Betriebssysteme und Plattformen vorgestellt (z.B. Android und
iOS). Android wird detaillierter dargestellt, so dass Programmieraufgaben mit
mobilen Geräten durchgeführt werden können. Die behandelten Themen umfassen
mobile Ein- und Ausgabetechnologien (z.B. Touchscreens), Multimodalität (visuell,
auditiv, haptisch), Ortsabhängigkeit & Kontext, Fußgängernavigation, drahtlose
Kommunikation, Szenarien und Evaluation im mobilen Kontext, Visualisierung und
Interaktionstechniken für kleine Displays, Kamera- und Sensor-basierte mobile
Interaktion, Touchscreen-Gesten, Bewegungs-Gesten, sowie
Anwendungskategorien und Entwurfsmuster. Der Übungsteil umfasst
Programmieraufgaben, die Entwicklung von mobilen Nutzungsszenarien mit
Papier-Prototypen, die Verarbeitung von Touchscreen-Gesten, sowie die Evaluation
im mobilen Kontext.

Mobile Interaction – Summer 2014 3

 Lernziele

§ Kompetenzen im Entwurf mobiler Anwendungen

§ Kenntnis der Spezifika der mobilen
Mensch-Computer-Interaktion
§ Kenntnis mobiler Interaktionstechniken
§ Modalitäten (visuell, auditiv, haptisch)
§ Touchscreen-Gesten, Bewegungs-Gesten
§ Kenntnis der Einsatzmöglichkeiten von Sensoren
§ Kamera, Bewegungssensoren, NFC
§ Kenntnis der Verwendung von
Orts- und Kontextinformationen

Mobile Interaction – Summer 2014 4

 Organizational Details

§ Time
§ Lecture: Tuesday 13:30-15:00 (2 SWS)
§ Exercises: Tuesday 15:15-16:00 (1 SWS)
§ 4 LP (benotet)
§ Lecturer
§ Prof. Dr. Michael Rohs, [Link]@[Link]
§ Sprechstunde: Monday 11:00-12:00
§ Lab assistant
§ Max Pfeiffer, [Link]., [Link]@[Link]
§ Einordunung
§ Master Informatik
§ Kompetenzbereiche MMK and SE
Mobile Interaction – Summer 2014 5
 Homepage and [Link]

§ Course Homepage
§ Course outline, references
§ [Link]

§ [Link]
§ Slides, exercises
§ Anouncements
§ Discussion, feedback
§ [Link]

Mobile Interaction – Summer 2014 6

 Assessment
Mobile Interaktion

§ Assignments Sommer 2013

Prof. Dr. Michael Rohs, [Link]. Henning Pohl

Übungsblatt 1
§ Done individually, if not stated
differently on the assignment Dieses Übungsblatt ist fällig am Dienstag, den 9.4.2013 um 13:00 Uhr. Das Übungsblatt darf auf
Deutsch oder Englisch bearbeitet werden. Bitte schicken Sie Ihre Lösung möglichst als pdf-Datei
(Word und OpenOffice werden auch akzeptiert) per Email an summer13-mobint@[Link]-

§ Programming Android
[Link] mit dem Subject “Übungsblatt 01: <Vorname> <Nachname>”. Sie sollten Ihre
Lösung in der Übung vorstellen können. Dieses Übungsblatt muss einzeln bearbeitet werden.

Aufgabe 1: [Mobile Nutzungsszenarien]

§ Plagiarism is a) [4 Punkte] Beschreiben Sie, wann Sie mobile Anwendungen nutzen. Welche Kategorien
von Anwendungen nutzen Sie? Was sind typische Aktivitäten? Welche mobilen oder

not allowed!
tragbaren Geräte verwenden Sie? In welchen Situationen und sozialen Kontexten ver-
wenden Sie diese Geräte und Anwendungen? Wie häufig treten externe Unterbrechun-
gen auf?
b) [4 Punkte] Beobachten Sie Nutzer mobiler Anwendungen im öffentlichen Raum (z.B. am
Bahnhof, im Bus, beim Warten). Welche Nutzungssituationen können Sie entdecken?

§ Grading Welche sozialen Kontexte sind erkennbar? Wie lang bleibt die Aufmerksamkeit beim mo-
bilen Gerät?
c) [2 Punkte] Welchen negativen Erfahrungen mit mobilen Interaktionen haben Sie bereits
gemacht (durch Ihre eigene Nutzung oder Nutzung anderer)?

§ Bonus points for Aufgabe 2: [Mobile Anwendungen in Entwicklungsländern]

passed assignments [6 Punkte] Viele Entwicklungsländer haben sehr hohe Wachstumsraten bei der Verbreitung mo-
biler Geräte. Finden Sie durch Recherche im Web heraus, welche mobilen Anwendungen einen
hohen Nutzen speziell für Menschen in Entwicklungsländern haben. Finden Sie zwei konkrete
Beispiele und beschreiben Sie diese kurz. Werden mobile Geräte in diesen Beispielen in erster

§ up to 15% of exam points Linie als Kommunikations- oder als Informationsgeräte verwendet?

Aufgabe 3: [Android SDK installieren]

§ Final exam [4 Punkte] Installieren Sie eine Android-Entwicklungsumgebung für Ihr System, wie unter
[Link] beschrieben. Erstellen Sie ein Android Virtual De-
vice (AVD) für Version 4.2 (API 17) von Android. Falls Sie ein Android-Gerät mit einer anderen

§ 90 minutes
Version besitzen, erstellen Sie für diese Android-Version ebenfalls ein AVD. Nennen Sie uns bitte
die Version Ihres Android-Geräts, so dass wir uns bei der Erstellung der Aufgaben darauf einstel-
len können. Legen Sie in Eclipse ein neues Android-Projekt an. Verwenden Sie einen zufällig ge-

Mobile Interaction – Summer 2014 7

 Aspects: Humans, Technology, Design, Evaluation

§ Understanding “human factors” and analyzing needs

§ Interaction when mobile
§ Models of interaction between people and their environment
§ Understanding technology
§ Programming mobile interactions
§ Sensors and their uses
§ Being creative
§ Rules and principles of iterative design
§ Prototyping techniques
§ Being humble
§ User studies and evaluation methods

Mobile Interaction – Summer 2014 8

 Lectures
# Date Topic
1 15.4. Introduction to Mobile Interaction
2 22.4. Mobile Device Platforms
3 29.4. Mobile Input and Output Technologies
4 6.5. Text Input
5 13.5. Touch Screen Gestures
6 20.5. Touch Screen Interaction
7 27.5. Visualization for Small Displays
8 3.6. Design, Prototyping, Evaluation
9 10.6. no class (spring break)
10 17.6. Mobile Communication
11 24.6. Location and Context
12 1.7. Linking Mobile Devices to the World
Klausur: 5.9.2014,
13 8.7. Sensor- and Camera-Based Mobile Interaction
13-15 Uhr, F102
14 15.7. Application Areas

Mobile Interaction – Summer 2014 9

 Exercises
# Date Topic
1 15.4. Mobile usage scenarios
2 22.4. Android GUI
3 29.4. Touch screen input, tactile feedback, no exercise session (CHI)
4 6.5. Text input
5 13.5. Touch screen gestures
6 20.5. Visualizing off-screen data, interacting with small targets
7 27.5. Back-of-device interaction, complex navigation structure
8 3.6. Designing and paper-prototyping a mobile application
9 10.6. no exercise session (spring break)
10 17.6. Exchanging data and communication
11 24.6. Location-based colors
12 1.7. Mobile tagging
13 8.7. Sensor- and camera-based interaction
14 15.7. Exam preparation

Mobile Interaction – Summer 2014 10

 Books

§ Jakob Nielsen, Raluca Budiu: Mobile Usability.

New Riders, 2012.

§ Steven Hoober, Eric Berkman: Designing Mobile

Interfaces. O'Reilly, 2011.

§ Greg Nudelman: Android Design Patterns: Interaction

Design Solutions for Developers. Wiley, 2013.

§ Matt Jones, Gary Marsden: Mobile Interaction

Design. Wiley, 2006.

Mobile Interaction – Summer 2014 11

 Preview

§ Statistics
§ History
§ Technological enablers

Mobile Interaction – Summer 2014 12

 How Times Have Changed

“In 1954, the Marquis of Donegal heard that the Duke of

Edinburgh possessed a mobile radio set with which he phoned
through to Buckingham Palace – and anyone else on the network
– while driving in London. The Marquis was more than a little
jealous, and enquired of the postmaster general whether he, too,
could have such a telephone. The polite but firm reply was “no”.
In the mid-1950s, if you were the husband of the Queen you
could have a mobile telephone connection to the public network.
But if you were a mere marquis, you could go whistle.”
Agar, J.: Learning from the mobile phone. Journal of
the Royal Society of Arts, pp. 26-27, January 2004.

Mobile Interaction – Summer 2014 13

 Today More Mobile Phones than PCs

§ Mobile subscribers
§ 6.8 billion 2013 (estimate)
§ cf. world population: 7.2 billion (2014)
§ 700 million households with a PC (worldwide)
§ of 1.8 billion households worldwide
§ BRIC countries over one third and fastest growing
§ Brazil, Russia, India, China
§ 2.5 billion mobile subscribers
§ Germany
§ 1.3 mobile subscriptions per inhabitant
§ 2.1:1 ratio mobile to fixed subscriptions
[Link]

Mobile Interaction – Summer 2014 14

 Mobile Phone Subscribers in China

Passerini et al., CACM Oct. 2007

Mobile Interaction – Summer 2014 15
 Mobile, Fixed and Broadband Subscriptions (ITU)

Source: ITU World Telecommunication/ICT Indicators database

Mobile Interaction – Summer 2014 16

 [Link] Source: Mary Meeker, KPCB

Mobile Interaction – Summer 2014 17

 [Link] Source: Mary Meeker, KPCB

Mobile Interaction – Summer 2014 18

 [Link] Source: Mary Meeker, KPCB

Mobile Interaction – Summer 2014 19

 [Link] Source: Mary Meeker, KPCB

Mobile Interaction – Summer 2014 20

 [Link] Source: Mary Meeker, KPCB

Mobile Interaction – Summer 2014 21

 [Link] Source: Mary Meeker, KPCB

Mobile Interaction – Summer 2014 22

 Discuss with Your Neighbor

§ How has communication changed?

§ Scenario: Meeting with friends in the city
§ Scenario: Travelling, train is late, being picked up

Mobile Interaction – Summer 2014 23

 Mobile Interaction is Usage in Context

Primary real-world task

Adapted from a slide by Albrecht Schmidt

Mobile Interaction – Summer 2014 24
 Attention

[Link] [Link]

Mobile Interaction – Summer 2014 25

 Mobiles “in the Wild” and “on the Go”

§ Interruptions
§ From environment or device itself
§ Short attention periods
§ Changing environments
§ Noise, lighting conditions
§ Full concentration on device impossible
§ Cognitive capacity shared with other tasks
§ Presence of others, social situation
§ Incoming call changes social situation
§ Importance of events in environment
§ Environment provides relevant information
§ Acting in the environment based on combination
Mobile Interaction – Summer 2014 26
 iPhone / Android: Mobile Web Share vs. Shipment Share

Source: Morgan Stanley, 2009

Mobile Interaction – Summer 2014 27
 “One more thing...”

Das iPhone und ich

Mensch und Internet werden
eins, dank des iPhones von
Apple. Die ersten vier Wochen
mit dem neuen Wundergerät.

ZEIT ONLINE, [Link]/2008/38/iPhone-38

[Link]/press/[Link]?article=20080318-iphonehype

Mobile Interaction – Summer 2014 28

 N=?
Your Mobile Device Uses?

§ Talking ?
§ Text messaging
§ News
§ Web browsing
§ YouTube
§ Facebook
§ Twitter
§ Music
§ Videos
§ Mobile TV

Mobile Interaction – Summer 2014 29

 Personal Devices and Context

§ Mobile applications closer to real-life needs than PC applications

§ Talk to others
§ Send messages to others
§ Read news
§ Find your way
§ Find a place to eat
§ Meet with friends
§ Get more information about a product
§ Reserve a ticket for an event
§ Pay
§ Remember to do something
§ Discover the surroundings

Mobile Interaction – Summer 2014 30

 Interaction?

Mobile and Wearable Devices

Mobile Interaction – Summer 2014 31

 Interaction?

Mobile Device?

Zátonyi Sándor, cc by-sa

[Link]

Mobile Interaction – Summer 2014 32

 Interaction?

Mobile Device?

§ A basketball that senses

motion and communicates
§ Accelerometer
§ Gyroscope
§ Bluetooth
§ Battery

[Link]

Mobile Interaction – Summer 2014 33

 Interaction?

Around-Device Interaction

§ Magnets as input devices (Harrison, Hudson)

§ Wireless, unpowered, high fidelity
finger input for mobile devices
§ Extending the input area beyond the device
§ User study: 92% selection accuracy for 16º radial targets

Harrison, Hudson: Abracadabra: Wireless, High-Precision, and Unpowered Finger Input for Very Small Mobile Devices. UIST 2009.

Mobile Interaction – Summer 2014 34

 Interaction?

Visual Search: Camera Phones

Recognize the World Around Us
§ Example: Google Goggles for Android
§ Visual search queries for the Web
§ Recognizes a wide range of artifacts
§ Landmarks, barcodes, book covers, etc.
§ Text translation
§ Future?
§ Plants, cars, faces?

Mobile Interaction – Summer 2014 35

 Interaction?

Research Prototypes

Karrer et al.: Pinstripe

Harrison et al.: OmniTouch

Baudisch, Chu: nanoTouch
Mobile Interaction – Summer 2014 36
 Interaction?

Research Prototypes

Song et al.: PenLight

Khalilbeigi et al.: Xpaaand

Pasquero et al.: Haptic Wristwatch Lahey et al.: PaperPhone

Mobile Interaction – Summer 2014 37
 ShoeSense: A New Perspective on
Hand Gestures and Wearable Devices

§ Depth-sensor embedded in shoe

§ Always-available gesture recognition
§ Casual and relaxed hand gestures
§ Acquisition-free and eyes-free input
§ Gesture sets: Triangle, Radial, Finger-Count

Bailly, Müller, Rohs, Wigdor, Kratz. ShoeSense: A New Perspective

on Hand Gestures and Wearable Applications. CHI 2012.
Mobile Interaction – Summer 2014 38
Mobile Interaction – Summer 2014 39
 Interaction?

Smart Artifacts

§ Example: MediaCup
§ [Link]
§ Add “self perception” to everyday things
§ Temperature, fill level, movement
§ Communicate their own state
§ e.g., Bluetooth, ZigBee
§ Determine activity where it occurs
§ “Meeting” if collocated cups with hot liquid
§ The artifact digitally supports
its own applications

Mobile Interaction – Summer 2014 40

 Interaction?

Ambient Umbrella

§ “Never forget your umbrella again.

The Ambient Umbrella lets you know
when rain or snow is in the forecast
by illuminating its handle. Light
patterns intuitively indicate rain,
drizzle, snow, or thunderstorms.
Automatically receives local weather
data from [Link] — no
setup, no sensors, no wet commute.
This intelligent umbrella has you
covered.”
[Link]

Mobile Interaction – Summer 2014 41

 Communication or Information Devices?

§ Information optimists
§ “Mobile phones […] have suddenly become platforms for
entertainment and commerce and tools for information
management and media consumption”
Christian Lindholm et al., Mobile Usability, 2003
§ Communication advocates
§ “…mobile devices will be first and foremost about offering users
the ability to keep in touch with friends, family and colleagues,
and [...] this will take precedence over technologies and
applications that will offer information access and use.”
Richard Harper, People versus Information, Mobile HCI 2003
§ Convergence
§ Communications power and information access

Mobile Interaction – Summer 2014 42

 Information Appliances

§ Support a specific activity

§ Designed for one application
§ Connected to other
information appliances

+
Accelerometer

Mobile Interaction – Summer 2014 43

 Information Appliance or Swiss Army Knife?

§ Many devices with one § Or one device with many

function? functions?

Mobile Interaction – Summer 2014 44

 Keitai Culture
Technology influences Content
§ keitai denwa = mobile phones
§ Read a book on a cell phone?

...

Georg Diez, Die Zeit

Mobile Interaction – Summer 2014 45

 Environmental Impact

§ Mobile phones contain many chemical elements

§ Millions of mobile devices discarded each year
§ Toxic electronic waste, ends up in landfills
§ Disposable technology paradigm
§ Usage lifetime often shorter than functional lifetime
§ Short upgrade cycles
§ Sustainable mobile phone design
§ Nokia 3110 Evolve “Eco-Friendly Device”
§ Cover made of 50% renewable materials
§ Package 60% recycled materials
§ Low energy consumption More on this topic:
Elaine Huang, Khai Truong:
§ Energy-efficient charger Situated Sustainability for Mobile
Phones, Interactions, 3+4/2008

Mobile Interaction – Summer 2014 46

 Impoverished Interactions?

§ Mobiles have tiny screens and keypads

§ Overcome size limitations
§ Output: larger screens, pico projectors
§ Input: multitouch, sensors
§ Use alternative modalities
§ Output: auditive, tactile, auto-stereoscopic 3D
§ Input: speech, gestures, pressure
§ Reduce need for interaction
§ Implicit interaction: by-products of normal behavior
§ e.g., distance-sensor in ear-piece
§ Recognize context: location, calendar, Bluetooth
§ Recognize objects: NFC tags, 2D barcodes, image recognition

Mobile Interaction – Summer 2014 47

 TECHNOLOGICAL ENABLERS
FOR MOBILE COMPUTING

Mobile Interaction – Summer 2014 48

 Technological Enablers for Mobile Computing

§ Processing & Storage § Displays

§ Cheap, fast, reliable, § Cheap, small, high quality,
small, large capacity, energy efficient, integrated
energy efficient
§ Moore’s Law

§ Networking § Sensors & Actuators

§ Cheap, fast, reliable, global, § Cheap, small, accurate, invisible,
local, wireless, many types, energy efficient
ad-hoc, low power

Mobile Interaction – Summer 2014 49

 Processing and Storage

§ Microelectronics and Moore’s “law”:

number of components that can be
integrated on a single chip
doubles every 18 months
§ Exponential improvement
§ Growth is slowing
§ Parallelism
§ Memory chips
have higher capacities
§ Energy per unit of computation falls
§ Size and energy consumption often
as important as processing power 1’’ 8GB (2007)
1’’ 340 MB (2001)
Moshe Y. Vardi: Is Moore's Party Over?, CACM, 54/11, p.5, 2011 2 TB USB drive for 79¤€

Mobile Interaction – Summer 2014 50

 Networking

§ Wireless communication technologies for mobile devices

§ Medium to long-range communication
§ WLAN (range 30-100m, bandwidth 11Mbps or 54 Mbps)
§ GSM (some tens of kbps)
§ UMTS (up to 1920 kbps)
§ LTE (up to 300 Mbps)
§ Low power short range communication
§ Bluetooth (range typically 10m, bandwidth typically 1 Mbps)
§ ZigBee (range 5-500m, bandwidth typically 250 kbps)

§ Sometimes just need to transfer a

few sensor readings over a short distance
§ Energy

Mobile Interaction – Summer 2014 51

 Output and Input Technologies

§ Displays / Output devices

§ LCD screens
§ Loudspeakers
§ Vibrotactile motors
§ Handheld projectors

§ Sensors as “eyes and ears” of mobile devices

§ Multitouch displays
§ Low-power MEMS sensors
§ Sound, acceleration, magnetic field, pressure, capacitance, temperature
§ CCD cameras
§ Powerful class of mobile sensors

Mobile Interaction – Summer 2014 52

 Batteries

§ Energy capacity does not grow exponentially

§ Lead Acid 30-40 Wh/kg toxic, large
§ Nickel Cadmium 40-60 Wh/kg toxic, memory effect
§ Nickel Metal Hydride 60-120 Wh/kg 1990s, self-discharge
§ Lithium-ion 100 to 265 Wh/kg 1991, flammable, expensive
§ Lithium-ion polymer 130 to 200 Wh/kg 1995, flammable, moldable
§ Future
§ Zinc-air batteries up to 470 Wh/kg not rechargeable,
used in hearing aids
§ Lithium-air batteries? up to 1kWh/kg
§ Fuel cells?
§ Harvest energy from environment?

Mobile Interaction – Summer 2014 53

 HISTORY OF MOBILE DEVICES

Mobile Interaction – Summer 2014 54

 History of Mobile Devices

§ Johann Philipp Reis (1834-1874)

§ Self-taught scientist and inventor
§ 1859: Paper “On the Radiation of Electricity”,
rejected, reviewer did not believe in the idea Johann Philipp Reis
§ Idea of sound transmitted by electricity (1834-1874)
§ „Über die Fortpflanzung von Tönen auf beliebige Entfernungen
durch Vermittlung des galvanischen Stroms“
§ „Über Telephonie durch den galvanischen Strom“
§ 1861: First telephone prototype
§ 100 m transmission distance
§ one-way, poor sound quality
§ Had difficulty to interest investors
§ Sold devices for 8-12 Taler

Mobile Interaction – Summer 2014 55

 History of Mobile Devices

§ Alexander Graham Bell saw early Reis

telephone in 1862
§ His father encouraged him to improve it Alexander Graham Bell
(1847-1922)
§ 1876 telephone patented
by Alexander Graham Bell
§ February 14, 1876:
“Improvement in Telegraphy”
was filed at the USPTO
§ A few hours later Elisha Gray
filed “Transmitting Vocal
Sounds Telegraphically”
§ Bell was the 5th entry of
that day, Gray was 39th

Mobile Interaction – Summer 2014 56

 History of Mobile Devices

§ 1894 Guglielmo Marconi invents

the radiotelegraph
§ 1909 Nobel Prize in Physics “in
recognition of contributions to the
development of wireless telegraphy”

§ 1921 combination of
telephone and radio
§ Officers at Detroit Michigan Police
Department communicate from
petrol car to petrol car

Mobile Interaction – Summer 2014 57

 History of Mobile Devices

§ 1938 Canadian Alfred J. Gross invents

the walkie-talkie (also invented
telephone pager and cordless telephone)
§ “I was born thirty-five years too soon. If I
still had the patents on my inventions, Bill
Gates would have to stand aside for me.”

§ 1946 AT&T first commercial

mobile telephone service for
private customers
§ Mounted in vehicles
§ Weighted 80 lbs

Mobile Interaction – Summer 2014 58

 History of Mobile Devices

§ 1962 Telstar first active communications satellite

§ Designed to transmit telephone and high-speed data
communications
§ 1968 Alan Kay’s Dynabook
§ Vision of a portable computer
§ “The Dynabook will have considerable local
storage and will do most computing locally,
it will spend a large percentage of its time
hooked to various large, global information
utilities which will permit communication with others
of ideas, data, working models, as well as the daily
chit-chat that organizations need in order to function.
The communications link will be by private and public
wires and by packet radio.” [Link]

Mobile Interaction – Summer 2014 59

 “Dynabooks” Today

§ Field work supported by tablet PCs

§ Example: Work in archaeological sites

§ Capture notes and images
§ Exchange data
§ Match items to databases

§ Source: [Link]/ipad/pompeii

Mobile Interaction – Summer 2014 60

 History of Mobile Devices

§ 1969 DARPA begins the Internet programme

§ 1971 Ray Tomlinson invents electronic mail
(including “@”)

§ 1971 James Fergason invents Liquid Crystal

Displays, first LCD watches
§ electro-optical effect discovered in 1962
§ 1970 “twisted nematic field effect” patented in
Switzerland
§ 1973 Sharp LCD calculator

Mobile Interaction – Summer 2014 61

 History of Mobile Devices

§ 1972 Motorola prototype of

Portable Radio Telephone
§ First mobile phone call April 3, 1973
§ DynaTAC 8000x first mobile telephone
Martin Cooper (consi-
§ could connect to the telephone network dered as the inventor
§ could be carried about by the user of the mobile phone)
§ [Link]/news/background/tech/cellphones/[Link]

§ 1978 Commercial mobile phone service

in Japan by NTT
§ First city-wide cellular network
§ 1979 Sony Walkman TPS-L2

Mobile Interaction – Summer 2014 62

 History of Mobile Devices

§ 1980 Nintendo “Ball”

§ First commercially successful
mobile LCD screen game

§ 1982 Digital phone exchange in Europe

§ 1984 Psion 1
§ First PDA (personal digital assistant)
§ Clock, calendar, address book, calculator

Mobile Interaction – Summer 2014 63

 History of Mobile Devices

§ 1987 Text message service is launched in Japan

§ 1989 First of 24 GPS satellites of current

constellation is put into orbit (Block II)

§ 1992 First mobile phone for digital networks

§ Motorola International 3200 (500g)

§ 1993 Apple Newton MessagePad 100

§ 5.5" screen, 240x320 pixels, touch screen

Mobile Interaction – Summer 2014 64

 History of Mobile Devices

§ 1996 Palm Pilot

§ 4" screen, 160x160 pixels
§ 1996 Nokia Communicator smartphone

§ 1999 DoCoMo launches i-mode

§ First mobile Internet service

§ 2000 first Bluetooth phone

§ Ericsson T36
§ 2000 first camera phone
§ Sharp J-SH04
§ 110k pixel CMOS sensor
Mobile Interaction – Summer 2014 65
 History of Mobile Devices

§ 2001 debut of the iPod

§ 2" screen, 160x128 pixels, 10000 songs

§ 2002 number of mobile phone subscribers

exceeds number of landline subscribers

§ 2004 PDA with OLED screen

§ Sony Clie VZ-90
§ 3.8" screen, 460x320 pixels

§ 2004 first device using e-paper

§ Sony LIBRIé ebook reader
§ 6" screen, 800x600 pixels, 170 dpi
Mobile Interaction – Summer 2014 66
 History of Mobile Devices

§ 2004 Playstation Portable

§ 4.3" 16:9 wide screen, 480x272 pixels

§ 2005 first mobile phone with motion control sensor

§ Sharp V603SH
§ 2.4" screen, 320x240 pixels

Mobile Interaction – Summer 2014 67

 History of Mobile Devices

§ 2006 First NFC-enabled phone

(Nokia 6131)
§ NFC: Near Field Communication
§ sharing, pairing, transactions between two
devices in close proximity (a few cm)
§ mobile payments, credit card information
§ read NFC tags on museum exhibits or retail displays
§ share contacts, photos, songs, applications
§ pair Bluetooth devices

Mobile Interaction – Summer 2014 68

 History of Mobile Devices

§ 2007 iPhone
§ GSM EDGE, WiFi, Bluetooth
§ 3.5" screen, 320x480 pixels
§ Multi-touch display, no keypad
§ Accelerometer to sense orientation
§ Slide and multi-touch interactions

cover flow

sliding multi-touch (“pinch out”)

Mobile Interaction – Summer 2014 69
 History of Mobile Devices

§ 2008 Android
§ T-Mobile G1
§ SDK 1.0 released
§ Android open sourced under
Apache’s open source license
§ 2005
§ Google buys startup company “Android Inc.”
§ Work on Dalvik VM starts

Mobile Interaction – Summer 2014 70

 History of Mobile Devices

§ 2007 Amazon Kindle

§ E-book reader by Amazon
§ Browse, buy, download, read e-books
§ Newspapers, magazines, blogs, etc.
§ Internet access via Wi-Fi / 3G included
§ E-Ink display, 6”, 1200x824 pixels, 16 grays
§ Very lightweight: 241g
§ 2010 iPad
§ Tablet computer, 10 finger multitouch
§ 75% of tablet computer sales end of 2010
§ 83% tablet computer market share in 2011
§ 1 GHz processor, 10”, 1024x768 pixel screen
§ Weight: 600g
Mobile Interaction – Summer 2014 71
 Future Communication – The Cloud

Example: Body Area Network

Mobile Interaction – Summer 2014 72

 Future Sensing and Context Recognition

§ Sensors for explicit and implicit interaction

§ Example: mobile phone

§ Microphone and camera
§ Acceleration sensors

§ Today: For entertainment (control

music player, sports applications)

§ In the future: Context recognition

and intelligent behavior
of the mobile device
Mobile Interaction – Summer 2014 73
 Future Output Technologies

§ E-paper
§ Flexible displays
§ Projection
§ Head-mounted displays Mistry, Maes, Chang. WUW – Wear Ur World:
A wearable gestural interface. CHI EA 2009.
§ Wall displays
§ Tabletop displays

Mobile Interaction – Summer 2014 74

 Google Glass

[Link]

Mobile Interaction – Summer 2014 75

 Smart Glasses: What Would You Use Them For?

§ Discuss with your neighbor for 2 minutes

§ Uses?
§ Interaction?

§ Better than phone? For what?

Mobile Interaction – Summer 2014 76

 Google Glasses: What Would You Use Them For?

[Link]

Mobile Interaction – Summer 2014 77

 Google Glasses: What Would You Use Them For?

[Link]

Mobile Interaction – Summer 2014 78

 Google Glasses: What Would You Use Them For?

[Link]

Mobile Interaction – Summer 2014 79

 Google Glasses: What Would You Use Them For?

[Link]

Mobile Interaction – Summer 2014 80

Mobile Interaction – Summer 2014 81
Mobile Interaction – Summer 2014 82
 “i’m Watch” Android Watch

§ Display: 240x240 pixels

§ Processor: freescale i.MX233 454 MHz
§ Memory: 128 MB RAM, 4 GB Flash
§ Connectivity: Bluetooth
§ Weight: 95 g
§ OS: Android (“i’m Droid”) [Link]
i-m-watch-kaufen/color-buy

§ Battery: Li-Polymer 450 mAh

§ 24 h standby with Bluetooth on, 5 h continuous use
§ Sensors: accelerometer, magnetometer
[Link]
[Link]
[Link]

Mobile Interaction – Summer 2014 83

 A Smart Watch: What Would You Use It For?

§ Discuss with your neighbor for 2 minutes

§ Uses?
§ Interaction?

§ Better than phone? For what?

Mobile Interaction – Summer 2014 84

 UBIQUITOUS COMPUTING

Mobile Interaction – Summer 2014 85

 Device Combinations

§ New possibilities from using different

devices in combination
§ Interaction flow between devices

§ Position on the body: Glasses,

Watches, Phones
§ Display size: Tabs, Pads, Boards
§ Mobile and fixed: Phone, table, wall,
public display
§ Phone and car

Mobile Interaction – Summer 2014 86

 Combining a Laser Rangefinder and a Mobile Phone?

§ Combination
§ What to use it for

§ Who could use it

§ Usage situation / scenario

Zátonyi Sándor, cc by-sa

§ UI design sketch

§ Relation to external media (e.g., paper)?

§ Connection to website?

§ 3 minutes

Mobile Interaction – Summer 2014 87

 Ubiquitous Computing
“The most profound
technologies are those that
disappear. They weave
themselves into the fabric of
everyday life until they are
indistinguishable from it.” Mark Weiser

§ Computers embedded in everyday things

§ Technology moves into the background
§ Computers in the world, instead of world in the computer

§ Mobile devices as always available mediators

§ Entry point into the digital world
Mobile Interaction – Summer 2014 88
 “Computational Surplus”

§ Limitations of computing systems no longer the limiting factor

§ processing, storage, bandwidth
§ Interaction becomes part of everyday life
§ UI design becomes Capacity of
discriminating feature human attention
§ More devices
§ Devices must t i o n al
become simpler m p uta 1 p u ting
Co per $ ly y o f co m
supp si t
Diver under $1
k
es
devic

2010 time
Graph adapted from: Lee: In Search of a Natural Gesture, XRDS, summer 2010, 16(4)

Mobile Interaction – Summer 2014 89

Mobile Interaction – Summer 2014 90