WO2014094699A1

WO2014094699A1 - Method for operating a device having a user interface with a touch sensor, and corresponding device

Info

Publication number: WO2014094699A1
Application number: PCT/DE2013/000605
Authority: WO
Inventors: Jörg Edelmann; Philipp Mock; Andreas Schilling
Original assignee: Eberhard Karls Universität Tübingen; Stiftung Medien in der Bildung
Priority date: 2012-10-19
Filing date: 2013-10-16
Publication date: 2014-06-26
Also published as: DE102012219129A1; DE102012219129B4

Abstract

In a method for operating a device having a user interface with a touch sensor, a virtual keyboard with virtual keys is generated on the touch sensor and touch sensor data are generated by the detection of the typing behaviour of a user from a multiplicity of typing actions of the user on the virtual keyboard. Using the touch sensor data, a user-specific keyboard model is generated which is used for classifying key presses during the use of a virtual keyboard. Position data and fingerprint data are determined during the determination of touch sensor data for each key press of a finger, wherein the position data represent the position of a touch of the surface of the touch sensor during a key press and the fingerprint data represent an image of the local sensor response to a touch of the surface of the touch sensor during a key press in the form of raw data of the touch sensor or modified raw data derived from the raw data in such a way that the classification-relevant information contained in the raw data is substantially maintained. The keyboard model is generated using the position data and the fingerprint data. The method makes it possible to reduce the frequency of incorrect inputs during the use of a virtual keyboard on devices having a user interface with a touch sensor.

Description

Method for operating a device having a user interface with a

Has touch sensor, and corresponding device

description

The invention relates to a method for operating a device, the one

A user interface with a touch sensor, according to the preamble of claim 1, as well as a device with a user interface having a touch sensor, according to the preamble of claim 1 first

There are a variety of computer-aided devices today with a

A user interface having a touch sensor. The device may be, for example, a stationary or portable computer or a computer-controlled machine. A touch sensor is an input device in which a program flow of the device can be controlled by touching certain areas of the surface. A touch sensor may be combined with a screen. A touch screen combines the electronic display of a screen with a screen

Touch sensor. A touch screen is therefore a combined input and output device. A controller connected to the touch screen generates image contents of the

Touch screen and serves to receive and process input signals obtained by touching touch areas on the surface, e.g. be generated by finger pressure.

If commands are expected in the form of letter and / or number sequences, the controller is typically configured to operate in at least one mode of operation

Touchscreen a virtual keyboard can be generated with a variety of virtual buttons. As "keyboard" an input device is referred to, which contains as controls a number of fingers to be pressed with keys. For example, are known

Typewriter keyboards with mechanical keys that move when pressed, triggering input signals. A "virtual keyboard" has "virtual keys". Virtual buttons are generated on a touch screen usually appear as a button-like appearing symbols (soft keys or icons) electronically. Virtual buttons can also be created on a touch sensor by printing, recording or otherwise. The keyboard layout or the keyboard layout of a mechanical or virtual keyboard describes both the coding of the individual keys as well as their position and number on the keyboard.

It has long been known that the text input by means of a virtual keyboard on a

Touchscreen is usually less accurate and more error-prone or slower than on a hardware keyboard with mechanical buttons. Unlike mechanical keyboards, the user does not receive any haptic feedback about the position of the fingers on the keyboard because the edges of the keys are imperceptible and the finger position can not be corrected without visual control. This makes fast typing difficult with the ten-finger system, and even experienced users often have to look at the virtual keys as they type. This results in lower writing speeds for text input and higher error rates.

Against the background of this problem, a method is described in WO 2012/048380 AI, according to which an adapted to the laid finger keyboard layout can be determined. For this purpose, a keyboard model is created in which "home keys" of the individual fingers as well as linked keys are defined. For example, with an English QWERTY keyboard layout or a German QWERTZ keyboard layout, the keys ASDF and JKL represent the home keys for an accomplished keyboard user now the hand in the

Positioned on the touch screen and recognized, the virtual keys move to the corresponding positions of the fingertips. Depending on how many fingers are placed, different keyboard layouts can be called up. The virtual keyboard thus adapts to the natural finger positions of the respective user by changing the position of the individual virtual buttons.

US 2009/023736 A1 describes another possibility for automatically adapting the keyboard layout to the personal characteristics of a user. The user places his hand on the touch screen so that one or more fingers and the palm of his hand touch the surface of the touch screen. The keyboard layout is then determined at least in part from the distance between the detected point of contact of the handball and the positions of the finger touch and adapted accordingly to the user. In the patent application US 2010/0259561 AI methods are described, which allow the keyboard layout of a virtual keyboard in a learning process to the respective user adapts. The computer system recognizes the typing pattern of a user and then changes the positions, the sizes and / or orientations of the virtual keys in such a way that the user gets comfortable working, whereby the typing errors are to be reduced.

The patent application US 2011/0254772 Al describes methods for detecting movements of one or more touches in the area of a virtual keyboard of a touch screen. In the method, certain characteristics of a touch, such as e.g. the intensity, the size of the contact surface or the support angle used to perform actions in addition to the text input with the virtual keyboard. Gestures are interpreted from successive touches. These can also be created by a user himself and are then recognized by the system. The gesture recognition allows, among other things, to distinguish between touch by means of a finger, a thumb or a palm. The keyboard layout is not adapted to the user in this procedure.

The article "Personalized Input: Improving Ten-Finger Touching Typing Through Automatic Adaptation" by Findlater, L., and Wobbrock, JO in: Proc. CHI '12 (2012), ACM Press, 815-824 (hereinafter referred to simply as "Findlater et al.) describes a method for adapting a virtual keyboard of a touch screen to the typing behavior of a user. The

Adaptation can be visible or not visible to the user. In addition to the location of the touch screen touches, additional geometrical characteristics of the touch, such as the size and orientation of the contact zone, and movement of the touch during key press are evaluated to assign keystrokes to individual virtual keys. It has been shown that customization of the virtual keyboard to the user can increase the input speed.

It is an object of the invention to reduce the incidence of misses when using a virtual keyboard on devices having a user interface with a touch sensor. To achieve this object, the invention provides a method having the features of claim 1 and a device having the features of claim 11.

Advantageous developments are specified in the dependent claims. The wording of all claims is incorporated herein by reference.

In the method, a virtual keyboard is generated with a plurality of virtual buttons on the touch sensor. If the touch sensor with a screen to a

Touchscreen is combined, the virtual buttons are generated in the form of electronically generated icons on the screen. It is also possible to generate virtual keys in other ways, e.g. by optical projection onto a surface of the touch sensor or in the form of permanent, invariable virtual buttons, e.g. can be generated by printing or recording or laying on the surface of the touch sensor. An example of this is known as "Touch Cover" for tablet computers from Microsoft.

Each virtual key has a given position and shape. By detecting the typing behavior of a user from a plurality of typing actions of the user on the virtual keyboard, user-specific touch sensor data is detected. These include information about individual peculiarities of the user in keyboard usage, such as the way certain virtual keys are hit by the user. The user-specific touch sensor data is used to provide a user-specific

generate (personalized) keyboard model. The keyboard model is thus the result of a learning process. Depending on the keyboard model generated in this manner, the response of virtual keys of the same virtual keyboard or other corresponding virtual keyboard to the user in use is adjusted. The

Keyboard model is thus used to classify keystrokes when using a virtual keyboard.

The step of associating a typing action and the input triggered by this typing action, ie, selecting the virtual key that is considered to be typed, is called a "classification." The classification is a fingerprint at a particular location of a particular key of a plurality of The hardware used for this purpose and the program components (software) that are active for this purpose form the The term "keyboard model" is an alternative term for a classifier.The task of the keyboard model is, for example, to interpret a keystroke as an input of a specific letter.The term "response" refers to the classification result or to the " Answer "of the virtual keyboard at a finger pressure.

In the claimed invention, in determining user-specific

Typing behavior data for each press of a finger both position data and

Fingerprint data determined.

The term "position data" refers to data representing the position of the contact zone of a finger with the surface of the touch screen when a key is pressed.

The term "fingerprint data" refers to data representing an image of the area of contact between the finger and the surface of the touch sensor when a button is pressed by a finger The area of contact covered by the image includes, but may be, the immediate area of contact between the finger and the surface

include adjacent areas where there is no physical contact. The captured

Touch area may e.g. rectangular or otherwise polygonal. The image is preferably two-dimensional, but in certain cases could also be one-dimensional. Instead of a single image per key press, the time course of the

Fingerprint data, so a series of multiple images are determined.

The term "image" hereby refers to the raw data of the area covered in the area of contact

Touch sensor and modified raw data derived from the raw data in such a way that the classification relevant information contained in the raw data is substantially preserved. In an optical touch sensor, the detected raw data may be e.g. represent the spatial distribution of light intensity in the area of the fingerprint. For a capacitive touch sensor, the detected raw data may represent the local distribution of the measured capacitance.

"Modified raw data" is data that also contains this information essentially unadulterated, but usually with a reduced amount of data. The modified raw data include, in particular, such data as is available from the raw data by compression methods (eg JPEG compression), filtering (eg low pass) or methods for dimensional reduction (eg Principal component analysis or discriminant analysis). Modified raw data can also be derived from the raw data by calibration or normalization.

Modified raw data in the sense of this application are to be distinguished from such data that are generated by evaluating raw data according to specific, pre-defined, reasonable criteria by feature extraction. In some conventional methods, raw data is e.g. in terms of geometric criteria, such as relative fingertip position or relative fingerprint center position or moving direction of the fingertip for a fingerprint, before the data processed according to pre-defined criteria are used to classify a classifier. According to the findings of the inventors, however, such feature extractions which seem plausible to humans can lead to a loss of classification-relevant information which is still contained in the raw data or the modified raw data. The claimed invention avoids such

Information loss largely by fingerprint data in the form of raw data or modified raw data are used directly to generate the keyboard model.

The fingerprint data is fed to a classifier without further pre-processing. Thus, the full primary information in a fingerprint is unadulterated to the

Classification available.

Among other things, the claimed invention is based on the insight that in general it can not be determined in advance which concrete information leads to a better distinction of the fingerprints. The learning algorithm separates the data based on the information given in the raw data or modified raw data. The criteria of separation need not always correspond to a writable property (e.g., size or orientation of a fingerprint assumed to be elliptical, etc.). In that regard, the classification works similar to human information processing. Again, it is not always possible to specify clear criteria according to which a person distinguishes objects.

Preferably also in the adaptation of the response of the keyboard in

intended use for each key press of a finger both fingerprint data and position data determined and evaluated using the keyboard model. The keyboard model could have the effect of changing the visible keyboard layout as a result of customization. However, it is considered advantageous if the customization of the virtual keyboard to the individual user is imperceptible to the user. In a preferred embodiment, the appearance of the virtual keyboard, ie, the visible keyboard layout, remains unchanged, so that the user can orient himself to the visible positions of the virtual keys as needed. The virtual keyboard is made by customizing the

Automatically based on the keyboard model to ensure that the frequency of incorrect entries is significantly reduced compared to methods without using the keyboard model. Thus, an improvement in the accuracy of the classification of the keys is achieved.

Such a method differs from known methods which adapt the keyboard size of the key likely to be next pressed using a language model, in that the adaptation of the response of the virtual keyboard is independent of the meaning of the information represented by the keyboard input. One difference to methods where creating a customized keyboard layout by placing fingers or other hand parts on the touch screen is that the virtual keyboard need not show any change visible to the user. Similar to a hardware keyboard, the virtual keyboard can have a timeless appearance, so that a user is in a working situation familiar from mechanical keyboards. This will not distract the user and prevent the user from adapting to a changed keyboard layout. To some extent, this can prevent the user's writing behavior from gradually deteriorating. Similar advantages also result over known methods, in which the position and / or size of individual keys is adapted to the typing position of the user.

In preferred embodiments, the position data is derived from the touch sensor data. For example, the geometric center of gravity of a fingerprint or a weighted according to certain criteria focus of a fingerprint can be determined as a position of the associated keystroke and further processed.

It has proved to be advantageous for the classification of the position data, the

custom distribution of keystrokes from a variety of previous ones

Key presses by a two-dimensional normal distribution (bivariate Gaussian function) too model. For new keystrokes, these normal distributions can be evaluated to calculate probabilities for each key.

For the processing of the fingerprint data, different classification methods can be used, which are able to classify the information encoded in the fingerprint data and thus assign a specific keypress to the most probable virtual key. In a variant of the method, a support vector machine (SVM) is used to classify the fingerprint data, which is essentially an algorithm for pattern recognition, which can be implemented in a computer program and is capable of detecting the recorded fingerprints on the basis of the Fingerprint data can be reliably assigned to different classes or to different virtual keys.Also other classification algorithms can be used, eg neural networks or classification with Gaussian processes or logistic regression.

As an alternative to two separate classification methods for position and fingerprint data, these two pieces of information can also be used in combination in a classification method.

The use of fingerprint data requires that the touch sensor be configured to exist within the typical size contact zone of a finger with the surface, a plurality of measurement points for a finger-size variable physical size, such that an image of the touch area using the spatially resolved measured values of the measured variable can be generated. The term "measuring point" here refers to a measuring area of small extent, which is not punctiform in the mathematical sense. The extent of a measuring point can e.g. in tenths of a millimeter range or less.

In some embodiments, the touch sensor is an optical touch sensor, in which changes in the sensor properties in the respective contact region can be detected spatially resolved optically by means of a two-dimensional image of picture elements (pixels). It may, for example, be a grayscale image in which measuring points which were exposed to a greater pressure when the key was pressed receive a different gray level in the computer-aided image than weaker pressure-loaded measuring points. With an optical touch sensor, if necessary, a shadow can also be detected in the immediate vicinity of the contact zone. It is also possible to use the method when using a capacitive touch sensor whose local resolving power is so large that within the typical size of a fingerprint multiple measurement points lie whose capacity can be determined independently of the capacity of adjacent measurement points. The measuring points of a capacitive

Touch sensors are e.g. formed by crossing points of perpendicular to each other in two staggered planes extending tracks, which form a local capacity at each crossing point, the size of which changes under pressure load and / or finger touch.

Preferably, the spatial density of measurement points or in one or more transverse to each other parallel to the surface of the touch sensor directions should be at least 0.5 / mm, in particular 1 / mm or more. As a result, a sufficiently good spatial resolution can be achieved.

The function of the keyboard model may, in some embodiments, be described as associating with each virtual key an area-defined active area having a different position and / or shape for one or more virtual buttons than the area of the virtual button visible on the touch-sensor.

The active area is usually larger than the visible area of the virtual button, so that even typing actions that are at the edge of the visible virtual button or even outside the visible area of the virtual button are correctly assigned (classified). Alternatively or additionally, the active area may be laterally displaced from the visible area of the virtual button in a particular direction, which may be the case, for example, when a user taps regularly in a particular direction adjacent to the correct position of the button. Typically, the active regions each have an irregular shape, so in particular are neither circular nor rectangular. The shape and size of the active areas active on the key press and their location are determined dynamically using the user-specific keyboard model. The generation of such active areas allows error-free input even with very imprecise keystrokes, where necessary, by a

Keystroke desired input also occurs when the assigned key is missed while typing. The classification using fingerprint data offers advantages beyond just assigning keystrokes to specific virtual keys. In one embodiment, a user classifier is generated that performs a user classification such that a keystroke based on the fingerprint data is assigned to a particular user. Then, further actions of the device may be controlled depending on the result of the user classification. For example, a

Prompt for entering a password are generated and the subsequent keystrokes are analyzed using the user classifier not only on whether the expected password (eg a pre-defined and stored sequence of letters, numbers and / or characters) is entered, but in addition Also, whether the correct or expected password is entered by an authorized user. The

Security settings may be set up to release certain or all other actions of the device only if the correct password is entered by an authorized user.

The invention also relates to a device with a user interface, the one

Touch sensor has. The device is for carrying out the method in the

Generation of the keyboard model and / or in the intended use under

Use of an already generated keyboard model configured.

In one formulation, the device has a user interface that has a

A touch sensor having within a typical size of a contact zone of a finger with the surface of the touch sensor, a plurality of measuring points for a through

Fingerprint and / or finger contact and / or approach of the finger to the surface variable physical size, so that touch sensor data can be generated using the spatially resolved measured values of the measured variable. A controller coupled to the touch sensor is configured to receive and process touch sensor data obtained by contacting touch areas on the surface of the touch sensor

Touch sensor can be generated. The device is configured such that in at least one operating mode, a virtual keyboard having a plurality of virtual keys can be created on the touch sensor. In this mode of operation, when acquiring touch sensor data for each press of a finger, the above-described "position data" and

The position data and fingerprint data are classified using a user-specific keyboard model, with each of the virtual keys under Using the user-specific keyboard model, associating an active area of the touch sensor that has a different position and / or shape to one or more of the virtual buttons than the area of the virtual button visible on the touch sensor, wherein a touch is at a touch location within an active area is interpreted and processed as touching the associated virtual key.

The invention can be implemented on different devices with touch sensor and appropriate control. The ability to carry out embodiments of the invention may be implemented in the form of additional program parts or program modules in the control software of such controllers.

Therefore, another aspect of the present invention relates to a computer program product which is stored, in particular, on a computer-readable medium or signal, wherein the computer program product, when loaded into the memory of a suitable computer and executed by a computer, causes the computer program product controlled device performs a method according to the invention or a preferred embodiment thereof.

It is thus e.g. it is possible to create, store and transfer the keyboard model to a first device by means of a computer program product on a second device and to implement it there.

These and other features will become apparent from the claims and from the description and drawings, wherein the individual features each alone or more in the form of sub-combinations in an embodiment of the invention and in other areas be realized and advantageous and protectable Can represent versions. Embodiments of the invention are illustrated in the drawings and are explained in more detail below.

Fig. 1 shows schematically components of an embodiment of a device, the one

User interface having an optical touch sensor;

Fig. 2 shows in Figs. 2A to 2D grayscale images of various fingerprints of a particular one

User for different keystrokes when writing the word "BANK"; FIG. 3 shows in FIG. 3A a two-dimensional normal spreading functions of key presses of a specific user on the virtual keyboard for determining a position model and in FIG. 3B the active areas of the virtual keys determined on the basis of the position model;

Fig. 4 shows in Figs. 4A to 4D the effect of the application of a keyboard model based on a

Combining a positional classifier and a fingerprint classifier relies on the shape and location of the active areas when writing the word "BANK";

Fig. 5 shows a diagram with a comparison of error rates with different

Keyboard models were achieved.

embodiments

In the following, aspects of the invention will be described using the example of an optical touch sensor, which forms an optical touch screen in combination with a screen.

Essential aspects of the embodiments are based on the following considerations. Virtual keyboards have the advantage over mechanical keyboards that they can be adapted at runtime, ie during use by a user. To the

Improving text input on touchscreens (Touchscreens), should be the

Adjust the response of the virtual keyboard to the typing behavior of the user. The captured input data of some touchscreens, in contrast to a regular keyboard, which consists of discrete switches, much more complex. So not only the pressed button can be determined, but also the exact position of the button press. For example, if you press between two buttons, additional information can be evaluated to determine the button your user wants.

In addition to the precise location of the touch area on the touch sensor, modern touch sensors can capture much more information about each touch point. In the case of some optical touch sensors, such as e.g. the "Microsoft Surface" or "Samsung

SUR40 ", captures a complete grayscale image of the surface in the infrared light spectrum become. For each touch on the touch sensor, an image of the brightness profile can be captured in two dimensions.

When typing with keyboards, experienced users use different fingers for different keys. Even if the same finger is used for another key, e.g. the support angle and possibly the pressure on the

Touch screen. This information can be derived from the readings provided by the touch screen or the corresponding raw data. The differences are reflected in the raw data.

The following is an example of how it is possible to provide a virtual keyboard that learns not only where, but also how a particular user

different virtual buttons presses. For this purpose, machine learning algorithms are used to calculate a user-specific (or user-specific or personalized) keyboard model. The adaptation of the keyboard to the user is here invisible to the user in order not to distract the user and to avoid that the user changes his typing behavior by a visibly changed keyboard layout.

A method step serves to determine user-specific touch sensor data by detecting the typing behavior of a user from a plurality of user's typing actions on the virtual keyboard. This training or training process usually takes place at least in part, possibly even completely, before the actual use phase. Of the

Training process can also extend into the phase of normal use, so that the keyboard model created or refined during normal use or

can be optimized. The teach-in process can be performed on the same touch sensor on which the subsequent intended use is made. It can also be another touch sensor, e.g. a substantially identical, construction-like or functionally identical variant. The trained keyboard model can be present as a computer program product, so that a transfer from one device to another device is possible.

In order to determine a keyboard model adapted to the respective user, that became

Typing behavior recorded by twelve subjects. Each user had to type a given text. This made it possible to simulate a "perfect" keyboard by being inaccurate Inputs anyway the correct letter (ie the letter that should follow in the given text next) was selected as input. Based on the inputs, a user-specific keyboard model was determined or developed for each user.

The data was recorded with an optical touch screen. FIG. 1 shows a schematic section through the touch screen equipped with an optical touch sensor 100 in the area of the virtual keyboard 150 displayed on the screen. A warp-resistant plane-parallel acrylic glass pane 102 carries on its planar upper side a thin projection film 104 whose free surface 106 is that of the user forms the touching surface of the touch sensor of the touch screen. At a distance behind the opposite rear side, an infrared-sensitive camera (IR camera) 120 is arranged. Infrared lamps 122 are used for uniform illumination of the acrylic glass pane.

The virtual buttons 140 of the virtual keyboard 150 were displayed on this touch screen. The layout of the keyboard corresponded to a standard QWERTY keyboard with im

Essentially square buttons with about 19 mm edge length. The virtual keyboard was generated by a computer program and projected from the back to the surface of the touch screen within the console using a projector 170.

The optical touch sensor works on the principle of frustrated total internal reflection (FTIR). In the area of the fingerprint, or in the area of contact between the finger and the surface when the button is pressed, the total internal reflection at the surface of the fingerprint becomes

Perturbed acrylic glass and there is scattered light 128, which is detected by the IR camera 120 spatially resolved. The intensity of the scattered light - represented by the length of the arrows - is u.a. Depending on the locally effective contact pressure, so that in the middle region of higher contact pressure, a higher scattered light intensity (longer arrows) results than in the edge region of lighter contact pressure (shorter arrows).

The infrared-sensitive camera was arranged so that it covered the surface in the area of the entire virtual keyboard from behind in its image field. For each touch, not only the position but also a two-dimensional grayscale image of the touch area in which the contact area 132 of the finger 130 lies on the surface is recorded. The camera had a resolution of 640 * 480 pixels at 60 frames per second. The image analysis was set up so that for each fingerprint a rectangular Grayscale image of the touch area with 42 * 42 pixels around a touch center in the area of the contact zone was detected.

2A to 2D show grayscale images of various fingerprints of a particular user on different virtual keys when writing the word "BANK." Here, the gray scale image in Fig. 2A corresponds to the letter "B", Fig. 2B represents the letter "A", Fig Fig. 2C represents the letter "N" and Fig. 2D represents the letter "K".

The complete information of the different keystrokes of each session in the training phase was stored in an xml file along with those obtained from image processing

Fingerprint data saved. The information included the location and a timestamp for each touch, as well as a reference to the associated fingerprint file. With this raw data, a learning algorithm implemented in a computer was trained, which generated an individual keyboard model for each user.

It is important, inter alia, that these touch sensor data provided by the touch sensor be in the form of raw data without prior analysis for particular features, e.g. geometric features such as shape of the fingerprint, orientation of the fingerprint, etc., were used directly for training (learning) of the classifier.

In one embodiment, to determine a user-specific keyboard model, a combination of two classifiers has been used to determine a high-fidelity association between a particular user's keystroke and that through it

Pressing a key probably meant to achieve virtual key. In addition to a position classifier, which processes position data and uses it for classification, an image-based fingerprint classifier was used, which takes into account the image information (fingerprint data) derived from the fingerprints to classify key presses.

The position of a fingerprint or a keypress is for the

Key classification is an important feature and should be properly mapped into a keyboard model to be really useful. An approach that would rely only on the limitations of the on-screen keyboard will bring only modest results in the inventors' experience, as keys are often not precisely hit and the actual touch positions of a key often deviate from the actual center of the virtual key. It has been found that the spatial distribution of the various key strokes of a person dedicated to a particular key can often be well described with a bivariate normal distribution. Such two-dimensional

Normal distributions were used in the determination of a position model. Fig. 3A shows the normal distribution functions of keystrokes of a particular user on the virtual keyboard. It can be seen that the local distribution functions are not symmetrical on all virtual keys and that some keys are statistically much more precisely hit (steeper distribution) than other keys (shallower distribution). For further data processing, two-dimensional normal distributions offer the advantage of being natural

Represent probability values for a classification. This is particularly advantageous when the results of the position classification with a classification according to other criteria

(Fingerprint data) should be combined.

On the basis of the two-dimensional normal distributions, each key can be assigned an active area which corresponds to the spatially limited area around a virtual key in which a keystroke is assigned to the respective key on the basis of the position model. Figure 3B is a schematic representation of the complete rectangular keypad which is divided into irregular shaped active areas (e.g., active areas 301-304). Directly adjacent active areas directly border, i. without gaps, so that there are no places in the rectangle area of the virtual keyboard without a clear assignment to a virtual key. In the tendency, those virtual keys precisely hit by the respective user with high repeatability obtain a relatively small active area, while those virtual keys hit with less precision and greater dispersion at the individual key presses receive a larger active area.

In the embodiment, a so-called "support vector machine" (SVM) has been used as the image-based fingerprint classifier An SVM is a classifier that classifies a set of objects into classes that are as wide as possible around the class boundaries An SVM therefore belongs to the class of so-called "Large Margin Classifiers". The starting point is a set of training objects, each of which knows which class they belong to. In the present case, the training objects are given by the keystrokes during teaching, while the virtual keys form the classes to which the keystrokes are to be assigned. Each object is represented by a vector in a vector space. The SVM works by fitting in this space a hyperplane that acts as a separation surface and divides the training objects into two classes, maximizing the distance of those vectors closest to the hyperplane. This wide margin around the class boundaries makes it possible to also classify (assign) the key presses that do not correspond exactly to the objects detected during training (key presses) as reliably as possible.

To combine the classifiers (position classifier and fingerprint classifier), decision probabilities for a keystroke x were calculated for each class. For the position _data x _pos , the probability of a decision for a particular key can be obtained directly from the normal distribution functions as:

Where f is the normal density function with parameters μ and σ of the key k at the coordinate x _pos and K represents the total number of keys.

For the image-based support vector machine, probabilities were calculated according to the article JCPlatt: "Probabilistic Outputs for Support Vector Machines and Comparisons to Regularized Likelihood Methods" in: Advanced in Large Margin Classifiers, pages 61 to 74, MIT press, 1999. This method is already integrated into the Matlab toolbox libSVM The class c with the highest multiplied probability is taken as the resulting decision: arg max (P _pos (A: | Xp _OS ) * P _img (k \ x _img ))

where Xjmg represents the fingerprint data of a keystroke x.

The effect of applying a combination of these two classifiers to the keyboard model or the active areas that can be generated thereby will be explained with reference to FIGS. 4A to 4D. As already mentioned, based on the keyboard model, an active area defined by area boundaries is assigned to each virtual key, the active areas of the respective keys generally having a different shape and / or position than the one on the other

Touch screen visible area of the virtual button. By way of example, FIG. 3B shows a distribution of active regions that are based exclusively on an individual

Position model, that is, only with the help of the position data, were determined for a particular user.

Now a keyboard model is used, which contains both the position data and the

Fingerprint data used for classification, resulting in typing dynamic active areas of other size and / or shape and / or position, since in addition to the position information using the fingerprint data and information about how are processed (eg with which finger) a user a particular Key presses or wants to press.

2, the user would like to write the word "BANK." To do this, he will first press the virtual key "B" with his index finger, for example. As soon as the

Fingertip touches the surface of the touch sensor, the fingerprint data and the position data are determined. The keyboard model recognizes from the fingerprint data to which key the fingerprint belongs. It does not need to be explicitly determined which finger is used. It only determines how similar the sensor information is. A key, e.g. the spacebar, can also be used with two fingers. The classifier would then correctly assign keystrokes of one finger and the other.

The combined information results in all finger touches occurring in the grayed-out active area 301 '(marked "B") in the usual way for the user being interpreted as input of the letter "B". When compared with FIG. 3B, it can be seen that the size and shape of the active area, which is determined based on the combination of position data and fingerprint data, has a different shape and area than the active area 301, which is based solely on of a position model. For example, the additional fingerprint information may cause a particular finger touch to still be interpreted as tapping the "B" key when it is relatively far from the center of the virtual key "B", but with the "right" finger. that is, with the finger with which the user usually presses the "B" key User operates different keys, thereby derived from the fingerprint data, which were detected and evaluated during the learning phase.

A corresponding dynamic definition of an active area also results in the subsequent key presses for the letters "A", "N" and "K", their dynamically generated during typing active areas 3012 ', 303' and 304 'in Fig. 4B until 4D are highlighted in gray.

It can be seen in each case in comparison with FIG. 3B that active areas are larger

Surface expansions and, where appropriate, other shapes result if, in addition to the position information, the fingerprint information in the keyboard model is also taken into account and becomes effective when using the virtual keyboard.

In the embodiment, neither the positions nor the shapes of the virtual keys change during learning or during intended use, so that the user-visible appearance of the keyboard layout remains unchanged, regardless of the position and shape of the active areas. This can promote high write speeds at low error rates.

The reliability of the classification using the described embodiment has been assessed inter alia by comparison with a classification made according to the proposal of Findlater et al. carried out, evaluated.

The proposed method was developed and tested using real user data. First, typing behavior data was recorded by twelve users. The touchscreen displayed a classic QWERTY keyboard. to

Touch detection, an optical touch sensor was used. After a

During the familiarization phase of 5 minutes, the subjects had to type in a given text for 3 x 10 minutes, whereby imprecise keystrokes were also accepted in order to allow as natural typing as possible. To data with different

To capture typing behavior, three users typing with two fingers, three users typing individually with 3-8 fingers, and three users typing on the ten-finger system were selected. For each keystroke, both the position and the local sensor image became

(Fingerprint data) recorded. On the basis of these data different individual keyboard models were developed and evaluated. In each case, the first half of the recorded data was used as training data for modeling and the other half used for evaluation. The following keyboard models were created and evaluated (see Fig. 5):

Model Ml A model based on the displayed keyboard

Model M2 A model based on individual tip positions

Model M3 A model based on the entire fingerprint data in the form of raw data Model M4 A combined model of individual tip positions and fingerprint data Model M5 To the presented method with Findlater et al. was an additional model with the extracted geometric used there

Properties of the fingerprint data and the individual typing positions evaluated.

The evaluation revealed significant differences in the error rate (ER). The error rates ER are shown graphically in FIG. 5 as a function of the models M1 to M5.

Individual position-based models (models M2, M4 and M5) result in a significantly lower error rate compared to a static model according to the keyboard actually represented (model M1).

Even the pure fingerprint data (without positional data) according to model M2 provide useful results.

The combination of individual position and fingerprint data model according to the embodiment of the invention described here (model M4) reduces the error rate unlike the other models.

The model M5, which is based on individual tip positions and the in Findlater et al. used extracted geometric properties (these represent size and

Alignment of the contact zone as well as the movement of the touch during the press of a button) does not lead to a significant improvement in the error rate compared to the model based on individual jogging positions. It is currently believed that these are in advance Specified selection of certain geometric criteria results in certain information relevant to key classification being lost during processing of the raw data provided by the touch sensor. The lost information can then not be taken into account in the classification. If, on the other hand, fingerprint data in the form of raw data are used directly for key classification, a significantly larger part of the information contained in the raw data, if appropriate all the relevant information, can be used.

The method exemplified here for generating a virtual keyboard on a touch screen can be used on touch sensors of different shape and size. Among other things, since the usefulness of the method in preferred embodiments depends on how differently the different virtual keys are pressed, the method is particularly advantageous for larger touch sensors that can be operated with multiple fingers. For example, tablet computers, such as the Apple iPad or tablet computer based on the Android operating system can be optimized by using the method with regard to the error rate when typing.

If the procedure is implemented in a device, a user can write notes quickly and without errors. Especially tablet computers are usually only used by one person for a long time, so that the virtual keyboard can be trained over a longer period of time, whereby the performance can be further improved.

A virtual keyboard of the type shown here can also be used as a replacement for mechanical keyboards. Particularly in the medical environment or in clean rooms, the use of mechanical keyboards is problematic because they are difficult to clean. One

Touch sensor, possibly even without screen, may be advantageous in the use of the method. By customizing the response of virtual buttons, text input can also be improved for this scenario.

The determination of user-specific touch sensor data by detecting the

Typing behavior of a user from a multiplicity of typing actions of the user on the virtual keyboard, ie the learning process or training procedure for positioning the keyboard model, does not have to be limited to the time before the actual use. The keyboard model can also be further optimized during use. The fingerprint data and the position data derived therefrom may also be recorded during normal use and stored for Creation or optimization of the keyboard model can be used. This may be done, for example, by capturing and storing fingerprint data and positional data while writing a text, and after the text has been generated, that is, when the user saves the text as error free, the detected keystrokes will be the correct letter or virtual key be assigned.

It may be sufficient to use only the position data and fingerprint data for creating the keyboard model and for later use for classification. However, in addition to position data and fingerprint data, further sensor information about each keystroke may also be determined and used in the classification, e.g. Data of an acceleration sensor, e.g. can detect the velocity of a keystroke.

The investigations based on the data sets generated by the test subjects have shown that especially the fingerprint data for individual keys differ greatly for different users. Therefore, these data can also be used to help users based on the

Keyboard input on a touch sensor to distinguish. For this purpose, a classifier is generated which can assign keystrokes to a user based on the fingerprint data. This method can be used, in particular, to make password entries more secure. The computer system then not only expects the correct letter sequence, but also the fingerprint data of the touch sensor corresponding to the expected user.

Claims

claims

A method of operating a device having a user interface with a touch sensor, comprising:

Generating a virtual keyboard with virtual buttons on the touch sensor;

Determining touch sensor data by detecting the typing behavior of a user from a plurality of typing actions of the user on the virtual keyboard;

Generation of a user-specific keyboard model using the

Touch sensor data;

Using the keyboard model to classify keystrokes when using a virtual keyboard,

characterized in that

In the determination of touch sensor data for each key press of a finger position data and fingerprint data are determined, wherein

the position data represents the position of a touch of the surface of the touch sensor at a keystroke and

the fingerprint data represent an image of the local sensor response to touch of the surface of the touch sensor at a keystroke in the form of raw data of the touch sensor or modified raw data derived from the raw data such that the classification relevant information contained in the raw data is substantially preserved; Keyboard model is generated using the position data and the fingerprint data.

2. The method of claim 1, wherein in the determination of modified raw data from the raw data at least one method of the following group is used:

a compression method applied to the raw data, in particular JPEG compression;

a filtering of the raw data, in particular a low-pass filtering;

a method for dimensionality reduction of the raw data, in particular a

Principal component analysis or a discriminant analysis;

a calibration of the raw data;

a normalization of the raw data.

3. The method of claim 1 or 2, wherein in the intended use for each key press of a finger both fingerprint data and position data are determined and classified using the keyboard model.

A method according to any one of the preceding claims, wherein adaptation of the virtual keyboard to the individual user based on the keyboard model is imperceptible to the user such that the appearance of the virtual keyboard remains unchanged.

5. The method according to any one of the preceding claims, wherein in the classification of position data, a user-specific distribution of keystrokes from a plurality of previous keystrokes on the basis of a probability function, in particular based on a two-dimensional normal distribution modeled.

6. The method according to any one of the preceding claims, wherein a Support Vector Machine (SVM) is used to classify the fingerprint data.

7. The method according to any one of the preceding claims, wherein for adjusting the

Responsiveness of the virtual keys is assigned to each virtual key based on the keyboard model, an active area that has a different position and / or shape for one or more keys than the touch sensor visible surface of the associated virtual key, wherein a touch at a touch position within a active area is interpreted and processed as touching the associated virtual key.

A method according to any one of the preceding claims, wherein a user classifier is generated that performs a user classification such that a keystroke based on the fingerprint data is assigned to a particular user, preferably further actions of the device depending on the result of the user -Classification be controlled.

The apparatus of claim 8, wherein a password prompt is generated and subsequent keystrokes are analyzed using the user classifier for whether the expected password is being entered by an authorized user, preferably releasing some or all of the further actions the device only takes place when the correct password is entered by an authorized user.

10. The method according to any one of the preceding claims, wherein an optical

Touch sensor or a capacitive touch sensor is used.

11. Device with:

a user interface comprising a touch sensor (100) having, within a typical size of a contact zone (132) of a finger with the surface (106), a plurality of measurement points for one of finger pressure and / or finger contact and / or approach of the finger to the surface has variable physical size, so that touch sensor data can be generated using the spatially resolved measured values of the measured variable; and a controller (160) coupled to the touch sensor for receiving and processing touch sensor data generated by touching touch areas on the surface of the touch sensor;

wherein the device is configured so that in at least one operating mode on the touch sensor, a virtual keyboard (150) with a plurality of virtual buttons (140) can be generated;

characterized in that the device is configured in the operating mode such that in determining touch sensor data for each press of a finger

Position data and fingerprint data are determined, wherein

the fingerprint data representing an image of the local sensor response to touch of the surface of the touch sensor at a keystroke in the form of raw data of the touch sensor or modified raw data derived from the raw data such that the classification relevant information contained in the raw data is substantially preserved, and that the position data and fingerprint data with a user-specific

Keyboard model are classified, each of the virtual keys using the user-specific keyboard model is assigned to an active area of the touch sensor that has a different position and / or shape for one or more of the virtual buttons than the touch sensor visible surface of the virtual Button,

wherein a touch at a touch position within an active area is interpreted and processed as touching the associated virtual key.

12. The apparatus of claim 11, wherein a spatial density of measurement points in one or more transverse to each other parallel to the surface of the touch sensor extending directions is at least 0.5 / mm, in particular 1 / mm or more.

13. The apparatus of claim 1 1 or 12, wherein the touch sensor (100) is an optical touch sensor or a capacitive touch sensor,

14. The apparatus of claim 1 1, 12 or 13, wherein the device is a tablet computer.

15. Device according to one of claims 11 to 14, wherein the device is configured for carrying out the method according to one of claims 1 to 9.

A computer program product stored or signaled in particular on a computer readable medium, wherein the computer program product, when loaded into the memory of a suitable computer and executed by a computer, causes the computer controlled device to perform a method according to any one of Claims 1 to 10 carried out.