US20130229375A1 - Contact Grouping and Gesture Recognition for Surface Computing - Google Patents

Contact Grouping and Gesture Recognition for Surface Computing Download PDF

Info

Publication number
US20130229375A1
US20130229375A1 US13/866,630 US201313866630A US2013229375A1 US 20130229375 A1 US20130229375 A1 US 20130229375A1 US 201313866630 A US201313866630 A US 201313866630A US 2013229375 A1 US2013229375 A1 US 2013229375A1
Authority
US
United States
Prior art keywords
data structure
contact
hand
contact signals
user
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Abandoned
Application number
US13/866,630
Inventor
Kuo Zhang
Xueqing Sun
Guanggen Zhang
KeCheng Huang
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Microsoft Technology Licensing LLC
Original Assignee
Microsoft Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Microsoft Corp filed Critical Microsoft Corp
Priority to US13/866,630 priority Critical patent/US20130229375A1/en
Publication of US20130229375A1 publication Critical patent/US20130229375A1/en
Assigned to MICROSOFT CORPORATION reassignment MICROSOFT CORPORATION ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: HUANG, KECHENG, SUN, XUEQING, ZHANG, GUANGGEN, ZhANG, Kuo
Assigned to MICROSOFT TECHNOLOGY LICENSING, LLC reassignment MICROSOFT TECHNOLOGY LICENSING, LLC ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: MICROSOFT CORPORATION
Abandoned legal-status Critical Current

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/048Interaction techniques based on graphical user interfaces [GUI]
    • G06F3/0487Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser
    • G06F3/0488Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser using a touch-screen or digitiser, e.g. input of commands through traced gestures
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/048Interaction techniques based on graphical user interfaces [GUI]
    • G06F3/0487Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser
    • G06F3/0488Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser using a touch-screen or digitiser, e.g. input of commands through traced gestures
    • G06F3/04883Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser using a touch-screen or digitiser, e.g. input of commands through traced gestures for inputting data by handwriting, e.g. gesture or text
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F2203/00Indexing scheme relating to G06F3/00 - G06F3/048
    • G06F2203/048Indexing scheme relating to G06F3/048
    • G06F2203/04808Several contacts: gestures triggering a specific function, e.g. scrolling, zooming, right-click, when the user establishes several contacts with the surface simultaneously; e.g. using several fingers or a combination of fingers and pen

Definitions

  • Tools and techniques for contact grouping and gesture recognition for surface computing are provided. These tools receive, from a finger grouping subsystem, data representing velocity parameters associated with contact points, with the contact points representing physical actions taken by users in interacting with software applications presented on a surface computing system. These tools also identify gestures corresponding to the physical actions by analyzing the velocity parameters, and provide indications of the identified gestures to the software applications.
  • FIG. 1 is a block diagram illustrating systems or operating environments suitable for contact grouping and gesture recognition for surface computing.
  • FIG. 2 is a schematic and block representation of user actions as directed to a hardware substrate provided by a surface computing system.
  • FIGS. 3 and 4 are flowcharts of processes for performing virtual hand grouping, as described in the foregoing examples.
  • FIG. 5 is a flow chart of processes for inferring gestures based on the virtual hand grouping shown in FIGS. 3 and 4 .
  • FIG. 6 is a block diagram illustrating reference frames that may be suitable for calculating or projecting gestures, as described above in FIG. 5 .
  • FIG. 1 illustrates systems or operating environments, denoted generally at 100 , related to contact grouping and gesture recognition for surface computing.
  • these systems 100 may include any number of surface computing systems 102 , with FIG. 1 illustrating one surface computing system 102 only for clarity of illustration.
  • Examples of the surface computing systems 102 may include any processor-based computing systems that enable users to interact with software applications 104 via physical actions.
  • Examples of the software applications 104 may include, are not limited to, digital mapping applications, navigation applications, image or photo viewing or editing applications, and the like.
  • the software applications 104 may present visual representations of a given geographic region onto the hardware substrate 106 .
  • the users may issue commands to the software applications 104 by directing particular physical actions to the hardware substrate 106 .
  • FIG. 1 generally denotes these physical actions at 108 . Without limiting possible implementations, and only to facilitate the present description, this discussion uses the term “gestures” to refer to physical actions that correspond to predefined commands recognized by the software applications 104 .
  • Examples of gestures recognized by the software applications 104 may include, but are not limited to, commands such as pan, zoom, rotate, and the like. Users may perform these gestures to using one hand or both hands, and may also use any number of fingers on either or both hands.
  • the software application 104 includes a mapping or navigation application
  • the mapping or navigation application may present an image on the hardware substrate 106 . The user may manipulate this image by, for example, placing one hand on the hardware substrate 106 in moving a thumb and at least one other finger closer together or farther apart.
  • this type of gesture may be interpreted as a zoom-in or a zoom-out command. Users may accomplish similar functions by placing two hands upon the hardware substrate 106 , and moving those two hands closer together or farther apart.
  • Users may issue commands to pan the image in any direction by placing one or both hands on the hardware substrate 106 , and sliding one or both hands in the desired direction along the hardware substrate 106 .
  • rotate commands users may rotate the image by placing one or both hands on the hardware substrate 106 .
  • users may place one hand on the hardware substrate 106 .
  • the users may issue a rotate command by rotating that hand on the hardware substrate 106 .
  • a user may leave his or her thumb relatively stationary on the hardware substrate 106 , and then rotate one or more fingers around the thumb, with the thumb serving as a type of pivot point.
  • users may place both hands on the hardware substrate 106 .
  • the users may issue rotate commands by leaving one hand relatively stationary on the hardware substrate 106 , and then rotating one or more fingers on the other hand relative to the stationary hand.
  • Users may also perform gestures to invoke particular functions provided by the software applications 104 .
  • the software applications 104 may interpret a question mark drawn by a user on the hardware substrate 106 as a command to invoke help utilities provided by the software applications 104 .
  • users may draw question marks or other symbols using one or both hands.
  • the user actions 108 may include, but are not limited to, the foregoing actions associated with the above examples of commands.
  • the surface computing systems 102 may detect and recognize the user actions 108 using any number of different technologies.
  • the hardware substrate 106 may provide a touch-sensitive surface, operative to output electrical signals indicating where and how a given user has physically interacted with the hardware substrate 106 .
  • the hardware substrate 106 or other hardware within the surface computing system 102 , may incorporate one or more cameras that are operative to capture representations of physical interactions with users.
  • These cameras may be combined with image or video processing technology, adapted as appropriate to process these representations of the user actions 108 , to determine where and how a given user has “touched” the hardware substrate 106 .
  • image or video processing technology adapted as appropriate to process these representations of the user actions 108 , to determine where and how a given user has “touched” the hardware substrate 106 .
  • the foregoing examples are provided only for the sake of illustration, and implementations of this description may include other technologies for the detecting and processing the user actions 108 without departing from the scope and spirit of this description.
  • FIG. 1 represents at 110 any hardware and/or software involved with detecting and processing the user actions 108 .
  • implementations of an input detection system 110 may include, but are not limited to, any of the foregoing technologies for detecting occurrences of the user actions 108 .
  • these systems may include one or more instances of processing hardware, with FIG. 1 providing a processor 112 as an example of such processing hardware.
  • the processors 112 may have a particular type or architecture, chosen as appropriate for particular implementations.
  • the processors 112 may couple to one or more bus systems 114 , having type and/or architecture that is chosen for compatibility with the processors 112 .
  • the surface computing systems 102 may also include one or more instances of hardware substrates 106 , coupled to communicate with the bus systems 114 .
  • the input detection systems 110 may also couple to communicate with the bus systems 114 . Accordingly, in some implementations of this description, signals representing the user actions 108 , as directed to the hardware substrate 106 , may travel along the bus systems 114 to be detected and processed by the input detection system 110 .
  • the hardware substrate 106 may be integrated with the input detection system 110 , and may communicate with other components using a single interface to the bus system 114 .
  • the surface computing systems 102 may include one or more instances of a physical computer-readable storage medium or media 116 , which couple to the bus systems 114 .
  • the bus systems 114 may enable the processors 112 to read code and/or data to/from the computer-readable storage media 116 .
  • the media 116 may represent apparatus in the form of storage elements that are implemented using any suitable technology, including but not limited to semiconductors, magnetic materials, optics, or the like.
  • the media 116 may represent memory components, whether characterized as RAM, ROM, flash, solid-state hard drive, or other types of technology.
  • the storage media 116 may include one or more modules of software instructions that, when loaded into the processor 112 and executed, cause the surface computing systems 102 to perform contact grouping and gesture recognition for surface computing. As detailed throughout this description, these modules of instructions may also provide various tools or techniques by which the surface computing systems 102 may participate within the overall systems or operating environments 100 using the components, message and command flows, and data structures discussed in more detail throughout this description.
  • the storage media 116 may include one or more software modules that constitute the software applications 104 .
  • the storage media 116 may also include software modules that provide a virtual contact system 118 .
  • the virtual contact system 118 may operate by receiving contact signals 120 that represent raw contact inputs, as generated an output by the input detection system 110 .
  • These contact signals 120 may represent electrical signals that represent the user actions 108 , as directed to the hardware substrate 106 .
  • the contact signals 120 may indicate contact points at which users are touching or sliding one or more hands, and/or one or more fingers, along the hardware substrate 106 . For example, if a given user is sliding all five fingers of both hands along the hardware substrate 106 , the contact signals 120 may indicate ten ( 10 ) contact points, sliding from one set of locations to another along the hardware substrate 106 at particular velocities. If the given user is sliding one finger from both hands along the hardware substrate 106 , the contact signals 120 may indicate two contact points. If the given user is sliding one finger from one hand along the hardware substrate 106 , the contact signals 120 may indicate one contact point, and so on.
  • the contact signals 120 may be said to be “raw”, in the sense that the signals themselves may not indicate which finger is associated with which contact signal, and may not indicate the hand with which different contact signals 120 are associated. However, the virtual contact system 118 may receive and process these contact signals 120 , as described in further detail below, to identify particular gestures 122 indicated by the user actions 108 .
  • the virtual contact system 118 may include a subsystem 124 for grouping the contact signals 120 into virtualized representations of human hands.
  • the subsystem 124 may group the contact signals 120 into virtual representations of two human hands.
  • implementations of this description may vary from these examples, without departing from the scope and spirit of this description.
  • the subsystem 124 may forward these grouped signals to another subsystem 126 , which is operative to infer the gestures 122 from these grouped signals. More specifically, as described below, the subsystem 126 may analyze characteristics of the virtual human hands to identify the gestures 122 .
  • FIG. 2 illustrates examples, denoted generally at 200 , of user actions 108 as directed to the hardware substrate 106 .
  • FIG. 20 does not include representations of the user actions 108 and the hardware substrate 106 .
  • FIG. 2 illustrates five examples of contact points 201 - 205 , which represent contact points associated with different fingers as the user actions 108 are performed along the hardware substrate 106 .
  • the contact signals 120 as generated by the input detection system 110 may provide representations of the contact points 201 - 205 . More specifically, the contact signals 120 may indicate position and velocity information associated with the different contact points 201 - 205 . Velocity information may include representations of magnitude and direction that are associated with a given movement.
  • the top of FIG. 2 denotes illustrative position information at 201 - 205 , and denotes velocity information by the vectors 206 - 210 . It is noted that the examples shown in FIG. 2 are not drawn to scale in terms of size or direction, and are provided only to facilitate the present description, but not to limit possible implementations of this description.
  • FIG. 2 illustrates five contact points 201 - 205 only as examples. However, implications of this description may operate with any number of contact points.
  • the velocity vectors 206 - 210 represent five respective velocity parameters associated with the five contact points 201 - 205 .
  • the subsystem 124 for virtual hand grouping may receive signals representing these velocity vectors 206 - 210 for the five contact points 201 - 205 , and may generate as output virtualized finger assignments, which may assign some the contact points 201 - 205 to a first virtual hand representation, in which may assign other contact points 201 - 205 to a second virtual hand representation.
  • FIG. 2 represents finger assignments for a first virtual hand at 212 , and represents finger assignments for a second virtual hand at 214 .
  • the contact points or fingers 201 and 203 are assigned to the virtual hand 212
  • the contact points or fingers 202 , 204 , and 205 are assigned to the virtual hand 214 .
  • the subsystem 124 for virtual hand grouping may assign positions and velocities for the virtual hands 212 and 214 , as distinguished from individual positions and velocities of the various contact points 201 - 205 that are assigned to those virtual hands 212 and 214 .
  • FIG. 2 represents velocity assignments for the virtual hands at 216 and 218 .
  • Location assignments for the virtual hands are shown schematically at the top of FIG. 2 at 220 and 222 .
  • assignments represent schematically in FIG. 2 may be implemented within suitable data structures maintained by the subsystem 124 for virtual hand grouping. These data structures may be created and manipulated according to the process flows now described with FIG. 3 .
  • FIG. 3 illustrates process flows, denoted generally at 300 , for performing virtual hand grouping, as described in the foregoing examples.
  • FIG. 3 is described in connection with the subsystem 124 for virtual hand grouping.
  • implementations of this description may perform at least part of the process flows 300 with other components, without departing from the scope and spirit of the present description.
  • decision block 302 represents awaiting action by a given user as directed to a surface computing system (e.g., 102 in FIG. 1 ).
  • FIG. 1 also provides examples of user actions at 108 .
  • the process flows 300 may take No branch 304 to loop at block 302 , until user actions 108 are detected. Once the user actions 108 are detected, the process flows 300 may take Yes branch 306 to block 308 .
  • the user actions 108 may be represented by any number of contact points (e.g., 201 - 205 in FIG. 2 ), which may in turn represent actions taken by different ones of the user's fingers.
  • Block 308 represents selecting representations of, for example, two fingers from among the contact points 201 - 205 .
  • the selection operations performed in block 308 may include calculating differentials in velocities between the various fingers as represented by the contact points 201 - 205 , as represented at 310 .
  • the fingers represented by the contact points 201 , 202 , 203 , 204 and 205 may be traveling at the respective velocities V F1 , V F2 , V F3 , V F4 , and V F5 .
  • block 310 may represent calculating velocity differentials for all pairs of contact points.
  • the operations represented in block 308 may also include identifying, for example, two fingers from those represented by the contact points 201 - 205 . More specifically, block 312 represents identifying two fingers having the greatest velocity differential, as calculated above in block 310 .
  • Block 314 represents associating the finger representations selected in block 308 with virtual hand representations.
  • block 314 may include defining two representations of virtual hands (e.g., 220 and 222 in FIG. 2 ), and associating or assigning one of the selected finger representations to one of the virtual hand representations, and assigning the other selected finger representation to the other virtual hand.
  • block 308 may include selecting those two contact points 201 and 205 .
  • block 314 may include associating the finger represented by the contact point 201 with the hand representation 220 , and may include associating the finger represented by the contact point 205 with the hand representation 222 .
  • block 316 represents assigning position and velocity parameters to the virtual hand representations. More specifically, block 316 may include assigning the individual velocities of the contact points selected in block 308 to the virtual hand representations. Referring briefly back to the top of FIG. 2 , block 316 may include assigning the velocity of the selected contact point 201 (i.e., V F1 ) as the velocity 216 of the hand representation 220 (i.e., V H1 ). Similarly, block 316 may include assigning the velocity of the selected contact point 205 (i.e., V F5 ) as the velocity 218 of the hand representation 222 (i.e., V H1 ).
  • block 318 represents assigning the other or remainder of the contact points to the virtual hand representations. More specifically, the operations represented in block 318 may include selecting one of the unassigned contact points, as represented in block 320 . For that selected unassigned contact point, block 322 represents calculating velocity differentials between that selected unassigned contact point and the different hand representations. Referring briefly back to the top of FIG. 2 , assuming that block 320 selected the contact point 203 , block 322 may include calculating differentials between the velocity (i.e., V F3 ) of the selected contact point 203 and the assigned velocities (i.e., V H1 and V H2 ) of the different hand representations. Put differently, block 322 may include calculating V F3 ⁇ V H1 and V F3 ⁇ V H2 .
  • block 324 represents associating the selected contact point (e.g., 203 in the previous example) with the hand representation with which it has the smaller velocity differential. For example, if the selected contact point (e.g., 203 ) has a velocity value (i.e., V F3 ) that is closer to the velocity value of the first hand representation (i.e., V H1 ), then the selected contact point will be associated with the first hand representation.
  • V F3 the velocity value of the first hand representation
  • Block 318 may be repeated for all remaining unassigned contact points (e.g., 201 - 205 in FIG. 2 ), assigning these contact points to the virtual hand representation with which they have the minimum velocity differential.
  • FIG. 2 provides at 212 examples of finger assignments for a first virtual hand representation, and provides at 214 examples of finger assignments for a second virtual hand representation.
  • FIG. 4 illustrates continuations, denoted generally at 400 , of the process flows 300 described above with FIG. 3 .
  • block 402 represents re-calculating the position and velocity values associated with the virtual hand representations.
  • FIG. 2 provides at 216 examples of velocity (V H1 ) and/or position assignments for a first virtual hand representation, and provides at 218 examples of velocity (V H2 ) and/or position assignments for a second virtual hand representation.
  • block 402 may include calculating the position and/or velocity of the virtual hand representations by averaging the velocity and position values of the contact points that are assigned respectively to the virtual hand representations. Accordingly, the position and/or velocity values recalculated for the virtual hand representations in block 402 may incorporate the contact point assignments made in block 318 above.
  • Block 406 represents regrouping the contact points between the different virtual hand representations, based on the hand positions and/or velocities as recalculated in block 402 . More specifically, block 406 may include reassigning contact points to the virtual hand representations, as represented in block 408 . Block 408 may include calculating, for each contact point, the velocity differential between that contact point and the two virtual hand representations. Block 408 may also include assigning the different contact points to the virtual hand representation with which they have the smallest velocity differential. Similar processing is represented above in blocks 322 and 324 .
  • Block 410 represents discarding, from each virtual hand representation, the contact point that has the greatest variance in position and/or velocity from the position and/or velocity assigned to that virtual hand representation. Block 410 may be repeated for each virtual hand representation defined at a given time. Block 410 may include comparing a given contact point within a virtual hand representation to the other contact points in that virtual hand representation, as represented at 412 . In this manner, block 410 may exclude from the virtual hand representations those contact points that are “outliers” as compared to the other contact points. This exclusion may help eliminate jitter or other aberrations that may be detected within the user actions 108 .
  • Block 414 represents evaluating whether a termination condition applicable to the process flows 300 and 400 has been met. Implementations of this description may incorporate any number of different suitable termination conditions. However, FIG. 4 illustrates at block 416 a non-limiting example of checking whether successive iterations of the process flows 300 and 400 change the velocities that are computed for the virtual hand representations. For example, if two successive iterations of the process flows 300 and 400 do not change the velocities computed for the virtual hand representations, in block 414 may determine that the process flows 300 and 400 have reached convergence, and that no further iterations are justified.
  • Block 126 is carried forward from FIG. 1 to represent inferring gestures from the user actions 108 , based on the virtual hand representations established above ID process flows 300 and 400 . More specifically, the virtual hand representations may be assigned a location parameter and a velocity parameter, based upon the contact points associated with those virtual hand representations. Processes for inferring gestures are described in further detail below with FIG. 5 .
  • block 314 may include initializing an updated or new hand structure for the next iteration of the processes shown in FIG. 3 .
  • block 314 may include initializing the updated or new hand structure with position and velocity values, so that the new hand structures inherit position and velocity values from the finger representations assigned thereto.
  • FIG. 5 illustrates process flows, denoted generally at 500 , for inferring gestures, as introduced previously in FIG. 4 at 418 .
  • gesture inference or transformation may begin with block 502 , which represents recording or storing previous positions of one or both of the virtual hands.
  • Block 504 represents calculating gestures as indicated by the velocity and/or position values assigned to the virtual hands 220 and 222 .
  • block 504 may also include comparing the previous positions of the virtual hands (as stored in block 502 ) with the present positions calculated for the virtual hand or hands.
  • FIG. 5 illustrates a two-hand scenario, implementations of this description may also operate with one virtual hand or any number of virtual hands.
  • Block 506 may also include projecting velocity parameters associated with the virtual hands onto a suitable reference frame, as described below in further detail with FIG. 6 .
  • Block 506 may therefore include identifying one or more gestures associated with a given user action (e.g., 108 in FIG. 1 ).
  • FIG. 5 illustrates non-limiting examples that include pan gestures 508 , zoom gestures 510 , rotate gestures 512 , and any number of other command gestures 514 .
  • Block 516 represents applying thresholding to the gestures identified in block 504 .
  • This thresholding may include tuning or filtering out gestures having minor magnitudes, because minor gestures may be inadvertent.
  • Establishing the threshold applied in block 516 may involve trading off latency of response versus overall user experience. For example, if the threshold is set too small, users may become frustrated when minor movements trigger unintended gestures. However, if the threshold is set too large, users may become frustrated when larger movements failed to trigger intended gestures. However, the threshold applied in block 516 may readily be established through reasonable experimentation.
  • Block 518 represents recalculating the gestures projected or calculated in block 504 , to account for the consequences of any thresholding applied in block 516 . For example, after the thresholding operations in block 516 , what may previously have appeared to be a gesture (e.g., zoom, pan, rotate, etc.) may be discarded as a relatively minor movement, and not output as a valid gesture.
  • a gesture e.g., zoom, pan, rotate, etc.
  • Block 520 represents outputting any gestures projected and calculated by the process flows 500 .
  • FIG. 5 carries forward from FIG. 1 examples of the gestures at 122 .
  • FIG. 6 illustrates example reference frames, denoted generally at 600 , that may be suitable for calculating or projecting gestures, as described above in FIG. 5 .
  • the reference frames 600 may be understood as elaborating further on block 504 in FIG. 5 .
  • FIG. 6 also carries forward examples of virtual hand representations 220 and 222 .
  • the reference frames at 600 may include an X-axis 602 that connects the approximate center points of the calculated positions of the virtual hand representations 220 and 222 .
  • An example Y-axis 604 is perpendicular to the X-axis 602 .
  • Block 504 may include projecting these velocity parameters 216 and 218 onto the X-axis 602 and the Y-axis 604 , to determine the X and Y components or weights of these velocity parameters along the axes 602 and 604 .
  • the weights or components of the velocity parameters 216 and 218 may indicate movement of the virtual and representations 220 and 222 , relative to one another. In turn, this relative movement may indicate which gesture was intended by the user actions 108 represented by the virtual hands 220 and 222 .
  • FIG. 6 denotes the X component of this velocity parameter at 606 , and denotes the Y component of this velocity parameter at 608 .
  • FIG. 6 denotes the X component of this velocity parameter at 610 , and denotes the Y component of this velocity parameter at 612 .
  • the gesture calculation may infer that the user intends a pan gesture in that direction.
  • the gesture calculation may infer that the user intends a zoom-out or zoom-in gesture, chosen as appropriate in different implementations.
  • the gesture calculation may infer that the user intends a rotation gesture.
  • the differential between the X components 206 and 210 may indicate a magnitude of any pan or zoom gestures
  • the differential between the Y components 608 and 612 may indicate a magnitude of any rotate gestures.
  • other examples of gestures are possible, with the foregoing examples provided only to illustrate possible, but not all, implementation scenarios.

Abstract

Tools and techniques for contact grouping and gesture recognition for surface computing are provided. These tools receive, from a finger grouping subsystem, data representing velocity parameters associated with contact points, with the contact points representing physical actions taken by users in interacting with software applications presented on a surface computing system. These tools also identify gestures corresponding to the physical actions by analyzing the velocity parameters, and provide indications of the identified gestures to the software applications.

Description

    CROSS REFERENCE TO RELATED APPLICATIONS
  • This application is continuation of co-pending U.S. application Ser. No. 12/435,419 filed May 5, 2009 entitled “Contact Grouping and Gesture Recognition for Surface Computing,” which is expressly incorporated herein by reference.
  • BACKGROUND
  • Surface computing systems enable users to interact with software applications by physical actions, rather than by issuing commands through a point-and-click menu hierarchy, or by speaking or typing commands. However, recognizing physical actions and correlating them correctly to commands intended by the users remains an ongoing challenge. Another challenge is reducing the impact of jitter inadvertently introduced into the physical actions by users.
  • SUMMARY
  • Tools and techniques for contact grouping and gesture recognition for surface computing are provided. These tools receive, from a finger grouping subsystem, data representing velocity parameters associated with contact points, with the contact points representing physical actions taken by users in interacting with software applications presented on a surface computing system. These tools also identify gestures corresponding to the physical actions by analyzing the velocity parameters, and provide indications of the identified gestures to the software applications.
  • It should be appreciated that the above-described subject matter may be implemented as a computer-controlled apparatus, a computer process, a computing system, or as an article of manufacture such as a computer-readable medium. These and various other features will be apparent from a reading of the following Detailed Description and a review of the associated drawings.
  • This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended that this Summary be used to limit the scope of the claimed subject matter. Furthermore, the claimed subject matter is not limited to implementations that solve any or all disadvantages noted in any part of this disclosure.
  • BRIEF DESCRIPTION OF THE DRAWINGS
  • FIG. 1 is a block diagram illustrating systems or operating environments suitable for contact grouping and gesture recognition for surface computing.
  • FIG. 2 is a schematic and block representation of user actions as directed to a hardware substrate provided by a surface computing system.
  • FIGS. 3 and 4 are flowcharts of processes for performing virtual hand grouping, as described in the foregoing examples.
  • FIG. 5 is a flow chart of processes for inferring gestures based on the virtual hand grouping shown in FIGS. 3 and 4.
  • FIG. 6 is a block diagram illustrating reference frames that may be suitable for calculating or projecting gestures, as described above in FIG. 5.
  • DETAILED DESCRIPTION
  • The following detailed description provides tools and techniques for contact grouping and gesture recognition for surface computing. While the subject matter described herein presents a general context of program modules that execute in conjunction with the execution of an operating system and application programs on a computer system, those skilled in the art will recognize that other implementations may be performed in combination with other types of program modules. Generally, program modules include routines, programs, components, data structures, and other types of structures that perform particular tasks or implement particular abstract data types. Moreover, those skilled in the art will appreciate that the subject matter described herein may be practiced with other computer system configurations, including hand-held devices, multiprocessor systems, microprocessor-based or programmable consumer electronics, minicomputers, mainframe computers, and the like.
  • The following detailed description refers to the accompanying drawings that form a part hereof, and that show, by way of illustration, specific example implementations. Referring now to the drawings, in which like numerals represent like elements through the several figures, this description provides various tools and techniques related to contact grouping and gesture recognition for surface computing.
  • FIG. 1 illustrates systems or operating environments, denoted generally at 100, related to contact grouping and gesture recognition for surface computing. Turning to FIG. 1 in more detail, these systems 100 may include any number of surface computing systems 102, with FIG. 1 illustrating one surface computing system 102 only for clarity of illustration.
  • Examples of the surface computing systems 102 may include any processor-based computing systems that enable users to interact with software applications 104 via physical actions. Examples of the software applications 104 may include, are not limited to, digital mapping applications, navigation applications, image or photo viewing or editing applications, and the like. In cases where the software applications 104 include navigation or mapping applications, the software applications 104 may present visual representations of a given geographic region onto the hardware substrate 106. In turn, the users may issue commands to the software applications 104 by directing particular physical actions to the hardware substrate 106. FIG. 1 generally denotes these physical actions at 108. Without limiting possible implementations, and only to facilitate the present description, this discussion uses the term “gestures” to refer to physical actions that correspond to predefined commands recognized by the software applications 104.
  • Examples of gestures recognized by the software applications 104 may include, but are not limited to, commands such as pan, zoom, rotate, and the like. Users may perform these gestures to using one hand or both hands, and may also use any number of fingers on either or both hands. For example, wherein the software application 104 includes a mapping or navigation application, the mapping or navigation application may present an image on the hardware substrate 106. The user may manipulate this image by, for example, placing one hand on the hardware substrate 106 in moving a thumb and at least one other finger closer together or farther apart. Depending on particular implementations of the software applications 104, this type of gesture may be interpreted as a zoom-in or a zoom-out command. Users may accomplish similar functions by placing two hands upon the hardware substrate 106, and moving those two hands closer together or farther apart.
  • Users may issue commands to pan the image in any direction by placing one or both hands on the hardware substrate 106, and sliding one or both hands in the desired direction along the hardware substrate 106. Regarding rotate commands, users may rotate the image by placing one or both hands on the hardware substrate 106. In some cases, users may place one hand on the hardware substrate 106. In such cases, the users may issue a rotate command by rotating that hand on the hardware substrate 106. For example, a user may leave his or her thumb relatively stationary on the hardware substrate 106, and then rotate one or more fingers around the thumb, with the thumb serving as a type of pivot point.
  • In other cases, users may place both hands on the hardware substrate 106. In such cases, the users may issue rotate commands by leaving one hand relatively stationary on the hardware substrate 106, and then rotating one or more fingers on the other hand relative to the stationary hand.
  • Users may also perform gestures to invoke particular functions provided by the software applications 104. For example, the software applications 104 may interpret a question mark drawn by a user on the hardware substrate 106 as a command to invoke help utilities provided by the software applications 104. As with the example commands described above, users may draw question marks or other symbols using one or both hands.
  • In light of the previous description, the user actions 108 may include, but are not limited to, the foregoing actions associated with the above examples of commands. The surface computing systems 102 may detect and recognize the user actions 108 using any number of different technologies. For example, the hardware substrate 106 may provide a touch-sensitive surface, operative to output electrical signals indicating where and how a given user has physically interacted with the hardware substrate 106. In other examples, the hardware substrate 106, or other hardware within the surface computing system 102, may incorporate one or more cameras that are operative to capture representations of physical interactions with users. These cameras may be combined with image or video processing technology, adapted as appropriate to process these representations of the user actions 108, to determine where and how a given user has “touched” the hardware substrate 106. However, the foregoing examples are provided only for the sake of illustration, and implementations of this description may include other technologies for the detecting and processing the user actions 108 without departing from the scope and spirit of this description.
  • In general, FIG. 1 represents at 110 any hardware and/or software involved with detecting and processing the user actions 108. it is understood that implementations of an input detection system 110 may include, but are not limited to, any of the foregoing technologies for detecting occurrences of the user actions 108.
  • Turning to the surface computing systems 102 in more detail, these systems may include one or more instances of processing hardware, with FIG. 1 providing a processor 112 as an example of such processing hardware. The processors 112 may have a particular type or architecture, chosen as appropriate for particular implementations. In addition, the processors 112 may couple to one or more bus systems 114, having type and/or architecture that is chosen for compatibility with the processors 112.
  • The surface computing systems 102 may also include one or more instances of hardware substrates 106, coupled to communicate with the bus systems 114. In addition, the input detection systems 110 may also couple to communicate with the bus systems 114. Accordingly, in some implementations of this description, signals representing the user actions 108, as directed to the hardware substrate 106, may travel along the bus systems 114 to be detected and processed by the input detection system 110. In other implementations, the hardware substrate 106 may be integrated with the input detection system 110, and may communicate with other components using a single interface to the bus system 114.
  • The surface computing systems 102 may include one or more instances of a physical computer-readable storage medium or media 116, which couple to the bus systems 114. The bus systems 114 may enable the processors 112 to read code and/or data to/from the computer-readable storage media 116. The media 116 may represent apparatus in the form of storage elements that are implemented using any suitable technology, including but not limited to semiconductors, magnetic materials, optics, or the like. The media 116 may represent memory components, whether characterized as RAM, ROM, flash, solid-state hard drive, or other types of technology.
  • The storage media 116 may include one or more modules of software instructions that, when loaded into the processor 112 and executed, cause the surface computing systems 102 to perform contact grouping and gesture recognition for surface computing. As detailed throughout this description, these modules of instructions may also provide various tools or techniques by which the surface computing systems 102 may participate within the overall systems or operating environments 100 using the components, message and command flows, and data structures discussed in more detail throughout this description. For example, the storage media 116 may include one or more software modules that constitute the software applications 104.
  • The storage media 116 may also include software modules that provide a virtual contact system 118. In general, the virtual contact system 118 may operate by receiving contact signals 120 that represent raw contact inputs, as generated an output by the input detection system 110. These contact signals 120 may represent electrical signals that represent the user actions 108, as directed to the hardware substrate 106. More specifically, the contact signals 120 may indicate contact points at which users are touching or sliding one or more hands, and/or one or more fingers, along the hardware substrate 106. For example, if a given user is sliding all five fingers of both hands along the hardware substrate 106, the contact signals 120 may indicate ten (10) contact points, sliding from one set of locations to another along the hardware substrate 106 at particular velocities. If the given user is sliding one finger from both hands along the hardware substrate 106, the contact signals 120 may indicate two contact points. If the given user is sliding one finger from one hand along the hardware substrate 106, the contact signals 120 may indicate one contact point, and so on.
  • The contact signals 120 may be said to be “raw”, in the sense that the signals themselves may not indicate which finger is associated with which contact signal, and may not indicate the hand with which different contact signals 120 are associated. However, the virtual contact system 118 may receive and process these contact signals 120, as described in further detail below, to identify particular gestures 122 indicated by the user actions 108.
  • Turning to the virtual contact system 118 in more detail, the virtual contact system 118 may include a subsystem 124 for grouping the contact signals 120 into virtualized representations of human hands. In the examples described herein, the subsystem 124 may group the contact signals 120 into virtual representations of two human hands. However, implementations of this description may vary from these examples, without departing from the scope and spirit of this description.
  • In turn, the subsystem 124 may forward these grouped signals to another subsystem 126, which is operative to infer the gestures 122 from these grouped signals. More specifically, as described below, the subsystem 126 may analyze characteristics of the virtual human hands to identify the gestures 122.
  • Having described the systems and overall operating environments 100, the discussion now turns to a more detailed description of the subsystem 124 for virtual hand grouping. This description is now provided with FIG. 2, and subsequent drawings.
  • FIG. 2 illustrates examples, denoted generally at 200, of user actions 108 as directed to the hardware substrate 106. For clarity of illustration, FIG. 20 does not include representations of the user actions 108 and the hardware substrate 106. However, FIG. 2 illustrates five examples of contact points 201-205, which represent contact points associated with different fingers as the user actions 108 are performed along the hardware substrate 106.
  • Initially, the contact signals 120 as generated by the input detection system 110 may provide representations of the contact points 201-205. More specifically, the contact signals 120 may indicate position and velocity information associated with the different contact points 201-205. Velocity information may include representations of magnitude and direction that are associated with a given movement. The top of FIG. 2 denotes illustrative position information at 201-205, and denotes velocity information by the vectors 206-210. It is noted that the examples shown in FIG. 2 are not drawn to scale in terms of size or direction, and are provided only to facilitate the present description, but not to limit possible implementations of this description. In addition, FIG. 2 illustrates five contact points 201-205 only as examples. However, implications of this description may operate with any number of contact points.
  • Turning to the bottom portion of FIG. 2, the velocity vectors 206-210 represent five respective velocity parameters associated with the five contact points 201-205. In overview, the subsystem 124 for virtual hand grouping may receive signals representing these velocity vectors 206-210 for the five contact points 201-205, and may generate as output virtualized finger assignments, which may assign some the contact points 201-205 to a first virtual hand representation, in which may assign other contact points 201-205 to a second virtual hand representation. FIG. 2 represents finger assignments for a first virtual hand at 212, and represents finger assignments for a second virtual hand at 214. In the non-limiting examples shown at the top of FIG. 2, the contact points or fingers 201 and 203 are assigned to the virtual hand 212, and the contact points or fingers 202, 204, and 205 are assigned to the virtual hand 214.
  • In addition, the subsystem 124 for virtual hand grouping may assign positions and velocities for the virtual hands 212 and 214, as distinguished from individual positions and velocities of the various contact points 201-205 that are assigned to those virtual hands 212 and 214. FIG. 2 represents velocity assignments for the virtual hands at 216 and 218. Location assignments for the virtual hands are shown schematically at the top of FIG. 2 at 220 and 222.
  • In providing the foregoing description of FIG. 2, it is noted that the assignments represent schematically in FIG. 2 may be implemented within suitable data structures maintained by the subsystem 124 for virtual hand grouping. These data structures may be created and manipulated according to the process flows now described with FIG. 3.
  • FIG. 3 illustrates process flows, denoted generally at 300, for performing virtual hand grouping, as described in the foregoing examples. For ease of reference and description, but for limiting possible implementations, FIG. 3 is described in connection with the subsystem 124 for virtual hand grouping. However, implementations of this description may perform at least part of the process flows 300 with other components, without departing from the scope and spirit of the present description.
  • Turning to the process flows 300 in more detail, decision block 302 represents awaiting action by a given user as directed to a surface computing system (e.g., 102 in FIG. 1). FIG. 1 also provides examples of user actions at 108. The process flows 300 may take No branch 304 to loop at block 302, until user actions 108 are detected. Once the user actions 108 are detected, the process flows 300 may take Yes branch 306 to block 308. As described previously, the user actions 108 may be represented by any number of contact points (e.g., 201-205 in FIG. 2), which may in turn represent actions taken by different ones of the user's fingers.
  • Block 308 represents selecting representations of, for example, two fingers from among the contact points 201-205. The selection operations performed in block 308 may include calculating differentials in velocities between the various fingers as represented by the contact points 201-205, as represented at 310. For example, the fingers represented by the contact points 201, 202, 203, 204 and 205 may be traveling at the respective velocities VF1, VF2, VF3, VF4, and VF5. The differential between the velocities VF1 and VF2 may be expressed as VF2−VF1, the differential between the velocities VF1 and VF3 may be expressed as VF3−VF1, and so on. Thus, block 310 may represent calculating velocity differentials for all pairs of contact points.
  • The operations represented in block 308 may also include identifying, for example, two fingers from those represented by the contact points 201-205. More specifically, block 312 represents identifying two fingers having the greatest velocity differential, as calculated above in block 310.
  • Block 314 represents associating the finger representations selected in block 308 with virtual hand representations. For example, block 314 may include defining two representations of virtual hands (e.g., 220 and 222 in FIG. 2), and associating or assigning one of the selected finger representations to one of the virtual hand representations, and assigning the other selected finger representation to the other virtual hand. For example, referring briefly back to the top of FIG. 2, assume that the fingers represented by the contact points 201 and 205 have the maximum velocity differential of any of the contact points 201-205. Thus, block 308 may include selecting those two contact points 201 and 205. In turn, block 314 may include associating the finger represented by the contact point 201 with the hand representation 220, and may include associating the finger represented by the contact point 205 with the hand representation 222.
  • Returning to FIG. 3, block 316 represents assigning position and velocity parameters to the virtual hand representations. More specifically, block 316 may include assigning the individual velocities of the contact points selected in block 308 to the virtual hand representations. Referring briefly back to the top of FIG. 2, block 316 may include assigning the velocity of the selected contact point 201 (i.e., VF1) as the velocity 216 of the hand representation 220 (i.e., VH1). Similarly, block 316 may include assigning the velocity of the selected contact point 205 (i.e., VF5) as the velocity 218 of the hand representation 222 (i.e., VH1).
  • Returning to FIG. 3, block 318 represents assigning the other or remainder of the contact points to the virtual hand representations. More specifically, the operations represented in block 318 may include selecting one of the unassigned contact points, as represented in block 320. For that selected unassigned contact point, block 322 represents calculating velocity differentials between that selected unassigned contact point and the different hand representations. Referring briefly back to the top of FIG. 2, assuming that block 320 selected the contact point 203, block 322 may include calculating differentials between the velocity (i.e., VF3) of the selected contact point 203 and the assigned velocities (i.e., VH1 and VH2) of the different hand representations. Put differently, block 322 may include calculating VF3−VH1 and VF3−VH2.
  • Returning to FIG. 3, block 324 represents associating the selected contact point (e.g., 203 in the previous example) with the hand representation with which it has the smaller velocity differential. For example, if the selected contact point (e.g., 203) has a velocity value (i.e., VF3) that is closer to the velocity value of the first hand representation (i.e., VH1), then the selected contact point will be associated with the first hand representation.
  • Block 318 may be repeated for all remaining unassigned contact points (e.g., 201-205 in FIG. 2), assigning these contact points to the virtual hand representation with which they have the minimum velocity differential. As described above, FIG. 2 provides at 212 examples of finger assignments for a first virtual hand representation, and provides at 214 examples of finger assignments for a second virtual hand representation.
  • For convenience of illustration and description, but not to limit possible implementations of the process flows 300, the description of these process flows 300 continues to FIG. 4 via off-page reference 326.
  • FIG. 4 illustrates continuations, denoted generally at 400, of the process flows 300 described above with FIG. 3. Beginning at off-page reference 326, block 402 represents re-calculating the position and velocity values associated with the virtual hand representations. As described above, FIG. 2 provides at 216 examples of velocity (VH1) and/or position assignments for a first virtual hand representation, and provides at 218 examples of velocity (VH2) and/or position assignments for a second virtual hand representation.
  • As represented by block 404, block 402 may include calculating the position and/or velocity of the virtual hand representations by averaging the velocity and position values of the contact points that are assigned respectively to the virtual hand representations. Accordingly, the position and/or velocity values recalculated for the virtual hand representations in block 402 may incorporate the contact point assignments made in block 318 above.
  • Block 406 represents regrouping the contact points between the different virtual hand representations, based on the hand positions and/or velocities as recalculated in block 402. More specifically, block 406 may include reassigning contact points to the virtual hand representations, as represented in block 408. Block 408 may include calculating, for each contact point, the velocity differential between that contact point and the two virtual hand representations. Block 408 may also include assigning the different contact points to the virtual hand representation with which they have the smallest velocity differential. Similar processing is represented above in blocks 322 and 324.
  • Block 410 represents discarding, from each virtual hand representation, the contact point that has the greatest variance in position and/or velocity from the position and/or velocity assigned to that virtual hand representation. Block 410 may be repeated for each virtual hand representation defined at a given time. Block 410 may include comparing a given contact point within a virtual hand representation to the other contact points in that virtual hand representation, as represented at 412. In this manner, block 410 may exclude from the virtual hand representations those contact points that are “outliers” as compared to the other contact points. This exclusion may help eliminate jitter or other aberrations that may be detected within the user actions 108.
  • Block 414 represents evaluating whether a termination condition applicable to the process flows 300 and 400 has been met. Implementations of this description may incorporate any number of different suitable termination conditions. However, FIG. 4 illustrates at block 416 a non-limiting example of checking whether successive iterations of the process flows 300 and 400 change the velocities that are computed for the virtual hand representations. For example, if two successive iterations of the process flows 300 and 400 do not change the velocities computed for the virtual hand representations, in block 414 may determine that the process flows 300 and 400 have reached convergence, and that no further iterations are justified.
  • From decision block 414, if the termination condition has been met at a given iteration of the process flows 400, the process flows 400 may take Yes branch 418 to block 126. Block 126 is carried forward from FIG. 1 to represent inferring gestures from the user actions 108, based on the virtual hand representations established above ID process flows 300 and 400. More specifically, the virtual hand representations may be assigned a location parameter and a velocity parameter, based upon the contact points associated with those virtual hand representations. Processes for inferring gestures are described in further detail below with FIG. 5.
  • Returning to decision block 414, if the termination condition has not yet been met at a given iteration, the process flows 400 may take No branch 420 to return to the processes 300 in FIG. 3 via off-page reference 422. Referring briefly back to FIG. 3, processing may resume at block 314, where indicated by the off-page reference 422. When processing returns to block 314 via the off-page reference 422, block 314 may include initializing an updated or new hand structure for the next iteration of the processes shown in FIG. 3. For example, block 314 may include initializing the updated or new hand structure with position and velocity values, so that the new hand structures inherit position and velocity values from the finger representations assigned thereto.
  • FIG. 5 illustrates process flows, denoted generally at 500, for inferring gestures, as introduced previously in FIG. 4 at 418. Turning to the process flows 500 in more detail, gesture inference or transformation may begin with block 502, which represents recording or storing previous positions of one or both of the virtual hands.
  • Block 504 represents calculating gestures as indicated by the velocity and/or position values assigned to the virtual hands 220 and 222. In turn, block 504 may also include comparing the previous positions of the virtual hands (as stored in block 502) with the present positions calculated for the virtual hand or hands. Although FIG. 5 illustrates a two-hand scenario, implementations of this description may also operate with one virtual hand or any number of virtual hands. Block 506 may also include projecting velocity parameters associated with the virtual hands onto a suitable reference frame, as described below in further detail with FIG. 6.
  • Projecting the velocity parameters onto a reference frame may enable the gesture inference process 418 to analyze relative movements of the hands, as represented by block 506. Block 506 may therefore include identifying one or more gestures associated with a given user action (e.g., 108 in FIG. 1). FIG. 5 illustrates non-limiting examples that include pan gestures 508, zoom gestures 510, rotate gestures 512, and any number of other command gestures 514.
  • Block 516 represents applying thresholding to the gestures identified in block 504. This thresholding may include tuning or filtering out gestures having minor magnitudes, because minor gestures may be inadvertent. Establishing the threshold applied in block 516 may involve trading off latency of response versus overall user experience. For example, if the threshold is set too small, users may become frustrated when minor movements trigger unintended gestures. However, if the threshold is set too large, users may become frustrated when larger movements failed to trigger intended gestures. However, the threshold applied in block 516 may readily be established through reasonable experimentation.
  • Block 518 represents recalculating the gestures projected or calculated in block 504, to account for the consequences of any thresholding applied in block 516. For example, after the thresholding operations in block 516, what may previously have appeared to be a gesture (e.g., zoom, pan, rotate, etc.) may be discarded as a relatively minor movement, and not output as a valid gesture.
  • Block 520 represents outputting any gestures projected and calculated by the process flows 500. Without limiting possible implementations, FIG. 5 carries forward from FIG. 1 examples of the gestures at 122.
  • FIG. 6 illustrates example reference frames, denoted generally at 600, that may be suitable for calculating or projecting gestures, as described above in FIG. 5. Without limiting possible implementations, the reference frames 600 may be understood as elaborating further on block 504 in FIG. 5. FIG. 6 also carries forward examples of virtual hand representations 220 and 222.
  • Turning to the reference frames 600 in more detail, the reference frames at 600 may include an X-axis 602 that connects the approximate center points of the calculated positions of the virtual hand representations 220 and 222. An example Y-axis 604 is perpendicular to the X-axis 602.
  • Recalling previous discussion, the virtual hand representations 220 and 222 are associated with calculated velocity parameters, with examples carried forward respectively at 216 and 218 (i.e., denoted as VH1 and VH2 in FIG. 6). Block 504 may include projecting these velocity parameters 216 and 218 onto the X-axis 602 and the Y-axis 604, to determine the X and Y components or weights of these velocity parameters along the axes 602 and 604. The weights or components of the velocity parameters 216 and 218 may indicate movement of the virtual and representations 220 and 222, relative to one another. In turn, this relative movement may indicate which gesture was intended by the user actions 108 represented by the virtual hands 220 and 222.
  • Turning to the velocity parameter 216 (VH1), FIG. 6 denotes the X component of this velocity parameter at 606, and denotes the Y component of this velocity parameter at 608. Turning to the velocity parameter 218 (VH2), FIG. 6 denotes the X component of this velocity parameter at 610, and denotes the Y component of this velocity parameter at 612.
  • In cases where the X components 606 and 608 indicate that the virtual hands 220 and 222 are both moving in approximately the same direction relative to the x-axis 602, the gesture calculation may infer that the user intends a pan gesture in that direction. In cases where the X components 606 and 608 indicate that the virtual hands 220 and 222 are moving approximately away from one another or approximately toward one another, the gesture calculation may infer that the user intends a zoom-out or zoom-in gesture, chosen as appropriate in different implementations. In cases where the Y components 608 and 612 indicate that the virtual hands 220 and 222 are rotating relative to one another, the gesture calculation may infer that the user intends a rotation gesture.
  • Generalizing the foregoing examples, the differential between the X components 206 and 210 may indicate a magnitude of any pan or zoom gestures, and the differential between the Y components 608 and 612 may indicate a magnitude of any rotate gestures. However, other examples of gestures are possible, with the foregoing examples provided only to illustrate possible, but not all, implementation scenarios.
  • The foregoing description provides technologies for contact grouping and gesture recognition for surface computing. Although this description incorporates language specific to computer structural features, methodological acts, and computer readable media, the scope of the appended claims is not necessarily limited to the specific features, acts, or media described herein. Rather, this description provides illustrative, rather than limiting, implementations. Moreover, these implementations may modify and change various aspects of this description without departing from the true spirit and scope of this description, which is set forth in the following claims.

Claims (20)

We claim:
1. A method comprising:
receiving, by a surface computing system, a plurality of touch inputs;
in response to receiving the plurality of touch inputs, generating, by the surface computing system, a plurality of contact signals corresponding to the plurality of touch inputs, wherein each contact signal of the plurality of contact signals has a velocity associated therewith;
identifying a first contact signal of the plurality of contact signals and a second contact signal of the plurality of contact signals that have a greatest velocity differential therebetween;
grouping, by the surface computing system, the first contact signal into a first data structure; and
grouping, by the surface computing system, the second contact signal into a second data structure.
2. The method of claim 1, wherein the plurality of touch inputs is associated with a single user.
3. The method of claim 2, wherein the first data structure is associated with a first hand of the single user, and wherein the second data structure is associated with a second hand of the single user.
4. The method of claim 3, further comprising:
associating a first remainder of the plurality of contact signals with the first data structure, wherein the first remainder of the plurality of contact signals is associated with the first hand of the single user; and
associating a second remainder of the plurality of contact signals with the second data structure, wherein the second remainder of the plurality of contact signals is associated with the second hand of the single user.
5. The method of claim 1, wherein the plurality of touch inputs is associated with multiple users.
6. The method of claim 5, wherein the first data structure is associated with a hand of a first user of the multiple users, and wherein the second data structure is associated with a hand of a second user of the multiple users.
7. The method of claim 6, further comprising:
associating a first remainder of the plurality of contact signals with the first data structure, wherein the first remainder of the plurality of contact signals is associated with the hand of the first user; and
associating a second remainder of the plurality of contact signals with the second data structure, wherein the second remainder of the plurality of contact signals is associated with the hand of the second user.
8. The method of claim 1, wherein the plurality of touch inputs is associated with at least one touch gesture.
9. A tangible computer-readable storage medium having instructions stored thereon which, when executed by a processor of a surface computing system, cause the surface computing system to perform operations comprising:
receiving a plurality of touch inputs;
in response to receiving the plurality of touch inputs, generating a plurality of contact signals corresponding to the plurality of touch inputs, wherein each contact signal of the plurality of contact signals has a velocity associated therewith;
identifying a first contact signal of the plurality of contact signals and a second contact signal of the plurality of contact signals that have a greatest velocity differential therebetween;
grouping the first contact signal into a first data structure; and
grouping the second contact signal into a second data structure.
10. The tangible computer-readable storage medium of claim 9, wherein the plurality of touch inputs is associated with a single user.
11. The tangible computer-readable storage medium of claim 10, wherein the first data structure is associated with a first hand of the single user, and wherein the second data structure is associated with a second hand of the single user.
12. The tangible computer-readable storage medium of claim 11, further comprising:
associating a first remainder of the plurality of contact signals with the first data structure, wherein the first remainder of the plurality of contact signals is associated with the first hand of the single user; and
associating a second remainder of the plurality of contact signals with the second data structure, wherein the second remainder of the plurality of contact signals is associated with the second hand of the single user.
13. The tangible computer-readable storage medium of claim 9, wherein the plurality of touch inputs is associated with multiple users.
14. The tangible computer-readable storage medium of claim 13, wherein the first data structure is associated with a hand of a first user of the multiple users, and wherein the second data structure is associated with a hand of a second user.
15. The tangible computer-readable storage medium of claim 14, further comprising:
associating a first remainder of the plurality of contact signals with the first data structure, wherein the first remainder of the plurality of contact signals is associated with the hand of the first user; and
associating a second remainder of the plurality of contact signals with the second data structure, wherein the second remainder of the plurality of contact signals is associated with the hand of the second user.
16. A surface computing system comprising:
a hardware substrate configured to receive a plurality of touch inputs from at least one user;
an input detection system configured to detect the plurality of touch inputs received via the hardware substrate and to generate a plurality of contact signals corresponding to the plurality of touch inputs, wherein each contact signal of the plurality of contact signals has a velocity associated therewith;
a processor; and
a computer-readable storage medium having instructions stored thereon which, when executed by the processor, cause the processor to perform operations comprising
identifying a first contact signal of the plurality of contact signals and a second contact signal of the plurality of contact signals that have a greatest velocity differential therebetween,
grouping the first contact signal into a first data structure, and
grouping the second contact signal into a second data structure.
17. The surface computing system of claim 16, wherein the instructions, when executed by the processor, cause the processor to perform further operations comprising:
associating a first remainder of the plurality of contact signals with the first data structure; and
associating a second remainder of the plurality of contact signals with the second data structure.
18. The surface computing system of claim 17, wherein the first data structure is associated with a first hand of a first user, and wherein the second data structure is associated with a second hand of the first user.
19. The surface computing system of claim 17, wherein the first data structure is associated with a hand of a first user, and wherein the second data structure is associated with a hand of a second user.
20. The surface computing system of claim 16, wherein the computer-readable storage medium also has an application stored thereon, and wherein the application provides visual content to the hardware substrate which is further configured to display the visual content.
US13/866,630 2009-05-05 2013-04-19 Contact Grouping and Gesture Recognition for Surface Computing Abandoned US20130229375A1 (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US13/866,630 US20130229375A1 (en) 2009-05-05 2013-04-19 Contact Grouping and Gesture Recognition for Surface Computing

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US12/435,419 US8427440B2 (en) 2009-05-05 2009-05-05 Contact grouping and gesture recognition for surface computing
US13/866,630 US20130229375A1 (en) 2009-05-05 2013-04-19 Contact Grouping and Gesture Recognition for Surface Computing

Related Parent Applications (1)

Application Number Title Priority Date Filing Date
US12/435,419 Continuation US8427440B2 (en) 2009-05-05 2009-05-05 Contact grouping and gesture recognition for surface computing

Publications (1)

Publication Number Publication Date
US20130229375A1 true US20130229375A1 (en) 2013-09-05

Family

ID=43062082

Family Applications (2)

Application Number Title Priority Date Filing Date
US12/435,419 Expired - Fee Related US8427440B2 (en) 2009-05-05 2009-05-05 Contact grouping and gesture recognition for surface computing
US13/866,630 Abandoned US20130229375A1 (en) 2009-05-05 2013-04-19 Contact Grouping and Gesture Recognition for Surface Computing

Family Applications Before (1)

Application Number Title Priority Date Filing Date
US12/435,419 Expired - Fee Related US8427440B2 (en) 2009-05-05 2009-05-05 Contact grouping and gesture recognition for surface computing

Country Status (1)

Country Link
US (2) US8427440B2 (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP2857951A1 (en) * 2013-10-01 2015-04-08 Samsung Electronics Co., Ltd Apparatus and method of using events for user interface

Families Citing this family (20)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20080168478A1 (en) 2007-01-07 2008-07-10 Andrew Platzer Application Programming Interfaces for Scrolling
US20080168402A1 (en) 2007-01-07 2008-07-10 Christopher Blumenberg Application Programming Interfaces for Gesture Operations
US7844915B2 (en) 2007-01-07 2010-11-30 Apple Inc. Application programming interfaces for scrolling operations
US8645827B2 (en) 2008-03-04 2014-02-04 Apple Inc. Touch event model
US8416196B2 (en) 2008-03-04 2013-04-09 Apple Inc. Touch event model programming interface
US8717305B2 (en) 2008-03-04 2014-05-06 Apple Inc. Touch event model for web pages
US8566045B2 (en) 2009-03-16 2013-10-22 Apple Inc. Event recognition
US8566044B2 (en) 2009-03-16 2013-10-22 Apple Inc. Event recognition
US9311112B2 (en) 2009-03-16 2016-04-12 Apple Inc. Event recognition
US9684521B2 (en) * 2010-01-26 2017-06-20 Apple Inc. Systems having discrete and continuous gesture recognizers
US8285499B2 (en) 2009-03-16 2012-10-09 Apple Inc. Event recognition
US10216408B2 (en) 2010-06-14 2019-02-26 Apple Inc. Devices and methods for identifying user interface objects based on view hierarchy
US20120210261A1 (en) * 2011-02-11 2012-08-16 Apple Inc. Systems, methods, and computer-readable media for changing graphical object input tools
JP5716502B2 (en) * 2011-04-06 2015-05-13 ソニー株式会社 Information processing apparatus, information processing method, and computer program
US9298363B2 (en) 2011-04-11 2016-03-29 Apple Inc. Region activation for touch sensitive surface
US9696867B2 (en) 2013-01-15 2017-07-04 Leap Motion, Inc. Dynamic user interactions for display control and identifying dominant gestures
US9910579B2 (en) * 2013-04-15 2018-03-06 Microsoft Technology Licensing, Llc Detection of pan and scaling during multi-finger touch interactions
US9733716B2 (en) 2013-06-09 2017-08-15 Apple Inc. Proxy gesture recognizer
US9977505B2 (en) * 2014-06-06 2018-05-22 International Business Machines Corporation Controlling inadvertent inputs to a mobile device
CN110402427B (en) * 2017-03-15 2022-05-31 三菱电机株式会社 Touch gesture determination device, touch gesture determination method, recording medium containing touch gesture determination program, and touch panel input device

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20070103452A1 (en) * 2000-01-31 2007-05-10 Canon Kabushiki Kaisha Method and apparatus for detecting and interpreting path of designated position
US20090091539A1 (en) * 2007-10-08 2009-04-09 International Business Machines Corporation Sending A Document For Display To A User Of A Surface Computer
US20090251439A1 (en) * 1998-01-26 2009-10-08 Wayne Westerman Contact tracking and identification module for touch sensing
US20090278997A1 (en) * 2008-05-02 2009-11-12 Sony Corporation Image display device, image display method, and image display program

Family Cites Families (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8479122B2 (en) * 2004-07-30 2013-07-02 Apple Inc. Gestures for touch sensitive input devices
US8077147B2 (en) * 2005-12-30 2011-12-13 Apple Inc. Mouse with optical sensing surface
US8564544B2 (en) * 2006-09-06 2013-10-22 Apple Inc. Touch screen device, method, and graphical user interface for customizing display of content category icons
US7956847B2 (en) * 2007-01-05 2011-06-07 Apple Inc. Gestures for controlling, manipulating, and editing of media files using touch sensitive devices

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20090251439A1 (en) * 1998-01-26 2009-10-08 Wayne Westerman Contact tracking and identification module for touch sensing
US20070103452A1 (en) * 2000-01-31 2007-05-10 Canon Kabushiki Kaisha Method and apparatus for detecting and interpreting path of designated position
US20090091539A1 (en) * 2007-10-08 2009-04-09 International Business Machines Corporation Sending A Document For Display To A User Of A Surface Computer
US20090278997A1 (en) * 2008-05-02 2009-11-12 Sony Corporation Image display device, image display method, and image display program

Cited By (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP2857951A1 (en) * 2013-10-01 2015-04-08 Samsung Electronics Co., Ltd Apparatus and method of using events for user interface
KR20150038877A (en) * 2013-10-01 2015-04-09 삼성전자주식회사 User interfacing apparatus and method using an event corresponding a user input
CN104516499A (en) * 2013-10-01 2015-04-15 三星电子株式会社 Apparatus and method of using events for user interface
US10310615B2 (en) 2013-10-01 2019-06-04 Samsung Electronics Co., Ltd. Apparatus and method of using events for user interface
CN110045829A (en) * 2013-10-01 2019-07-23 三星电子株式会社 Utilize the device and method of the event of user interface
US10838508B2 (en) 2013-10-01 2020-11-17 Samsung Electronics Co., Ltd. Apparatus and method of using events for user interface
KR102203810B1 (en) * 2013-10-01 2021-01-15 삼성전자주식회사 User interfacing apparatus and method using an event corresponding a user input

Also Published As

Publication number Publication date
US20100283739A1 (en) 2010-11-11
US8427440B2 (en) 2013-04-23

Similar Documents

Publication Publication Date Title
US8427440B2 (en) Contact grouping and gesture recognition for surface computing
Shriram et al. Deep learning-based real-time AI virtual mouse system using computer vision to avoid COVID-19 spread
JP6641306B2 (en) Touch classification
US8581901B2 (en) Methods and apparatus for interactive rotation of 3D objects using multitouch gestures
US8878800B2 (en) Information processing apparatus, information processing method, and computer-readable storage medium
JP2020052991A (en) Gesture recognition-based interactive display method and device
US9612675B2 (en) Emulating pressure sensitivity on multi-touch devices
Lee et al. Finger identification and hand gesture recognition techniques for natural user interface
CN102768595B (en) A kind of method and device identifying touch control operation instruction on touch-screen
CN111625157B (en) Fingertip key point detection method, device, equipment and readable storage medium
TW201232331A (en) Computing device for performing functions of multi-touch finger gesture and method of the same
KR102224932B1 (en) Apparatus for processing user input using vision sensor and method thereof
Hung et al. Free-hand pointer by use of an active stereo vision system
CN110727383A (en) Touch interaction method and device based on small program, electronic equipment and storage medium
US20170315615A1 (en) Gesture library
US10162518B2 (en) Reversible digital ink for inking application user interfaces
Zaliva 3D finger posture detection and gesture recognition on touch surfaces
Ghodichor et al. Virtual mouse using hand gesture and color detection
Lakshmi et al. A novel air writing recognition system using Raspberry Pi for the control and interaction of digital systems
CN110248023A (en) Intelligent terminal control method, device, equipment and medium
JP4901670B2 (en) Electronic board system
US20120299837A1 (en) Identifying contacts and contact attributes in touch sensor data using spatial and temporal features
Seong et al. Design and implementation of user interface through hand movement tracking and gesture recognition
CN108415571A (en) A kind of somatosensory device implementation method moving caused data analysis based on thumb
Islam et al. Hand Gesture Recognition Based Human Computer Interaction to Control Multiple Applications

Legal Events

Date Code Title Description
AS Assignment

Owner name: MICROSOFT CORPORATION, WASHINGTON

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:ZHANG, KUO;SUN, XUEQING;ZHANG, GUANGGEN;AND OTHERS;REEL/FRAME:033771/0948

Effective date: 20090504

AS Assignment

Owner name: MICROSOFT TECHNOLOGY LICENSING, LLC, WASHINGTON

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:MICROSOFT CORPORATION;REEL/FRAME:034544/0541

Effective date: 20141014

STCB Information on status: application discontinuation

Free format text: ABANDONED -- FAILURE TO PAY ISSUE FEE