WO2011115572A1

WO2011115572A1 - An apparatus for enabling control of content on a display device using at least one gesture, consequent methods enabled by the apparatus and applications of the apparatus

Info

Publication number: WO2011115572A1
Application number: PCT/SG2010/000108
Authority: WO
Inventors: Ho Chee Yue
Original assignee: Xyz Wave Pte Ltd
Priority date: 2010-03-19
Filing date: 2010-03-19
Publication date: 2011-09-22
Also published as: SG184183A1

Abstract

There is provided an apparatus for enabling control of content on a display device with the use of at least one gesture. Using the apparatus also enables both a method to count an incidence of actions and a method for consolidating gestures used on controlling the content on the display. Some applications of the apparatus are also provided.

Description

AN APPARATUS FOR ENABLING CONTROL OF CONTENT ON A DISPLAY DEVICE USING AT LEAST ONE GESTURE, CONSEQUENT METHODS ENABLED BY THE APPARATUS AND APPLICATIONS OF THE APPARATUS

FIELD OF INVENTION

The present invention relates to an add-on apparatus which enables gesture recognition and applications of the add-on apparatus. In addition, the present invention relates to methods of gesture database consolidation which is enabled by the add-on apparatus.

BACKGROUND In the era where users constantly have to interact with computers for prolonged periods of time for either work or leisure, the forms of the user- computer interaction interface has correspondingly evolved in a continual manner in order to meet expectations of users at large. Throughout the course of history, the user-computer interaction interface has typically centred around usage of an alphanumeric keyboard, with additional peripheral devices such as, for example, mice, joysticks, trackballs, scroll pads, and so forth being used when applications on the computer enabled usage of any of the aforementioned peripheral devices.

Developments in the field of the user-computer interaction interface has led to non-physical interaction interfaces being introduced to the market, such as, for example, voice control and gesture control. A significant reason for the development of non-physical interaction interfaces is to address the issue of strains experienced by users at the various joints of their arms resulting from continuous pounding at the physical interaction interface. Another reason for the development of non-physical interaction interfaces is for hygiene reasons to prevent transmission of contagious viruses like SARS and avian flu. .

l 0108

Gesture control interaction interfaces are increasing in popularity although such interfaces are considered to be still in their early stages of infancy in relation to their path of development. Gesture control interaction interfaces are still at a phase where a novelty factor of such interfaces is high, and that invariably also drives the popularity of such interfaces. However, it will only be a matter of time before gesture control interaction interfaces will be commonplace as a user-computer interaction interface.

In this regard, it would be desirable if there was an apparatus which could be used with existing computers without a gesture recognition interface to enable gesture recognition by the existing computers so as to delay obsolescence of those existing computers for a period of time. Such an apparatus may also be used as a bridging device for a user adapting from a physical user-computer interaction interface to a non-physical user-computer interaction interface. There are clearly advantages resulting from use of such an apparatus.

SUMMARY

In a first aspect, there is provided an apparatus for coupling with a display device for enabling control of content shown on the display device using at least one gesture. The apparatus includes a casing for encasing components of the apparatus; a data processing unit for controlling the apparatus; an image capture apparatus coupled to the data processing unit; a data storage unit coupled to the data processing unit; a data connector coupled to the data processing unit, the data connector being for allowing a functional connection of the display device to the apparatus; and at least one mirror for reflecting an image onto the image capture apparatus. The image capture apparatus may advantageously be fully integrated within the casing such that incidence of damage to the image capture apparatus is minimised.

The data storage unit may store gestures captured by the image capture apparatus. It is advantageous that a number of gestures stored in the data storage unit increases over time. The at least one gesture may be selected from, for example, hand gestures, facial gestures, leg movements, body movements and so forth.

The data processing unit may associate controls for the content shown on the display device with respective gestures stored on the data storage unit. The associated controls for the content shown on the display device with respective gestures may also be stored on the data storage unit. The data processing unit may include a counter which counts an incidence of the associated controls. The counter data may also be stored on the data storage unit.

The apparatus may also include a network connection unit, the network connection unit being coupled to the data processing unit. The network connection unit may allow data stored on the data storage unit to be transmitted to a central data server. It is advantageous that the data is shared with any of the apparatus connected to the central data server. The apparatus may also include a mounting mechanism on the casing, with the mounting mechanism being for attaching the apparatus to the display device in a secure manner.

The display device may be either a flat panel display or a CRT display. The display device may be used for displaying digital content. The digital content may be used for advertising purposes. The image capture apparatus being employed may be dependent on the type of images being captured.

The aforementioned apparatus may be used with a product vending machine to enable purchase of at least one product using at least one gesture. Alternatively, the aforementioned apparatus may also be used with a jackpot machine to enable a single play using at least one gesture.

In a second aspect, there is provided a method for counting an incidence of actions from a plurality of display devices. The method includes counting an incidence of each action controlling content shown on the display device occurring at each display device; sending the count for each action at each display device to a central data server; and consolidating the count for each action at the central data server. Advantageously, the consolidated count for each action may indicate a popularity of each action. The display device may be either a flat panel display or a CRT display with the display device being used for displaying digital content. The digital content may be used for advertising purposes. In a third aspect, there is provided a method for consolidating gestures used for controlling content shown on a plurality of display devices which are connected on a network. The method includes consolidating gestures used at each display device; and sending the consolidated gestures at each display device to a central data server. It is advantageous that the consolidated gestures may be usable at each of the plurality of display devices. The display device may be either a flat panel display or a CRT display with the display device being used for displaying digital content. The digital content may be used for advertising purposes. The gestures may be selected from, for example, hand gestures, facial gestures, leg movements, body movements and so forth.

DESCRIPTION OF DRAWINGS

In order that the present invention may be fully understood and readily put into practical effect, there shall now be described by way of non-limitative example only preferred embodiments of the present invention, the description being with reference to the accompanying illustrative drawings.

Figure 1 shows a schematic view of an apparatus of the present invention. Figure 2 shows a perspective view of the apparatus of the present invention during use.

Figure 3 shows a process flow of a first method of the present invention.

Figure 4 shows a process flow of a second method of the present invention. Figure 5 shows an overview of a system where a plurality of the apparatus of Figure 1 is employed.

Figure 6 shows a look-up table used to store gestures in the apparatus of the present invention.

Figure 7 shows a perspective view of the apparatus of the present invention during use in a vending machine application.

Figure 8 shows a perspective view of the apparatus of the present invention during use in a jackpot machine application. DESCRIPTION OF PREFERRED EMBODIMENTS

It is appreciated that there are various aspects to the present invention, and the description in the following paragraphs merely illustrates the various aspects of the present invention, and should not be construed to be limiting in any form or manner.

A first aspect of the present invention relates to an apparatus 20 for coupling with a display device for enabling control of content shown on the display device using at least one gesture. Figure 1 shows a schematic view of the apparatus 20 of the present invention. Figure 2 shows a perspective view of the apparatus 20 during use. Reference will be made to both Figures 1 and 2 when describing the first aspect of the present invention. The display device may be either a flat panel display (50 in Figure 2) or a CRT display. The display device 50 may be used for displaying digital content, such as, for example, content 52 relating to food as shown in Figure 2. A further example of the digital content may be in relation to images used to promote movies. It should be appreciated that the digital content may relate to content used for advertising purposes. The apparatus 20 includes a casing 22 for encasing components of the apparatus 20. The casing 22 may be made from materials such as, for example, metals, plastics, composites, and so forth. The casing 22 may be of a form which prevents tampering and theft of components of the apparatus 20. The casing 22 may also provide all weather protection for components of the apparatus 20. The aforementioned purposes of the casing 22 are essential because the apparatus 20 may be used either in indoor or outdoor deployments without any supervision of the apparatus 20 during durations of deployment.

There may be a mounting mechanism 34 on the casing 22, where the mounting mechanism 34 is for attaching the apparatus 20 to the display device 50 in a secure manner. The mounting mechanism 34 may be lockable such that the apparatus 20 is not removable from the display device 50 without unlocking the mounting mechanism 34. It should be appreciated that the apparatus 20 need not be attached to the display device 50 if the apparatus 20 is able to be positioned at a location which is able to effectively capture the gestures initiated by a user 90 in front of the display device 50. Figure 2 shows the apparatus 20 being mounted onto a top edge 54 of the display device 50 so as to capture gestures initiated by the user 90.

There is also included a data processing unit 24 for controlling the apparatus 20. The data processing unit 24 may be considered to be a brain for the apparatus 20 which controls all functions and processes of the apparatus 20. The data processing unit 24 includes a counter 26 which may be used for making counts of, for example, instances of invoked functions, instances of invoked processes, and so forth. An image capture apparatus 28 may be coupled to the data processing unit 24. The image capture apparatus 28 may be a typical lens-light sensor digital imaging device. It should be appreciated that the image capture apparatus 28 may be varied to allow for variations in relation to the type of images that are captured by the image capture apparatus 28. For example, cameras with a fish-eye lens may be used if wide- angle images are required, while cameras with a plurality of lens may be used if stereoscopic images are required, and so forth. At least one mirror 30 may be used for reflecting an image onto the image capture apparatus 28. The at least one mirror 30 may be adjustable to ensure that an optimal image is captured by the image capture apparatus 28. A possible arrangement of mirrors may be a "periscope" configuration where an even number of mirrors are used to correct laterally inverted images and where the image capture apparatus 28 is fully integrated within the casing 22 such that incidence of damage to the image capture apparatus 28 is minimised. Referring to Figure 2, a practical embodiment of the apparatus 20 only includes a slot 56 for the capture of images, with the at least one mirror 30 and the image capture apparatus 28 being encased within the casing 22 of the apparatus 20.

The apparatus 20 also includes a data storage unit 32 coupled to the data processing unit 26. The data storage unit 32 may preferable be a solid state memory module for durability purposes. The data storage unit 32 may store gestures captured by the image capture apparatus 28 in a look-up table 200 form as shown in Figure 6. Referring to Figure 6, a first column 202 shows associated controls for specific hand gestures. A second column 204 shows the specific hand gestures in question. For illustration purposes, a first row 206 shows a hand gesture for "flipping an object upwards". Similarly, a second row 208 shows a hand gesture for "flipping an object downwards". A third row 210 shows a hand gesture for "moving an object to top left" while a fourth row 212 shows a hand gesture for "moving an object to bottom right". It should be appreciated that while Figure 6 denotes a use of hand gestures, the term "gestures" should not be limited to only hand gestures. Gestures may also include, for example, facial gestures, leg movements, body movements, and so forth.

A number of gestures stored in the data storage unit 32 increases over time. This also means that the multiple look-up tables 200 may be stored or a number of entries into the look-up table 200 continually increases. The data processing unit 26 may carry out processes to associate controls for the content shown on the display device with respective gestures (as shown in look-up table 200) stored on the data storage unit 32. The associated controls for the content shown on the display device with respective gestures may also be stored on the data storage unit 32 in a form other than the look-up table 200 form. The data storage unit 32 may also store data from the counter 26 of the data processing unit 24. Data from the counter 26 may include counts for an incidence of invoking the associated controls. There may also be a data connector 36 coupled to the data processing unit 24 of the apparatus 20, where the data connector 36 is for allowing a functional connection of the display device 50 to the apparatus 20. The data connector 36 may be a conduit for instructions to be sent from the apparatus 20 to the display device 50 such that the content 52 on the display device 50 appears to be controlled by gestures of the user 90 (as shown in Figure 2). A cable 58 of the data connector 36 may be seen in Figure 2.

A network connection unit 38 which is coupled to the data processing unit 24 may also be included in the apparatus 20. The network connection unit 38 may enable connection of the apparatus 20 via either a direct wired connection or a wireless connection. The network connection unit 38 may allow data stored on the data storage unit 32 to be transmitted to a central data server. The central data server 252 is shown in Figure 5, and may comprise more than a single data server. A more detailed description for Figure 5 will be provided in subsequent portions of the description. It should be appreciated that the data on the central data server 252 may be shared with any of the apparatus 20 connected to the central data server 252 either directly or via a network. Transmission of data from the apparatus 20 to the central data server 252 may bring forth advantages which will be described in subsequent portions of the description.

It should be noted that even though Figure 2 shows the apparatus 20 being used in a display device 50 used for advertising purposes, the apparatus 20 may be used for other purposes such as, for example, control of televisions, control input into computers for work/pleasure, and so forth. An advantage brought forth by the use of the apparatus 20 may at least include at least minimising strains experienced by users at the various joints of their arms resulting from not having to continuously pound at a physical interaction interface. An additional advantage relates to how the apparatus 20 enables non-contact interaction with the display device 50 which is more hygienic for the users. Further examples of the use of the apparatus 20 is shown in Figures 7 and 8. Figure 7 shows the apparatus 20 used in a vending machine application, while Figure 8 shows the apparatus 20 used in a jackpot machine application. In Figure 7, the vending machine 60 includes a display panel 62. The display panel 62 may be used to show graphics which interact with the user 90 while the user 90 is deciding which products to purchase and may also be used to show products being sold by the vending machine 60. The user 90 may use gestures such as, for example, hand gestures, facial gestures, leg movements, body movements, and so forth to both interact with the vending machine 60 and also to select products for purchase at the vending machine 60. The interaction between the user 90 and the vending machine 60 may depend on the graphics shown on the display panel 62. For example, the graphics may be in a form of a concierge character guiding the user 90, a game which ends with a purchase of a product, a basic product display interface, and so forth. The vending machine 60 would also have a dispenser compartment 64 for the products purchased at the vending machine 60.

In Figure 8, the jackpot machine 180 also includes a display screen 182. The display screen 182 may be used to show graphics for every play the user 90 executes. The user 90 may use gestures such as, for example, hand gestures, facial gestures, leg movements, body movements, and so forth to both interact with the jackpot machine 80 and also to manage playing credits for use at the jackpot machine 180. The graphics shown on the display screen 182 may depend on the gestures used by the user. For example, virtual cylinders shown on the display screen 182 may rotate at different rates depending on the gestures used by the user 90. In this manner, the user 90 may believe that an outcome for each play is dependent on the user's 90 gestures and not merely determined by probabilities dictated by a computer program. The jackpot machine 180 would also have a dispenser compartment 184 for winnings at the jackpot machine 180.

Figure 5 shows an overview of a system 250 where a plurality of the apparatus 20 is employed. In the system 250, each of the plurality of the apparatus 20 is connected to the internet 254 using the network connection unit 38 of each of the apparatus 20. There is also the earlier mentioned central data server 252 which is also connected to the internet 254. Understanding components of the system 250 will aid in understanding further aspects of the present invention in subsequent portions of the description. It should be noted that even though Figure 5 shows the apparatus 20 being used in a display device 50 used for advertising purposes, the apparatus 20 may be used for other purposes such as, for example, control of televisions, control input into computers for work/pleasure, and so forth.

In a second aspect, Figure 3 shows a process flow for a method 70 for counting an incidence of actions from a plurality of display devices 50. The display devices 50 may be either a flat panel display or a CRT display as described earlier. Referring to Figure 5, the method 70 may be used for counting an incidence of controls invoked by gestures at each of the display devices 50. It should be appreciated that the term "gestures" should not be limited to only hand gestures. Gestures may also include, for example, facial gestures, leg movements, body movements, and so forth. The method 70 includes counting an incidence of each action (72) controlling the content 52 shown on the display device 50 occurring at each display device 50. The counting of the incidence may be carried out using the counter 26 included in the data processing unit 24 of the apparatus 20.

The method 70 then includes sending the count (74) for each action at each display device 50 to the central data server 252 using the network connection unit 38 of each of the apparatus 20. Finally, the count for each action for each display device 50 is consolidated (76) at the central data server 252, where it can be ascertained that the consolidated count for each action indicates a popularity of each action.

It should be noted that being able to ascertain the consolidated count for each action will lead to the obtaining of information in relation to user behaviour at each display device 50 which may be used to infer a receptivity of the content 52 shown on each display device 50. For example, a high consolidated count for a "tell me more" action indicates high receptivity of the content 52, while a low consolidated count for a "tell me more" action indicates low interest of the content 52. In a third aspect, Figure 4 shows a process flow for a method 80 for consolidating gestures used for controlling content shown on a plurality of display devices 50 which are connected on a network. The display device 50 may be either a flat panel display or a CRT display as described earlier. Referring to Figure 5, the method 80 may be used for dynamically building up a database of a variety of gestures used by different users at each of the display devices 50 for invoking a variety of actions for controlling the content 52 shown on the display device 50. It should be appreciated that the term "gestures" should not be limited to only hand gestures. Gestures may also include, for example, facial gestures, leg movements, body movements, and so forth.

The method 80 includes consolidating gestures (possibly a number of the look-up tables 200 of Figure 6) used at each display device 50 (82) using the data processing unit 24 of each of the apparatus 20. Subsequently, the method 80 includes sending the consolidated gestures (84) at each display device 50 to the central data server 252 using the network connection unit 38 of each of the apparatus 20. When the consolidated gestures from each of the display devices 50 are stored on the central data server 252, the consolidated gestures at the central data server 252 may be used as a reference database for all the display devices 50, such that all gestures which have been used at the various display devices 50 may be recognized and usable at each of the plurality of display devices 50 (86).

It should be noted that being able to dynamically build up a database of a variety of gestures used by different users at each of the display devices 50 for invoking a variety of actions for controlling the content 52 shown on the display device 50 will lead to improved interactivity with improved intuitiveness (less "teaching" of gestures is required) for users at each display device 50. Correspondingly, the improved interactivity may lead to greater interest being directed at the content 52 shown at the display devices 50.

Whilst there has been described in the foregoing description preferred embodiments of the present invention, it will be understood by those skilled in the technology concerned that many variations or modifications in details of design or construction may be made without departing from the present invention.

Claims

1. An apparatus for coupling with a display device for enabling control of content shown on the display device using at least one gesture, the apparatus including:

a casing for encasing components of the apparatus;

a data processing unit for controlling the apparatus;

an image capture apparatus coupled to the data processing unit;

a data storage unit coupled to the data processing unit;

a data connector coupled to the data processing unit, the data connector being for allowing a functional connection of the display device to the apparatus; and

at least one mirror for reflecting an image onto the image capture apparatus,

wherein the image capture apparatus is fully integrated within the casing such that incidence of damage to the image capture apparatus is minimised.

2. The apparatus of claim 1 , wherein the data storage unit stores gestures captured by the image capture apparatus.

3. The apparatus of claim 2, wherein a number of gestures stored in the data storage unit increases over time.

4. The apparatus of claim 3, wherein the data processing unit associates controls for the content shown on the display device with respective gestures stored on the data storage unit.

5. The apparatus of claim 4, wherein the associated controls for the content shown on the display device with respective gestures are also stored on the data storage unit.

6. The apparatus of claim 5, further including a network connection unit, the network connection unit being coupled to the data processing unit.

7. The apparatus of claim 6, wherein the network connection unit allows data stored on the data storage unit to be transmitted to a central data server.

8. The apparatus of claim 7, wherein the data is shared with any of the apparatus connected to the central data server.

9. The apparatus of claim 1 , wherein the display device is either a flat panel display or a CRT display.

10. The apparatus of claim 9, wherein the display device is used for displaying digital content.

11. The apparatus of claim 10, wherein the digital content is used for advertising purposes.

12. The apparatus of claim 1, further including a mounting mechanism on the casing, the mounting mechanism being for attaching the apparatus to the display device in a secure manner.

13. The apparatus of claim 4, wherein the data processing unit includes a counter which counts an incidence of the associated controls.

14. The apparatus of claim 13, wherein the counter data is also stored on the data storage unit.

15. The apparatus of claim 1 , wherein the image capture apparatus being employed is dependent on the type of images being captured.

16. The apparatus of claim 1 , wherein the at least one gesture is selected from a group consisting of: hand gestures, facial gestures, leg movements and body movements.

17. A method for counting an incidence of actions from a plurality of display devices, the method including:

counting an incidence of each action controlling content shown on the display device occurring at each display device;

sending the count for each action at each display device to a central data server; and

consolidating the count for each action at the central data server;

wherein the consolidated count for each action indicates a popularity of each action.

18. The method of claim 17, wherein the display device is either a flat panel display or a CRT display.

19. The method of claim 18, wherein the display device is used for displaying digital content.

20. The method of claim 19, wherein the digital content is used for advertising purposes.

21. A method for consolidating gestures used for controlling content shown on a plurality of display devices which are connected on a network, the method including:

consolidating gestures used at each display device; and

sending the consolidated gestures at each display device to a central data server;

wherein the consolidated gestures is usable at each of the plurality of display devices.

22. The method of claim 21 , wherein the display device is either a flat panel display or a CRT display.

23. The method of claim 22, wherein the display device is used for displaying digital content.

24. The method of claim 23, wherein the digital content is used for advertising purposes.

25. The method of claim 21 , wherein the gestures are selected from a 5 group consisting of: hand gestures, facial gestures, leg movements and body movements.

26. A product vending machine using the apparatus of claim 1 to enable purchase of at least one product using at least one gesture.

o

27. A jackpot machine using the apparatus of claim 1 to enable a single play using at least one gesture.