Human activity recognition

Pal-SPOT* Project

 

1. Project definition

1.1 Background

Mobile and pervasive computing scenarios are characterized by the heterogeneity of mobile devices, network infrastructures, and usage conditions. Hence, automatically recognizing the situation under which a user is accessing a given service is of key importance for adapting that service to the user’s needs, requirements, and expectations. This theme is a hot research topic in the field of context-awareness; existing techniques can be roughly classified in ones based on symbolic reasoning and ones based on statistical reasoning.

1.2 Objectives

The aim of this project is to investigate the use of statistical reasoning for deriving high-level situations (e.g., the current activity of the user) from raw sensor data that characterize basic user’s parameters. In particular, the ultimate goal is to devise techniques for recognizing physical activities on the basis of data gathered from a restricted set of sensors that will be eventually integrated in many commercial mobile phones (e.g., accelerometer, GPS receiver, environmental sensors).

The expected outcome of this work is the integration of statistical reasoning with an existing middleware for context-awareness based on symbolic reasoning (CARE middleware [CARE]). A demonstrator application (called Pal-SPOT) will be released, that will be in charge of collecting raw data from Sun SPOT sensors and other sensors that a user brings in her pocket, automatically deriving situations such as the user’s physical and social activity, and interacting with an existing service to proactively notify the user about her closeness to interesting points-of-interest.

1.3 Use cases

UC1: Alice is travelling abroad for business, carrying her Pal-SPOT in her bag. At noon she leaves the train station towards the center of the city, where she’s expected for a business meeting at 1 pm. She has only a few minutes for having lunch before the meeting begins, so she walks at a quick pace. Her Pal-SPOT (on the basis of accelerometer data) recognizes that she’s walking fast; hence, it infers that Alice is in a hurry. Moreover, since it is noon, the Pal-SPOT derives that she is probably interested in finding a restaurant. Hence, on the basis of location data gathered from the GPS receiver, Pal-SPOT retrieves a list of nearby fast-foods from the remote point-of-interest service, and notifies Alice about their presence through an alert on her smart phone.

UC2: Bob trains in the park according to a personalized training plan, carrying a Pal-SPOT in his pocket. On the basis of sensor data, Pal-SPOT recognizes Bob’s speed and activity (jogging), and transmits the data to a virtual trainer integrated in his smart phone that is in charge of calculating the caloric consumption and suggesting exercises. On the basis of those data, the virtu al trainer suggests starting an exercise session in order to reach the expected caloric consumption amount. Hence, Bob asks the remote point-of-interest service for the nearest fitness trail, and obtains directions for that spot.

1.4 References

[CARE] C. Bettini, D. Maggiorini, D. Riboni. "Distributed Context Monitoring for the Adaptation of Continuous Services". World Wide Web Journal (WWWJ), Special issue on Multi-channel Adaptive Information Systems on the World Wide Web, 10(4):503-528, Springer, 2007.

 

2. Preliminary architecture

The image below shows the overall system architecture.

In this version, due to technical problems in directly connecting SPOTs and PDAs, the information flow between sensors and the end-user device is mediated by a portable device equipped with a USB port (in our case, a Sony Vaio UX Series). The portable device is also in charge of providing Internet connectivity to the end-user device. However, in an ideal setting, sensor data are directly acquired by the end-user device, that is also in charge of sensor fusion, reasoning, and direct communication with the service provider.

Image

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

3. Datasets

Using the proposed system we have collected 5-hours activity data (10 activities) for the experimental evaluation of our activity recognition techniques. The dataset has been used in [2], [3] and [5]. It can be downloaded here.

The annotated dataset used in [7] and [8] can be downloaded here.

4. Activity ontologies

The activity ontology used in papers [2], [3] and [5].

We have defined other (partial) activity ontologies for the experimental evaluation of [4]:

The multilevel activity ontology used in papers [7] and [8] for the Opportunity dataset is available here.

 

5. Publications

  • [8] Rim Helaoui, Daniele Riboni, Heiner Stuckenschmidt, "A Probabilistic Ontological Framework for the Recognition of Multilevel Human Activities". In Proceedings of the 2013 ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp), pp. 345-354, ACM, 2013. PDF
  • [7] Rim Helaoui, Daniele Riboni, Mathias Niepert, Claudio Bettini, Heiner Stuckenschmidt, "Towards Activity Recognition using Probabilistic Description Logics". In Activity Context Representation: Techniques and Languages, AAAI Technical Report WS-12-05, pp. 26-31. AAAI, 2012. PDF
  • [6] Daniele Riboni, Claudio Bettini, "OWL 2 Modeling and Reasoning with Complex Human Activities". Journal of Pervasive and Mobile Computing, special issue on Knowledge-driven activity recognition, 7(3): 379-395, Elsevier, 2011. (DOI: 10.1016/j.pmcj.2011.02.001).
  • [5] Daniele Riboni, Claudio Bettini, "COSAR: Hybrid Reasoning for Context-aware Activity Recognition". Personal and Ubiquitous Computing, special issue on Context-aware Middleware and Applications, 15(3):271-289, Springer, 2011. (DOI: 10.1007/s00779-010-0331-7).
  • [4] Daniele Riboni, Linda Pareschi, Laura Radaelli, Claudio Bettini, "Is Ontology-based Activity Recognition Really Effective?". In Proceedings of CoMoRea'11, 8th IEEE Workshop on Context Modeling and Reasoning. IEEE Computer Society, 2011. PDF
  • [3] Daniele Riboni, Linda Pareschi, Claudio Bettini, "Towards the Adaptive Integration of Multiple Context Reasoners in Pervasive Computing Environments". In Proceedings of CoMoRea'10, 7th IEEE Workshop on Context Modeling and Reasoning, pp. 25-29. IEEE Computer Society, 2010. PDF
  • [2] Daniele Riboni, Claudio Bettini, "Context-aware Activity Recognition through a Combination of Ontological and Statistical Reasoning". In Proceedings of the 6th International Conference on Ubiquitous Intelligence and Computing (UIC-09), pp. 39-53, LNCS 5585, Springer, 2009. PDF
  • [1] Daniele Riboni, "Towards the Combination of Statistical and Symbolic Techniques for Activity Recognition". Invited talk at the 6th IEEE Workshop on Context Modeling and Reasoning (CoMoRea'09), colocated with PerCom'09, Galveston, Texas, 13 March 2009. Extended abstract (pdf)

 

6. Contacts

For more information please contact Daniele Riboni.

* This project was partially supported by a grant from Sun Microsystems.