The present paper describes the construction of a multimodal database, referred to as the PROMETHEUS database, which contains recordings from heterogeneous sensors. The main purpose of this database is the development of a framework for monitoring and interpretation of human behavior in unrestricted environments of both indoor and outdoor type. It contains single-person and multi-person scenarios, but also covers scenarios with interactions between groups of people. It is devoted to detection of typical and atypical events, while care has been to taken for the recordings to be as close to real-world conditions as possible. The uniqueness of the PROMETHEUS database comes not only from the unique sensor sets but is due primarily to its generic design, which allows for embracing a wide range of real-world applications (including smart-home and human- robot interaction interfaces, indoors/outdoors public areas surveillance etc). © 2009 IEEE.