Publication: Context detection and identification in multi-agent reinforcement learning on non-stationary environments
Abstract
Etmenin öğrendiği ortamın durağan olması varsayımı birçok pekiştirmeli öğrenme yön- temi tarafından benimsenmiştir. Fakat, doğal ve gerçek hayat uygulamalarında ortam durağan değildir, devingendir. Devingen ortam, birçok durağan ortamın bir araya gelme- siyle oluşur. Ortam ile aynı anda birden fazla etmen etkileşim kurabilir ve etmenler de ortamın devingen hale gelmesine sebep olabilir. Pekiştirmeli öğrenme-bağlam sezme yön- temi, etmenin önsel bir bilgisi olmadan devingen ortamları öğrenmesini, bağlam değişim noktasını sezmesini ve bağlamı tanımlamasını sağlayan yaklaşımdır. Bu yaklaşımın temelinde tek etmen bulunur ve çok etmenli öğrenim için eksiklikleri bulunmaktadır. Bu çalışmada çok etmenli devingen ortamlarda; bağlam değişim noktalarını tespit eden, bağlamları tanımlayabilen ve etmenlerin ortamı öğrenmesine olanak sağlayan çok etmenli pekiştirmeli öğrenme-bağlam sezme adında yeni bir yaklaşım geliştirilmiştir. Pekiştirmeli öğrenme - bağlam sezme yöntemini temel alan bu yaklaşım; çok etmenli öğrenmede, ortam üzerindeki etmenlerden kaynaklı devingenliği sezmesi ve bağlam değişim nok- tasını tespit etmesi yönüyle daha verimlidir. Ortam dinamiklerinden kaynaklı bağlam değişikliklerinin yanı sıra ortamdaki etmenlerin politikalarının değişmesi sonucu oluşan bağlam değişimlerini de sezmesini sağlar. Bu çalışmadaki yaklaşımda, etmenler ener- jilerini %16 daha az harcaması ve değişim noktalarını daha doğru ve erken sezmesi açısından pekiştirmeli öğrenme - bağlam sezme yöntemine göre daha verimli olduğu deney sonuçları ile gösterilmiştir.
The assumption that the environment the agent learns is stationary has been adopted by many reinforcement learning methods. However, in natural and real-life applications, the environment is non-stationary. One possibility is that non-stationary environments are composed of several stationary components (i.e. context or sub-environment). More than one agent can interact with the environment at the same time, and agents can cause the environment to become non-stationary. The Reinforcement Learning - Context Detection method is an approach that enables the agent to learn non-stationary environments without prior knowledge, detects context change points, and identifies contexts. The basis of this approach is single-agent and it has shortcomings for multi-agent learning. In this study a new approach called Multi-Agent Reinforcement Learning - Context Detection has been developed that can detect context change points, identify contexts and allow agents to learn the multi-agent non-stationary environment. This approach is based on the reinforcement learning - context detection method; In multi-agent learning, it is more efficient in terms of detecting non-stationarity originating from agents in the environment and detecting context change points. In addition to the context changes caused by the environment dynamics, it also allows detecting the context changes that occur as a result of the changes in the policies of the agents in the environment. In the approach in this study, it has been shown by the experimental results that the agents spend their energy %16 less and are more efficient than the reinforcement learning - context detection method in terms of detecting the context change points more accurately and earlier.
The assumption that the environment the agent learns is stationary has been adopted by many reinforcement learning methods. However, in natural and real-life applications, the environment is non-stationary. One possibility is that non-stationary environments are composed of several stationary components (i.e. context or sub-environment). More than one agent can interact with the environment at the same time, and agents can cause the environment to become non-stationary. The Reinforcement Learning - Context Detection method is an approach that enables the agent to learn non-stationary environments without prior knowledge, detects context change points, and identifies contexts. The basis of this approach is single-agent and it has shortcomings for multi-agent learning. In this study a new approach called Multi-Agent Reinforcement Learning - Context Detection has been developed that can detect context change points, identify contexts and allow agents to learn the multi-agent non-stationary environment. This approach is based on the reinforcement learning - context detection method; In multi-agent learning, it is more efficient in terms of detecting non-stationarity originating from agents in the environment and detecting context change points. In addition to the context changes caused by the environment dynamics, it also allows detecting the context changes that occur as a result of the changes in the policies of the agents in the environment. In the approach in this study, it has been shown by the experimental results that the agents spend their energy %16 less and are more efficient than the reinforcement learning - context detection method in terms of detecting the context change points more accurately and earlier.