Machine Learning in Video Games

Introduction

In recent times, the video games industry has ascertained machine learning (ML) as a significant ingredient to make games more realistic and challenging. Additionally, machine learning has discovered computer video games as an interesting and rewarding area. Also, the video game market has become a prominent and proven global industry.

From technical experiments at colleges in the 1950s, video games have evolved into one of the most influential recreational genres of the twenty-first century. Furthermore, a variety of elements distinguish video games, such as Pong from 1972 or today's various online multiplayer video games. Despite this, computer-controlled opponents have existed since the beginning of time and they are becoming more prevalent. On the other hand, AI and machine learning have grown in prominence across a wide variety of application domains in recent years. Non-player subjects in games are almost controlled by AI and machine learning. Additionally, these tasks are frequently performed using a predefined pattern that reacts to specific player behavior but does not learn beyond that and does not adapt to new circumstances [1]

The health of the video game industry, aided by the variety of genres and the availability of cutting-edge technologies, suggests that computer video game concepts are being implemented and applied in a variety of domains, including serious games.

Whereas, serious games depict a genre that is supposed to be more than just fun [2-3]. Additionally, the scholastic value affiliated with serious games has advanced beyond learning purposes, determining their focus on skill practice and entertainment value through exposure [4]. Thus, a serious game is designed to target a broad audience and to achieve certain goals of education. Therefore, these games are split into multiple categories which are based on different schemes of classification. These schemes involve purpose and market-based classifications respectively. The multiple segments recognized in these domains are as follows:

• Educational, healthcare, and military games respectively [5].
• Game evaluation, training and simulation, health, and tactical communication [6].
• Defense, training and learning and advertising. [7]

Decision-tree and path-finding techniques are frequently utilized. In addition to the standard difficulty levels (easy, medium, and hard), learning artificial intelligence would allow the player to create their own unique gaming experience.

History of Machine Learning Use in Video Games

Machine learning is a critical component of today’s business and research. It makes use of algorithms and neural network models to assist computers in continuously improving their performance. Machine learning algorithms generate a mathematical model from sample data – commonly referred to as "training data" – in order to make decisions in real-time without being explicitly programmed to do so.

Samuel’s [8] pioneering work on developing a Checkers playing game in the 1950s. Particularly, the use of symbolic, search-based artificial intelligence, automated cues selection, and a beta version of reinforcement learning, laid the groundwork for much-resulting machine learning and symbolic AI research, particularly in the domain of game-playing applications. Similarly, in the beginning, researchers focused their efforts on perfect-information games such as Chess and Checkers, because they required the development of game-playing solutions within large, completely specified state spaces.

Later research efforts expanded the application domain to involve imperfect information and stochastic games such as Backgammon and Poker, thus expanding the challenge to include state spaces with hidden information [9]. Furthermore, by the late 1990s, several significant milestones had been achieved, including the defeat of the world Checkers champion, Marion Tinsley, by the computer program named Chinkook [10] in 1994. Possibly the most important result in 1997 was when IBM's Deep Blue beat the world chess champion, Garry Kasparov, in a friendly match [11]. Furthermore, game-playing algorithms have also been built for the Poker game, most notably Loki, which is capable of offering an intermediate-level challenge to human opponents but still can not play at a world-class level [12].

The development of game-playing systems for conventional board and card games have been vital to the advancement of AI research, resulting in the development of a plethora of novel AI and machine learning techniques. Additionally, the concurrent development of low-cost, high-performance home computing, along with the growth of the computer games industry, has also contributed to the introduction of entirely new game-play genres and game-play types [13].

Video games, in contrast to traditional games, make use of more complex and comprehensive virtual settings, frequently integrating human-controlled protagonists and a large number of computer-controlled adversaries, which are also referred to as game agents. Additionally, these agents must be modeled more realistically and credible set of behaviors for a game AI engine subsystem. In addition, as evidenced by the plethora of genres arising from video games, such as Person Shooting Games, Action games, Adventure games, strategic and simulation games [14]. Thus, the requirements and specifications for the aforementioned games involve machine learning for both class-specific goals and low-level static behaviors. Also, by exploiting a fraction of total machine resources in a game.

Artificial intelligence in games enables the independent option of behaviors for agents via game perceptrons and decision-making subsystems [15]. Furthermore, game developers have adopted both traditional and machine learning methods and incorporated them into game AI, involving the use of algorithms such as finite state machines and neural networks for activity and behavioral control, inference engines and rule-based models for goal planning, and genetic and evolutionary methods for tuning and testing the game encoding parameters.

Procedural Content Generation (PCG) in Video Games via Machine Learning

Machine Learning has been studied for its potential application in recommendation systems and generation. Therefore, a method is required to create data algorithmically rather than the manual generation of data. As a result, PCG [16] has been used to generate various types of content in a variety of video games, such as weapons in Borderlands, layouts in Minecraft, and so on.

Search-based algorithms, grammars, and logic-based programming are among the popular PCG methodologies. However, these methods rely on humans to manually define the set of viable content. This implies human developers determine which descriptors constitute a valid amount of generated content. However, machine learning is capable of learning these characteristics when examples are provided to train it on them, which drastically minimizes the overhead tasks of developers.

Super Mario Bros

Many researchers have utilized Super Mario Bros [17] to counterfeit PCG-level buildings. However, various attempts have been made, each employing a different strategy. In 2014, a version was developed that used n-grams to generate levels that were uniform with the ones it was trained on. Moreover, this version was later upgraded by incorporating Monte Carlo tree search (MCTS) to guide generation. However, these additions to generations are not often optimal movement of players taken into consideration. A related research study conducted in 2017 attempted to overcome this problem by creating levels depending on player movement using Markov Chains.

Galactic Arms Race

It is a space shooter video game that utilizes unique weaponry for the player, a neuro-evolution powered PCG [18]. In 2010, this game remained a runner-up in the Indie Game Challenge, and its related paper won the best game paper award. The developers of this game employ a technique called neuro-evolution called cgNeast in order to generate new content or recommender system on each individual preference.

Grand Theft Auto (GTA) V

Grand Theft Auto V is a 2013 action-adventure video game developed and released by Rockstar North and Rockstar Games. In addition, to progress through the game, you play as either a third-person character, who views the game world from a distance, or a first-person character, who explores the game world through exploration and interaction. Furthermore, the game engine utilizes machine learning bots in the game as well as the control flow, which is a collection of node-to-node graphs that permits event handling methods.

However, with recent advancements in state-of-the-art technologies, many game developers have added extra functionalities and features in order to improve the game experience for the players. In the context of GTA 5, it has provided full account access at epic games store to increase the global access rate of players.

Issues And Constraints with Machine Learning in Video Game Environments

Here, we discuss the challenges and restrictions associated with the application of learning in video games.

Machine Learning (ML) related issues

According to research, a player's interest in a game can be partially explained by interactions with game agents within a game environment that is typically non-deterministic and dynamic, involving several game agents with potentially disrupted inputs. In addition, video games involve dynamic game environments, therefore agents are necessary to respond to changes in the game environment in real-time [19].

However, a significant consideration for incorporating learning into video games is the demand for computing efficiency and robustness throughout both the learning and classification phases of the learning process. Also, the utilization of knowledge by the game AI will be apparent to game creators and developers via the use of a high-level, symbolic knowledge representation method. Therefore, a high-level approach, on the other hand, maybe considered appropriate only when the computational cost of acquiring and manipulating the information is low enough to permit its efficient and effective use [20]. Furthermore, a problem that comes as a result of learning is over-fitting in video games [21]. This occurs when a game agent has trained to adjust its performance to a certain set of states which is unable to generalize. This results in inefficient performance whenever new game states are experienced. However, the exploitation of a set of validation examples is mostly used in machine learning in order to reduce over-fitting, but this technique may not be appropriate for use in a game environment due to the substantial amount of time required to learn an example. As a result, opting for training examples for a gaming AI-based learning task might be challenging.

Video games, whether designed for information, entertainment, or simulation, offer numerous opportunities for ML. The variety of viable virtual worlds and the ensuing machine learning-related problems presented to agents in these settings are restricted by their imagination. Furthermore, the gaming industry is not only large and growing (it has recently outperformed the film industry in terms of revenue), but it also has a high demand for new products that are difficult to meet. Against this background, the ML's well-directed successes would attract a high focus in the field. Unexpectedly, though, the less impact learning has had, the more business the game has had to date.

The most significant subjects in effective gaming applications involve learning how to play well, the modeling of players, adaptability, model analysis, and performance of course. These needs might be redefined as a requirement for new pragmatic and theoretical tools.

Learning to Play the Game

Game worlds offer ideal testbeds to examine the potential to enhance the abilities of agents through learning. This environment can be built with several properties, ranging from deterministic and discrete to non-deterministic and continuous, like a classical board, card, and action games. Learning-based algorithms for these tasks were researched very carefully. Backgammon, a two-player game, is probably the most well-known example of a learning player [22].

Learning about Players or Characters

Games like poker, where opponent simulation is important to enhance the game's performance, which is one of the main challenges in this work line [23].

Behavior analysis of players:

Developing a strong avatar based on the behavior of a player is a fascinating and demanding task for supervised machine learning. For example, in online video games, an avatar that is trained to mimic the game-playing behavior may take the place of his creator when the human player can’t attend to the main character in the game.

Machine Learning, Video Games, and the Future of Intelligence

The incorporation of machine learning into gaming has the potential to have a positive and profound impact on both the games themselves and on a variety of other areas. Such as with the introduction of computers professional chess was reshaped, which revealed existing distinct flaws in analysis; permit players to get instant access and train with datasets of millions of previous games played by participants, and delivered new tools that enabled the new generation of emerging players to enhance even more quickly than their predecessors.

However, the real attraction of ML in modern games is that such video games require thinking for expert play. A way to solve this problem will force us either to recognize the idea of independent thinking or to further limit our understanding of thinking. Therefore, it is self-evident that machines can not win by brute force in complex, flawed games. Will AI or machine learning be able to play the aforementioned games not only better than us, but also like a human?

References

• [1] K. Merrick and M. L. Maher, "Motivated reinforcement learning for non-player characters in persistent computer game worlds", Proc. ACM SIGCHI Int. Conf. Adv. Comput. Entertain. Technol., 2006.
• [2] T. Dumbleton and J. Kirriemuir, "13 digital games and education", Understanding Digital Games, pp. 223, 2006.
• [3] F.L. Greitzer, O.A. Kuchar and K. Huston, "Cognitive Science Implications for Enhancing Training Effectiveness in a Serious Gaming Context", ACM J. Educational Resources in Computing, vol. 7, no. 3, August 2007.
• [4] C. Girard, J. Ecalle and A. Magnan, "Serious games as new educational tools: how effective are they? A meta-analysis of recent studies", J. Comput. Assist. Learn., vol. 29, no. 3, pp. 207-219, Jun. 2013.
• [5] S. Chen and D. Michael, Serious Games: Games that Educate Train and Inform, Boston: Thomson Course Technology PTR, 2006.
• [6] M. Zyda, "From visual simulation to virtual reality to games," in Computer, vol. 38, no. 9, pp. 25-32, Sept. 2005.
• [7] U. Ritterfeld, M. Cody and P. Vorderer, "Serious Games: Mechanisms and Effects", Routledge London, 2009.
• [8] A. L. Samuel, "Some studies in machine learning using the games of checkers", IBM J., vol. 3, pp. 211-229, 1959.
• [9] J. Schaeffer, "The games computers (and people) play" in Advances in Computers, New York: Academic, pp. 189-266, 2000.
• [10] J. Schaeffer, “One Jump Ahead: Challenging Human Supremacy in Checkers”, New York: Springer-Verlag, pp. 97, 447, 1996.
• [11] M. Campbell, A.J. Hoane, F-H Hsu “Deep blue”, Artif Intell vol. 134, pp. 57–83, 2002.
• [12] D. Billings, A. Davidson, J. Schaeffer and D. Szafron, "The challenge of poker", Artificial Intelligence Journal, vol. 134, no. 1-2, pp. 201-240.
• [13] R. Miikkulainen, B. D. Bryant, R. Cornelius, I. V. Karpov, K. O. Stanley and C. H. Yong, "Computational intelligence in games" in Computational Intelligence: Principles and Practice, NJ, Piscataway: IEEE, 2006.
• [14] J. E. Laird and M. van Lent, "Human-level AI's killer application: Interactive computer games", Proc. 17th Nat. Conf. Artif. Intell. and 12th Ann. Conf. Innov. Appl. Artif. Intell., pp. 1171-1178, 2000.
• [15] B. Schwab, AI Game Engine Programming, Charles River Media, 2004.
• [16] S. Risi and J. Togelius, "Increasing generality in machine learning through procedural content generation", 2019.
• [17] G. Volkmar, N, Mählmann, and M. Rainer "Procedural content generation in competitive multiplayer platform games." Joint International Conference on Entertainment Computing and Serious Games. Springer, Cham, 2019.
• [18] E. J. Hastings, R. K. Guha and K. O. Stanley, "Evolving content in the Galactic Arms Race video game," 2009 IEEE Symposium on Computational Intelligence and Games, 2009, pp. 241-248.
• [19] J. Duan, N.E. Gough and Q. H. Mcdhi, "Multi-agent reinforcement learning for computer game agents," in Proceedings of the 3rd International Conference on Intelligent Games and Simulation, 2002, pp. 104-109.
• [20] K. O. Stanley, B. D. Bryant and R. Miikkulainen, "Real-time neuroevolution in the NERO video game," in IEEE Transactions on Evolutionary Computation, vol. 9, no. 6, pp. 653-668, Dec. 2005.
• [21] N. Kirby, 'Getting around the limits of machine learning," in Al Game Programming Wisdom 2, 1st edition. S. Rabin, Ed. Hingham, MA: Charles River Media, 2004, pp. 603-611.
• [22] G. Tesauro, "Temporal difference learning and TD-Gammon," Commun. ACM, vol. 38, no. 3, pp. 58-68, 1995.
• [23] D. Billings, A. Davidson, J. Schaeffer and D. Szafron, "The challenge of poker", Artificial Intelligence Journal, vol. 134, no. 1-2, pp. 201-240.