Cfrtjryk

I am in the process of learning Python (background in C++ and R). So after the obligatory "Hello World", I decided that my first non-trivial program would be a port of a Java implementation of the counter-factual regret minimization algorithm for a simple dice game called Liar Die [original source, Joodle online compiler].

The program runs a million simulations of the dice game and computes the optimal bluffing/calling frequencies. It does this by creating Node class instances for all decision points in the game, and keeping track of the various actions the player to move can make, as well as the expected values of those actions.

I then tried to translate this into Python as faithfully as possible:

import numpy as np



class LiarDieTrainer:

    DOUBT, ACCEPT = 0, 1



    class Node:

        u, pPlayer, pOpponent = 0.0, 0.0, 0.0



        def __init__(self, numActions):

            self.regretSum = np.zeros(numActions)

            self.strategy = np.zeros(numActions)

            self.strategySum = np.zeros(numActions)



        def getStrategy(self):            

            self.strategy = np.maximum(self.regretSum, 0)

            normalizingSum = np.sum(self.strategy)

            if normalizingSum > 0:

                self.strategy /= normalizingSum

            else:

                self.strategy.fill(1.0/len(self.strategy))

            self.strategySum += self.pPlayer * self.strategy

            return self.strategy



        def getAverageStrategy(self):

            normalizingSum = np.sum(self.strategySum)

            if normalizingSum > 0:

                self.strategySum /= normalizingSum

            else:

                self.strategySum.fill(1.0/len(self.strategySum))

            return self.strategySum



    def __init__(self, sides):

        self.sides = sides

        self.responseNodes = np.empty((sides, sides+1), dtype=self.Node)

        for myClaim in range(sides):

            for oppClaim in range(myClaim+1, sides+1):                

                self.responseNodes[myClaim, oppClaim] = self.Node(1 if oppClaim == sides else 2)

        self.claimNodes = np.empty((sides, sides+1), dtype=self.Node)

        for oppClaim  in range(sides):

            for roll in range(1, sides+1):

                self.claimNodes[oppClaim , roll] = self.Node(sides - oppClaim)



    def train(self, iterations):

        regret = np.zeros(self.sides)

        rollAfterAcceptingClaim = np.zeros(self.sides, dtype=int)

        for it in range(iterations):

            for i in range(len(rollAfterAcceptingClaim)):

                rollAfterAcceptingClaim[i] = np.random.randint(self.sides) + 1

            self.claimNodes[0, rollAfterAcceptingClaim[0]].pPlayer = 1

            self.claimNodes[0, rollAfterAcceptingClaim[0]].pOpponent = 1



            for oppClaim in range(self.sides+1):

                if oppClaim > 0:

                    for myClaim in range(oppClaim):

                        node = self.responseNodes[myClaim, oppClaim]

                        actionProb = node.getStrategy()

                        if oppClaim < self.sides:

                            nextNode = self.claimNodes[oppClaim, rollAfterAcceptingClaim[oppClaim]]

                            nextNode.pPlayer += actionProb[1] * node.pPlayer

                            nextNode.pOpponent += node.pOpponent



                if oppClaim < self.sides:

                    node = self.claimNodes[oppClaim, rollAfterAcceptingClaim[oppClaim]]

                    actionProb = node.getStrategy()

                    for myClaim in range(oppClaim+1, self.sides+1):

                        nextClaimProb = actionProb[myClaim - oppClaim - 1]

                        if nextClaimProb > 0:

                            nextNode = self.responseNodes[oppClaim, myClaim]

                            nextNode.pPlayer += node.pOpponent

                            nextNode.pOpponent += nextClaimProb * node.pPlayer



            for oppClaim in reversed(range(self.sides+1)):

                if oppClaim < self.sides:

                    node = self.claimNodes[oppClaim, rollAfterAcceptingClaim[oppClaim]]

                    actionProb = node.strategy

                    node.u = 0.0

                    for myClaim in range(oppClaim+1, self.sides+1):

                        actionIndex = myClaim - oppClaim - 1

                        nextNode = self.responseNodes[oppClaim, myClaim]

                        childUtil = - nextNode.u

                        regret[actionIndex] = childUtil

                        node.u += actionProb[actionIndex] * childUtil

                    for a in range(len(actionProb)):

                        regret[a] -= node.u

                        node.regretSum[a] += node.pOpponent * regret[a]

                    node.pPlayer = node.pOpponent = 0              



                if oppClaim > 0:                    

                    for myClaim in range(oppClaim):

                        node = self.responseNodes[myClaim, oppClaim]

                        actionProb = node.strategy

                        node.u = 0.0

                        doubtUtil = 1 if oppClaim > rollAfterAcceptingClaim[myClaim] else -1

                        regret[self.DOUBT] = doubtUtil

                        node.u += actionProb[self.DOUBT] * doubtUtil

                        if oppClaim < self.sides:

                            nextNode = self.claimNodes[oppClaim, rollAfterAcceptingClaim[oppClaim]]

                            regret[self.ACCEPT] += nextNode.u

                            node.u += actionProb[self.ACCEPT] * nextNode.u

                        for a in range(len(actionProb)):

                            regret[a] -= node.u

                            node.regretSum[a] += node.pOpponent * regret[a]

                        node.pPlayer = node.pOpponent = 0



            if it == iterations // 2:

                for nodes in self.responseNodes:

                    for node in nodes:

                        if node:

                            node.strategySum.fill(0)

                for nodes in self.claimNodes:

                    for node in nodes:

                        if node:

                            node.strategySum.fill(0)                  



        for initialRoll in range(1, self.sides+1):

            print("Initial claim policy with roll %d: %s" % (initialRoll, np.round(self.claimNodes[0, initialRoll].getAverageStrategy(), 2)))       

        print("nOld ClaimtNew ClaimtAction Probabilities")            

        for myClaim in range(self.sides):

            for oppClaim in range(myClaim+1, self.sides+1):

                print("t%dt%dt%s" % (myClaim, oppClaim, self.responseNodes[myClaim, oppClaim].getAverageStrategy()))

        print("nOld ClaimtRolltAction Probabilities")

        for oppClaim in range(self.sides):

            for roll in range(1, self.sides+1):

                print("%dt%dt%s" % (oppClaim , roll, self.claimNodes[oppClaim , roll].getAverageStrategy()))



trainer = LiarDieTrainer(6)

trainer.train(1000)

Working example on the Ideone online compiler (factor of 1000 less iterations, apparently Python is way slower than even Java). Unfortunately, the algorithm works by randomly throwing dice, and the Java/Python random number generators give different sequences, and the dice game may not have a unique equilibrium anyway. This means I can't directly compare the outcomes.

Questions:

1. how can I make my code more Pythonic?


2. which other idioms / coding style should I apply?


3. which other useful libraries (besides NumPy) could I have used for this exercise?

Your function names and variable names are lowerCamelCase when the convention for Python is snake_case.

You have some inconsistent spacing here:

for oppClaim  in range(sides):

A linter would catch both of these issues.

This:

        self.claimNodes[0, rollAfterAcceptingClaim[0]].pPlayer = 1

        self.claimNodes[0, rollAfterAcceptingClaim[0]].pOpponent = 1

should use a temporary variable:

node = self.claim_nodes[0, roll_after_accepting_claim[0]]

node.p_player = 1

node.p_opponent = 1

These two loops:

            for nodes in self.responseNodes:

                for node in nodes:

                    if node:

                        node.strategySum.fill(0)

            for nodes in self.claimNodes:

                for node in nodes:

                    if node:

                        node.strategySum.fill(0)                  

can be refactored into one set of nested loops:

for node_source in (self.response_node, self.claim_nodes):

    for nodes in node_source:

        for node in nodes:

            if node:

                node.strategy_sum.fill(0)

Strings such as this:

print("t%dt%dt%s" % (myClaim, oppClaim, self.responseNodes[myClaim, oppClaim].getAverageStrategy()))

are good candidates for being converted to f-strings:

ave_strategy = self.response_nodes[my_claim, opp_claim].get_average_strategy()

print(f't{my_claim}t{opp_claim}t{ave_strategy}')

Also, since you're printing tabular data, you should apply fixed field widths to both your heading string and your row strings. To learn more about field widths, read about the Format Specification Mini-Language.

You should consider adding a main function instead of calling train from global code.