Title: Gym

Q: env.step()
A: Arguments:
- action: actType
Returns:
- obs: ObsType
- reward: float
- done: bool
  - can be used for early episode termination
- info: dict
  - can contain anything extra that doesn't fit into the uniform interface
  - useful for storing "cheating" information

Q: env.reset()
A: Arguments:
- seed: Optional[int] = None
- return_info: bool = False
- options = None
Returns:
- obs: Union[ObsType, tuple[ObsType, dict]

Q: env.render()
A: Render is only used for debugging or entertainment, and what it does is up to the environment.

Q: Registering an environment
A: 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
class MultiArmedBandit(gym.Env):
    pass
gym.envs.registration.register(
    id="ArmedBanditTestbed-v0",
    entry_point=MultiArmedBandit,
    max_episode_steps=max_episode_steps,
    nondeterministic=True,
    reward_threshold=1.0,
    kwargs={"num_arms": 10, "stationary": True},
)
if MAIN:
    env = gym.make("ArmedBanditTestbed-v0")

Q: agent.observe()
A: This is useful for online learning.
Arguments:
- obs: ObsType
- action: ActType
- reward: float
- new_obs: ObsType
- new_act: ActType
- info: dict
Return: None

Q: agent.run_episode()
A: We repeatedly call env.step(), choosing actions and updating state:
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
def run_episode(self: Agent, seed: int) -> List[float]:
    '''
    Simulates one episode of interaction, agent learns as appropriate
    Inputs:
        seed : Seed for the random number generator
    Outputs:
        The rewards obtained during the episode
    '''
    rewards = []
    obs = self.env.reset(seed=seed)
    self.reset(seed=seed)
    done = False
    act = self.get_action(obs)
    while not done:
        (new_obs, reward, done, info) = self.env.step(act)
        new_act = self.get_action(new_obs)
        self.observe(obs, act, reward, new_obs, new_act)
        rewards.append(reward)
        obs = new_obs
        act = new_act
    return rewards

Q: agent.get_action()
A: Arguments:
- obs: ObsType
Returns:
- act: ActType

Q: env.np_random
A: - Fast and good quality RNG
- Same as np.random.default_rng()
- From the PCG family (Permuted Congruential Generator)

Q: Reward Shaping
A: One solution to sparse rewards is to use human knowledge to define auxillary reward functions that are more dense

Q: Reward Hacking
A: When an agent learns to obtain the auxiliary reward without completing the desired task

Q: Probe Environments
A: Extremely simple environments used to debug an agent

Q: Action spaces
A: - Continuous
  - gym.spaces.Box(low: np.ndarray, high: np.ndarray)
- Discrete
  - gym.spaces.Discrete(n: int)