Trying to make it work for Single inverted pendulum custom environment

[maazashraf23]

Hey simon, I was trying to use the model to train on my custom Single inverted pendulum environment with 5 inputs and single action but its not balancing. It is balancing on left side but not upwards. Do you have any suggestions that I can apply to make it work for my environment.

[SimonRennotte]

Hi,

It probably ends up in a local minimum.
It's difficult to say for sure without more information but I would try:

• Higher sampling rate. Lower the time between each control step
• Better initialization for the hyperpriors of the gaussian processes
• Put bounds on the states so that it does not reach the extremity.
• Check how the length of the control horizon impact the control, and make sure you have a correct value
• Try an higher exploration factor

Does it help ?
Can you share the animation of the dynamic 2d plot of the control so that I can see better what is happening?

[maazashraf23]

This is the dynamic 2d plot that I am getting right now. It's really messy. In my environment there are technically 6 outputs(My states were like this(['$\alpha$ [rad]'(angle), '$d \alpha$ [rad/s]'(angular velocity), '$x$ m', '$v$ [m/s]'(Velocity), '$F$ [N]'(Force), '$I$'(Inertia))) in which I am using 4 states(['$\alpha$ [rad]'(angle), '$d \alpha$ [rad/s]'(angular velocity), '$x$ m', '$v$ [m/s]'(Velocity)). because in pendulum environment there is only angular velocity we donot have to worry about the position of the cart and velocity of the cart. But in my case there is a pole attached to a cart and I have to balance it.

.

I have one more question that how did you assign values in the config pendulum file? specifically to the target state norm and target action norm ?because in your code the values are 1 .5 and .5 and because I have 4 states I assigned 1,0.5,0.5,0.5.

this is the my configuration.

def get_config(len_horizon=15, include_time_model=False, num_repeat_actions=1):
observation_config = ObservationConfig(
obs_var_norm = [1e-6, 1e-6, 1e-6, 1e-6]
)

reward_config = RewardConfig(
	target_state_norm=[1, 0.5, 0.5 ,0.5],  

	weight_state=[1, 0.1, 0.1, 0.1], 
	weight_state_terminal=[5, 2, 2, 2], 

	target_action_norm=[0.05], 
	weight_action=[1e-3],

	exploration_factor=1,

	use_constraints=False,
	state_min=[-3,-3,-3,-3],
	state_max=[3,3,3,3],
	area_multiplier=1, 

	clip_lower_bound_cost_to_0=False,
)

actions_config = ActionsConfig(
	limit_action_change=False,
	max_change_action_norm=[0.3]
)

model_config = ModelConfig(
	gp_init = {
		"noise_covar.noise": [1e-5, 1e-5, 1e-5, 1e-5], # variance = (std)²
		"base_kernel.lengthscale": [0.5, 0.5, 0.5, 0.5],
		"outputscale": [5e-2, 5e-2, 5e-2, 5e-2]
	},
	# [[0.75, 0.75, 0.75, 0.75],
	# [0.75, 0.75, 0.75, 0.75],
	# [0.75, 0.75, 0.75, 0.75]]
	min_std_noise=1e-3,
	max_std_noise=1e-2,
	min_outputscale=1e-2,
	max_outputscale=0.95,
	min_lengthscale=4e-3,
	max_lengthscale=10.0,
	min_lengthscale_time=10,
	max_lengthscale_time=10000,
	init_lengthscale_time=100,
	include_time_model=include_time_model
)

memory_config = MemoryConfig(
	check_errors_for_storage=True,
	min_error_prediction_state_for_memory=[3e-4, 3e-4, 3e-4, 3e-4],
	min_prediction_state_std_for_memory=[3e-3, 3e-3, 3e-3, 3e-3],
	points_batch_memory=1500
)

training_config = TrainingConfig(
	lr_train=7e-3,
	iter_train=15,
	training_frequency=25,
	clip_grad_value=1e-3,
	print_train=False,
	step_print_train=5
)

controller_config = ControllerConfig(
	len_horizon=len_horizon,
	actions_optimizer_params = {
		"disp": None, "maxcor": 4, "ftol": 1e-15, "gtol": 1e-15, "eps": 1e-2, "maxfun": 4,
		"maxiter": 4, "iprint": -1, "maxls": 4, "finite_diff_rel_step": None
	},
	num_repeat_actions=num_repeat_actions
)

config = Config(
	observation_config=observation_config,
	reward_config=reward_config,
	actions_config=actions_config,
	model_config=model_config,
	memory_config=memory_config,
	training_config=training_config,
	controller_config=controller_config, 
)

return config

[maazashraf23]

Can you also tell me how are you calculating the reward? So we have a reward that comes from the environment and you have also calculated a reward(cost). I am finding it hard to imagine it can you please help me with that and also during the training I was getting reward -1 from the custom reward function made by me. So I placed in the penalty output bounds so if any of my 4 states are outside the penalty bounds then the reward will be -1 otherwise the reward will be calculated based on some other factors. But when the states are within the penalty bounds I am getting nan value and when it is out of bounds then I am getting -1 but technically I should get a certain value for reward if it is inside the penalty bounds.

[maazashraf23]

In my case the controlled output is the cart position so technically my model have to control the cart position to balance the pole on it. Any suggestions?