Browsing > By speaker > Ko Fu-Yao

Structural optimization has attracted considerable interest over the last three decades due to limited energy and material resources [1]. For combined sizing and shape optimization of truss structures, the design variables are the nodal coordinates of the joints and the cross-sectional areas of the members [2]. This optimization problem has been recognized as a more important task than purity sizing optimization because it can provide more reduction in weight [3]. However, this problem is considered to be more challenging because of the different natures of the two types of variables. Combining these may produce a different rate of convergence and ill-conditioning problems. Because the constraints and objective function become more complex, it may lead to solutions far from the global optimum or even in infeasible domains [4]. To solve these difficulties, two classes of optimizers are used to solve structural optimization problems of truss structure, including mathematical programming approach [5] and metaheuristic algorithms [6].

Reinforcement learning (RL) is an area of machine learning, which trains an action taker called agent to take appropriate actions to maximize the cumulative numerical reward signal. RL task must be formulated into a Markov decision process (MDP). The MDP is a mathematical tool used for sequential decision making where state transition and reward function solely depend on the current state and action and are independent of the sequence of events that preceded it [7]. RL method has achieved remarkable success in game playing, autonomous driving, intelligent control, and planning and scheduling [8]. However, few studies focus on the area of engineering optimization and its applications.

Recently, RL method has been successfully implemented in structural optimization problems. Hayashi and Ohsaki [9] presented a combined approach using Q-learning and graph embedding for binary truss topology optimization for volume minimization under stress and displacement constraints. Huynh et al. [10] integrated Q-learning into differential evolution algorithm. This method was validated by truss optimization on sizing and shape with multiple frequency constraints. All design variables are considered in continuous search space (CSS). Kupwiwat et al. [11] proposed a combined method of deep deterministic policy gradient (DDPG) and graph attention network for geometry optimization of latticed shells described by Bézier surface. Kupwiwat et al. [12] developed a novel method using multi-agent RL and DDPG and applied to multi-objective optimization problems of truss structures. RL approach can be used for structural design problems where the optimal solution is not known beforehand because it does not need input–output pairs as training data. However, the disadvantage of this method is the high computational load for training the agent [13].

An RL algorithm based on Monte Carlo tree search (MCTS) is proposed to solve the truss layout optimization problems in the last three years. Luo et al. [14] proposed an RL algorithm named KR-UCT to handle continuous variables for solving truss layout design problems using kernel regression. Luo et al. [15] developed a novel MCTS-based RL method known as AlphaTruss to produce the optimal truss layout considering sizing, shape, and topology. AlphaTruss uses a two-stage strategy to determine the optimal solution and does not introduce accelerating technique. Therefore, it tends to get trapped in local optimum and cannot be directly used for practical engineering problems. Both KR-UCT and AlphaTruss are applied to CSS. Ko et al. [16] presented an RL approach called Improved Monte Carlo Tree Search (IMCTS) for discrete sizing optimization of truss structures. This technique with update process and accelerating technique for discrete variable, the best reward, and terminal condition utilizes the search tree with multiple root nodes. This method is employed to minimize the weight of the truss subject to stress and displacement constraints. The results show that IMCTS can determine the globally optimal solution within a reasonable time, stably produce optimal designs, and solve large-scale truss structures and multi-objective optimization problems. IMCTS only deals with one type of design variable with discrete case. Few studies employ RL methods to deal with different types of design variables with mixed continuous/discrete system.

In this paper, Mixed Variable System Monte Carlo Tree Search (MVSMCTS) is proposed to handle sizing and shape variable at the same time. MVSMCTS revolves around two important characteristics: update process and accelerating technique in CSS and combined scheme in single and mixed system. The substantive contributions of this paper are the following:

(1) Search region is defined as certain part of the CSS. The main aim is to find the optimal solution. Search region is determined by the design variable of the initial state in current round, which is calculated from the design variable of the final state in previous round.

(2) Search region is composed of all possible points in the CSS. Three important points, including the left endpoint, the midpoint, and the right endpoint are specified in search region. The midpoint is chosen as the design variable of the initial state to achieve global optimality.

(3) Nonuniform meshes are generated in search region to add the child node in the search tree. The meshes are randomly distributed. In other words, the interval between two adjacent meshes is not a constant. The optimal solution found by nonuniform meshes are slightly better than that by uniform meshes.

(4) Accelerating technique incorporates decreasing the size of the search region and the width of search tree as the update process proceeds. The width of search tree is based on the number of meshes generated for the current search tree.

(5) Three types of accelerating techniques including geometric decay, linear decrease, and step reduction are discussed. Regardless of the size of the search region and the width of search tree, geometric decay performs better than linear decrease and step reduction concerning computational efficiency.

(6) In combined scheme, the search tree couples different types of design variables to solve mixed variable structural optimization (MVSO).

(7) MVSMCTS proposes MDP framework, four steps in search tree, UCB, the best reward, policy improvement, update process and accelerating technique for discrete variable, and terminal condition to guarantee to find an optimal solution path.

The results demonstrate that this approach can find the optimal solution within a reasonable time, stably produce optimal design, and solve real-sized truss structures. In conclusion, RL-based MVSMCTS is a powerful optimization technique for MVSO. This study also shows evidence that this algorithm has the potential to solve optimization problems with mixed continuous/discrete system for practical engineering problems.

References

[1] G. N. Vanderplaats, Structural optimization—past, present, and future, AIAA Journal, 20 (7), 992-1000, 1982.

[2] B. H. V. Topping, Shape optimization of skeletal structures: a review, Journal of Structural Engineering, 109 (8), 1993-1951, 1983.

[3] U. Kirsch and B. H. V. Topping, Minimum weight design of structural topologies, Journal of Structural Engineering, 118 (7), 1770-1785, 1992.

[4] M. Zhou and R. Xia, An efficient method of truss design for optimum geometry, Computers & Structures, 35 (2), 115-119, 1990.

[5] D. Wang, W. H. Zhang, and J. S. Jiang, Combined shape and sizing optimization of truss structures, 29, 307-312, 2002.

[6] S. J. Wu and P. T. Chow, Integrated discrete and configuration optimization of trusses using genetic algorithms, Computers & Structures, 55 (4), 695-702, 1995.

[7] R. S. Sutton and A. G. Barto, Reinforcement learning, second edition: an introduction, MIT Press, Cambridge, 2018.

[8] W. Du and S. Ding, A survey on multi‑agent deep reinforcement learning: from the perspective of challenges and applications, Artificial Intelligence Review, 54, 3215-3238, 2021.

[9] K. Hayashi and M. Ohsaki, Reinforcement learning and graph embedding for binary truss topology optimization under stress and displacement constraints, Frontiers in Built Environment, 6, 59, 2020.

[10] T. N. Huynh, D. T. T. Do, and J. Lee, Q-learning-based parameter control in differential evolution for structural optimization, Applied Soft Computing, 107, 107464, 2021.

[11] C. Kupwiwat, K. Hayashi, and M. Ohsaki, Deep deterministic policy gradient and graph attention network for geometry optimization of latticed shells, Applied Intelligence, 53, 19809-19826, 2023.

[12] C. Kupwiwat, K. Hayashi, and M. Ohsaki, Multi-objective optimization of truss structure using multi-agent reinforcement learning and graph representation, Engineering Applications of Artificial Intelligence, 129, 107594, 2024.

[13] K. Hayashi and M. Ohsaki, Graph-based reinforcement learning for discrete cross-section optimization of planar steel frames, Advanced Engineering Informatics, 51, 101512, 2022.

[14] R. Luo, Y. Wang, Z. Liu, W. Xiao, and X. Zhao, A reinforcement learning method for layout design of planar and spatial trusses using kernel regression, Applied Sciences, 12 (16), 8227, 2022.

[15] R. Luo, Y. Wang, W. Xiao, and X. Zhao, AlphaTruss: Monte Carlo tree search for optimal truss layout design, 12 (5), 641, 2022.

[16] F. Y. Ko, K. Suzuki, and K. Yonekura, Improved Monte Carlo tree search formulation with multiple root nodes for discrete sizing optimization of truss structures, Engineering Optimization, 2024.