Grain refinement is one of the most important strengthening mechanisms in metallic materials. It is described by the Hall-Patch (H-P) relationship as a homogenized single variable function on average grain size. However, in modern additive manufactured (AM) alloys, strong heterogeneous microstructures have been commonly observed. In the modelling of the mechanical properties of materials, usually an average value of grain size determined from the equivalent spherical diameter (ESD) is used. Owing to the complex grain size distribution caused by the temperature gradient, a sole value might not represent complex microstructures of AM materials adequately. In this work, we consider the grain size as a variable for each grain and each slip system based on the input microstructure. This variable is further used in a dislocation density based constitutive law in crystal plasticity finite element model (CPFEM). Thus, the dislocation density accumulation is affected by the directional grain size corresponding to specific slip system. Therefore, the H-P relationship is not directly applied in the constitutive law but embedded in the dislocation strengthening mechanism. The results indicate that the H-P relationship is still valid with this modified model. Furthermore, the results also show that the anisotropy of Young's moduli and yield strength can be well simulated with the same microstructure. Finally, the current model can show a more realistic strain distribution compared to experimental results on complex microstructures.