1: Start
2: Input: Input the set of task
          $\mathit{T}={\mathit{t}}_1,{\mathit{t}}_2,{\mathit{t}}_3,{\mathit{t}}_4,\dots ,{\mathit{t}}_{\mathit{m}}$
3: Output: Output task scheduling policies
         $\mathsf{\Pi }\mathit{P}={\mathsf{\Pi }\mathit{P}}_1,{\mathsf{\Pi }\mathit{P}}_2,{\mathsf{\Pi }\mathit{P}}_3,{\mathsf{\Pi }\mathit{P}}_4,{\mathsf{\Pi }\mathit{P}}_5\dots ,{\mathsf{\Pi }\mathit{P}}_{\mathit{p}}$
4: Initialize $\mathit{Q}\left(\mathit{s},\mathit{a}\right),\forall \mathit{s}\in \mathit{S},\mathit{a}\in \mathit{A}\left(\mathit{s}\right),\mathbf{a}\mathbf{r}\mathbf{b}\mathbf{i}\mathbf{t}\mathbf{r}\mathbf{a}\mathbf{r}\mathbf{i}\mathbf{l}\mathbf{y} \mathbf{a}\mathbf{n}\mathbf{d} \mathbf{Q}\left(\mathbf{t}\mathbf{e}\mathbf{r}\mathbf{m}\mathbf{i}\mathbf{n}\mathbf{a}\mathbf{l} \mathbf{s}\mathbf{t}\mathbf{a}\mathbf{t}\mathbf{e}\right)=0$
5: Initialize LLM heuristic Q buffer $\mathit{D}\left(\mathit{G}\right(\mathit{p})=“>\; <mrow>\; <mi\; mathvariant="bold-italic">p</mi>\; </mrow>$
6: For each episode S, perform
7:   Training phase of LLM_SARSA
8:        For every task in training task set ${\mathit{t}}_{\mathit{i}}\in \mathit{T},$ perform
9:        Initialize state S, Action A
10:      Choose Action A from state S using the policy derived from $\in \mathit{g}\mathit{r}\mathit{e}\mathit{e}\mathit{d}\mathit{y}$
11:        For each step of episode, perform
12:         Take action A, observe reward R, and go to next step ${\mathit{S}}^{\prime }$
13:         Choose action ${\mathit{A}}^{\prime }$ from state ${\mathit{S}}^{\prime }$ using the policy derived from $\in \mathit{g}\mathit{r}\mathit{e}\mathit{e}\mathit{d}\mathit{y}$
11:      $\mathit{Q}\left(\mathit{S},\mathit{A}\right)=\mathit{Q}\left(\mathit{S},\mathit{A}\right)+\mathit{\alpha }(\mathit{R}+\mathit{\gamma }\ast \mathit{Q}\left({\mathit{S}}^{\prime },{\mathit{A}}^{\prime }\right)-\mathit{Q}(\mathit{S},\mathit{A})$
12:          Update $\mathit{S}\leftarrow {\mathit{S}}^{\prime }$, $\mathit{A}\leftarrow {\mathit{A}}^{\prime }$
12:          Compute the LLM heuristic value $\mathit{D}(\mathit{G}\left(\mathit{p}\right)=\mathit{D}(\mathit{G}\left(\mathit{p}\right)\cup \mathit{Q}\left(\mathit{S},\mathit{A},\mathit{R},{\mathit{S}}^{\prime }\right)$
13:          Update the $\mathit{Q}\left(\mathit{S},\mathit{A}\right)$ with LLM $\mathit{D}(\mathit{G}\left(\mathit{p}\right)$
14:          ${\mathit{Q}}^{\mathit{*}}\left(\mathit{S},\mathit{A}\right)=(\mathit{Q}\left(\mathit{S},\mathit{A}\right)+\mathit{\alpha }(\mathit{R}+\mathit{\gamma }\ast \mathit{Q}\left({\mathit{S}}^{\prime },{\mathit{A}}^{\prime }\right)-\mathit{Q}\left(\mathit{S},\mathit{A}\right))+\mathit{D}(\mathit{G}\left(\mathit{p}\right))$
15:          Employ L2 loss to approximate the ${\mathit{Q}}^{\mathit{*}}$ value
16:          L2(${\mathit{Q}}^{\mathit{*}}\left(\mathit{S},\mathit{A}\right))=\mathit{E}\mathit{r}\mathit{r}\mathit{o}\mathit{r}\left(\mathit{S},\mathit{A},{\mathit{Q}}^{\mathit{*}}\left(\mathit{S},\mathit{A}\right)\right)\ast \mathit{D}(\mathit{G}\left(\mathit{p}\right)\ast {({\mathit{Q}}^{\mathit{*}\mathit{*}}\left(\mathit{S},\mathit{A}\right)-{\mathit{Q}}^{\mathit{*}}\left(\mathit{S},\mathit{A}\right))}^2$
17:        ${\mathit{Q}}^{\mathit{*}\mathit{*}}\left(\mathit{S},\mathit{A}\right)=\mathbf{L}2({\mathit{Q}}^{\mathit{*}}\left(\mathit{S},\mathit{A}\right))+({\mathit{Q}}^{\mathit{*}}\left(\mathit{S},\mathit{A}\right)+\mathit{\alpha }(\mathit{R}+\mathit{\gamma }\ast {\mathit{Q}}^{\mathit{*}}\left({\mathit{S}}^{\prime },{\mathit{A}}^{\prime }\right)-{\mathit{Q}}^{\mathit{*}}\left(\mathit{S},\mathit{A}\right))+\mathit{D}(\mathit{G}\left(\mathit{p}\right))$
18:           End For of episode until S is terminal
19:    End For
20:    Testing phase of LLM_SARSA
21:       For every task in testing task set ${\mathit{t}}_{\mathit{i}}\in \mathit{T},$ perform
22:          Initialize state S, Action A
23:          Choose Action A from state S using the policy derived from $\in \mathit{g}\mathit{r}\mathit{e}\mathit{e}\mathit{d}\mathit{y}$
24:           For each step of episode, perform
25:              Execute the action ${\mathit{A}}^{\prime }$ from state ${\mathit{S}}^{\prime }$ with updated heuristic value and L2 loss value
26:              ${\mathit{Q}}^{\mathit{*}\mathit{*}}\left(\mathit{S},\mathit{A}\right)=\mathbf{L}2({\mathit{Q}}^{\mathit{*}}\left(\mathit{S},\mathit{A}\right))+({\mathit{Q}}^{\mathit{*}}\left(\mathit{S},\mathit{A}\right)+\mathit{\alpha }(\mathit{R}+\ast \mathit{*}{\mathit{Q}}^{\mathit{*}}\left({\mathit{S}}^{\prime },{\mathit{A}}^{\prime }\right)-{\mathit{Q}}^{\mathit{*}}\left(\mathit{S},\mathit{A}\right))+\mathit{D}(\mathit{G}\left(\mathit{p}\right))$
27:           End For of episode until S is terminal
28:          End For
29: End For
30: Output $\mathsf{\Pi }\mathit{P}={\mathsf{\Pi }\mathit{P}}_1,{\mathsf{\Pi }\mathit{P}}_2,{\mathsf{\Pi }\mathit{P}}_3,{\mathsf{\Pi }\mathit{P}}_4,{\mathsf{\Pi }\mathit{P}}_5\dots ,{\mathsf{\Pi }\mathit{P}}_{\mathit{p}}$
34: Stop